Data Science, Analytics, and AI Behind the Save Warrior Nun Campaign

The Netflix TV series Warrior Nun was cancelled on December 13, 2022. As with so many other shows and franchises, fans of the series took to social media to protest it – but the Warrior Nun fandom took a different approach. Rather than uncoordinated yelling, fans self-organized in what was one of the most interesting self-management phenomena of recent times. On December 28, a friend asked me a simple question that led to an adventure like I’ve never experienced before:

“I know you’re pretty busy, and there’s no obligation, but there’s some people really getting in the trenches with organizing the movement to try and get warrior nun picked up by other networks and they’re like serious serious about with marketing teams and making contacts with executives and stuff like that. They’re asking for anyone’s help who has marketing or data experience to help compile data into info that can help them give pitches to other services. Would you have time/be willing to lend any of your data skills to them?”

What started out as an inquiry into how to process Twitter data turned into a 6 month campaign to save a show that ended on June 28, 2023 with a tweet from showrunner Simon Barry:

How did this happen? How did a ragtag group of volunteers take on one of the largest tech companies and entertainment companies in the world – and win? What can other fan efforts (and any volunteer campaign) learn and apply to improve your chances of succeeding?

I won’t speak to the entire campaign, only the parts that I had direct involvement in, but those parts might lend some insights into similar campaigns, similar situations. If you’re involved in those campaigns, hopefully you’ll find inspiration from what we did.

The Early Days: Reporting

The moment I walked in the virtual door, the folks in The Order (the organization and Discord server coordinating a lot of fan activities) asked for this specifically:

okay so basically we’re looking to compile twitter data, like amount of #SWN tweets that exist, the amount of unique accounts tweeting about it, and maybe some more specific metrics that have to do with the phrases we’ve trended this last week. last one for this 7 days was today, so tomorrow is for brainstorming and data gathering. i basically want to compare each quote per day that we trended to each other to gauge the increase in participation

Despite the lack of formality, this is a pretty decent user story that lends itself well to building basic requirements. The challenge with this ask is that most of the social media analysis tools on the market will not do this. Even the very best tools like Talkwalker deliver aggregated data for the most part because of Twitter’s restrictions on how much data you can download at once – 50,000 tweets. And the campaign was generating hundreds of thousands of tweets a day at that point from thousands of accounts.

What’s a marketing technologist to do? You build your own Twitter data extraction system, of course. Using a spare Trust Insights server that had capacity, I wrote some custom code in the R programming language that would query the Twitter API every 5 minutes and download the data from it, then clean it up and store it in a database.

Once we had the data, then it was a matter of analyzing it. I asked the team leads how often they wanted to make decisions with data, and the answer was daily.

If you’ve ever done any kind of data analysis, even something basic in a spreadsheet, you know it takes time. Doing an ad hoc analysis once is not a big deal. You spend the time, you dig through the data, you produce the analysis. That’s fine…

… once. Daily? You’d better be able to automate the process or you’re going to burn a lot of hours every day for who knows how many days. (197 days, to be clear)

The only sane choice for that kind of reporting cadence is automation, full automation. You can’t afford to spend an hour a day or even half an hour a day recreating the same charts and graphs over and over again for more than a few days. But writing code to do that? That dramatically shortens the time to minutes, if not seconds, and that’s what we did. We wrote code in the R programming language to ingest all the Twitter data and process it into charts every day. The entire time it takes to run the code? Under 40 seconds, allowing us to produce snapshots like this:

Once the campaign teams had this information, they were able to make informed, data-driven decisions about what to do, what topics to pursue, when to ease up on the pace of content, and when to push harder.

Those first easy wins opened the door to more tasks, more things to do to build up the campaign. The next chapter in the campaign was infrastructure.

Building the Machinery

There are foundational pieces every campaign needs to succeed, from a digital marketing infrastructure perspective. You need a home base, an outlet, and a water cooler. Sometimes these things can be one system, but very often they’re individual point solutions. In the case of volunteer campaigns they’re almost always point solutions because campaigns typically have very little budget to work with, and it’s easier to find a suite of free solutions than try to fundraise for a paid all-in-one solution.

So let’s talk through these parts. If you expect the campaign to last for more than a few weeks, a home base is essential. This is normally a website of some kind, which you can host on places like Wix, Squarespace, or WordPress (My personal choice is usually WordPress because of its plugin ecosystem). Why? You want some place that will inherit search traffic. As the campaign takes root, you want to capture casual searches about it.

As an aside, remember to name your campaign something obvious. While it’s tempting to use cultural references from your franchise of choice, no one except people already in your fan base will ever search for those terms, whereas going after the show name is much more obvious and sensible.

The second piece you need is an outlet, a publication. Social media is nice, but social media does not give us reliable reach – the ability for us to reach out and contact fans at a moment’s notice and be sure our communication has arrived. Instead, with social media, we are constantly competing with AI systems trying to guess whether our content is less or more worthy than all the other content a user could be shown at any given time. Relying on social media is a substantial risk, one which played out later in the campaign. An email newsletter? That’s a solid bet. We chose the Substack platform because it was free and reliable.

The third piece you need is a water cooler, a community, a place for people to have conversations in an unrestricted format. Again, you might be tempted to say, “Just have conversations on social media!” And while there is definitely an important component there, you also want to ensure people have a safe space that isn’t public social media. The sheer amount of trolling on places like Twitter and Reddit can quickly frighten away some of your fan base. The better place for a community is a private social network like Discord, and early on in the campaign, several Discord communities popped up; the largest is the Warrior Nun Discord, which is still highly active today with over 3,000 members.

The meta piece across much of this is measurement. Unsurprising for anyone who reads my stuff on a regular basis, measurement is the linchpin that holds these pieces together, because if you can’t measure it, you can’t manage it. For that, I deployed Google Tag Manager and Google Analytics 4, then built a convenient dashboard using Google Looker Studio:

Because the campaign’s objectives were largely around attention and awareness, the dashboards and reporting all focused on those objectives, making it relatively straightforward to measure.

Within the first couple of weeks after I joined the campaign, once reporting the basics was squared away, I set up the Substack email newsletter, joined the various Discord communities, and did the IT infrastructure for the website.

A sidebar on IT infrastructure: a website is a bit more complex than it used to be. If you’re part of a fanbase that is in any way controversial, you want to ensure your site is as safe as you can make it from hackers, malware, denial of service, and other malicious actions. Part of setting up the website is standing up a service like Cloudflare. The free edition provides robust security and intercepts the most obvious attacks and denials of service.

So, reporting of the basics is in place, there’s a newsletter and a website, and everything’s all set, right? Well… no. Because something has to fuel those properties, especially when you have a very fast cadence like a daily newsletter.

Content: Data-Driven Provides High Speed Cadence

The most reliable content you can have for a very fast cadence marketing program is data, particularly data from and about your audience. The same data we were extracting for campaign reporting, we made public in the daily newsletter as a way to highlight the good work the fans were doing. People love nothing more than to hear how they’re doing, and especially if there’s a chance for them to be in the limelight among their peers.

This was the secret that kept the newsletter operating every day. There were five fundamental sections to the newsletter, three of which were assembled by different pieces of code. A typical newsletter on any given day opened with a piece of content from the fanbase, from The Order’s marketing team, or from me about ways to make campaign efforts more impactful.

The second portion of the newsletter was code-driven charts and reporting. One of the practices the campaign started very early on was picking a daily phrase of some kind for people to use in their social media posts, especially on Twitter. These phrases are a great way to focus people and help those who want to contribute but don’t know what to write/post to participate more easily, as well as prevent some services from flagging a message as repetitive if you’d just shared the campaign hashtags. Tracking these phrases, as well as overall participation in the campaign hashtags, made up the charts in the second part of the newsletter, and each day, we encouraged fans to keep participating, to keep posting more – especially as showrunner Simon Barry attached rewards to different milestones.

The third portion of the newsletter was code-driven media coverage reporting. From the very early days, the Save Warrior Nun campaign attracted a lot of press coverage. We wanted to be able to track that coverage, archive it, and be able to report on it.

Using the Talkwalker Conversational Intelligence software platform (Trust Insights has a license for it, which is what we used for the campaign), I wrote some code that processed the outputs, scored them based on the reputation of the news source, and then created daily newsletter snippets of coverage:

This mattered a great deal for the campaign because the current media landscape is one in which writers for publications are paid on how well their content performs. The more eyes on their content, the more they get paid; some publications even pay based on pageviews. We wanted it to be blatantly obvious to writers that if they covered Warrior Nun, we would reward them with traffic to their content – and featuring that content in the daily newsletter was a great way to achieve that.

The fourth portion of the newsletter was upcoming fandom events, fed largely from the various Discord servers such as SWN, Warrior Nun Netflix, and The Order. Fans would post watch parties, music parties, Twitter spaces, and other events that would be appealing to the community, and we highlighted those as much as possible. This section, due to the disparate nature of the data, was done manually but didn’t change as often as the other sections.

The final section of the newsletter featured the highest performing content from the past 24 hours on social media (almost always Twitter). These posts were gathered programmatically from the Twitter API as well as the Talkwalker system, then published in the newsletter for others to view and share. These posts were usually cited by handle in the newsletter’s daily tweets as well to encourage the authors to share the newsletter and amplify each issue.

The time it took to assemble the newsletter each day varied, but got progressively shorter over time as our code became more and more efficient. By the conclusion of the campaign, a newsletter could be assembled and sent in about 15 minutes. You can read a sample issue here.

With the mechanics of the data-driven marketing program taken care of, we could turn out attention to the juicy part of the campaign: data and research.

Campaign Optimization With Data

Part of our remit for the campaign, once the operational portion was settled and in maintenance mode, was to use data to investigate ways to make the campaign more impactful, ways to create unique content, and ways to create enough pressure on Netflix and other streaming services to convince a corporate entity with the means to do so to save the show.

First, when it came to campaign efficiency, we made a few immediate improvements. The Order’s PR outreach team was going through reporters one by one, Googling to find contact information for outreach. Using site analysis SEO software, we looked at the websites of major publications, exported the data from those websites, and wrote code to isolate those reporters who were responsible for covering television at publications like Hollywood Reporter, Variety, Deadline, and many others. From that data, we were able to assemble up-to date media lists similar to what companies like Muckrack and Cision charge thousands of dollars a month for (which we clearly could not afford).

Finding influencers is also a key part of any campaign. One of the best measures of influence is not who’s doing the most talking, but who’s most talked about. Using Twitter data and a set of machine learning algorithms called network graphs, we identified everyone participating in the conversations about the Save Warrior Nun campaign and then examined who was the most talked about. From that data, we assembled outreach lists on Twitter, Instagram, Tiktok, and Twitch for outreach teams to reach out and involve people in the campaign.

We also did large-scale data analysis to set social media strategy. Early on, a lot of folks were sharing anecdotal information about what to do to maximize reach on social networks like Twitter, but that information wasn’t grounded in anything scientifically or mathematically proven. We extracted several million tweets and performed multiple regression analysis on those social posts to determine whether or not any of the recommendations people were sharing had any statistical validity, and it turned out many of them did not. This process repeated itself several times throughout the campaign as Twitter’s management continued to make adversarial changes to the service:

Billboards were another part of the campaign’s early tactics. Prior to the purchase of the first billboard, we tackled the question of whether billboards actually worked in terms of increasing awareness of a show and interest in it. Using a technique from biostatistics called propensity score modeling, we looked at several fandoms in the past that had commissioned billboards and how much search volume each show received after the billboard, as well as media coverage. We found that on average, shows saw a minimum of a 7% increase in general audience interest after a fan-led billboard promotion went up, reassuring the Save Warrior Nun campaign organizers that a billboard was indeed the right way to go.

Above and beyond the social media campaign milestones, a number of events occurred that we wanted to understand the impact of. For example, along the way, USA regional convenience store chain Sheetz joined in the campaign, producing content, sending Warrior Nun-themed merch, and generally promoting the show to its fan base as well. Using the same biostatistics techniques as we used for billboards, we examined whether or not this welcome, unplanned ally had any impact, and it clearly did in terms of media coverage.

This set of techniques is incredibly valuable for any kind of public relations work where you want to know what kind of lift a set of activities had while taking into account the things that would have happened anyway.

Once we’d helped optimize the operations of the campaign, we turned our attention to the most exciting, interesting data of all: how to make a case to the public, and to entertainment companies, that Warrior Nun was worth saving.

Research: Non-Obvious Insights Garner Attention

It’s one thing to make a claim, like Warrior Nun was a popular, well-watched TV series that was canceled by Netflix for spurious reasons. It’s another thing to be able to back up that claim and many other claims with objective data, especially data sourced from credible third parties (some of which included Netflix itself). The Order’s data team started asking very good, very interesting questions: can we prove that the show did better than Netflix claimed? Is there demand for the show? How might we do apples-to-apples comparisons of Warrior Nun to other similar shows?

Robin, one of our lead analysts, purchased a subscription to the entertainment industry’s Parrot Analytics software which measures platform-agnostic demand. Demand, according to Parrot, is a combination of several different sets of data such as viewing information, social media, video content creation, and reviews. We immediately began parsing and publishing that data to illustrate how Warrior Nun was competitive to, or even exceeded demand of other shows that had not been cancelled.

That got us thinking… how else could we highlight Warrior Nun’s popularity? We began to examine other third party, neutral data sources like IMDB ratings, which are available from them in very large datasets for download. We downloaded the massive databases, wrote custom code to process them, and found that they were exceptionally good for comparing the show to other shows based on ratings and reviews, a set of findings we shared widely.

But then came the big inference, one which was a pivotal moment for the campaign. Back in 2021, Netflix co-CEO Ted Sarandos talked about how Netflix measures content on stage with Kara Swisher at a Vice Media event. He showed a few examples, enough that we were able to construct a basic model from the data he shared. From that, we bonded the data to search volume data from Google so that we could infer Warrior Nun’s viewership from Season 1 as well as Season 2, then create a data-driven projection of what Season 2’s performance should have been, had Netflix promoted the show at all (which it did not). Fan reactions to this data were significant, and it showcased just how large the Warrior Nun audience truly was – and what a future partner could potentially expect to see if they picked up the franchise and continued it.

These various research projects eventually led to full-fledged white papers published on the Warrior Nun website and distributed not only to fans but also media outlets. We used them as the basis for outreach, for campaigning to specific streaming services to highlight the value of Warrior Nun’s fandom as a customer base, and in media pitches.

The Power of AI

Some problems were too big for us to tackle alone. For those, we needed a little help from our friends… the machines. Artificial intelligence (AI) has been part of the Save Warrior Nun campaign from the early days. From simple regression analysis to models for forecasting campaign performance and alternate scenarios, we used AI liberally wherever it made sense, wherever it accelerated the work. But few things tested us like a massive change made to Twitter in March of 2023. At the end of that month, Twitter’s new management published the source code to their recommendation engine, the software that decides what users on Twitter see. Knowing exactly how Twitter worked would be a boon to any campaign that used the platform to reach new audiences and motivate an existing fanbase.

However, there was a twist. Twitter’s code was written in Scala, Java, and Thrift – three different programming languages all cobbled together rather messily, and in such a way that it made understanding what the various components of this huge code base did. Fortunately, we didn’t have to go it alone. Using OpenAI’s GPT-4 model and the Twitter code from GitHub, we fed the relevant portions of the code base into the GPT-4 large language model and asked it to interpret the code for us. After several days of deep analysis, we were able to distill the components relevant to our campaign (and to marketing in general) from the AI-driven code analysis. That reshaped our recommendations and ensured that we continued to make the most of our social media efforts. We also did similar analysis of engineering content and academic papers for Instagram as well.

The power of AI wasn’t limited to just code interpretation. Throughout the campaign, we found ourselves communicating with multiple stakeholders, from corporate entities (like the convenience store chain Sheetz!) to Warrior Nun cast and crew, to the thousands of fans around the world. We integrated AI in as many places as it made sense, such as multi-language captions on content from events to automated translation of content and publications in 7 different languages for fans around the world to creating social media content:

For example, a portion of each day’s Substack tweets were AI generated (the Twitter handle citations) because it saved a few minutes of time to process them with a GPT model rather than code that part manually.

We even built AI-based summarization of key fan feedback as well as news articles, so that we could get a handle on what the media was saying as well as what other fans were saying. By using natural language processing and large language models, we dramatically reduced the time needed to get the lay of the land before starting each day’s campaigning. The same is true for handling things like Twitter Spaces transcription and meeting notes; with AI tools, we were able to provide summarizes and transcripts very quickly from Spaces, often within a day or less of the actual meeting. This communicated to fellow fans that we wanted to operate in the spirit of transparency, sharing our content and conversations as quickly as possible.

Lessons Learned

The marketing technology, data science, and AI portions of the Save Warrior Nun campaign provided a strong foundation for the campaign to thrive on. Certainly, fans would have continued to be passionate about the show no matter what, but by infusing marketing technology and data into The order’s campaign operations from very early on, we ensured the campaign would have the necessary pieces in place to go the distance – and 197 days later, it did. Let’s wrap up by talking through some lessons learned about 5 key areas of the campaign. These 5 areas are derived from the Trust Insights 5P Framework – purpose, people, process, platform, performance.

Purpose

By far, the most important aspect of the campaign is its purpose. On the surface, the purpose almost seems juvenile – convince the entertainment industry’s key players to renew a canceled franchise. However, behind this straightforward premise was a host of deeper, more important issues. Entertainment franchises that feature women-led casts, particularly non-heterosexual women-led casts, are relatively rare. Part of the purpose, part of what deeply motivated fans of the show to want to save it was that representation, a rare reflection of themselves in an otherwise barren entertainment landscape.

A purpose that is bound to people’s identity is a strong motivator; determining how any cause or purpose ties into identity is a critical success factor. If your campaign’s purpose doesn’t resonate emotionally with the core of the fans, it will not succeed.

People

One key differentiator in the campaign, and in our participation of it, is the people organizing it. Identifying which people needed which data and providing that data to them on a regular and frequent basis improved decision-making. Throughout the campaign, various teams in The Order would ask for ad hoc reports to make quick decisions about things like daily phrases or whether a particular piece of content was doing well or not. Learn who needs what, and provide it to them quickly and correctly. Part of the success formula for the Save Warrior Nun campaign was that agility, that ability to pivot on a dime to whatever needed to happen, which comes from a culture that’s focused on results and highly collaborative.

One of the unique aspects of both the fandom as a whole and The Order in particular is that it is overwhelmingly female. The entire leadership team is female, all the department heads are female, and virtually all of the team was. That led to a very different “workplace” environment than many of us are used to. Decisions were made by consensus many times, which is usually a recipe for disaster, but when everyone’s laser-focused on the purpose and willing to put ego aside, it works amazingly well.

Process

As with almost all marketing, process was the most critical tactical part of the campaign’s success (whereas purpose is strategic). Motivating people to participate in a campaign with no obvious signs of progress over a long period of time requires strong, solid habits, habits that people will do. The daily distributions of data and the use of marketing technology like email newsletters helped to create and reinforce positive habits, habits that people stuck to until the campaign succeeded. Make sure fans have guidelines and habits if they want them, so that when the emotional impact of a cancelation begins to fade, their willingness to continue participating is baked into their days and is as easy as possible for them to keep contributing.

Platform

Platforms and technologies are actually the least important part of the campaign. Technologies come and go, but the key lesson here is that you do the best you can with what you have. In the case of the Save Warrior Nun campaign, much of what we needed technology to do did not exist as boxed products or ready-made services that we could just go out and obtain. Instead, because of the very unique needs of the campaign, we ended up creating the technology platforms we needed. We took existing data sources, like the Twitter API, the Crowdtangle social data service (for Instagram), Talkwalker’s outstanding conversational intelligence tools, Parrot Analytics, and many other data platforms, then wrote over 100 pieces of code to glue the various components together into useful insights.

The advice we usually give applies here: buy it if you can, build it if you can’t.

Performance

The final P, performance, is the most straightforward. The outcome of the campaign was binary – either the franchise would be saved or it would not be. Showrunner Simon Barry would either announce success, or thank fans one last time and ask them to move on.

And on June 28, 2023, the announcement everyone was hoping for came. We saved Warrior Nun.

Final Thoughts

One of the key lessons I took away from my work in this campaign is how important it is for people – especially in the professional sectors – to participate and volunteer in activities that have nothing to do with work. We get stuck in ruts, seeing the same kinds of customers, doing the same kinds of work over and over again. When you participate in a cause – any kind of cause – you get to spread your wings, test yourself, expand your horizons. This is especially true the more senior you are in an organization. After a time, you can doubt yourself, doubt whether the success you’ve experienced is because you have the skills to make it happen. When you volunteer, when you start over, when you get scrappy, you can put yourself to the test in a safe environment and validate that you know what you’re doing, as well as gain new skills, new insights, new professional relationships and friendships. Putting your skills to the test in a worthy cause is almost always a good idea and I recommend it to virtually everyone.

Acknowledgements

This piece would not be complete without offering thanks to the many folks who made it possible. First and foremost, so much of my time as well as several technology resources were donated by my company, Trust Insights. While I did most of my work on evenings and weekends, it was still time doing something that I could have been doing other things instead, and after 197 days, that time added up. In a recent assessment internally, if the Save Warrior Nun campaign had been a commercial client, the cost of services rendered would have totaled approximately US$770,000. So thank you, especially to my partner and CEO, Katie Robbert, for indulging this rather off-beat adventure. I’ll take a moment to offer a commercial plug – if any of what we did for the campaign is of interest for your company’s marketing, contact us.

Second, my thanks go to the members of The Order for letting themselves be my guinea pig for so much of this. As much as the campaign benefited from the work I did, so too did I benefit from facing new, different challenges than I normally face at work. And the solutions I had to create will have long-lasting, portable applications well outside the Warrior Nun fandom. Some solutions, like the AI-based news summarization technology I wrote, are already in use in different contexts. In particular, I want to thank the data team led by “Mom boss” Laura, Robin, Charlie, and Jereczko for letting me tag along, as well as Kelsey, Sarah, and Ari for enduring hours of what was technobabble to you. Additional thanks, in the context of what we did with the marketing technology, goes especially to Adrienne and the OCS Newsletter which provided at least 40% of the content of the Substack newsletters. Any time I sat looking at the blank first section of the Substack and wondering what I would put as the feature item, I knew Adrienne had some content ready to go. Even more additional thanks go to Lucy and Reagan from the design team; many of our charts and graphics were professionally polished by the team for distribution, and they would not have looked nearly as impressive without it.

Finally, none of the success would have happened without the fandom at large. You can organize and plan all you want, but if there aren’t enough people who care, who show up, who never back down, then your plans are just daydreams. Every person who participated in the campaign in any tangible way deserves credit for the campaign’s success.

---

Need help with your marketing AI and analytics? Get help with your AI strategy » Learn generative AI for marketers with our new course » Prove the ROI of your marketing with our advanced analytics services » Just got a "silly" question you want an answer to? Ask us here »	You might also enjoy: In-Ear Insights: The Trust Insights Podcast About Trust Insights Trust Insights Services : Data & Analytics Help, Predictive Analytics Disclosure of AI and Protection of Copyright Instant Insights: The Beginner’s Generative AI Starter Kit
Get unique data, analysis, and perspectives on analytics, insights, machine learning, marketing, and AI in the weekly Trust Insights newsletter, INBOX INSIGHTS. Subscribe now for free; new issues every Wednesday! Click here to subscribe now »	Want to learn more about data, analytics, and insights? Subscribe to In-Ear Insights, the Trust Insights podcast, with new episodes every Wednesday. Click here for Google Podcasts » Click here for Apple Podcasts » Link for all other podcasting software »