Let’s talk about ChatGPT’s new API

This data was originally featured in the March 8, 2023 newsletter found here: https://www.trustinsights.ai/blog/2023/03/inbox-insights-march-8-2023-starting-over-chatgpt-apps/

In this week’s Data Diaries, let’s talk about ChatGPT’s new API. Announced last week, the API allows technically-sophisticated marketers and companies to perform the same tasks we’ve all had fun experimenting with using the web interface, but at a much greater scale than a single person copying and pasting prompts and results.

What are some of the tasks we might put this new API to work on? One of the tasks that’s been persistently problematic for less advanced tools to handle is sentiment analysis that’s textually aware. Older sentiment analysis methods typically rely on procedures like “Bag of Words”, where words with known sentiment are used to score a passage. “Sucks”, “refund”, “angry” would be examples of such a lexicon. However, this approach is fairly naive and very problematic when it comes to more complex topics.

In general, large language models tend to do much better about scoring sentiment accurately; in production tests, ChatGPT’s model was about 84% accurate with regard to sentiment analysis. That’s a substantial improvement over models using bag of words and similar approaches, which often score in the 70% range for accuracy or lower.

We wanted to test two things: first, can ChatGPT’s API be used efficiently for sentiment analysis at scale, and second, what could we do with the data? We created new code that ingested fifty articles about International Women’s Day (a complex topic poorly suited to simplistic sentiment models) and score them all with ChatGPT.

Let’s take a look at the process. First, we need the content itself, which we extract from the Talkwalker media monitoring system. Note that Talkwalker also includes its own sentiment measure, which either shows -5 (negative), 0 (neutral), or +5:

From here, we load the text of each of the articles into OpenAI’s API with the following prompt programmatically built in as the instructions:

You will act as a linguistics expert. You are familiar with language, semantics, sentiment analysis, tokenization, word order, and other language analysis. You specialize in sentiment analysis and sentiment scoring. Your first task will be to analyze text and return a sentiment score from -10 to +10. If no sentiment score is possible, return 0. Return results in a pipe-delimited format with the score as the first column and the explanation as the second column with a pipe separating the columns.

What comes back is a simple table:

What’s fascinating about the returned data is that not only does ChatGPT return the requested score, but it also returns an explanation of how it derived that score. This is helpful for checking the results, to see if ChatGPT’s model made correct judgements about sentiment.

Now, what could we do with this data? One obvious application would be for content curation; if we wanted to ensure that content shared for the day met a specific standard of sentiment, we could use this data to filter content that met our requirements.

The bigger picture lesson here is one I talked about previously; every prompt that you’ve successfully run in the past in ChatGPT has the potential to be an application, a piece of software in its own right now that could scale far beyond one person typing into a chat box. If you’ve got prompts that you’ve been very successful with, now is the time to think big about what you could do with them at scale.