ChatGPT Code Interpreter

This data was originally featured in the July 12, 2023 newsletter found here: https://www.trustinsights.ai/blog/2023/07/inbox-insights-ai-efficiencies-ripple-effects-chatgpt-code-interpreter/.

In this week’s Data Diaries, let’s take a brain-melting look at the newest AI data analysis tool, GPT-4 Code Interpreter from OpenAI. Housed within the familiar ChatGPT interface, Code Interpreter is part of the $20/month paid plan, and after you see it in action, you’ll wonder why it isn’t $2,000/month. First, we need some data to analyze. Since we opened Trust Insights Academy, we’ve followed a pretty well-known industry strategy: offer up free courses and upsell into paid courses.

The question we want to know the answer to is: does this strategy actually work? We see plenty of course registrations, and we’ve been successful at selling courses, but does this particular marketing strategy work? Let’s have Code Interpreter answer the question.

Before we begin, Code Interpreter is a stupid name for the service. What they should actually call it is Automated Junior Data Scientist, because that’s exactly what it does.

Our first step is to export our orders from the Academy and load them into ChatGPT, along with some basic prompting. Here’s what we said:

Today we’re going to be doing an analysis on the included file. The file is a list of orders for courses, by customer. The Customer ID is the email address, noted in the email field. The courses are in the product field. Some courses are free. Other courses cost money. The way to determine whether something is free or not is in the payment method field, which denotes “no charge” for a free course or a payment method if the course is not free.

The analysis we want to understand is twofold. First, we give away access to free courses to hopefully encourage upsells into our paid courses. We want to know if this strategy is effective. What analysis method would best answer this question?

It responded:

Given your question, one potential analysis approach would be to perform a Cohort Analysis. This type of analysis could provide insights into the behavior of different groups of customers who started their journey with free courses. We could then track if they subsequently purchased paid courses.

Here are the steps we might follow:

Data Cleaning: First, we’ll import the data and clean it as necessary, making sure that the fields are in the correct formats, handling missing values, and so on.

Cohort Identification: We’ll identify different cohorts of customers based on the date when they first enrolled in a free course. These cohorts will represent users who started their journey with a free course in a specific month.

Cohort Metrics: For each cohort, we’ll calculate metrics like:

The number of customers who started with a free course The number of customers who later purchased a paid course The conversion rate from free to paid courses (number of customers who purchased a paid course / total number of customers who started with a free course) Cohort Analysis: We’ll analyze the metrics across different cohorts to understand the effectiveness of the upsell strategy. For instance, we might look at whether conversion rates are improving over time.

However, before we proceed with this approach, let’s first import the data and perform an exploratory data analysis to understand the structure of the data and validate if the proposed method is suitable.

That’s a terrific start. Then it began working – and it does so by writing code:

 

It does all the work of loading the file, then cleaning and processing the data:

 

Then it writes the executable Python code to do the cohort analysis:

 

And finally, it presents us with the analysis:

 

Oh.

Well… so much for that strategy. What we see is that hundreds and hundreds of people have signed up for our free courses, but of them, exactly 4 have become paying customers. That’s… not a great track record at all. And this is over more than a year’s time, which is plenty of time to accumulate evidence. While this strategy may work for others, it clearly does not work for Trust Insights. Katie and I will have to have a conversation about this strategy’s lack of results.

While this in itself is a useful insight, the big picture takeaway here is what Code Interpreter was able to do. I intentionally did not use the RACE prompt structure we recommend, to better simulate what a non-technical person might ask of the system. It did all the heavy lifting – choosing the analytical method, cleaning and processing the data, and presenting the analysis.

And from start to finish, it took 6 minutes.

I am a skilled data engineer and data scientist. It would take me longer than 6 minutes to perform this analysis, and I would have followed very similar steps (albeit in R, not Python). This tool did a fantastic job for the most part, only needing a couple corrections along the way.

So, am I out of a job? Well… no. The couple corrections I made were subtle math errors. See the final column in the table, Started_Paid? I had to ask it for that specifically because it didn’t create it, and the first iteration of the table, the numbers didn’t add up. So I inspected the code and found that it had made a logical error, which it corrected when I told it to do so. That’s why skilled workers are still needed, because when the machines make errors, the errors aren’t always obvious.

But do I need to hire a junior data scientist now? Not with Code Interpreter. I can have it do a lot of the heavy lifting – the first draft – and then edit, tune, and improve on what it did. Tools like Code Interpreter will be powerful allies to skilled employees, dramatically shortening the time needed to produce powerful analyses. Now that it’s open to the public, you should start testing it yourself, asking it the marketing analytics questions you’ve always wanted answers to, but didn’t have the coding or technical skills yourself to answer.