ChatGPT and Job Losses

This data was originally featured in the August 2, 2023 newsletter found here: https://www.trustinsights.ai/blog/2023/08/inbox-insights-artisanal-vs-ai-chatgpt-and-job-losses-8-2/.

In this week’s Data Diaries, let’s look at the supposed impact of ChatGPT and similar tools on jobs. One of the great fears people have is that AI is going to devour all the jobs, and while there will be many, many tasks that we will turn over to AI, the technology is still insufficiently advanced to do entire jobs (assuming a job is more than one task).

So, how would we know AI is becoming a problem? What would be a good indicator? A candidate source would be the number of open positions in jobs that would be heavily impacted by generative AI. For this newsletter, we’ve picked three categories: information jobs, marketing jobs, and software development jobs. Generative AI impacts all three; by some estimates, 70% of ChatGPT usage is by developers, so all three should be the proverbial canary in the coal mine.

To do this, we can turn to the data provided by Indeed.com to the US Federal Reserve Bank of St. Louis, the FRED data system. Indeed has published a relative index of job openings in a variety of fields since just before the pandemic, with data normalized at 100 for February 2020.

Let’s take a look at the big picture:

What we see in the first chart here are the three fields we’ve chosen, and their relative indices throughout and after the pandemic. Note the black vertical line: that’s the day ChatGPT arrived on the scene.

If ChatGPT had a disastrous effect on jobs, we would see a substantially sharper decline in the post-ChatGPT period than we do. Now, one of the challenges with a dataset like this is that the pandemic itself imposes enormous distortions on the data. These effects are impossible to see with the unaided eye; to identify it clearly, we need to do a mathematical comparison of periods in the data when the pandemic wasn’t happening to periods when it was – as well as account for the presence or absence of ChatGPT. To do this, we use a statistical method known as propensity score matching, which is a form of retroactive A/B testing.

When we perform this test, what do we see?

We see the pandemic control, which makes perfect sense. There are fewer pandemic days since the rollout of ChatGPT than there were prior, so that number should be in the negative. That tells us the basic mechanism is working correctly.

And we see that marketing jobs, developer jobs, and information jobs on average have INCREASED proportional to similar time periods after the rollout of ChatGPT rather than before.

So what? What does this tell us? As is normal for humans, we overestimate the short term impact of a new technology and we underestimate the long term impact. This system of measurement helps us to discern changes over time, and is something – assuming the data remains available – to monitor in the months and years to come. Techniques like this will help us to see the changes as they happen, and they will be slower at first until they’re not. By running an analysis like this regularly and frequently, we can monitor for these changes and know to take action sooner rather than later.