So What? Getting Started Being A Data Prepper

Christopher Penn – 00:37

Happy Thursday, folks. This is So What, the Marketing Analytics and Insights live show. Katie is going to be along in just a few moments. She is just getting out of the Smarter X — I think it’s the masterclass. She was teaching in two different places at the same time: one pre-recorded and one live.

I’m here along with John. This week, we’re talking about something interesting: getting started being a data prepper. If you’re not familiar, this has nothing to do with traditional data preparation for ETL and data science. This is actually tinfoil hat territory prepper stuff, but for data and AI.

John, before we get started, when we say data prepper, what does that bring to mind other than a bunker?

John Wall – 01:28

Exactly. This is someone who has tapes stored in a vault underneath Fort Knox in the salt mines. You’re going far and beyond so that no matter what — even if you survive an EMP — you’re good to go.

Christopher Penn – 01:43

Exactly. We’re going to talk about some of the different software and methodologies. Let’s start by talking about — and we’ll channel Katie here — why we would do this. There she is.

Katie Robbert – 02:03

Was I summoned? Are you talking about the 5P Framework here?

Christopher Penn – 02:06

I was just about to start talking about the 5P Framework.

Katie Robbert – 02:09

I don’t know what you’re talking about, but I can talk about the 5P Framework all day long.

Christopher Penn – 02:13

We’re talking about why you would become a data prepper. What is the purpose of it in the 5P Framework? There are a bunch of different reasons. One: stuff vanishes from the internet all the time. Companies run out of funding, websites go down, governments change, and censorship changes. Stuff that was valuable might just go missing. There are a number of different web archives. I just saw 538, which was acquired by ABC, is just gone. ABC wiped it from the internet. All that history of Nate Silver’s stuff is just gone from their website.

Second reason: internet access is not always reliable, as evidenced by how many hacks and other things are happening right now. The very fabric of the internet is potentially at risk, so having copies of important stuff is a good idea. Third, for all of us AI nerds, AI is much faster if the data is local. If Claude doesn’t have to go out and get data because you’ve got it right on your machine, it’s faster.

Fourth: backups. Follow the 3-2-1 rule. Three copies of your data in two different places on two different types of media, one of which is off-site. My father, who was in the army, always used to say, “If you have two, you have one. If you have one, you have none.” If you have two backups and one of them fails, you at least have one left. If you have only one backup and it fails, you don’t have any backups. Being a data prepper or a data hoarder is about making sure we have copies of our stuff.

Katie Robbert – 03:58

The question we would naturally get is: when is that too much? There are three copies of your data: two backups and one live version. Why not three versions? Why not four versions? At what point is it too much?

Christopher Penn – 04:16

When you can’t afford it anymore.

Katie Robbert – 04:20

That’s fair. It comes down to proper planning. Do I have the space for it? Can I afford those backups? If I can, let me make the most use of them. We have stuff on — I don’t remember the name — the “frozen tundra” of where data lives. I don’t know necessarily where that is or how to access it, but I know we have stuff in there just in case.

John Wall – 04:48

Glacier, right?

Christopher Penn – 04:50

Amazon AWS Glacier. We’ll talk about that a little bit. One of the things I was putting together for today’s episode is full-on prepper tinfoil. Suppose all the cloud services and big tech companies just go kaboom. Maybe they were hacked, maybe they were rooted, maybe they ran out of money. What would you have?

There are three forms of media generally recommended for data prepping and data hoarding: SSDs (solid-state drives), regular mechanical hard drives, and optical media like CDs and DVDs. In terms of shelf life, if you were to just put an SSD in a box, the data will last about five years. If you plug it in and run diagnostics every so often, you can get more life out of it.

Typically, you can get five to 10 years out of a hard drive, especially if it’s not on all the time. If you have it in a box somewhere secure and you power it up twice a year to run a disk check, you can get 10 years out of it. Optical media that you burn yourself — like those of us who were 90s kids remember — will get about five years off-the-shelf, but it is very sensitive to light, heat, and moisture. If you keep it in a cool, dry, dark place, you can get five to 10 years out of that. There are some forms of archival burnable CDs and DVDs that are more expensive, but you can get 25 to 100 years out of them if properly stored.

Katie Robbert – 06:38

That goes into proper planning. I live in Massachusetts, and despite the amount of humidifiers I run, I can have dark places, but they’re not cool. They’re humid regardless of where I am in my house. That’s something I would have to factor in. Do I need to spring for an external storage place — maybe a storage box at an external location — because of the environment in my home? That’s the thinking ahead. My brain immediately goes to the fact that I don’t have any place dark, cool, and temperature-controlled. I have places that are dark, but they’re going to be warm and humid.

Christopher Penn – 07:22

It depends on the importance of the data. If it’s all your favorite MP3s from when you were a kid, you can just check it every year and make sure it’s still good. If it’s the financials of your billion-dollar company, you’re going to spring for the storage facility.

Katie Robbert – 07:46

You really do have to prioritize. When you start to think about data preppers or data hoarders, we tend to think everything is important. Not everything is important. You really have to focus. If you’re talking about your business, it’s a good opportunity to do a KPI map to see what data informs whether we’re meeting our goals. That’s likely the data you want to start with in terms of backing up most often.

John Wall – 08:22

I think we could get a Netflix special out of Data Hoarders. There’s something there, definitely.

Katie Robbert – 08:29

Directed by John Wall.

John Wall – 08:32

Open up the drive. What’s in there?

Christopher Penn – 08:35

That is a genuine thing. There are folks over on the Reddit DataHoarder platform who will buy old hard drives at yard sales just to see what’s on them. There are some surprises.

Katie Robbert – 08:49

I’m sure. I could probably give you at least three or four thumb drives out of the top drawer of my desk. I couldn’t tell you what’s on them. Guaranteed nothing salacious, but also nothing interesting.

Christopher Penn – 09:04

I picked up a flash drive after last year’s Fourth of July fireworks that was in the projector. The city left the flash drive behind on the lawn. On there were budget documents because they didn’t format the drive beforehand, which was fantastic.

We’re going to go through a lot of stuff today. This show will have an accompanying PDF if you want to get it for free. Pop by our free Slack group at TrustInsights.ai/analyticsformarketers. There’s a lot to go through. Any questions before I start foaming at the mouth?

Katie Robbert – 09:47

Not a question, but a comment: I will interrupt you to ask my questions when I have them.

Christopher Penn – 09:54

First things first: what software should you generally have to do data hoarding properly? There are about 10 pieces of software I think are essential. Some of these are command-line tools. If you are not comfortable with them, this is a great time to learn.

The first one is called W3M. This is a text-based web browser you would use with a tool like Claude Code. You can say, “Use this tool to go download this webpage or this website,” and it will know how to grab all that data and store it safely. If there’s a website or even just a page you really like, this tool will go and grab the actual HTML and store it locally on your computer.

Next, built into virtually every system, is a system called Git. Git is version control software. When it comes to data hoarding, you have to be aware that data can be corrupted online. For example, if you were to go to certain pages about COVID-19, those pages have changed in the last year and a half to have a completely different version of what happened. Version control software lets you essentially have a timeline. As you download things over and over again, you can see how things change. If you’re using any kind of agentic AI tool, please have Git installed. It will save your bacon. At one point or another, Claude or Open Code will do something and you’ll realize it is completely not what you asked it to do. If it compacts the chat, you can’t get the working software back because it wrecked it.

Katie Robbert – 12:04

We’re seeing Claude do that more often. If you have a long chat or a project that goes on over several days, it’s forcing you into compacting things.

Christopher Penn – 12:18

Exactly. Next up on the list is Transmission. If you do not have a BitTorrent client, this is one of my favorites. This allows you to get torrent files of all sorts of different media. If you look at the Internet Archive, you can download entire archives of the internet using torrents. Torrents allow you to do peer-to-peer file sharing where everyone shares the load of certain types of data. Earlier this year, when the U.S. Department of Justice released a whole bunch of files, one of the formats available was a torrent, which takes the load off of having one central file server. It forms a peer community.

John Wall – 13:14

Be mindful of your settings on that. Don’t open it up wide to let the whole world have unlimited data if you’re on a limited data plan.

Christopher Penn – 13:23

Exactly. Rsync and Rclone are command-line utilities. Rsync will synchronize two directories. Every Monday and Friday, I have a script on my computer that backs up my entire code directory to the Trust Insights Google Drive so that I no longer have to remember to back up mission-critical code. It does it automatically.

Rclone does that with cloud-based services like AWS S3. If you have data that you want to back up, these scripts — which a tool like Claude Code can write for you — will allow you to sock away your data on a scheduled task. You will never have to remember to run backups because it will just do it for you.

Next up is one of my favorites for travel: an app called Kiwix. Kiwix is a free, open-source set of applications that allow you to download big archives of data. The Kiwix reader is probably the best thing to use. It will download a two-gigabyte file on your laptop or phone, and then you can use all of Wikipedia inside the app. You can get a nine-gigabyte version of Wikipedia which will easily fit in a modern phone, and then it’s offline. If you’re on a plane with no Wi-Fi and want to read, it’s there for you. Public domain things like Project Gutenberg are also available. There are 100,000 works on Project Gutenberg, which are all public domain. It’s a great tool.

John Wall – 15:10

It looks like it has packages already. If you just want to grab all the TED Talks, can it do that? Is it all HTML or are there other things it can handle?

Christopher Penn – 15:21

It’s HTML and ebooks.

Christopher Penn – 15:24

The next one is an open-source package called Czkawka, which is Polish for hiccup. This is deduplication software, which is super helpful. If you’ve got big data archives — say your own photo library from the last 20 years — and you want to know which photos are duplicates, this is one of the best packages. It’s super fast, built in Rust, and open-source.

If you want to operate your own local file server, OpenMediaVault is great. If you bought a couple of eight-terabyte drives on sale, you can install your own networking software at home to have your own data hoard ready to go. Multiple devices on your intranet can access it. If you are going to be doing super heavy AI work, this is really helpful for grabbing large files on your local network rather than storing them all on your laptop, because your laptop will eventually run out of space.

If you like ebooks, Calibre has been around forever and can convert pretty much anything to an ebook. If you have HTML webpages, RSS feeds, Kindle books, or PDFs, it will tear them apart and reassemble them as ebooks. I use this software when we’re making Trust Insights ebooks because it does a great job of formatting. It can also manage your library for you. If you give it an entire website worth of stuff, it can download stories and turn them into nice ebooks that are totally yours.

Open Code is another one. If you are familiar with Claude Code, Open Code is very similar, except it’s not run by a corporation; it’s run by the community. It is a command-line coding agent that can hook into pretty much any LLM. This is a good one to have because Claude Code is supported by Anthropic. While it works great today, if Anthropic runs out of money, Claude Code stops working. Open Code will continue to work for as long as the community supports it.

You want to pair this with a tool like Ollama. Ollama is another open-source tool that serves AI models. You might download an open-weights model like Google’s Gemma or Alibaba’s Qwen. Depending on how much hardware you have, you could even put things like DeepSeek on it. Open Code plus Ollama gives you generative AI on your machine. If the internet blows up, you still have AI capabilities at your fingertips.

Those are the 10 core pieces I think everybody should have if they want to get into data prepping. Before I go into multimedia, are there any questions?

Katie Robbert – 18:36

My dog is now attacking me because I’ve been sitting at my desk for too long, so apologies for the chaos. These are really technical tools. If you had to narrow it down to one or two tools for a beginner to start with, what would they be? You’d want to think about your use cases and your 5P Framework. If I wanted to back up all of my web analytics data — which is a lot of information — where would I start?

Christopher Penn – 19:08

Web analytics data depends on whose system you have. If you have Adobe or Google Analytics, you’re going to be using an AI tool like Claude Code or Open Code to write software that will connect to those systems and download all your data for you.

Katie Robbert – 19:34

Where do I put it? Are these systems for extraction or for storage?

Christopher Penn – 19:42

This is mostly storage. We’re going to talk about extraction in just a second.

Katie Robbert – 19:47

If I was a brand new babe in the woods and I wanted to back up my Google Analytics data, how would I know where to start among those 10 options?

Christopher Penn – 20:03

I would start with Open Code and an AI model to write the software, because there is no out-of-the-box free product where you press a button to get your analytics data right now.

Katie Robbert – 20:17

I understand that, but you said these are storage systems. Take the AI out of it for a second. How do I know which storage system to choose? You gave me 10 options and it’s overwhelming.

Christopher Penn – 20:34

This goes back to the storage medium: hard drives, solid-state drives, and optical media. It depends on your use case. If you were just getting started, I’d say look for a hard drive or solid-state drive that’s cheap and at least one terabyte. Your analytics data is actually really tiny. When I downloaded all of the Trust Insights Google Analytics 3 data, five years worth of data was about 10 megabytes compressed. That fits on a thumb drive. When I downloaded my own data — 15 years worth — it was about 15 megabytes compressed. It’s super small data.

Katie Robbert – 21:18

Web analytics data was just one example. 10 options is a lot, and they all seem very technical. If I know nothing about any of these, how do I know where to start?

Christopher Penn – 21:38

You start with the 5P Framework. Based on that, you talk to your AI agent and say, “Here’s what I want to do. Here are a bunch of different options.” You can even take the PDF from this episode and ask, “What of this makes the most sense?” Be clear in your purpose and performance needs. Tell it you don’t want a cloud subscription or to pay more. It can help you understand the best choices or even do a web search to help line up the best fit. It always comes back to the 5P Framework.

Katie Robbert – 22:15

That makes sense. That’s more helpful.

Christopher Penn – 22:19

That’s a lot of the utility stuff: copying, deduplicating, and storing certain formats. Now, let’s talk about extraction. One of the best tools for video is a project called yt-dlp. You can ask Claude Code or Open Code to help you install this. This will download video, audio, or subtitles from over 100 different websites. Not just YouTube, but if you can see it, it can usually grab data from it if you have the appropriate credentials. You can download from Wistia or Vimeo as long as you have the rights to view the content.

This is helpful when there’s a video you want to save. For example, maybe you want to back up all of The Late Show with Stephen Colbert before CBS removes it from YouTube. You just give it the playlist and say, “Go and grab all the things.” For folks using AI, this is a phenomenal tool. If there’s a channel on YouTube with cool AI tips — maybe it’s the In-Ear Insights podcast from Trust Insights — you could give it that playlist URL and say, “Just grab the subtitles.” You’ll get hundreds of files that you can put into NotebookLM or the AI agent of your choice to have a knowledge base that’s yours. Even if Katie decides one day that we are now a pizza company and removes all our YouTube content, you still have those episodes stored.

Katie Robbert – 24:15

Don’t tempt me. That sounds great.

Christopher Penn – 24:22

If you have rich media in one format and want it in another, FFmpeg is a free command-line tool that converts pretty much anything to anything. Every week when I edit the Trust Insights podcast, I use this tool to immediately make the MP3. It’s written into a little script on my desktop. If you want to take really big videos and make them more compact — for example, if you made a copy of a Blu-ray DVD which is 28 gigabytes — you can use this tool to convert it to the smallest available file format. You’ll end up with a one-gigabyte file that looks great and works on your hardware.

If you have DVDs and other media, Handbrake will do exactly that. I literally did this not too long ago. For those who are 90s and 2000s kids, you know what the binder of CDs and DVDs is. Handbrake will convert them all into the digital format of your choosing, which makes it a lot easier to archive so you don’t have 80 of those binders laying around.

Katie Robbert – 25:50

Am I even really living if I’m not speeding down the highway with one hand on the wheel, trying to flip through my CD book to find the right tune and maybe not crash my car? Am I even living if I digitize everything?

Christopher Penn – 26:05

Honestly, you’re better off that way than some of the people I see driving while staring at their phones and not seeing the road at all.

John Wall – 26:12

Surfing Spotify.

Katie Robbert – 26:16

It’s not the same as seeing “Jams” written on the CD with a black Sharpie.

Christopher Penn – 26:26

If you want to play that media, Jellyfin is probably the single best local network tool to serve up your video, audio, and images. It’s completely local and open-source. It even works over Tailscale, so if you set up a Tailnet, you can access your home server while you’re on the road without opening holes in your firewall.

If you just want to manage images, Immich is a tool that runs on a local server or laptop. It’s like Google Photos without the Google part. Gallery-dl is similar to yt-dlp, except it takes any page worth of images and grabs them all. If you go to the Smithsonian Institution’s galleries, you can get a copy of all that. Obviously, please use these tools only for things you have permission for or that are licensed for you to do. Do not perform illegal activities.

Spotify has a downloader that will attempt to find the YouTube versions of your playlist songs and grab those. Most importantly for AI today, GitHub Desktop allows you to find code repositories and download them. If a project goes away or the maintainer stops supporting it, you still have a copy. With your own AI coding tools, you can make your own changes. There are so many good pieces of software that are open-source projects; I grab a copy once a month just to make sure I still have them.

Are there any questions on extraction before we move on to formats and data sources?

Katie Robbert – 29:02

No, let’s keep rolling.

Christopher Penn – 29:04

There are three formats you should really care about. Number one is MBOX, which is the UNIX mailbox format. Tools like Outlook have their own proprietary formats that make it impossible to use them a couple of years later because they keep changing. If you had email from 2007, you often can’t open that in Outlook today without several levels of conversion. Apps like Thunderbird store email in MBOX format, which is a plain text archive. I have MBOX files from college in 1995 that are viewable today because they’re just plain text. One of the things you want to do is have your email in a system like Thunderbird so those files are usable in 100 years.

The second format you should know and love is Markdown. Markdown is a plain text format that has styling information but remains plain text. Today’s show notes are in Markdown. It’s text with little marks, and you can see what it looks like when it renders. You’ll still be able to open Markdown in 100 years.

The third format, particularly for data and AI, is YAML. It stands for “YAML Ain’t a Markup Language.” It’s a way to represent tabular data in a linear format so AI can read it more easily, because AI often struggles with tables. It also allows you to store data in a very compact format.

Finally, there are seven places I like to have copies of data from. Number one: Wikipedia. You can download all of Wikipedia, and it works out to about 115 gigabytes. You can get a one-terabyte flash drive now, so this is not a burden. It’s not a bad idea to have that available offline.

The biggest archive is archive.org, the Internet Archive. Please donate to them; they are community-funded. If there’s any page you want to look up, you can see it. You can even see how the Trust Insights website has changed over time. The third is the Kiwix archive. These are modules like all of Wikipedia, the Bitcoin Wiki, or TED Talks that you can download into the Kiwix reader on your mobile device.

Fourth is Hugging Face AI models. You want to find the models that make the most sense for you. There are 2.9 million AI models out there. All the ones on Hugging Face can be downloaded and installed on your computer. This is where models for Ollama come from. There are models for image recognition, audio, video generation, and OCR. Once you download it, it’s yours. If the company goes out of business, you don’t have to worry about them changing the model on you. Enterprises love stability, and while AI typically doesn’t deliver that, using open-weights models allows you to standardize on something like Gemma 4 and run it locally. You don’t have to change until you are ready.

Katie Robbert – 34:30

If you take nothing else away from this episode, that is the best pro-tip for companies exhausted by the constant functionality changes of large language models. You don’t have control over how or why they change, but if you find a version that works well for you, download it and use it. You don’t always have to have the latest bells and whistles to do your job effectively.

Christopher Penn – 35:06

Exactly. The last two worth pointing out are Project Gutenberg, which has 75,000 books in the public domain, and OpenStreetMap. OpenStreetMap is a free alternative to Google Maps. You can download a snapshot of the entire planet — about 162 gigabytes — and have a map for any place on Earth. If Google decided to charge everyone $3 for maps, you could just use this instead.

Katie Robbert – 36:00

How does that stay updated?

Christopher Penn – 36:03

The OpenStreetMap database is a foundation. People submit corrections, there’s a whole team behind it, and they use publicly available data from various governments.

Katie Robbert – 36:24

I’m realizing through this episode that I am not a data hoarder or a data prepper. I’m happy to let a lot of stuff go. I’ve used the Wayback Machine primarily when I forget to take “before” shots of pages I’ve changed. Going through the 5P Framework to find your “why” is essential. This takes time and resources to maintain. Make sure you’re clear about why you’re backing up data in the first place. I can’t think of much in my life I’d want backed up besides my wedding photos.

Christopher Penn – 37:43

There is value as a private citizen to having a text-only version of Wikipedia, which is about nine gigabytes. Storage is cheap. In the event that media companies start trying to rewrite history, you can say, “This is canonically what the point of view was at this point in time.” From a library sciences perspective, this will be useful in 20 or 40 years.

At any point, a cloud service can pull the rug and either charge you to get your data or just make it go away. Flickr did a rug pull years ago and started charging much more for the same features. If you look at Google Photos, it is free today, but will it stay that way? AWS Glacier is low-cost today, but we don’t know if Amazon will be here in 10 years. It’s never a bad idea to have multiple copies of important stuff.

Katie Robbert – 39:43

So you either are a data prepper or you’re friends with a data prepper and you let them do it for you.

John Wall – 39:54

It’s the “friend with a boat” model.

Katie Robbert – 39:58

John gets it. You’re the friend who brings the baked goods and the wine, and you let the other friend do all the data prepping.

John Wall – 40:09

Setting up a media server to access music in every room is a rabbit hole. Sometimes it’s a good thing, and other times you’re like, “Maybe I’ll just switch back to CDs.”

Christopher Penn – 40:38

Gen Z and Gen Alpha folks talk about subscription overload. An entire generation has realized they don’t own anything. When you stop paying for Apple Music or Spotify, your stuff goes away. If you are doing the data hoarding thing and making copies, then your music doesn’t vanish. These are challenging economic times. If you have the ability to do this now while the content is available, it pays off. Some shows are no longer on streaming services. If you bought the DVDs and ripped copies, you can still watch them. For everyone else who doesn’t own anything, those shows are just gone.

Katie Robbert – 42:04

I can see the use cases. It’s fascinating. You could prep everything, but you might also look for organizations already doing that work and participate by donating or getting involved so you don’t have to take on all the storage yourself.

Christopher Penn – 42:52

The more data you bring to AI, the less it hallucinates. If you have good quality data stockpiled, you don’t have to go looking for it. I have often dragged a PDF out of my archives and dropped it into Claude to show it how to do something. The same is true of the information products you’re making with generative AI. Don’t just delete the chat and throw away the work. You burned tokens — which is electricity and water — to make that output. If it’s useful, store it so it can be reused without burning more resources.

Katie Robbert – 44:06

Before this livestream, I was talking about finding efficiencies and scalabilities with systems like Claude. The big takeaway was making sure you’re doing those requirements and the 5P Framework upfront. You need to build reusable skills, like plugins and connectors, before you unleash the whole team. I’ve already built the SOW creator so John doesn’t have to recreate that every time. He’s saving tokens, usage, and the planet.

John Wall – 44:55

Free tokens. That’s what I’m all about. Ever since Chuck E. Cheese, I’ve known.

Christopher Penn – 45:02

Yesterday, Kelsey and I were working on Google Apps Script and we built a rule set. We told Claude how to write Google Apps Script that goes right into our Google Drive for the company so everyone can use it. You’re not constantly reinventing the wheel. Reinventing the wheel is one of my biggest bugaboos because it’s inefficient and wasteful.

Katie Robbert – 45:33

I agree.

Christopher Penn – 45:38

That is getting started as a data prepper. All the tools we covered today will be in our Analytics for Marketers Slack group. You can join for free at TrustInsights.ai/analyticsformarketers. Any final thoughts?

Katie Robbert – 45:59

Be clear on your use cases. If you try to save every piece of data, it’s going to get overwhelming quickly. What’s the point if you’re not maintaining it or checking on it? Build that into your planning.

John Wall – 46:20

Test your backups. That’s mine.

Christopher Penn – 46:26

Exactly. Thanks for tuning in, folks. Be sure to subscribe to our show and check out the Trust Insights podcast and our weekly email newsletter. If you have questions, join our free Slack group. See you next time.