So What? Local AI Model Roundup

Christopher Penn – 00:00

Happy Thursday, everyone. This is So What?, the Marketing Analytics and Insights live show from Trust Insights. Katie is staring at the mountains of Vermont. Does Vermont even have mountains?

John Wall – 00:37

Vermont, Green Mountain—that’s what it means.

Christopher Penn – 00:39

I mean, yeah, but are they actually mountains?

John Wall – 00:41

Yeah, there are a few well over five, I think, but Bromley is over 8,000.

Christopher Penn – 00:47

I always think of a rolling hills person, but okay, the green hill. I’m Chris. We have John here, and Katie is out staring at apparently actual mountains. This week we are talking about the local AI model roundup. This is something that is by popular request.

In fact, Brian in our Analytics for Marketers Slack group—if you are not a member, it is Trust Insights AI Analytics for Marketers—was asking what is the latest and greatest about local AI models. Before we begin, John, when you are looking at the AI world, do local models even factor in on your radar? Or are you strictly cloud-only? Or are you like, “You know what? I’m just gonna bypass this whole AI thing.”

John Wall – 01:37

No, cloud is good enough for what I’ve seen. In fact, I’m not gonna lie about this, Kelsey. You and Katie are like, “I need this fixed. Make this happen,” and you guys just show up with the finished file. It works because you guys have been testing and playing with stuff. The stuff I’ve been seeing in the last two or three months has just been amazing.

Cloud is definitely enough for me. For everything I’ve seen, it’s basically if you get to a point where you’re having to ration your work because every week you’re burning through all your tokens, that’s when you really want to start looking at this. Or, of course, security is the other thing if you don’t want to go to the outside world. That would be my reason because there are a lot of things that I don’t do because I’m not putting my data in there. I don’t want that to go there.

That’s part of the mix, but I’m excited about this because it is the same thing with every new technology. You can play around with it and learn how to use it and do different stuff, but until you cross that line of actually setting up and configuring it yourself, that’s a whole different level of learning how things work and being able to do stuff too. It opens your eyes to possibilities when you have to dig down in the blood and guts.

I’m super thrilled most of all because normally this is the kind of thing that people just have to do at 2:00 in the morning by themselves and find help in Reddit groups. It’s completely and entirely painful every step of the way. I’m getting a guided tour here, so I’m thrilled to death about this.

Christopher Penn – 03:08

We should probably do a bit of table setting. We’ve done it in the past on past episodes of the show, which you can find at the Trust Insights AI YouTube channel. You can look for local models. We should differentiate because there is a bunch of terminology: cloud models, open weight models, closed weight models, and local models.

Let’s talk about open versus closed first. Closed weight models are models where you are not allowed to download them. They are locked behind closed doors. This is like Claude Opus, OpenAI GPT-5.5, or Google Gemini. These models are good, but you don’t get access to them. You can buy API access, but you will never have them on hardware of your own. They are closed and under lock and key.

Open models are models where the training weights are available. When we talk about models, all AI models are basically made of statistics and that big pile of statistics are called weights—basically the probability of this or that. It is almost like a big spreadsheet. OpenAI, Google, and Anthropic are not giving you the spreadsheet.

Whereas open weights models like Kimi K 2.6, Minimax, DeepSeek, and Alibaba Qwen have said you can have the spreadsheet. You can have it and then, if you want, you can make changes to it. That is the difference between closed weights and open weights. With open weights models, you are allowed to download them and modify them. You don’t get the training data they were made with, but you do get the spreadsheets they use to assemble things.

In terms of cloud versus local, for today’s show, “local” means you can download it and run it on hardware that you own. This is going to vary based on how much money you spent on hardware. At the low end, there are some models that can run on things as small as a phone. On the high end, you could blow 50 grand on a DGX supercluster in your house.

There’s this one dude on YouTube who has 100 grand worth of Mac Minis all chained together so he can run DeepSeek at home. I’m like, “Well, it’s your money, man.” Whereas “cloud” means the model is being hosted somewhere else that has the computational facilities.

This is where it can get messy because open weights models can be served up by third parties called inference providers. Companies like Groq with the “q,” Cerebras, and Deep Infra download open weights models and use several million dollars’ worth of hardware to host them. You can use their hardware essentially like hosting your own website on their service. You’re still using an open weights model, but you’re just not putting it on your machine.

Today we’re talking about local models that you can run on nice hardware—a nice MacBook with decent memory. Essentially, if you can play Call of Duty at full ultra resolution, you can run local models pretty easily.

John Wall – 06:45

When you’re buying this, you’re pretty much going to dedicate this machine to it, right? If you’re going to be doing that much stuff, do we assume it’s just going to be running all the time and you’re not going to be able to get any other use out of this box?

Christopher Penn – 07:00

It depends on how nice a computer you have. If you have a top-of-the-line MacBook, you could be doing other things. The fans will be spinning, but you can do other things. On the other hand, if you’re doing it on a MacBook Air, yeah, that is all that computer is going to be doing. There are a lot of options for that hardware if you want to invest in it, but from what I’ve seen, you’re not going to find anything less than $4,000 for a device because GPUs are in very short supply.

Every AI company on the planet is trying to buy more of them—good old economics. In general, closed models are always cloud because you’re never going to get a company to release those. Open models can either be cloud or local. Today we’re going to be focusing on what those are.

When we talk about local models, they come in a bunch of different sizes. All local models are measured in a size called parameters. This is basically how big that spreadsheet is. In general, when you look at the parameter counts, the easiest way is to look at how much disk space it takes up. That is how much memory it’s going to need, plus an extra 20%. If you go to a model like DeepSeek and see it is 500 gigabytes of disk space, you’re going to need a lot of memory to run that thing.

The next thing is how do you know which model to use? Spoiler: if you go to Hugging Face, which is probably the planet’s best repository of models, and you tap on the model selector, you can see that as of today, there are 2.87 million models. Just a few. We see everything from Supertonic to Unsloth models with MTP and all these really cool things with lots of jargon in them.

In general, there are some really good review sites. One of my favorites is called Artificial Analysis, which looks at the big families and the headline models. First, we have to figure out what you want to use it for. There are models for text, for coding, for writing specifically—especially if you’re doing fiction writing—for running agents, for generating music, for generating the spoken word, and for doing speech recognition and transcription.

There are all these different things you want to do. The first part is figuring out what you would be doing. That goes back to the 5P Framework: what do you want to do? There are different models for different things. When you’re thinking about how you might use local AI, John, of all the different use cases, which ones make you say, “Oh, yeah, I want to do that”?

John Wall – 10:15

Text analysis is the most useful stuff so far. We get transcripts of reports and then we’ll get requests for proposal. We have our scopes, our agreements, and all of our processes. Getting all that data together and synthesized so it can say, “Hey, here’s a scope of work that’s done,” is a useful application for us.

It seems like all this model selection stuff is all about the use cases. You need to know which one goes with what you want. Is it the same with normal software in that there’s a range of models that have been running for a year and are pretty much error-free versus one that was released yesterday that might burn your whole house down? You have to know how much you can stomach.

Christopher Penn – 11:08

Essentially, yes. The extra challenge is that as new technologies come out, model makers adapt them very quickly. If you say, “I’m going to standardize on this model and not change it for a year because I don’t want to deal with the crazy pace of change,” and then someone comes up with a new paper that says they can make your model six times faster, you’re wondering if you should change.

Text analysis is probably the easiest one to get started with and there are a lot of choices. Text analysis and coding are probably the two most useful things. If a model is good at both of those by default, it is also going to be good at being what’s called an agentic model—a model that can run agents.

There are two more considerations in addition to the model. The model is like the engine of a car. There is the harness, which is the rest of the car. Depending on the kind of harness you have, sometimes you can put in a smaller engine and still get good performance. If you’re driving NASCAR, you probably need the beefiest engine you can cram into your Honda Civic.

If you’re just going to the grocery store, you don’t need an eight-cylinder engine. You can probably run on a two-cylinder engine because you’re not going to be going 89 miles an hour. That means the container we put the model in is just as important. These are containers like Claude Code and Claude Cowork.

With local models, there’s an additional twist: you need a server. You need a piece of software that will serve up the model, which you don’t have with cloud providers because cloud providers are the servers by default. These would be tools like LM Studio, Ollama—which is very popular because it’s so easy to use—vLLM if you’re a company and you want to make a model available to multiple employees, or AnythingLLM.

There is the model, there’s the server, and then there’s the harness that uses the model for things like text analysis and loading documents. The better your server and harness are, the smaller, lighter, and faster a model can be. One of the catches is the bigger a model gets, the smarter it gets, but the slower it gets and the more memory it takes up.

John Wall – 13:53

So there’s a lot of optimization going on. You kind of need to play around and see what kind of results you get. I understand that the harness is actually calling all kinds of stuff and has the UI. Are there other functions in the harness that you have to worry about or tweak?

Christopher Penn – 14:10

The things that it connects to are key. For example, if you’re using a harness like Open Code or Claude Code, you might have web search utilities or knowledge graphs. These things will make AI function better because they provide more guardrails. A great harness has lots of guardrails built into it that you configure.

When you’re using Claude Code for Anthropic, it has thousands of guardrails and little tricks to make things work better. Whereas if you were to use the model raw off of an Ollama server, it’s going to go pretty wild and be very unpredictable. It’s like a tornado in a factory. The more rules you have and the better the harness is, the better the model is going to perform.

John Wall – 15:03

That makes sense.

Christopher Penn – 15:04

In general, what we recommend if you are just getting started with local models is Ollama. The free open-source package is probably the easiest server to use. Then, pick a robust harness. For most people, that’s going to be Claude Code or Claude Cowork. The Claude desktop app is probably the easiest one to use because so much has been done to optimize it and make it as non-technical as possible. It’s terrific.

That brings us back to the model. In general, for a tool like Claude Cowork, you want a model that is the most skilled at agent tasks because that is really what that system is. If we go back to the Artificial Analysis bar chart, they have three indices: how smart is the model, how good is it at coding, and how good is it at running agents?

Currently, the absolute best model—and it will cost you an arm and a leg to run it as an agent tool—would be OpenAI GPT-5.5. It does things really well and it spins the meters as your bill goes up and up. The first model on here that is an open weights model is Xiaomi MIMO 2.5. However, this is not a local model because it requires a lot of hardware to run. You’re talking 10 grand.

The first model on here that you could realistically run on a laptop is Alibaba Qwen 3.6, the 27 billion parameter model. This came out three weeks ago and it is an absolutely incredible model that will run on a decently sized MacBook. If your MacBook has 64 gigabytes of RAM and a modern processor—M2, M3, M4, or M5—you can run this and it will do really well. It will be nice and fast.

The next model you could run conceivably on a laptop would be the Qwen 3.6 mixture of experts model. One of the things I think is worth pointing out on this index is that I put last summer’s OpenAI GPT-5 flagship model on here. This was the state of the art, and you now have models you can run on a laptop that beat the pants off of that super big, expensive model from last summer. That is how far these things have come.

John Wall – 17:53

That’s amazing. I was noticing it is not even a 15% difference based on this scale between what you can run on hardware and the best thing you can buy with unlimited hardware. That’s crazy.

Christopher Penn – 18:07

I also have Nemotron Super, which powers OpenClaude and Nemo Claude. Google Gemma just came out not too long ago. But really, the Alibaba Qwen model family is the most accessible. This is an important point because a lot of people get this confused. Yes, it is made by Alibaba. If you use Qwen in their web interface in the cloud, that cloud runs inside the People’s Republic of China.

If you work with any kind of sensitive data or private data, you should not use that cloud with it. However, if you download the model and run it on your own hardware, it is perfectly safe to use. You will still get some refusals—like asking about what happened in Tiananmen Square—but that is no different than the censorship in any other model. There is a difference between a Chinese cloud model run in China—don’t use their cloud—and a Chinese model you run on your own hardware. Totally safe.

John Wall – 19:18

All right, so Qwen is our choice then. We’re going to fire that up and see what we can do.

Christopher Penn – 19:23

Now we get into even messier stuff. You have to decide what size you want because there are many different flavors. If I go into Hugging Face and type in Qwen 3.6, there are 1,949 choices.

John Wall – 19:49

How does that work? Is it open source too, or is there the official version that comes from the maker and then all these forks?

Christopher Penn – 20:01

There is the official version. You can see the 27 billion parameter model and a 35 billion parameter model from Qwen AI itself. Then you have all the variants the open weights community has made that have been tuned for specific architectures or specific things.

For example, there is one here from a group called Unsloth. This is two Korean dudes who have come up with an absolutely insane laboratory where they take models and do what’s called quantization. They make them smaller and lighter while trying not to reduce their accuracy. For the most part, they do such a good job that their quantizations are better than the manufacturers’.

You can run a model modified by these guys and it’s faster, just as smart, and takes up less memory. They have started implementing some really cool new technology. If you’re going to use a local model, see if Unsloth has a version of it. That would be my tip there.

John Wall – 21:17

So the basic idea is they’re optimizing for hardware that isn’t the best on earth? They’re making a human version?

Christopher Penn – 21:25

That’s a good way of putting it. They’re basically taking a model and trimming the fat because there is a lot in the base model that you may not need. Their stuff is very trustworthy and high performing. When we talk about quantization, it really is like taking a model and applying compression to it, trimming out stuff you don’t need.

There are specialized tools for that. Your average person is not going to do that, but you do want to look at the different versions. You will see all the different quantizations and they get smaller in file size as the numbers go down. But as the numbers go down, they get dumber. The smartest version in this folder is going to be the 8-bit version at 39 gigabytes. You’re gonna need a lot of memory to run it, but it’s going to be absolutely pristine in its capabilities. Once you get below four bits and get into the three, two, and one bits, they start getting really stupid. They’re fast, but they’re stupid.

John Wall – 22:40

So you would only use that for some specific, very narrow use case.

Christopher Penn – 22:47

Or you would use their studio software to tune it for just one specific task because a lot of the knowledge has been lost. For the text space in general, take a look at Artificial Analysis for the family of model you’re thinking about using and see how it scores on the specific type of tasks you’re doing.

The other thing you want to look at is “omniscience,” which is how badly does this model make stuff up. Lower bars are better. For example, DeepSeek GPTs hallucinate 95% of the time. The higher the hallucination rate, the more you need to be bringing your own data to the party.

It is astonishing when you look at the Qwen 3.6 models—at 48% to 50%, that is half of what a big model like DeepSeek or OpenAI GPT-5.5 hallucinates at. None of these are still reliable enough that you would want to use them without a harness that incorporates a web search so that when the model says something, it can go check that out or make sure you are providing all the data.

As a simple example, let me show you this. I’m going to use Qwen 3.6 and turn off web search. This model can only rely on its own knowledge. I’m going to say, “Who is Katie Robbert?” It’s going to go through its logic and it completely hallucinates this. I don’t even know who Laura Thompson is.

John Wall – 24:58

Right.

Christopher Penn – 25:01

Completely wrong because it doesn’t have access to outside knowledge. If I turn web search back on, it will think and say, “Maybe I should search this.” It knows web search is available. It then says Katie Robbert is the CEO and co-founder. What’s interesting is the language is the same—it knew enough from its training data to know the title, but it still 100% needed web search to pull this off.

That’s when we’re talking about local AI. You have to have a harness that incorporates things like web search and background data because otherwise it’s going to make things up.

Another score to look at is called GDP Val: can a model execute an economically valuable task? This is where agents land. Last summer, GPT-5 scored a 1,292 on this test. If we look at the results here, Qwen 3.6 scored 200 points higher. The little model that you run on your machine scores significantly higher. This is insane that it has these capabilities.

We use the Singapore-based Minimax which, because it’s a cloud provider, is a Chinese company based in Singapore. You should not put sensitive information into that, but we use it a lot on their cloud because it requires a lot of compute.

Text generation aside, there’s a whole bunch of other models out there. We have a handy page on the Trust Insights website in the Instant Insight section on which models you should use today. In general, for open weights, if you have the hardware, use GLM 5.1. If you don’t have $10,000 worth of hardware, use Qwen 3.6, the 27 billion parameter model. It’s so smart and it’s a great coding and text model.

For video generation, the current best model you can download and run is LTX 2.3. It’s not great; it has some work to do. For image generation, the best local model is Black Forest Labs Flux 2 Dev. This model is very capable and generates decent images. I can demo this one using a provider called Deep Infra. They’re one of those cloud-based providers. We’ll use them today because it’s way faster than waiting for this stuff to start up on my laptop.

Let’s look at Flux 2 Dev and put in: “A photorealistic image of a CEO and their sidekick in a modern office conference room.” I’ll use 1,024 by 768 landscape format and hit go. It came up with a dude and a dude with a man bun.

John Wall – 29:01

Man bun, all right.

Christopher Penn – 29:05

It’s a choice, but for an open weights model, that’s not terrible and you can run it on your computer.

For text-to-speech, one of the shocking new ones in this space is Supertonic. This model just came out a couple of weeks ago and it runs entirely locally with 10 different voices. I generated a very short clip last night: “The world’s best marketing podcast with Christopher Penn and John Wall. You’re listening to Marketing Over Coffee.” That’s not terrible for something that runs entirely locally.

If there is text content you want to turn into spoken word, the best in class is Google Gemini Flash 3.1, but you get charged for it. When we did the Generative AI for SEO and PPC book, we made an audiobook version that cost about 10 dollars. That’s not a huge amount of money, but if you have a lot of text, that adds up. Supertonic is pretty terrific, but the downside is you have to use it with Python code. It doesn’t have a harness like Claude Code; you have to build the harness yourself.

John Wall – 31:17

How much of a lift is that? Are you just throwing commands in the IDE or do you have to configure other stuff?

Christopher Penn – 31:27

It has robust documentation. You would typically use the 5P Framework to build a project plan: I need a Python application that will use Supertonic to render this text. I’ll put the text in this folder, and here’s how you process it. Here is who the audience is and how you’re going to clean up the text first to make it speakable. Then you check how you did it right by giving it to transcription software like Parakeet to see how many words it screwed up. You can use these two kinds of tools in parallel.

John Wall – 32:17

I didn’t understand where you were coming from originally. It’s not that you drive it with Python; it’s that you need to use Python to build the thing to make it run.

Christopher Penn – 32:27

That’s right. But if you do, it is pretty fantastic. I was using it earlier today for a client who wants to turn thousands of pages into audio. That would be a bill of hundreds of dollars in the cloud. Instead, I can put their page content into the Python app and just make MP3s at very low cost.

The best current music generation model you can run is called Ace Step, and it’s not great right now. However, it is better than open weights models used to be, which sounded like cats screaming. Ace Step sounds like where Suno was two years ago. Suno is currently best in class but it is cloud-based. Ace Step is limited only by your computational power.

For speech recognition, one of the best and fastest models is Parakeet by NVIDIA. This is one of those rare ones where the open weights choice is also the first choice. It is insanely fast. I use it in an open-source tool called TypeWhisper. If you are a Mac owner and want nice text-to-speech transcription for free, this is the tool to use. You download it and specify the Parakeet engine.

If you’ve seen Whisper Flow or Super Whisper where people pay 20 dollars a month for dictation, you could be doing this for zero dollars a month. Install TypeWhisper, which uses the Parakeet model, and enjoy the same utility. Version 2 of Parakeet is English only but has extremely low word error rates. Version 3 has slightly higher error rates but is multilingual and speaks eight different languages.

John Wall – 36:03

That opens doors for agents as far as translating things.

Christopher Penn – 36:09

For agentic operations, Qwen 3.6 is the open weights model to use. It is a peer of Claude Sonnet, which is Anthropic’s second-best model. If you look at Claude Opus 4.5, which was best in class in January, this model is better than that. It is difficult for people to wrap their heads around the fact that it beats a foundation model from four months ago.

If you use a system like Hermes Agent or OpenClaude, you would run the Qwen model locally in a tool like LM Studio. You would hook your agent into this and, instead of spending a gazillion dollars on Anthropic costs or token plans, it’s just the cost of electricity. I’m using the Qwen 3.6 mixture of experts model—the 35B model—and I can run my Hermes agent off of it and spend no money at all.

Remember last week when we talked about using Hermes Agent for sales prospecting? I could power that whole thing with a model running on my laptop and have it working 24/7 just doing sales prospecting for me.

John Wall – 38:31

I can see that. You just have the thing running in the background and as long as it’s plugged in, you’re good. You’re not paying huge expenses the whole time.

Christopher Penn – 38:40

Exactly. So that’s the local model roundup: Qwen 3.6 for text and agents, Flux 2 Dev for images, Supertonic for text-to-speech, Ace Step for audio, Parakeet for speech recognition, and LTX 2.3 for video. This list changes weekly.

Keith is asking a basic question. Keith, feel free to ask anything. John, our friend Tom Webster bought a DGX Spark, which is a high-end piece of AI hardware for about $5,000. Do you see yourself buying hardware like that to run your own models?

John Wall – 39:48

No. If you’re talking about five grand, you have to ask how long it’s going to take to burn a thousand bucks in the cloud. I would have to hit a use case where I was burning tokens at an insane rate. Five grand is not play-around money for me.

Christopher Penn – 40:44

In the early days of OpenClaude, some folks got bills from Anthropic for 8,000 to 10,000 dollars because the agents went crazy. That five-grand box suddenly looks real cheap.

John Wall – 41:01

Keeping your job suddenly looks awesome.

Christopher Penn – 41:04

Keith’s question was, “I often hear about protecting data, but how do I use, formulate, and organize that data?” That’s a data governance question. You have to have your data in formats that generative AI tools can read, especially local models because they’re not as smart as their cloud counterparts.

Text files are your friend, especially in markdown format. If you’re doing tabular data, use a format called YAML—rhymes with camel. It takes a CSV file and turns it into a long list that an AI model can understand. Avoid any file format that requires you to open an application other than a text editor. If you have to open Microsoft Word, PowerPoint, or Photoshop, it’s not a good format for AI. If you can open it in Notepad, it will work for you.

John Wall – 42:40

There are tons of markdown tools. It’s not difficult to get to .md.

Christopher Penn – 42:45

Exactly. And if you don’t have any, ask Claude or the agent of your choice to write you one. John, any final parting thoughts?

John Wall – 43:01

It’s interesting to think about use cases, but for me, 5,000 bucks is still too much.

Christopher Penn – 43:11

One thing for an organization to think about is that you might spring for that box and use a technology called Tailscale. It’s in the Trust Insights newsletter this week at trustinsights.ai/newsletter. Tailscale allows you to create a VPN no matter where you are in the world. We could have the box living in my basement or Katie’s basement, and we could all connect to it as though we were using a cloud-based server. If Anthropic went bankrupt or OpenAI went out of business, we would still have options.

John Wall – 44:13

That’s a super idea. Networking a single box like that makes it a fractional cost. That’s a real interesting proposition.

Christopher Penn – 44:22

The other thing is a harness called Exo. It looks on your LAN and uses all the GPUs of the computers on it. If you had four people with nice MacBooks, you could share the load across all the machines to load really big models.

John Wall – 45:08

That’s nuts. I imagine there’s some crazy setup where you could buy 65 Xboxes and put them in a closet, but there’s always a commensurate level of headache with a lower price.

Christopher Penn – 45:23

Some YouTubers have 10 Mac Studios all chained together in a closet with a cooling fan. It’s pretty crazy.

John Wall – 45:37

I won’t be heating my house with servers this winter, but we’ll see how it goes.

Christopher Penn – 45:44

Apparently, that is a thing in France. French AI companies give people wall-mounted units with a couple of GPUs. People use them as literal space heaters in their houses while the company uses them for distributed processing. It’s a fantastic idea to use the waste heat to heat a French farmhouse.

That’s going to do it for this week. Next week Katie will be back and we are doing “Getting Started with Paperclip, the AI Agency.” Until next week, thanks for tuning in. Be sure to subscribe to our show, check out the Trust Insights podcast at trustinsights.ai/tipodcast, and our weekly email newsletter at trustinsights.ai/newsletter. Join our free Analytics for Marketers Slack group at trustinsights.ai/analyticsformarketers. See you next time.