Responsible AI Part 3

Our Trust Insights RAFT Responsible AI framework asks every organization to “minimize bias as much as possible” under its Fairness pillar. That language sounds responsible. It also sets no threshold. A standard without a threshold cannot fail. A company can always claim it tried its best. No one outside that company can prove otherwise.

Fairness needs a test, not a promise. This is the third piece in our four-part series on RAFT: Respect, Accountability, Fairness, Transparency. Part 1 examined the values your AI must honor. Part 2 examined who owns the outcome when AI gets something wrong. Fairness asks a harder operational question: how do you measure bias, and how much of it can you tolerate before you call a system broken?

I want to borrow an idea from publishing to answer that question, because publishing solved a version of this problem long before anyone trained a language model.

A Sensitivity Reader for Your AI

Publishers who work across cultures hire sensitivity readers: people paid to read a manuscript from the perspective of a background different from their own, watching for anything unintentionally harmful. Every organization deploying AI needs the same function, whether a person or a process fills it. It can catch the blatant problems, like slurs, and the subtle ones, like every example in a document using only male-coded names, a blind spot that seems harmless until someone notices it. The goal is to reduce implicit bias, not eliminate it, because you can never get rid of it completely. Walking through the desert, you’re going to get sand in your shoes. The goal is as little sand as possible, not none.

Katie has lived this firsthand. She once asked a language model, fine-tuned on years of her own writing, to draft content under her name. The model filled it with anecdotes about shoe shopping, closets, and mall trips: details she has never written about, chosen because the model associated a feminine name with those interests. As she put it on our podcast:

“It’s kind of disappointing that the large language model, despite having probably years’ worth of examples of my writing with no examples of those specific anecdotes, made those choices and assumptions about me because I have a feminine name.”

The failure doesn’t announce itself. A sensitivity reader, human or automated, is what catches it before a customer does.

Notice what that framing doesn’t do. It doesn’t name a number. RAFT’s Fairness pillar deliberately avoids prescribing a universal bias score, because your risk tolerance depends on your industry, your audience, and what you’ve publicly promised about your values. A healthcare AI vendor and a meme-generation app shouldn’t answer to the same curve. Set your own bar, write it down, and hold your systems to it consistently. That standard beats borrowing someone else’s number and hoping it fits your situation.

What a Rubric Can Look Like

At Trust Insights, we built a scoring rubric using our PAIR framework, asking a model to identify what a professional sensitivity reader looks for: slurs, cultural appropriation, and quieter problems, like every example in a document skewing toward one gender. We score content against that rubric, and anything below 90 goes back for revision.

That number, 90, is ours. It reflects what we’ve decided we’re willing to tolerate, and your organization’s threshold may land somewhere else entirely. The rubric matters more than the specific number attached to it. Build one, apply it to every piece of content consistently, and revise it as you learn where your gaps are, not where a published paper says they should be.

If you need a lower-effort starting point, check whether your tools autocorrect the names of the people on your team. As Katie has pointed out, that’s a genuinely useful gut check: a system that consistently “fixes” an unfamiliar name into something more common is telling you how much bias runs underneath, and if it does that, you know the problem goes deeper. That check takes a few minutes to run, before you build anything more formal.

The Larry/Lena Test, Now Backed by Published Data

A rubric tells you what to look for. The Larry/Lena test tells you whether your systems actually treat people differently, and it takes minutes to run. I described the original experiment on our podcast:

“We did a test with a sales example, a marketing example, and an HR example where we had the exact same prompt, but we just changed one word. We changed the word Larry to the word Lena, and we got very different results, including results that were substantially more condescending or derogatory for the female-coded name.”

That informal test turned out to be a preview of something we later measured rigorously. Katie and I published “Gender Bias in LLaMA-3 Embeddings: Implications for LinkedIn-Style Retrieval Systems” (TrustInsights.ai, December 18, 2025), testing 406 paired LinkedIn-style posts with identical content but different author names.

We found systematic bias: a mean cosine similarity of 0.994 instead of the expected 1.0, roughly a 0.6 percent embedding deviation, but a large effect size (Cohen’s d = -0.93, p < 0.0001). Small numbers compound. Across a retrieval system serving millions of professionals, that differential treatment adds up. We published the dataset, code, and methodology so anyone can run this audit themselves. Full citation: Christopher Penn and Katherine Robbert, “Gender Bias in LLaMA-3 Embeddings: Implications for LinkedIn-Style Retrieval Systems” (TrustInsights.ai, December 18, 2025), DOI 10.5281/zenodo.17982122, available at https://zenodo.org/records/17982122.

That effect size is not a rounding error. It’s evidence that identical content earns different treatment based on the author’s name alone, at a scale most teams never think to measure. Running this test yourself doesn’t require access to embeddings research. It requires a habit: test qualitatively with name-swap prompts, but also use real mathematical methods, like cosine similarity of embeddings, for any system you build.

Your vendors should be providing the same due diligence: has this model been tested, and what’s the severity of the bias found? In the EU, that disclosure is mandatory. Test against protected classes specifically: age, gender, ethnic identity, sexual orientation, veteran status, and disability. Most jurisdictions define these already, and your AI can’t discriminate on those grounds either.

Try this yourself: the same logic extends to image generation. Prompt a model for a senator, a doctor, a nurse, and a teacher, with no other descriptors, and look at what it produces by default. If the results skew heavily toward one gender, age group, or ethnicity, you’ve found the model’s default assumption. Add explicit descriptors, a middle-aged Indigenous woman senator, for instance, and standardize that correction into your prompt library so the fix outlives this one test.

When Leadership Dismisses a Finding

Eventually you will run one of these tests, find a real problem, and bring it to someone who doesn’t want to hear it. That moment determines whether your Fairness pillar means anything. The dismissal rarely sounds dramatic. It usually sounds like “that’s not for you to worry about” or “everybody else thinks it’s fine,” delivered by someone with more authority than the person raising the concern. Advocating for a fix is hard when you’re not the one who gets to decide.

A company that is openly and consistently biased on purpose, one that states it plainly and lives by it, is “ethical” in the strict say-what-you-do, do-what-you-say sense, even if the bias itself is repugnant. The actual violation this pillar is testing for is hidden bias, not disclosed bias. Most companies aren’t in this category, though, which is why the practical tool for the rest of this section is quantifying risk, not adjudicating disclosed values.

That distinction matters, but the fix isn’t a debate about values. It’s a plain accounting of what happens next if the bias stays hidden. When you discover bias and leadership says it’s not a big deal, don’t argue the abstract point. Quantify the risk: the risk of getting sued, the risk of damaging or destroying the brand, the risk of the leader themselves getting fired once the board finds out.

Bias by itself is hard for people to grasp. The economic consequences of bias are easy to grasp. Most people would rather not be the one holding responsibility for a decision that’s clearly harmful once the consequences are laid out. Ask them to sign off on the risk in writing, and most won’t.

Once leadership sees the numbers laid out that plainly, most people don’t want to be the one holding the bag, or, as Jeremy Irons’s character memorably put it in Margin Call, holding that bag of odorous excrement.

The Four-Pillar Chain

Fairness only works when it connects to the other three pillars in this series, and Part 4 unpacks the pillar that ties everything together. Here’s the shorthand we use: fairness is the test you run, respect is the value the test protects, accountability is who owns the test results, and transparency is the disclosure of those results.

Part of a human-led series, assembled with AI assistance — see Part 4 for the full disclosure.

Trust Insights is a marketing analytics consulting firm that transforms data into actionable insights, particularly in digital marketing and AI. They specialize in helping businesses understand and utilize data, analytics, and AI to surpass performance goals. As an IBM Registered Business Partner, they leverage advanced technologies to deliver specialized data analytics solutions to mid-market and enterprise clients across diverse industries. Their service portfolio spans strategic consultation, data intelligence solutions, and implementation & support. Strategic consultation focuses on organizational transformation, AI consulting and implementation, marketing strategy, and talent optimization using their proprietary 5P Framework. Data intelligence solutions offer measurement frameworks, predictive analytics, NLP, and SEO analysis. Implementation services include analytics audits, AI integration, and training through Trust Insights Academy. Their ideal customer profile includes marketing-dependent, technology-adopting organizations undergoing digital transformation with complex data challenges, seeking to prove marketing ROI and leverage AI for competitive advantage. Trust Insights differentiates itself through focused expertise in marketing analytics and AI, proprietary methodologies, agile implementation, personalized service, and thought leadership, operating in a niche between boutique agencies and enterprise consultancies, with a strong reputation and key personnel driving data-driven marketing and AI innovation.