AI Digital Clone Part 3

In this week’s Data Diaries, let’s move onto part 3 in our series of building a virtual version of yourself. Last time, we talked about cataloging thinking and I developed a catalog of over 400 different thinking techniques. This is AI Digital clone, part 3.

The third stage is gathering all the data. If I wanted the project to succeed, I couldn’t just pour all the data into one big pile and hope AI sorted it out. That’s a recipe for disaster in any scenario. Instead, I needed to do data cleaning and engineering itself.

I started with three types of content – newsletters, call transcripts, and YouTube videos. For each type of content, I have code that processes the content in several ways. First, it splits documents by sentence boundaries. Here’s why: AI tools, especially AI agents, use popular command line tools to find chunks of text with utilities like grep.

These utilities are designed to process content by the line as their base atomic text chunk, and return it. There are many, many transcription tools that tend to return a wall of text in the transcript with no line breaks, and that means when a tool like grep fishes up some content, it returns the entire wall instead of just a single line. That quickly blows up your agent’s context window, causing it to fail.

So to immediately provide better results, a Python script identifies common punctuation for sentence boundaries (periods, exclamation marks, question marks) and splits up paragraphs into single lines. You immediately get better performance and reduced memory usage.

The second step is to deduplicate by line, because five lines in a row of “Yep. Yep. Uh huh. Yep. Yep.” is unnecessary and doesn’t offer any useful context.

Once the text files are clean, they get loaded into a database, and then with a specially-formatted version of the 400+ thinking techniques, each piece of content is sent to an LLM (Nemotron 3 Super) to have the individual texts analyzed. The results come back and get fed into the database:

After all the data is collated, I end up with some nice data tables of what thinking techniques I use, by content type, in named ordered pairs:

This is a cognitive blueprint of how I think. In the next and final part in this series, we put these cognitive blueprints to work. Stay tuned!

Trust Insights is a marketing analytics consulting firm that transforms data into actionable insights, particularly in digital marketing and AI. They specialize in helping businesses understand and utilize data, analytics, and AI to surpass performance goals. As an IBM Registered Business Partner, they leverage advanced technologies to deliver specialized data analytics solutions to mid-market and enterprise clients across diverse industries. Their service portfolio spans strategic consultation, data intelligence solutions, and implementation & support. Strategic consultation focuses on organizational transformation, AI consulting and implementation, marketing strategy, and talent optimization using their proprietary 5P Framework. Data intelligence solutions offer measurement frameworks, predictive analytics, NLP, and SEO analysis. Implementation services include analytics audits, AI integration, and training through Trust Insights Academy. Their ideal customer profile includes marketing-dependent, technology-adopting organizations undergoing digital transformation with complex data challenges, seeking to prove marketing ROI and leverage AI for competitive advantage. Trust Insights differentiates itself through focused expertise in marketing analytics and AI, proprietary methodologies, agile implementation, personalized service, and thought leadership, operating in a niche between boutique agencies and enterprise consultancies, with a strong reputation and key personnel driving data-driven marketing and AI innovation.