Last month, I posted about a system I built: a Topical Clustering Engine that structures 5.3 million raw search queries into a 5-level taxonomy - Categories, Subcategories, Intents, Topics, and Keywords at under 1 millisecond per classification.
The post got 254 reactions, 89 comments, and 11 reposts.
Venkata Pagadala AI Product Manager | 10+ Years in Search & SEO Building AI systems that actually work at enterprise scale.
🌐 venkatapagadala.com - Deep dives, tools, and resources 🔗 [Follow me on LinkedIn for weekly insights between editions →]
But the most common question wasn't about the technology. It was this:
"How did you actually build this?"
So let me tell you the story.
The Problem Nobody Talks About
Every large enterprise I've worked at has the same invisible problem: they're drowning in search data but starving for search intelligence.
Here's what I mean.
Imagine a company that operates over 40 million web pages. Millions of people search for things related to their products every single day. And like every large enterprise, they have keyword data. Spreadsheets of it. Dashboards of it. Terabytes of it.
But when leadership asked a simple question - "What are ALL the topics our customers care about?" nobody could answer it.
Not because the data didn't exist. Because nobody could organize it at scale.
A skilled SEO analyst can manually categorize about 1,000 keywords in 2 hours. At that rate, organizing 5.3 million keywords would take approximately 10,600 hours. That's 5 years of full-time work. For one snapshot. That's already outdated by the time you finish.
This is the dirty secret of the search industry: we have more data than ever and less understanding than ever.
Embedded LinkedIn post: view it on LinkedIn
Why Traditional AI Approaches Failed
My first instinct was to throw an LLM at it. GPT-4, Claude - just feed in the keywords and ask it to categorize them.
It didn't work. Here's why:
Cost. Classifying 5.3M keywords through GPT-4 API calls would cost thousands of dollars per run. And you need to run it regularly because search behavior changes.
Speed. Even with batching, LLM-based classification took hours. Business decisions can't wait hours for a taxonomy to rebuild.
Consistency. LLMs are creative, which is exactly what you don't want in classification. Ask Claude to categorize "iPhone 16 Pro Max case" ten times and you might get ten slightly different category paths.
Hallucination. LLMs would sometimes invent categories that didn't exist in our taxonomy, or merge categories that should stay separate.
I needed something different. Something that combined AI intelligence with engineering precision.
The Architecture That Actually Works
What I built is a hybrid system - not a single model, but a pipeline:
Layer 1: Taxonomy Design (Human + AI) I worked with domain experts to define the initial category structure. This is the one step where human judgment is irreplaceable. You need to know what "Devices," "Internet," "Accessories," and "Support" mean in your business context. AI helped expand and refine, but humans set the foundation.
Layer 2: Training Data Generation (AI-Assisted) Using a combination of LLMs and rule-based systems, I generated high-quality training examples for each category at every level of the taxonomy. The key insight: use LLMs for training data generation (where creativity is an asset), not for real-time classification (where consistency is everything).
Layer 3: Classification Engine (Purpose-Built ML) The actual classifier is a fine-tuned model optimized for one thing: speed and accuracy at scale. No API calls. No token costs. Sub-millisecond inference. It runs locally and processes millions of keywords in seconds.
Layer 4: Quality Assurance (Automated) A separate validation layer checks for edge cases, ambiguous classifications, and category drift over time. This is where I use LLMs again - as a quality auditor, not as the primary classifier.
The result:
5,331,768 keywords classified
913 L1 categories
746 L2 subcategories
882 L3 intents
<1ms classification speed
What used to take a team 5 years now takes a machine under a minute.
The Lesson That Changes Everything
Here's what building this system taught me - and it's the core thesis of this newsletter:
The companies winning with AI aren't using AI to replace thinking. They're using AI to make thinking possible at a scale that was previously impossible.
Before this system, nobody at a company managing tens of millions of pages could answer "what do our customers actually care about?" Not because they were lazy. Because it was humanly impossible.
Now we can see the entire landscape. 1.6 million topics. 789 categories. Every keyword mapped to an intent, every intent mapped to a business opportunity.
That's not automation. That's augmentation. And it's a fundamentally different philosophy than "let AI write your blog posts."
What's Coming Next
This is Edition #1 of Signal Over Noise. Here's what I'm building toward:
Edition #2: How Netflix, Amazon, and Google architect their knowledge graphs - and what every company can learn from their approaches.
Edition #3: From taxonomy to strategy - how to turn 5 million classified keywords into a 3-4 year content and optimization roadmap.
Edition #4: The internal linking recommendation engine - using AI to connect content across millions of pages.
Every edition will follow the same principle: less hype, more signal. Real systems. Real architectures. Real results.
Your Turn
I built this system for a Fortune 500 company, but the architecture works for any industry with scale.
Drop a comment below: What industry are you in, and what would you do with a complete map of every topic your customers search for?
I'll pick the most interesting industry from the comments and show how this system would work for it in a future edition.
If this was useful, share it with someone who's tired of AI hype and wants to see what AI actually looks like in production.
And if you're new here - welcome to Signal Over Noise. Subscribe to get the next edition delivered to your inbox. For more resources and deep dives, visit venkatapagadala.com.
