Machine Learning Keyword Research: Build a Clustering, Intent and Scoring Pipeline

Context: The Team With 22,000 Keywords and Five Pages Fighting for One Head Term

Picture a scaled South African e-commerce or B2B SaaS marketing team. They have accumulated 22,000 keywords across a decade of agency handovers, tool exports and Search Console downloads. Five URLs on their site receive impressions for the same head term. The content lead wants to prioritise product pages, the paid search manager wants to protect a high-converting Search campaign, and neither can agree on which cluster deserves the next sprint. The bottleneck is not keyword volume - it is classification, prioritisation and the missing link between query groups and revenue.

This is the problem that machine learning keyword research is genuinely equipped to solve. Not 'type a topic into an AI tool and get 500 ideas', but a modelling pipeline that embeds every query, groups it semantically, classifies its intent, scores its commercial opportunity and routes it into content, paid search and CRO.

Why 'ML Keyword Research' Is Not the Same as Typing a Prompt Into an AI Tool

Generative Assistance vs. the Modelling Layer

A generative assistant can produce a list of related terms in seconds. That output is useful for brainstorming but it is not a research system. It has no grounding in your Search Console queries, no awareness of which URL already ranks, no idea what your margin looks like per product category, and no memory of which clusters you decided to deprioritise last quarter. Generative assistance answers 'what might people search?'. The modelling layer answers 'which of the 22,000 queries we already have data for should we act on first, and with what asset?'.

What Genuine ML Contributes: Clustering, Classification, Forecasting

Four ML capability families matter here. First, embedding and clustering converts each query into a numeric vector and groups semantically similar queries without a human reading all 22,000. Second, classification assigns each query an intent label and a funnel stage (TOF, MOF, BOF) using a small hand-labelled training set. Third, forecasting and anomaly detection flag rising or decaying terms before a seasonal spike - useful in South African retail where the BFCM window and January 'back to school' patterns behave differently to Northern Hemisphere cycles. Fourth, SERP-overlap clustering uses shared ranking URLs as Google's own signal for topic boundaries.

The Four Data Sources That Actually Feed a Keyword Model

Search Console Queries, Pages, Country and Device

The Performance report in Google Search Console gives you measured impressions, clicks and average position - the only first-party source that links a query to a URL. Export 16 months of data segmented by country and device so you can distinguish a South African head term from the same term searched in the UK.

Google Ads and Amazon Ads Search-Term Reports

The Google Ads search terms report contributes commercial-intent signals and conversion data that Search Console cannot see. Amazon search-term reports do the same for marketplace shoppers, where query phrasing is shorter and more transactional.

First-Party On-Site Search Logs and CRM Lead-Source Fields

On-site search logs reveal what visitors wanted but could not find - often the sharpest content gap available. For B2B SaaS, the CRM lead-source field connects queries to pipeline value, so a cluster can be scored by opportunity-to-close, not just traffic.

Stage One - Data Hygiene Before Any Model Runs

Deduplicating, Normalising and Stripping Branded Terms

Lowercase everything, strip diacritics, collapse whitespace and remove your own brand plus known competitor brands into a separate list. A common mistake is leaving branded queries inside the corpus - they cluster together artificially and distort every similarity score. Also strip internal query parameters before deduplication.

Separating Measured Data From Tool-Estimated Volumes

Stage Two - Embeddings and Vector Storage for Semantic Grouping

How Sentence Embeddings Represent a Query

A sentence embedding model, in the tradition of Sentence-BERT, maps a query like 'best running shoes for flat feet' to a 384- or 768-dimension vector. Queries with similar meaning sit close together in that vector space even when they share no exact keywords. Store the vectors in a vector store keyed by query ID; the store now becomes the substrate for every subsequent clustering and scoring step.

Choosing a Similarity Threshold (and Why It Breaks Both Ways)

Cosine similarity between two query vectors ranges roughly from 0 to 1 for semantically related queries. A threshold that is too loose merges commercial and informational intent into one cluster; too tight and 'running shoes' fragments into a dozen near-singletons that never justify a dedicated page. There is no universally correct number - it must be tuned against a manually reviewed sample.

A Worked Threshold Comparison Table

The following table is a worked example with stated assumptions, not a benchmark. It represents observed behaviour when tuning on an e-commerce corpus of 20,000+ mixed-intent queries.

Stage Three - Training a Lightweight Intent Classifier

Building a Labelled Seed Set of a Few Hundred Queries per Class

Hand-label 200-400 queries per intent class - informational, commercial investigation, transactional, navigational. A simple logistic regression or gradient-boosted tree on top of the embeddings is often enough; a large language model is not required for this step. Retrain quarterly, because intent drift is real: a query that meant 'what is X' this year may be dominated by 'buy X' next year.

Mapping Intent Labels to TOF, MOF and BOF Funnel Stages

Stage Four - SERP-Overlap Clustering as the Deciding Vote

How Shared Ranking URLs Reveal Google's Own Topic Boundaries

Semantic embeddings group queries by meaning. SERP-overlap clustering groups them by which URLs actually rank. If two queries share three or more of the same top-10 URLs, Google is treating them as closely related - often the same topic. Pull the top-10 results for a sample of queries (respecting crawl budget and rate limits), build a query-by-URL matrix, and cluster on Jaccard similarity. Where embeddings and SERP overlap disagree, SERP overlap usually reflects the current competitive reality and should take precedence for content decisions.

One Page or Many? Turning Overlap Scores Into Content Decisions

A high overlap score (for example, above 0.4 in a worked example) suggests one page should target the cluster. A low score suggests separate pages are warranted. Treat the threshold as a heuristic and validate against your own Search Console data - if Google already shows two of your URLs ranking for the same cluster, you have a cannibalisation problem rather than a consolidation opportunity.

Stage Five - The Opportunity Scoring Formula

Pseudo-Code for a Weighted Opportunity Score

The following is a worked example with stated assumptions - no benchmark implied. Each input is normalised to a 0-1 scale before weighting.

# Worked example only. Weights must be set by a strategist, not the model.
# Inputs are normalised 0-1 across the cluster set.

opportunity = (
    demand_score          # e.g. min-max normalised impressions or estimated volume
    * commercial_value    # strategist-assigned per cluster, tied to margin or pipeline
    * (1 - current_coverage)  # 0 if a dedicated page exists, 1 if none
) / difficulty_proxy      # e.g. avg. domain authority of ranking URLs, or KD

# Example weights (illustrative):
# demand 1.0 | commercial_value weighted 0.0-1.0 | coverage 0/1 | difficulty clip at 0.1
opportunity = (demand * commercial_value * (1 - coverage)) / max(difficulty, 0.1)

Setting Commercial-Value Weights With a Strategist, Not a Model

Demand, coverage and difficulty can be derived from data. Commercial value cannot - it requires a human to say 'this cluster maps to our highest-margin category' or 'this cluster feeds the enterprise pipeline'. Set weights in a workshop with the commercial owner, document them, and revisit them quarterly. The model applies the weights; it does not invent them.

The Cannibalisation Audit: Finding Query-Page Collisions in Search Console

Flagging Multiple URLs Receiving Impressions for One Cluster

Export the Search Console Pages dimension, join it to your query-cluster assignment, and flag every cluster where two or more URLs receive meaningful impressions. This is the operational definition of cannibalisation. In South African retail, seasonal spikes (BFCM, January) often expose these collisions because both URLs start competing for spiking demand.

Consolidation, Redirect or Re-Brief? Three Resolution Paths

Consolidation: merge the weaker page's content into the stronger URL and redirect. Redirect: when one URL has no unique value, 301 it to the primary page. Re-brief: when both URLs legitimately serve different audiences, rewrite both briefs to sharpen topical separation and internal linking. The pipeline should output a recommended resolution path per collision, subject to strategist approval.

Routing Clusters Into Content, Paid Search and CRO

Content Briefs, Paid Search Structure and Landing Page Tests

TOF clusters route to content briefs with AI-retrieval-ready structure. BOF clusters feed paid search campaign structure - one ad group per transactional cluster, with negative keywords drawn from adjacent clusters to prevent overlap. Where organic and paid both land on the same URL, coordinate message match across the two channels. Clusters with high traffic but weak conversion become candidates for conversion rate optimisation tests, not for more content.

Why Cluster-Level Attribution Beats Last-Click Reporting

A cluster often drives assisted conversions that last-click reporting attributes to a brand term or a paid search click. Reporting performance only at last-click systematically undervalues the top-of-funnel clusters doing the discovery work. This is why we build measurement around cluster-level and assisted-conversion attribution, as described in our custom reporting approach.

Where This Pipeline Breaks: Five Traps and How to Avoid Them

Estimated Volumes, Intent Drift and Seasonal Distortion

Three recurring traps. First, treating tool-estimated volumes as measured - they are not, and for South African and EMEA terms the error is often largest. Second, intent drift: label sets expire within a few quarters and must be refreshed. Third, seasonal distortion: a term that looks flat on a 12-month average may spike hard in BFCM or January, and any demand score computed on annual averages will miss it. Use rolling windows and flag known seasonal terms explicitly.

Consent, POPIA and GDPR Constraints on First-Party Query Data

On-site search logs and CRM lead-source fields are personal data when they can be linked to an individual. Under South Africa's POPIA and the EU GDPR, you must have a lawful basis for processing, apply data minimisation and, where possible, aggregate or pseudonymise query data before it reaches the model. Consent-mode and server-side implementations should be reviewed by a legal advisor - this is not a marketing team decision alone.

Measuring Whether the System Works: Coverage, Cannibalisation, Revenue

Coverage and Cannibalisation Metrics

Two leading indicators tell you whether the pipeline is functioning. Coverage: the percentage of priority clusters that have a dedicated, well-optimised page. Cannibalisation: the count of query-page collisions from Search Console, tracked month over month. Both should move in the right direction within a quarter if the routing and audit stages are working.

Traffic-to-Revenue, ROAS and Lead Quality Signals

For e-commerce, tie cluster-level traffic to revenue via a clean data pipeline and review ROAS for clusters used in paid search - not blended, cluster-specific. For B2B SaaS, measure lead quality (MQL-to-SQL conversion) per cluster rather than raw form fills. Assisted conversions should be reported alongside last-click, with the reporting page linked so stakeholders understand the model. When the pipeline is working, the content team stops arguing about priority because the opportunity score, cannibalisation audit and commercial weights already answer the question.

Sources

• Google Search Console Help - Performance report (queries, pages, countries)
• Google Ads Help - Search terms report
• Reimers & Gurevych - Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (arXiv)
• Devlin et al. - BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (arXiv)
• Information Regulator (South Africa) - POPIA official regulator site