Your web estate did not grow on purpose. It grew one campaign, one acquisition, one "the docs team wants its own platform" at a time. Now you have a dozen subdomains, three or four content management systems, and a worry that AI engines are pulling the wrong pages, or none of yours. If that feels like a lot, take a breath. You do not have to fix it all this quarter.
This is a working playbook for the marketing lead who owns enterprise AI search across a sprawling site. By the end you will know how to inventory every property, set a crawler policy, pick subdomains over subfolders on purpose, canonicalize across systems, seal off staging so it never leaks, and govern the whole thing so it stays clean. We are structuring a live estate here, not running a migration cutover. Let's take it one step at a time.
Step 1: Inventory every hostname, CMS, and page type
You cannot govern what you have not counted. So the first move is boring and non-negotiable: build one table that lists everything.
Run a server-side crawler across every subdomain and every CMS. Combine that crawl with a CMS export and a DNS inventory so nothing hides. For each hostname, capture the CMS, template family, page type, content owner, primary topic and audience, language or region, production status (prod, staging, dev, QA, UAT), whether it needs auth, and when it was last meaningfully updated. Then bucket every URL into one of five page classes: marketing, product or landing, blog or resource, documentation, transactional. Tag each subdomain by business purpose too: corporate, brand, regional, support community, knowledge base, dev portal, careers.
You know this step is done when a single source-of-truth table exists with one row per subdomain, one per CMS, and one per page class. Every hostname, including the legacy and parked ones you forgot about, has a named business owner and a named SEO or AEO owner.
Here is where most teams trip. Inventories stop at marketing URLs. The documentation portal, the community forum, the dev portal, the careers microsite all get skipped, yet AI crawlers still visit and cite them. Old campaign subdomains pile up because no one decommissioned them. And ownership stays fuzzy ("marketing owns it"), so when something breaks, no one is accountable. Name a real human for every row. That discipline pays for itself all year.
Step 2: Map the AI crawler landscape and set a policy
Now that you know what you have, decide who gets to read it. The bot universe is not one thing, and treating it as one is how good content gets blocked.
Split the crawlers into groups, because each has a different reason to visit. AI search assistants that cite sources (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, Google-Extended) can earn you citations, so the default is allow. Training crawlers that do not drive citations (GPTBot, ClaudeBot, CCBot and friends) are a real decision: allow, block, or rate-limit, and write down why. Classic search bots (Googlebot, Bingbot) stay allowed. For each category, set a default policy, publish it in robots.txt with the known bots named and a wildcard fallback, and reinforce it at the edge with your WAF or CDN.
Common mistake: conflating ChatGPT-User with GPTBot. They are different bots with different jobs. GPTBot is the training crawler; ChatGPT-User is the on-demand fetcher that browses when someone asks ChatGPT a live question. Block the wrong one and you quietly cost yourself citations.
You know this step is done when a maintained allow, block, or charge list lives in version control, robots.txt mirrors it in plain language, and every block has a written rationale. A quarterly review catches new bots before they hit production at scale, which matters because multi-CMS AI crawlers touch every property you own, not just the one you watch most.
Two cautions to hold onto. First, vendors add and rename user agents constantly, so keep a living list, not a static one, and verify any block by reverse DNS or published IP ranges rather than trusting the user-agent string, which anyone can spoof. Second, robots.txt is a polite request, not a wall. A disallowed page can still get indexed and linked from elsewhere. When you need something truly hidden, that is a job for Step 5, not robots.txt.
Step 3: Decide subdomains versus subfolders on purpose
This decision shapes your whole estate, and most sprawl comes from making it by accident. So make it on purpose.
Default to a subfolder (example.com/blog/, example.com/de/). Subfolders consolidate authority, simplify analytics, and are easiest for AI crawlers to treat as one source. Reach for a subdomain (blog.example.com, de.example.com) only when there is a real technical, geographic, or brand reason: a docs portal on a different stack, a region served from a separate origin, or a genuinely separate brand. Reserve a separate domain for a truly separate brand or legal entity, because every extra domain dilutes authority and adds another surface to police.
Walk the decision like this. Same authority and audience? Subfolder. Different infrastructure stack, hosting region, or product team ownership? Subdomain. Separate brand or legal entity with no intent to share authority? Separate domain. Many markets with distinct local teams? Country-code domains with hreflang can fit; otherwise subfolders stay simpler.
You know this is done when every active subdomain has a written rationale, a named owner, and an AEO review, and when your subdomain count is flat or falling year over year, never quietly growing. That last metric matters more than it looks. Each new subdomain is a new governance surface and a new footprint that your subdomain AI visibility work has to cover.
Where teams go wrong: a team wanted its own platform, so a subdomain appeared. A region got a subdomain when a subfolder with hreflang would have done. A tangential product got a brand subdomain that diluted the parent, and split subdomain AI visibility across two footprints instead of one. One honest note from the research: subfolder-versus-subdomain is not always a clean SEO win for subfolders anymore, and the engines have softened their language on it. The bigger factor is operational. Who owns it, what CMS runs it, how often it ships.
Step 4: Canonicalize across every property
You almost certainly have the same content living at more than one URL. Print views, tracking parameters, a page that exists in two CMSs after an acquisition. Canonicalization tells engines which version is the real one, and doing it well is quiet, unglamorous, and hugely protective of your large site AI citation footprint.
Lean on the signal hierarchy, strongest to weakest. A 3xx redirect (301 for permanent, 308 to preserve the method) is the strongest signal that the target is canonical. A rel="canonical" annotation in the head is a strong signal. Sitemap inclusion is a weak one. These stack: a redirect plus a self-referencing canonical plus a sitemap listing is the gold standard. Canonicals can point across domains too, so a print version on a separate host can canonicalize to the real web page. One firm exception: a staging subdomain should never canonicalize to production. It gets sealed off instead.
Pro tip: always use absolute URLs in your canonical tags. Relative canonicals break the moment they cross a protocol or a host, and on a multi-property estate they will cross both.
You know this is done when a canonical audit reports zero issues across the usual suspects: relative URLs, missing self-references, chains, canonicals pointing at redirects or non-indexable pages, multiple canonical tags on one page, noindex and canonical fighting each other, and sitemaps that list a URL which canonicalizes elsewhere.
The traps are predictable. Treating canonical as an on-off switch when it is only a signal. Pointing canonicals across subdomains at content that does not actually match. Forgetting the self-referencing canonical on every page, the cheapest win here and the one most often skipped. And mixing noindex with canonical on the same page, which sends two contradictory instructions. Pick one.
Step 5: Seal every non-canonical surface so nothing leaks
Here is the fear that keeps enterprise teams up at night: a staging URL surfaces in an AI answer, or a half-finished preview page gets crawled and cited. It happens more than anyone admits, and the fix is layered.
First, know your leaky surfaces. Staging, dev, QA, UAT, and preview subdomains at predictable names. CMS preview links from headless systems. Internal search result pages. Print views. Faceted navigation combinations. Pagination past page one. Thin tag, category, and author archives. Legacy-CMS tag URLs on an old pattern. Full-content RSS feeds. Each is a place a duplicate or an unfinished page can leak.
Then defend in layers, because any single layer fails. Put Disallow: / in robots.txt on the staging hostname (the polite request). Add real authentication (HTTP basic auth, SSO, an IP allowlist, or VPN) on that hostname (the actual wall). Enforce noindex with a meta robots tag or the X-Robots-Tag header on every staging page, set at the template level so an editor cannot strip it. And make sure no production page links to a non-production host, because one stray link pulls a staging URL into an index.
You know this is done when a scripted test from production hits every known non-canonical hostname and gets back a 401, 403, or noindex, a weekly staging crawl returns zero indexable URLs, and no production page links to a non-production host.
The classic failure is relying on robots.txt alone. Disallowed staging pages can still surface in results and training sets without snippets, which is worse, not better, because you lose control without gaining privacy. The other one: letting people publish from staging and watching a staging link go live by accident. Lock the door with auth, not with a note on the door.
Step 6: Govern the multi-CMS sprawl with one shared contract
This is the heart of it. Multiple CMSs are fine. Multiple sets of rules are not. The point of enterprise site governance AI programs is to make three or four systems behave like one, so that multi-CMS AI crawlers see a single, consistent estate instead of a patchwork.
Write one shared contract that every CMS maps to. One controlled vocabulary for topic, audience, product, region, intent, and funnel stage. One schema.org generator that emits JSON-LD from those controlled fields, not from free text, so your structured data never drifts page to page. One metadata spec (title, description, canonical URL, OG image, author, publish and last-reviewed dates, language). One URL pattern (lowercase, hyphenated, a defined allow-list of query parameters). One redirect policy (301 for permanent, 302 only for time-boxed tests, single hops, never chains). When every system obeys the same contract, the sprawl stops mattering.
Then name who does what, because a policy with no owner is a wish. A simple RACI works: Marketing Ops and the SEO or AEO lead are responsible day to day, the VP of Marketing is accountable, Engineering and Legal and regional leads are consulted, and the executive team is informed through a monthly dashboard. Add Legal for bot policy, because training-data terms carry contractual weight, and add Engineering because CDN and WAF changes touch production.
Give content a lifecycle with real gates: brief, draft, review, publish, refresh, retire. Set refresh cadence by type (stats pages quarterly, product pages on release, thought leadership annually). Retire cleanly: 410 when something is gone for good, 301 to a successor when one exists, and remove the URL from sitemaps and internal links within 30 days.
This is also where a platform earns its keep. DeepSmith stores your brand context, voice, personas, and content types as structured data in Deep IQ, then produces publish-ready articles that carry consistent metadata, schema, and internal links no matter which CMS they land in. Instead of hoping four teams hand-apply the same rules, the rules live in the system that writes. That is the difference between a governance doc people ignore and governance that holds.
You know this is done when every CMS publishes into one search index and one analytics view, a vocabulary change propagates within a single release cycle, and a retired URL leaves your sitemaps and internal links within 30 days with proof in the crawl log.
Step 7: Engineer pages for citation, not just ranking
Ranking and getting cited are not the same job. AI systems break your pages into chunks, index each on its own, then assemble an answer from several sources. So a page can rank fine and never get quoted, because no single passage cleanly answers the question. Fixing this at the template level is how you solve the large site AI citation problem across thousands of pages at once.
Structure each page so the citeable passage is obvious. Open every H2 with a direct, declarative answer in the first 50 to 80 words, because that opening chunk is what an engine is most likely to lift. Phrase the H2 the way your audience asks the question, so "How to map subdomains for AI search" beats "Subdomain considerations." Then back it with substance: statistics with named sources, named examples, a clear next step. Keep one H1 per page, let H2s answer and H3s decompose, and show a visible last-updated date.
Research into generative engine optimization backs this up. Studies testing on-page techniques against answer engines found that citing sources, adding statistics, quoting named experts, and clear authorship all lifted visibility, while keyword stuffing did nothing or hurt. The moves that make a passage genuinely useful to a reader are the same moves that make it citeable.
Common mistake: letting a template redesign strip author bios, dates, and schema to "modernize" the look. Those are exactly the trust signals AI engines read. And never ship JSON-LD without validating it, because invalid schema is worse than none.
You know this is done when a spot-check of 20 representative pages shows each H2's first paragraph standing alone as a coherent answer, every page carrying a visible author, date, and sources, and schema validating with zero errors. At enterprise scale, this is a template job, not a page-by-page one. DeepSmith produces articles with answer-first structure, heading hierarchy, and schema built in during creation, so the citeable shape is baked in rather than retrofitted across an estate you could never hand-edit.
Step 8: Handle sitemaps, hreflang, and redirects at scale
Big estates break the small-site playbook, so the mechanics need their own step. Get these right and crawlers move through your site efficiently instead of wasting budget.
Respect the sitemap limits: 50,000 URLs and 50 MB per file, up to 500 sitemap indexes per site. Split sitemaps by content type (products, articles, images, video) so diagnostics stay useful, include only canonical 200-status URLs, and rebuild nightly. For hreflang across regions, give each variant a self-referencing tag plus a link to every other variant, use language-region codes like en-US and en-GB, include an x-default, and confirm the return links, because one-way hreflang is only a hint. For redirects, use 301 or 308 for permanent moves, keep every redirect to a single hop, never redirect a redirect, and update internal links to point at the final destination instead of routing through a chain.
One low-cost experiment: an llms.txt file at your root that lists your most important pages in clean Markdown for LLM tools to ingest. Treat it as a discovery aid, not a ranking factor. Major model vendors have not adopted it as a required file, so host it, keep it current, and do not expect miracles.
You know this is done when all canonical sitemaps submit error-free, the hreflang cluster is bidirectional with an x-default, and a live crawl finds no redirect chain longer than one hop. The classic enterprise failures are one mega-sitemap of unsegmented URLs, hreflang clusters missing return links, and redirect chains accumulated over years of migrations.
Step 9: Track AI citations, not just search rankings
You have done the structural work. Now you need to see whether it moved anything, and rankings alone will not tell you. This is the measurement layer of enterprise AI search, where the loop finally closes.
Measure the things that describe AI answers, not blue links. Mention rate: how often engines name your brand for the prompts you care about. Citation rate: how often they link your pages as sources. Share of voice against named competitors on the same prompts. Add per-prompt trend, per-page attribution, and a per-platform split across ChatGPT, Perplexity, Gemini, Claude, and Google AI Mode, because the engines do not all cite the same way. ChatGPT leans on top-ranking pages, Perplexity shows source panels, and Claude often cites directly when it browses.
This is where DeepSmith does the work you cannot do by hand. It tracks mention rate, citation rate, and share of voice across ChatGPT, Gemini, Perplexity, Claude, and Google AI Mode, shows a competitor leaderboard and which of your pages earn citations, then ties the weak prompts straight back into content production. Measurement that feeds production is the whole point. Tracking a gap you never close is just a prettier way to lose.
You know this is done when a weekly report ranks your top prompts and shows movement, a competitor leaderboard gets reviewed monthly by a named owner, and your production queue is aimed at the prompts where visibility is weakest. Where teams go wrong: watching only Google rankings while AI citations erode, tracking a single platform, or reporting AI visibility without feeding gaps back into what you write next.
What to do next
Do not try to do all nine steps this week. Pick the one that scares you most, which for most teams is the staging leak in Step 5, and close it first. Then build the inventory in Step 1, because every other step reads from it. Momentum matters more than perfection. Strong enterprise site governance AI is not one heroic project, it is one clean subdomain, one sealed staging host, one governed CMS at a time, until the sprawl turns into a system.
You are closer than it feels. A large estate is not a curse for AI search, it is an advantage once governed, because you already have the pages engines want to cite. The work is making them findable, consistent, and clean.
If you want the measurement and production halves handled in one place, start a free DeepSmith trial and see where your estate shows up in AI answers before you change a thing.
