Technical SEO & Crawlability
Understand whether search engines and AI agents can discover, crawl, index, canonicalize, and understand the right version of your site.
Ludwig Makhyan · Founder, WebsiteCoach ·
Jump to section
Executive Overview
Technical SEO checks verify that your pages are discoverable and unambiguous before content quality even matters. A missing canonical, accidental noindex, blocked robots.txt rule, stale sitemap, or absent llms.txt / AGENTS.md can prevent search engines and AI agents from finding the content you intended them to use.
Business Impact
A technical indexing mistake can remove revenue pages from search, split ranking authority across duplicate URLs, or leave AI discovery channels blind to your best content. These issues are often small text or configuration fixes, but the business impact can be large because they affect every crawler visit.
Source: Lighthouse SEO data plus WebsiteCoach checks of canonical, robots, hreflang, sitemaps, and llms.txt / AGENTS.md signals.
Technical SEO Signals Explained
- Page title — presence & length (30–60 chars)
- Use a descriptive, page-specific title. Character ranges are editorial guidance, not Google's fixed display limits.
- Meta description — presence & length (120–160 chars)
- Summarize the page accurately. Google can choose another snippet, and there is no guaranteed character cutoff.
- Canonical tag — present & self-referencing
- A canonical suggests the preferred URL among duplicates. Ensure it agrees with redirects and internal links.
- Noindex detection (meta robots / googlebot)
- A noindex directive requests exclusion from search. Confirm it is intentional and visible in the response or rendered page.
- hreflang — x-default & self-reference validation
- Language alternates should point to equivalent translated pages and include reciprocal references. Check URLs and language codes.
- robots.txt — crawl rules & Sitemap: declaration
- robots.txt controls crawling, not secure access or guaranteed removal from search. Keep the sitemap URL reachable.
- XML sitemap — URL count & lastmod freshness
- List canonical, indexable URLs. Use lastmod only for genuine content changes rather than every rebuild.
- Sitemap index — child sitemap resolution
- A sitemap index points to child sitemaps. Verify each child can be fetched and contains valid URLs.
- llms.txt / AGENTS.md — AI agent discoverability
- These optional files may document a site for some tools. Google does not require them for AI-feature eligibility.
Role-Specific Action Plan
- →Review noindex, robots.txt, sitemap, and canonical findings before content work
- →Prioritise fixes on money pages, landing pages, and international versions
- →Track index coverage changes in Google Search Console after deployment
- →Ensure every page template outputs a unique title, meta description, and canonical URL
- →Generate hreflang alternates from routing data instead of manual copy-paste
- →Add llms.txt / AGENTS.md and keep it aligned with important public content
- →Decide which duplicate or filtered pages should be canonical, indexed, or excluded
- →Keep sitemap URLs limited to canonical, indexable, valuable pages
- →Keep robots.txt rules, sitemap declarations, and CDN/WAF bot rules aligned
- →Regenerate sitemap indexes automatically after large content or product updates
Frequently asked questions
What is Technical SEO & Crawlability?
Understand whether search engines and AI agents can discover, crawl, index, canonicalize, and understand the right version of your site.
Why does Technical SEO & Crawlability matter?
A technical indexing mistake can remove revenue pages from search, split ranking authority across duplicate URLs, or leave AI discovery channels blind to your best content. These issues are often small text or configuration fixes, but the business impact can be large because they affect every crawler visit.
Official references
Related topics