# What Is Technical SEO? The Complete 2026 Guide to the Foundation Everything Sits On
The short answer#
Technical SEO is the work of making sure search engines and AI systems can find, read, understand and trust your website. It covers crawling, indexing, rendering, speed, structure, security and machine-readable data. Unlike content or links, it is mostly invisible when it is working - and capable of erasing everything else when it is not.
The pillar that is binary#
Content quality is a spectrum. A page can be quite good, or very good, or the best on the internet, and it will rank accordingly.
Technical SEO does not work like that. A noindex tag is on or off. A robots.txt file either blocks Googlebot or it does not. Your redirects either exist or they were forgotten during the redesign. There is no partial credit, and a single line of text in a file most people never open can take a site from ranking well to ranking nowhere.
That asymmetry is why this pillar deserves attention even from businesses with no technical staff. You do not need to implement any of it yourself. You need to know enough to ask whether it has been checked - because when it fails, nothing else you paid for matters.
And in 2026 there is a new version of that failure, quieter than the old one: sites that are perfectly visible to Google and completely invisible to the AI systems people increasingly ask for recommendations. Section 15 covers it, and it is the part of this guide most likely to apply to you right now.
Prerequisites: What Is SEO? for the fundamentals. What Is On-Page SEO? for the pillar you control completely. What Is Off-Page SEO? for the authority layer. This one is the foundation all three sit on.
Table of Contents#
1. What technical SEO covers 2. Crawling: how search engines find your pages 3. Indexing: what gets stored 4. Rendering and JavaScript 5. Site architecture 6. URLs, canonicals and duplicate content 7. Redirects 8. robots.txt and sitemaps 9. Speed and Core Web Vitals 10. Mobile 11. HTTPS and security 12. Structured data, technically 13. International and multilingual 14. Log files and crawl budget 15. AI crawlers: the 2026 frontier 16. Migrations: the biggest risk in SEO 17. Diagnosing common problems 18. The technical SEO audit checklist 19. Tools 20. When to do it yourself and when to hire 21. Frequently asked questions
1. What Technical SEO Covers#
1.1 The scope#
| Area | The question it answers |
|---|---|
| Crawling | Can search engines reach your pages? |
| Indexing | Do they store them? |
| Rendering | Can they see the content after JavaScript runs? |
| Architecture | Is the site organised so importance is clear? |
| Canonicalisation | Do they know which version is the real one? |
| Redirects | Does old value flow to new addresses? |
| Performance | Does the page load acceptably? |
| Mobile | Does it work on a phone? |
| Security | Is the connection encrypted and the site uncompromised? |
| Structured data | Can machines parse what the page contains? |
| International | Do the right language versions reach the right people? |
| Crawler policy | Which bots are allowed, and which have you blocked by accident? |
1.2 The four-stage pipeline#
Almost every technical problem lives at one of four stages, and knowing which one saves enormous diagnostic time.
Discover → Crawl → Render → Index → Rank
noindex, duplicate of another page, judged too thin to store1.2.1 The diagnostic value of this sequence
When a page is not performing, work down the list in order. Most people start at the bottom and rewrite content on a page that Google never indexed. Check indexation first; it takes thirty seconds in Search Console.2. Crawling: How Search Engines Find Your Pages#
2.1 What a crawler does#
Automated software - Googlebot, Bingbot, and now a growing list of AI crawlers - requests pages, reads them, follows the links it finds, and repeats. This is how the web is discovered.
2.2 How crawlers find your pages#
1. Links from pages they already know - the primary mechanism 2. XML sitemaps you submit 3. Internal links from your own pages 4. External links from other sites
2.3 What blocks crawling#
| Blocker | Effect |
|---|---|
robots.txt disallow rule | Crawler does not request the page |
| Server errors (5xx) | Crawler cannot retrieve it |
| Very slow responses | Crawler backs off and crawls less |
| Login or paywall | Content invisible |
| No internal links pointing to it | Never discovered in the first place |
| IP or user-agent blocking at firewall or CDN | Silently refused, often without anyone knowing |
2.3.1 The classic catastrophe
A staging site is built withDisallow: / in robots.txt to keep it out of search. At launch, the file is copied to production unchanged. Organic traffic goes to zero, and because nothing looks broken to a human visitor, it can take weeks for anyone to work out why.
This is not a rare edge case - it is one of the most common serious technical failures in the industry, and it costs businesses months.
##### The thirty-second check that prevents it
Open yourdomain.com/robots.txt in a browser. Right now. If you see Disallow: / under User-agent: *, you have found your problem.
3. Indexing: What Gets Stored#
3.1 Crawled is not indexed#
Google visiting your page does not mean Google kept it. Search Console distinguishes these clearly, and "Crawled - currently not indexed" is one of the most common and most misunderstood statuses in SEO.
It usually means Google looked at the page and decided it was not worth storing: too thin, too similar to something else, or not sufficiently useful.
3.2 Reasons a page is not indexed#
| Status | Meaning | Fix |
|---|---|---|
Excluded by noindex | You told Google not to index it | Remove the tag if unintended |
| Duplicate, Google chose different canonical | Google thinks another page is the real version | Consolidate or clarify canonicals |
| Crawled, currently not indexed | Judged insufficiently valuable | Improve or consolidate the page |
| Discovered, currently not indexed | Queued, often a crawl budget signal | Improve internal linking, reduce low-value URLs |
| Soft 404 | Page looks empty or error-like | Add real content or return a proper 404 |
| Blocked by robots.txt | Crawler cannot see it | Unblock if it should be indexed |
| Redirect | The URL redirects elsewhere | Usually correct |
3.3 Controlling indexation deliberately#
Not everything should be indexed. Thank-you pages, internal search results, filtered parameter URLs, tag archives with one post, print versions, and staging environments should all be excluded.
3.3.1 The important distinction
robots.txt blocks crawling. noindex blocks indexing. They are not interchangeable, and combining them incorrectly is a common error: if you block a page in robots.txt, Google cannot crawl it to see the noindex tag, so a previously indexed page can stay in the index indefinitely.
To remove a page from the index: allow crawling, add noindex, wait for recrawl, then block in robots.txt afterwards if you want.
##### Index bloat Sites that let every parameter combination, tag page and filter state get indexed end up with tens of thousands of near-worthless URLs competing with their real pages for crawl attention. Pruning index bloat is frequently the highest-impact technical work available on a large site.
4. Rendering and JavaScript#
4.1 Why rendering is a separate stage#
Modern websites often deliver an almost-empty HTML shell, then build the visible page with JavaScript in the browser. A crawler that reads the raw HTML sees nothing. It must execute the JavaScript to see what a human sees.
Google does this, in a second pass, with a delay. Many other crawlers do not do it at all.
4.2 The rendering options#
| Approach | How it works | SEO impact |
|---|---|---|
| Server-side rendering (SSR) | HTML built on the server, delivered complete | Safest |
| Static generation (SSG) | Pages pre-built at deploy time | Safest and fastest |
| Client-side rendering (CSR) | Browser builds the page from JavaScript | Riskiest |
| Dynamic rendering | Crawlers get pre-rendered HTML, users get the app | Workable but a legacy workaround |
| Hybrid / partial hydration | Critical content server-rendered, the rest client-side | Good compromise |
4.3 The 2026 reason this matters far more than it used to#
Google renders JavaScript. Most AI crawlers do not.
Research by Vercel and MERJ found that 69% of AI crawlers cannot execute JavaScript. If a site relies on client-side rendering, AI bots see a blank page regardless of what robots.txt says.
This is one of the most consequential technical findings of the last two years, and it is barely discussed outside developer circles. A React or Vue site that ranks perfectly well in Google can be entirely invisible to the systems generating AI recommendations - not blocked, not penalised, simply unreadable.
4.3.1 How to test it yourself
1. In your browser, disable JavaScript and load a key page. What remains?
2. View the raw page source (Ctrl+U / Cmd+Option+U) and search for a sentence from your main content. If it is not there, it is being injected by JavaScript.
3. Use Google's URL Inspection tool and look at the rendered HTML versus the raw HTML.
##### The fix, in order of preference Server-side render or statically generate the content that matters. Failing that, ensure critical text - headings, main copy, product details, prices, contact information - is present in the initial HTML even if the interactive layer loads afterwards.
5. Site Architecture#
5.1 What good architecture looks like#
5.2 Why depth matters#
Pages buried deep receive less internal link value, get crawled less often, and are found less easily by humans. Depth is a proxy for importance, and search engines read it that way.
5.3 The flat vs deep trade-off#
A completely flat site - everything linked from the homepage - dilutes signals and confuses hierarchy. A very deep site buries things. The workable middle is a clear hierarchy of two to three levels, with cross-linking between siblings.
5.3.1 Faceted navigation
Filter and sort options on e-commerce and listing sites can generate millions of URL combinations. Left uncontrolled, this consumes crawl budget, creates duplicate content, and buries the pages that matter.Control it with: canonical tags pointing to the main category, noindex on filter combinations, robots.txt rules for parameters that never need crawling, and links to filtered states rendered so they are not crawlable when they should not be.
##### Orphan pages Pages with no internal links pointing at them. They may exist, they may even be in the sitemap, but nothing on your site says they matter. Audit for these quarterly with a crawler.
6. URLs, Canonicals and Duplicate Content#
6.1 URL structure#
6.2 The duplicate content problem#
Most duplicate content is accidental. The usual sources:
| Source | Example |
|---|---|
www vs non-www | example.com and www.example.com both resolving |
http vs https | Both versions live |
| Trailing slashes | /page and /page/ |
| URL parameters | ?sort=price, ?utm_source=..., ?sessionid=... |
| Uppercase variants | /Page and /page |
| Pagination | Page 2 duplicating page 1's content |
| Printer-friendly versions | Same content, different URL |
| Faceted filters | Every combination a separate URL |
6.3 Canonical tags#
A canonical tag names the preferred version of a page:
``html
`
6.3.1 Canonical rules
##### The Google-chose-different-canonical warning When Search Console reports this, Google disagreed with your choice. Usually it means two pages are too similar to be distinguishable. The answer is rarely to fight it with tags - it is to make the pages genuinely different, or merge them.
7. Redirects#
7.1 The types#
| Code | Meaning | Use for |
|---|---|---|
| 301 | Moved permanently | Permanent URL changes; passes signals |
| 302 | Found / temporary | Genuinely temporary moves only |
| 307 | Temporary redirect | HTTP/1.1 equivalent of 302 |
| 308 | Permanent redirect | HTTP/1.1 equivalent of 301 |
| Meta refresh | Client-side delay redirect | Avoid |
| JavaScript redirect | Redirect after script execution | Avoid where possible |
7.2 The rules#
not A → B → C → D7.2.1 Why redirecting everything to the homepage fails
Google treats a redirect to an irrelevant page as a soft 404 and passes little or no value. If the old page has no equivalent, a proper 404 or 410 is often the better answer.##### Redirect chains and crawl waste Each hop costs crawl budget and loses a small amount of signal. On a large site, thousands of chained redirects meaningfully reduce how much of your site gets crawled.
8. robots.txt and Sitemaps#
8.1 robots.txt#
A plain text file at the root of your domain telling crawlers where they may go. It is standardised via RFC 9309.
`
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /*?sessionid=
Sitemap: https://example.com/sitemap.xml
`
8.1.1 What robots.txt does and does not do
##### The rule for anything sensitive Never rely on robots.txt to hide private content. Use authentication. The file is publicly readable and functions as a map of your admin paths.
8.2 XML sitemaps#
A machine-readable list of URLs you want indexed.
dates accurate; inaccurate ones get ignored8.2.1 What sitemaps do not do
They do not guarantee indexing. They aid discovery. A page in your sitemap that Google chooses not to index has a content problem, not a sitemap problem.9. Speed and Core Web Vitals#
9.1 The three metrics#
| Metric | Measures | Target |
|---|---|---|
| LCP - Largest Contentful Paint | When the main content becomes visible | Under 2.5 seconds |
| INP - Interaction to Next Paint | Responsiveness to user input | Under 200 milliseconds |
| CLS - Cumulative Layout Shift | Visual stability while loading | Under 0.1 |
These are measured from real users in field data, not from a lab test. A perfect score in a testing tool means little if actual visitors on mid-range phones over mobile networks experience something worse.
9.2 The usual causes of poor scores#
LCP problems: slow server response, large unoptimised hero images, render-blocking CSS and JavaScript, fonts loading late.
INP problems: heavy JavaScript execution, long tasks blocking the main thread, excessive third-party scripts.
CLS problems: images without width and height attributes, ads and embeds inserted without reserved space, fonts causing reflow, banners injected above existing content.
9.3 The fixes that usually matter most#
1. Compress and correctly size images; serve WebP or AVIF 2. Set explicit dimensions on all images and embeds 3. Defer non-critical JavaScript 4. Audit third-party scripts ruthlessly - chat widgets, heatmaps, tag managers 5. Use a CDN with edge locations near your audience 6. Preload the LCP image and critical fonts 7. Reduce server response time; upgrade hosting if it exceeds 600ms
9.3.1 The honest weighting
Core Web Vitals are a modest ranking factor and an enormous conversion factor. Fix them because slow sites lose sales, and take the ranking benefit as a bonus.##### The third-party script problem Most sites are slow because of code somebody else wrote. Open your tag manager and list what fires on every page. There is almost always something nobody remembers adding.
10. Mobile#
10.1 Mobile-first indexing#
Google indexes the mobile version of your site. If content exists on desktop but not mobile, for indexing purposes it does not exist.
10.2 The requirements#
10.2.1 The hidden-content trap
Content collapsed behind accordions and tabs on mobile is indexed normally - that is fine. Content removed entirely from the mobile version is not indexed at all. Check that your mobile template is not stripping sections your desktop version shows.##### The interstitial rule Pop-ups covering the main content immediately on arrival are explicitly penalised. Legally required cookie banners are exempt when reasonably sized. A full-screen newsletter modal before the reader sees a sentence is not.
11. HTTPS and Security#
11.1 The baseline#
11.2 Why security is an SEO issue#
A hacked site is an SEO catastrophe. Injected spam pages, hidden links and malicious redirects trigger manual actions, browser warnings and deindexation. Recovery is slow even after the compromise is cleaned.
11.2.1 Signs of compromise worth watching for
##### The prevention basics Keep the CMS and plugins updated, remove unused plugins and themes, enforce strong credentials with two-factor authentication, limit admin accounts, and keep offsite backups you have actually tested restoring.
12. Structured Data, Technically#
12.1 Implementation#
Use JSON-LD in the
- Google's stated preference, and the easiest to maintain because it is separate from your visible markup.
`html
`
12.2 The technical rules#
Only mark up content visible on the page
Keep values accurate - false prices, ratings or availability get markup ignored or penalised
Validate with the Rich Results Test and the Schema.org validator
Deploy at template level so every page of a type gets it consistently
Monitor the Enhancements reports in Search Console for errors
Nest entities properly rather than scattering disconnected blocks
12.2.1 Why structured data matters more in 2026
Two reasons. Rich results improve click-through in a world where clicks are scarcer. And structured data is machine-readable by definition - which makes it directly useful to systems deciding whether they can confidently state facts about your business.
13. International and Multilingual#
13.1 hreflang#
Annotations telling search engines which language and region each version of a page serves.
`html
`
13.2 The rules that get broken#
Every page must reference every alternate, including itself. Missing return tags are the most common hreflang error.
Use correct ISO codes: pt-PT and pt-BR are different; en-GB and en-US are different
Include x-default for your fallback
All referenced URLs must be indexable and canonical
hreflang and canonical must not contradict each other
13.3 URL structure options#
Structure Example Notes ccTLD example.pt Strongest geo signal, most expensive Subdirectory example.com/pt/ Easiest to manage; recommended default Subdomain pt.example.com Workable, weaker consolidation Parameters example.com?lang=pt Avoid
##### The two rules that prevent most international failures
Never auto-redirect by IP address - let users and crawlers choose, and offer a visible language selector. And never machine-translate without review; for Portugal specifically, translation tools frequently produce Brazilian Portuguese, which is both a quality failure and a visible signal that nobody native checked.
14. Log Files and Crawl Budget#
14.1 What crawl budget is#
How much crawling a search engine allocates to your site, determined by how much it can crawl without straining your server and how much it wants to crawl based on perceived value.
14.2 Who needs to care#
Sites under a few thousand pages: generally not an issue
Large e-commerce, marketplaces, publishers: significant
Sites with faceted navigation generating URL explosions: critical
14.3 What wastes it#
Infinite URL spaces from filters and parameters
Redirect chains
Soft 404s and error pages
Duplicate content
Very slow server responses
Low-value pages nobody should be indexing
14.4 Log file analysis#
Server logs record every crawler request. They answer questions no other tool can:
Which pages does Googlebot actually visit, and how often?
Which crawler is consuming your server capacity?
Are crawlers wasting time on parameter URLs?
Are important pages being crawled at all?
Which bots are hitting you that you had not considered?
14.4.1 Why logs matter more in 2026
Because the crawler population has changed dramatically. The number of active AI bots has more than doubled since 2023. Your logs are the only place you can see which of them are actually reaching your site - and whether something in your stack is turning them away.
15. AI Crawlers: The 2026 Frontier#
This is the newest area of technical SEO, the least understood, and the one most likely to be quietly costing you visibility right now.
15.1 The distinction almost everyone gets wrong#
Training crawlers and search crawlers are now separate bots. This is the single most important thing in this section.
A training crawler collects pages to build the dataset a model is trained on - your content becomes part of the model's weights, usually without attribution or referral. A search crawler indexes pages so an AI assistant can cite you in answers and link visitors back to your site. These now run as separate bots: GPTBot for training versus OAI-SearchBot for search at OpenAI, ClaudeBot versus Claude-SearchBot at Anthropic, Amazonbot versus Amzn-SearchBot at Amazon.
That split is what makes a precise policy possible - and what makes the old block-everything advice a quiet, self-inflicted loss of visibility.
15.2 The three separate decisions#
One robots.txt rule cannot make all three of these decisions correctly:
Decision What blocking means Training Opting your content out of future model training - an IP and privacy decision Search and retrieval Removing yourself from AI answers - a visibility decision User-triggered fetching Preventing AI assistants from completing user requests on your site - an access decision
15.3 The scale of the accidental damage#
An audit of B2B sites in early 2026 found 41% still blocking at least one major AI bot - usually a leftover from the "block everything" panic of 2023 and 2024. Among news publishers, 62% block GPTBot and 69% block ClaudeBot, and most are not making a strategic choice - they configured robots.txt years ago and moved on.
Blocking GPTBot has no measurable impact on Google Search rankings, but blocking OAI-SearchBot removes you from ChatGPT search answers entirely.
15.3.1 The CDN problem nobody checks
Cloudflare, which protects roughly a fifth of all websites, began blocking AI crawlers by default on new domains in 2024. Many technical SEO teams have perfectly configured robots.txt files that are being silently overridden at the CDN layer.
Your robots.txt is not the whole story. Check your CDN's bot management settings, your WAF rules, and any security plugin with a "block bad bots" feature. Then check your server logs to confirm what is actually getting through.
15.4 A defensible starting policy#
For most businesses: allow search and retrieval crawlers, make an explicit decision on training crawlers, and recognise that some agents ignore robots.txt entirely and require server-side controls.
`
# Allow AI search and retrieval - these can cite you
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Explicit decision on training crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
`
##### Read this before copying it
The block above assumes you want AI citations but not to contribute to training. That is a common position, not a universally correct one. A publisher whose business is content may block more; a business that wants maximum AI visibility may allow everything. Decide deliberately, then implement. Note also that
Google-Extended and Applebot-Extended are opt-out tokens for generative training and do not affect Googlebot or Google Search.
15.5 llms.txt#
A proposed convention - a markdown file listing your most important content for AI systems.
robots.txt controls crawling and is standardised; llms.txt is a newer, not yet standardised convention intended to guide AI systems toward relevant content for specific queries. They operate at different levels and are not alternatives to each other.
15.5.1 Is it worth implementing?
Adoption remains low - roughly one in ten domains. It costs very little to add, no major AI provider has committed to honouring it, and evidence of benefit is currently anecdotal. Reasonable position: add it if you have a well-structured content library and ten spare minutes; do not prioritise it over rendering, crawler policy or speed.
15.6 The audit, in five steps#
1. Read your robots.txt. Note every AI user-agent mentioned and what it is allowed.
2. Check your CDN and WAF. Bot management may be overriding the file entirely.
3. Check your server logs. Which AI crawlers actually reach you, and how often?
4. Test rendering. If your content requires JavaScript, most AI crawlers see nothing.
5. Decide deliberately on training versus search, then document the decision so the next developer does not undo it.
##### A note on the claimed gains
You will see specific figures attached to unblocking - percentage citation gains, traffic multiples. These come from small vendor cohorts and self-reported case studies rather than independent research. The directional claim is sound and the mechanism is obvious. The precise numbers are marketing.
16. Migrations: The Biggest Risk in SEO#
More organic traffic has been destroyed by well-intentioned website redesigns than by any Google update.
16.1 What counts as a migration#
A redesign that changes URLs
Moving to a new CMS or platform
Changing domain name
Moving from HTTP to HTTPS
Restructuring site architecture
Consolidating multiple sites into one
Changing from subdomain to subdirectory
16.2 The pre-launch requirements#
☐ Full crawl of the current site, saved as a baseline
☐ Export of all Search Console data - 16 months maximum, so do it early
☐ A URL-by-URL redirect map, old to new, reviewed by a human
☐ Preservation of high-value content - do not quietly delete pages that rank
☐ Staging site blocked from indexing, with a documented plan to unblock at launch
☐ Analytics and tracking configured on the new site before it goes live
☐ Structured data, canonicals, hreflang and meta data carried across
☐ A rollback plan
16.3 Launch day#
Verify robots.txt does not contain Disallow: / - the single most common launch failure
Test a sample of redirects immediately
Submit the new sitemap in Search Console
Confirm analytics is recording
Crawl the live site and compare against the baseline
16.4 The weeks after#
Monitor indexation daily for the first fortnight
Watch for 404 spikes and fix them as they appear
Expect a temporary ranking dip; it is normal
Do not make further large changes for a month
If traffic has not recovered within six to eight weeks, something is wrong - investigate rather than wait
##### The rule that prevents most migration disasters
Never let anyone tell you the redirect map can be done after launch. It cannot. Every day without redirects is accumulated loss, and the traffic does not come back automatically when they are finally added.
17. Diagnosing Common Problems#
17.1 "My site is not appearing on Google"#
Work through in order:
1. Is it indexed? Search
site:yourdomain.com
2. Does robots.txt block it?
3. Is there a noindex tag?
4. Is the site new? New domains take time
5. Are there any links pointing to it at all?
6. Check Search Console for manual actions
17.2 "Traffic dropped suddenly"#
1. Check Search Console for a manual action
2. Check the date against known Google update timelines
3. Check whether it is seasonal - compare year on year
4. Check whether analytics or consent configuration changed
5. Check whether the site was redesigned or migrated recently
6. Check which pages and queries lost, sitewide versus sectional
7. Check server logs and uptime for the period
17.3 "Pages are indexed but get no traffic"#
Usually not a technical problem. Likely intent mismatch, insufficient authority, or content that does not compete. Check average position in Search Console before assuming anything technical is broken.
17.4 "Google indexed the wrong version of my page"#
Canonical conflict. Check for duplicate URLs, contradictory canonical tags, and whether the canonical target is itself indexable.
17.5 "Rich results disappeared"#
Check the Enhancements reports in Search Console for structured data errors, and verify the markup still matches visible page content after any template change.
17.6 "AI systems never mention us"#
New in 2026, and often technical rather than reputational:
1. Are AI search crawlers blocked in robots.txt?
2. Is your CDN blocking them regardless of robots.txt?
3. Does your content require JavaScript to be visible?
4. Is your entity information consistent across the web?
5. Only after all four: is it an authority problem?
18. The Technical SEO Audit Checklist#
18.1 Crawling and indexing#
☐ robots.txt reviewed line by line
☐ No unintended Disallow: / tags
☐ Index bloat identified and pruned
18.2 Rendering#
☐ Critical content present in raw HTML
☐ Rendered HTML checked in URL Inspection
☐ JavaScript-dependent content identified
18.3 Structure#
☐ No important page deeper than three clicks
☐ No orphan pages
☐ Faceted navigation controlled
☐ Breadcrumbs implemented
18.4 URLs and duplication#
☐ One canonical version of the domain enforced
☐ Self-referencing canonicals present
☐ Parameter handling configured
☐ No duplicate title tags across pages
18.5 Redirects#
☐ No chains longer than one hop
☐ No redirect loops
☐ Internal links point to final URLs
☐ 404s reviewed and important ones redirected
18.6 Performance#
☐ Core Web Vitals passing on mobile field data
☐ Images optimised and correctly sized
☐ Third-party scripts audited
☐ Server response under 600ms
18.7 Mobile#
☐ Responsive, same content as desktop
☐ No intrusive interstitials
☐ Tap targets adequate
18.8 Security#
☐ Valid certificate, HTTPS enforced
☐ No mixed content
☐ No security issues in Search Console
☐ CMS and plugins current
18.9 Structured data#
☐ Correct types deployed at template level
☐ Validated without errors
☐ Matches visible content
18.10 International#
☐ hreflang complete with return tags
☐ Correct locale codes, pt-PT not pt-BR where relevant
☐ x-default present
☐ No IP-based auto-redirects
18.11 Crawler policy#
☐ AI crawler rules reviewed and deliberate
☐ Training versus search crawlers distinguished
☐ CDN and WAF bot settings checked
☐ Server logs confirm which bots reach the site
19. Tools#
Tool Purpose Cost Google Search Console Indexation, performance, errors, manual actions Free URL Inspection Rendered HTML, indexing status per page Free PageSpeed Insights Core Web Vitals, field and lab data Free Rich Results Test Structured data validation Free Screaming Frog Site crawling and auditing Free tier, then paid Sitebulb Crawling with prioritised recommendations Paid Ahrefs / Semrush Site audits alongside other data Paid Chrome DevTools Rendering, performance, network inspection Free Server log analysis What crawlers actually do Varies Wayback Machine Comparing historical versions Free
##### The honest minimum
Search Console and one crawler will find the overwhelming majority of technical problems on a normal business website. Everything else is refinement.
20. When to Do It Yourself and When to Hire#
20.1 What a non-technical owner can genuinely do#
Read robots.txt and check for Disallow: /
Set up Search Console and read the index coverage report
Check whether pages are indexed with site: searches
Run PageSpeed Insights and pass the results to a developer
Verify HTTPS works and the certificate is valid
Test the site on a phone
Check whether content is visible with JavaScript disabled
20.2 What genuinely needs expertise#
Log file analysis
JavaScript rendering diagnosis and fixes
Site migrations
Faceted navigation and crawl budget on large sites
hreflang at scale
Diagnosing traffic drops with multiple plausible causes
Structured data at template level
20.3 The realistic split#
Most small business sites need a one-off audit and fix rather than a monthly technical retainer. Technical SEO is largely finite: you fix the issues, then monitor. An agency billing for "technical SEO" every month for a year with no changelog is billing for observation.
20.3.1 The exception
Large e-commerce, publishers and any site shipping code weekly need continuous technical attention, because every deploy is an opportunity to break something.
##### The question to ask any agency
"What technical changes did you make this month, and to which URLs?" If the answer is vague for three consecutive months on a site with known issues, you are funding a placeholder.
21. Frequently Asked Questions#
21.1 What is technical SEO in simple terms?#
Making sure search engines and AI systems can find, read, understand and trust your website. It covers crawling, indexing, rendering, speed, structure, security and machine-readable data.
21.2 What is the difference between technical and on-page SEO?#
Technical concerns your site's infrastructure - how it is built and served. On-page concerns individual pages - content and how it is presented. They overlap at page speed and structured data.
21.3 Do I need technical SEO for a small website?#
Yes, but usually as a one-off audit rather than ongoing work. A ten-page brochure site has few technical problems, and the ones it has tend to be serious: blocked crawling, missing redirects, no HTTPS.
21.4 What is crawling and indexing?#
Crawling is search engines discovering and reading your pages. Indexing is deciding to store them. Crawled does not mean indexed - Google can read a page and choose not to keep it.
21.5 What is robots.txt and can it hurt me?#
A file telling crawlers where they may go. It can hurt you badly: a single Disallow: / line blocks your entire site, and this is one of the most common serious errors in SEO, usually introduced when staging configuration is copied to production.
21.6 Does robots.txt stop pages appearing in Google?#
No. It stops crawling, not indexing. A blocked URL can still appear if other sites link to it. To remove a page from the index, allow crawling and use a noindex tag.
21.7 What are Core Web Vitals?#
Three measurements from real users: LCP for loading, INP for responsiveness, CLS for visual stability. Targets are 2.5 seconds, 200 milliseconds and 0.1 respectively.
21.8 Does site speed affect rankings?#
Modestly. It affects conversions considerably more. Optimise it primarily for revenue and accept the ranking benefit as secondary.
21.9 Is JavaScript bad for SEO?#
Not inherently - Google renders it. But most AI crawlers do not, with research finding around 69% cannot execute JavaScript. If your content only appears after scripts run, you may rank in Google while being invisible to AI systems.
21.10 What is a canonical tag?#
An annotation naming the preferred version of a page when several URLs show similar content. It is a hint, not a directive, and Google can override it.
21.11 What is the difference between 301 and 302 redirects?#
301 is permanent and passes ranking signals. 302 is temporary. Use 301 for anything that will not revert.
21.12 Will changing my URLs hurt my rankings?#
Yes, unless every old URL 301-redirects to its closest equivalent, mapped before launch. This is the single most common cause of catastrophic traffic loss.
21.13 What is hreflang and do I need it?#
Annotations telling search engines which language and region each page version serves. You need it if you publish in more than one language or target more than one country.
21.14 What is crawl budget and should I worry about it?#
How much crawling a search engine allocates to your site. Below a few thousand pages, generally not a concern. For large e-commerce or publishers with faceted navigation, it matters considerably.
21.15 Should I block AI crawlers?#
It depends which ones and why. Training crawlers and search crawlers are now separate bots - blocking GPTBot does not stop ChatGPT citing you, but blocking OAI-SearchBot does. Decide the training and visibility questions separately.
21.16 Is my site accidentally blocking AI crawlers?#
Possibly. An early 2026 audit found 41% of B2B sites blocking at least one major AI bot, usually left over from 2023–24. Check robots.txt, and also check your CDN - Cloudflare began blocking AI crawlers by default on new domains in 2024, which can override your file entirely.
21.17 What is llms.txt and do I need it?#
A proposed convention listing your key content for AI systems. It is not standardised, adoption is around one in ten domains, and no major provider has committed to honouring it. Low cost, uncertain benefit - do it after the fundamentals.
21.18 How often should I do a technical audit?#
Annually for a stable small site. Quarterly for e-commerce. Continuously if you deploy code frequently. Always before and after a migration.
21.19 How much does a technical SEO audit cost?#
Typically 500€–3,000€ depending on site size and complexity, delivered as a prioritised action list with effort estimates. A crawler export with no prioritisation is not an audit.
21.20 Can technical SEO fix bad rankings on its own?#
No. It removes obstacles. If a page is technically perfect but does not answer the query as well as competitors, it will not rank. Technical work makes good content findable; it does not make weak content good.
21.21 What is the single most valuable technical check?#
Open your robots.txt and confirm you are not blocking your own site. It takes thirty seconds, and when it is wrong it is the most expensive error in this guide.
22. Recommended Schema Markup for This Page#
`json
{
"@context": "https://schema.org",
"@type": "Article",
"headline": "What Is Technical SEO? The Complete 2026 Guide",
"description": "What technical SEO is, how crawling and indexing actually work, and the 2026 crawler settings quietly costing sites their visibility in AI search.",
"keywords": [
"what is technical SEO",
"technical SEO audit",
"crawling and indexing",
"Core Web Vitals",
"robots.txt",
"AI crawlers"
],
"author": {
"@type": "Organization",
"name": "Umarella",
"url": "https://umarella.org"
},
"publisher": {
"@type": "Organization",
"name": "Umarella"
},
"dateModified": "2026-08-01",
"about": [
{
"@type": "Thing",
"name": "Technical search engine optimization"
},
{
"@type": "Thing",
"name": "Web crawling"
}
]
}
`
Pair with
FAQPage markup on section 21 and BreadcrumbList` on the page hierarchy.
23. Conclusion: The Foundation Nobody Notices#
Technical SEO is the only pillar that can take a business from ranking well to ranking nowhere with a single line of text. It is also the only one that produces no visible reward when done correctly - a site that crawls cleanly, indexes fully and loads fast simply works, and nobody congratulates anybody.
That asymmetry is why it gets neglected until something breaks, and why the breakages tend to be expensive. The staging robots.txt copied into production. The redesign launched without a redirect map. The framework that renders content only in the browser. None of these are exotic. All of them are recoverable. All of them cost months when nobody checked.
And there is now a second, quieter version of the same failure. Sites that Google reads perfectly and AI systems cannot read at all - because the content requires JavaScript, or because a crawler rule written in 2023 is still blocking bots that did not exist in their current form. Nothing looks broken. Rankings hold. The business simply stops appearing in the answers people increasingly rely on, and there is no notification.
23.1 The three checks worth doing this week#
1. Open your robots.txt. Confirm you are not blocking your own site, and read every AI crawler rule as a deliberate decision rather than an inherited one. 2. Disable JavaScript and load your most important page. If the content vanishes, most AI crawlers see what you just saw. 3. Open Search Console and read the page indexing report. Compare pages you think should be indexed against pages that actually are.
Three checks, under an hour, no technical skill required beyond following the steps. They will not improve anything by themselves - but they will tell you whether the foundation everything else sits on is actually there.
23.2 The completed picture#
This guide finishes the four-pillar set. On-page is what you control. Off-page is what you earn. Local is what proximity constrains. Technical is what makes the other three legible.
None of them works alone. A technically flawless site with nothing worth reading ranks for nothing. Brilliant content on a site Google cannot crawl reaches nobody. The pillars are not a menu - they are a sequence, and technical comes first because everything else depends on it.
Want to know whether anything is quietly broken? Request a free technical audit covering crawling, indexing, rendering, performance and AI crawler policy - [umarella.org](https://umarella.org)
Verified 1 August 2026. AI crawler user-agent names reflect provider documentation current at the time of writing and change frequently - verify against OpenAI, Anthropic, Google and Perplexity documentation before implementing. The JavaScript rendering figure comes from Vercel and MERJ research; crawler-blocking prevalence figures come from vendor cohort audits with limited sample sizes and should be treated as indicative. Percentage gains claimed for unblocking AI crawlers are self-reported vendor case studies, not independent research.