# Technical SEO Checklist 2026: From Core Web Vitals to AI Crawlers
Source: https://oldportindia.com/insights/technical-seo-checklist-2026/
Author: Athul Iyju Jacob, Founder of Old Port (https://www.athuliyjujacob.com)
Publisher: Old Port Design LLP, Bengaluru and Kochi, India
Published: 2026-10-15 | Updated: 2026-11-05
Category: SEO & GEO
## Short answer
A 2026 technical SEO checklist covers six areas: Core Web Vitals (LCP within 2.5 seconds, INP at or under 200 milliseconds, CLS at or under 0.1, measured at the 75th percentile), crawlability and indexing, canonical tags, hreflang for regional versions such as en-IN and en-AE, valid structured data, and content present in server-rendered HTML. Then set deliberate robots.txt rules for AI crawlers such as GPTBot and OAI-SearchBot.
## Key takeaways
- Good Core Web Vitals are LCP of 2.5 seconds or less, INP of 200 milliseconds or less and CLS of 0.1 or less, at the 75th percentile of page loads.
- INP became a stable Core Web Vital in 2024, replacing First Input Delay; above 500 milliseconds is poor.
- Hreflang annotations must be reciprocal: the en-IN and en-AE pages must each list themselves and each other, or Google ignores them.
- Google-Extended is a robots.txt control token, not a separate crawler, and blocking it does not affect Google Search inclusion or ranking.
- OpenAI's OAI-SearchBot (ChatGPT search) and GPTBot (training) are controlled independently, so a site can stay visible in ChatGPT search while opting out of training.
A technical SEO checklist for 2026 has six core checks: Core Web Vitals, crawlability and indexing, canonical tags, hreflang for regional versions, structured data, and making sure important content is in the server-rendered HTML. On top of that, decide deliberately which AI crawlers may use your content. Technical SEO does not make weak content rank, but problems here can stop strong content from being found, indexed or cited by Google and AI assistants. This checklist follows the order we use in audits for Indian and Gulf sites.
## What are the Core Web Vitals thresholds in 2026?
Good means LCP of 2.5 seconds or less, INP of 200 milliseconds or less and CLS of 0.1 or less, measured at the 75th percentile of real page loads, separately for mobile and desktop. The thresholds come from Google's [web.dev Web Vitals guide](https://web.dev/articles/vitals) and the individual metric pages for [LCP](https://web.dev/articles/lcp), [INP](https://web.dev/articles/inp) and [CLS](https://web.dev/articles/cls).
| Metric | What it measures | Good | Needs improvement | Poor |
| --- | --- | --- | --- | --- |
| Largest Contentful Paint (LCP) | Loading of the main content | 2.5 s or less | 2.5 to 4 s | Over 4 s |
| Interaction to Next Paint (INP) | Responsiveness to taps, clicks and key presses | 200 ms or less | 200 to 500 ms | Over 500 ms |
| Cumulative Layout Shift (CLS) | Unexpected movement of content | 0.1 or less | 0.1 to 0.25 | Over 0.25 |
INP measures the delay between a user interaction and the next frame the browser paints, across the whole visit. It became a stable Core Web Vital in 2024 as the successor to First Input Delay, and it is the metric Indian sites most often fail, because mid-range Android phones struggle with heavy JavaScript, chat widgets, tag managers and sliders.
- Break up long JavaScript tasks and defer non-essential third-party scripts.
- Load chat, review and analytics widgets on idle or on first scroll.
- Serve correctly sized, modern-format images and preload the LCP image.
- Reserve space for images, embeds and ad slots to protect CLS.
- Use field data in Search Console's Core Web Vitals report, not just lab scores.
Keep perspective. Google's [Core Web Vitals documentation](https://developers.google.com/search/docs/appearance/core-web-vitals) says good scores align with what its core ranking systems reward, without giving them a fixed weight. Fix failures, but do not chase a perfect lab score at the expense of content.
## Can search engines crawl and index the pages that matter?
Check that nothing important is blocked, that sitemaps list only canonical pages, and that status codes tell the truth.
- **robots.txt** does not block CSS, JavaScript or important sections by accident, a frequent leftover from staging sites.
- **XML sitemaps** list only canonical, indexable, 200-status URLs and are submitted in Search Console.
- **Status codes** are honest: real 404s for removed pages, 301s for permanent moves, no soft 404s.
- **noindex** is used deliberately on thin utility pages (internal search, filters, thank-you pages) and nowhere else.
- **Internal links** are plain HTML anchors with descriptive text, so every important page is reachable in three or four clicks.
- **The Page indexing report** is reviewed monthly for "Crawled, currently not indexed" patterns.
## Are canonical tags pointing to the right URL?
Every indexable page should carry a self-referencing canonical, and every signal should point to the same URL. A canonical tag tells search engines which URL is preferred when several show the same or similar content. Common problems on Indian e-commerce and WordPress sites include tracking parameters, filter combinations, http and https or www and non-www duplicates, and paginated archives.
- Each indexable page carries a self-referencing canonical with the absolute https URL.
- Parameter and filter variants canonicalise to the clean version.
- Canonicals, sitemaps, internal links and hreflang all agree. Mixed signals make Google choose for you.
- Never canonicalise one regional page (en-AE) to another (en-IN). That tells Google to drop it.
## How should hreflang work for India and UAE versions?
List every regional version on every version, including itself, using codes such as en-IN and en-AE, plus an x-default fallback. Google's [localised versions documentation](https://developers.google.com/search/docs/specialty/international/localized-versions) requires an ISO 639-1 language code, optionally followed by an ISO 3166-1 alpha-2 region code, and says pages that do not link back are ignored. A business with Indian and UAE sites, such as Old Port's India site and its [UAE web development site](https://oldport.ae/services/web-development/), is a typical case: both in English, with different prices, contacts and services.
- ``
- ``
- ``
| Rule | Correct | Common error |
| --- | --- | --- |
| Reciprocity | en-IN and en-AE pages each list both, plus themselves | Only the main site lists the alternates |
| Code order | Language then region: en-IN, en-AE | in-en, or a bare region code |
| Fallback | x-default for users who match neither | No fallback set |
| Method | One method: HTML link tags, HTTP headers or XML sitemap | Conflicting annotations in two places |
| Regional languages | ml-IN for Malayalam, kn-IN for Kannada, hi-IN for Hindi | Using en-IN on translated pages |
For the wider strategy of selling in both markets, see our guide to [India and UAE expansion marketing](/insights/india-uae-business-expansion-marketing/).
## Is structured data valid and honest?
Use JSON-LD that matches what is visible on the page, and validate it. Prioritise Organization (with logo and sameAs links), LocalBusiness for each branch, Article with author details, Product with offers, BreadcrumbList and, where genuine, FAQPage. Validate with the Rich Results Test and the Schema Markup Validator. Google's guidance on [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) says there are no additional requirements or special schema needed to appear in AI Overviews or AI Mode, and lists structured data that matches visible text among the practices that still matter. Treat it as clarity for machines, not a shortcut.
## Does your JavaScript hide content from crawlers?
Make sure headings, body copy, links and prices are in the initial HTML. Google can render JavaScript, but rendering is queued and resource-limited, and many AI crawlers fetch raw HTML without running scripts.
- View source (not the inspector) and confirm your main heading, copy, links and prices are present.
- Use server-side rendering or static generation for React, Vue and similar frameworks on pages that need to rank.
- Do not hide core content behind tabs or "load more" buttons that fetch only on click.
- Use the URL Inspection tool in Search Console to compare rendered HTML with what you expect.
## How should robots.txt handle AI crawlers?
Decide separately for search, training and user-triggered fetching, because the major AI companies now use different user agents for each.
| Company | Search or answers | Training | User-triggered |
| --- | --- | --- | --- |
| OpenAI | OAI-SearchBot | GPTBot | ChatGPT-User (robots.txt may not apply) |
| Anthropic | Claude-SearchBot | ClaudeBot | Claude-User |
| Perplexity | PerplexityBot | Not used for training | Perplexity-User (generally ignores robots.txt) |
| Google | Googlebot (Search, AI Overviews, AI Mode) | Google-Extended token | Not applicable |
- **OpenAI:** its [crawler documentation](https://developers.openai.com/api/docs/bots) says each setting is independent, so you can allow OAI-SearchBot while disallowing GPTBot.
- **Google:** Google-Extended is a product token, not a separate crawler. It controls whether content is used to train future Gemini models and for grounding in Gemini Apps and Vertex AI, and Google says it does not affect inclusion or ranking in Search, as [Search Engine Journal reported](https://www.searchenginejournal.com/google-updates-gemini-vertex-ai-user-agent-documentation/545409/).
- **Anthropic:** its [help centre](https://support.claude.com/en/articles/8896518) says ClaudeBot, Claude-SearchBot and Claude-User honour robots.txt, and supports the non-standard Crawl-delay directive.
- **Perplexity:** its [crawler page](https://docs.perplexity.ai/docs/resources/perplexity-crawlers.md) says PerplexityBot surfaces sites in search and is not used to train foundation models.
A common stance for a business that wants AI visibility but not training use:
- `User-agent: OAI-SearchBot` then `Allow: /`
- `User-agent: PerplexityBot` then `Allow: /`
- `User-agent: Claude-SearchBot` then `Allow: /`
- `User-agent: GPTBot` then `Disallow: /`
- `User-agent: ClaudeBot` then `Disallow: /`
- `User-agent: Google-Extended` then `Disallow: /`
Each pair goes on its own two lines, separated by a blank line, with existing Googlebot and Bingbot rules left untouched. This is a business decision, not a default; many brands allow training too, reasoning that presence in model knowledge helps. Record why you chose. Our [generative engine optimisation guide](/insights/generative-engine-optimization-guide/) explains how crawler access feeds AI citations.
## What technical SEO mistakes do we see most?
- A staging robots.txt with `Disallow: /` left live after launch.
- Blocking all AI user agents by copying a list, then wondering why ChatGPT search never cites the site.
- Regional pages canonicalised to the main site, so only one country version is indexed.
- Prices and product details loaded by JavaScript after the page renders.
- FAQ or review schema describing content that is not on the page.
## How often should you run this checklist?
- **Monthly:** Search Console indexing and Core Web Vitals reports, crawl errors, sitemap status.
- **Quarterly:** full crawl, canonical and hreflang validation, structured data checks, robots.txt review against current AI user agents.
- **Before every redesign or migration:** URL mapping, redirect plan and a staging robots.txt that is removed at launch.
## How we approach this at Old Port
We audit from field data and server responses first, not from a tool's score. We view raw HTML the way an AI crawler would, fix anything that blocks indexing before polishing speed, and agree a written AI crawler policy with the client rather than copying a template. Technical fixes create the conditions for content to perform; our guide to [building topical authority](/insights/content-strategy-topical-authority/) covers the content side. Our [SEO and GEO service](/services/seo-geo/) and [web development team](/services/web-development/) work together on fixes. For the Gulf view, see our UAE team's guide to [getting cited by ChatGPT, Gemini and Google AI Mode](https://oldport.ae/insights/ai-search-geo-2026-uae/).
## Frequently asked questions
### What is a good INP score?
An Interaction to Next Paint (INP) of 200 milliseconds or less at the 75th percentile of page loads is good. Above 200 and up to 500 milliseconds needs improvement, and above 500 milliseconds is poor. INP became a stable Core Web Vital in 2024, replacing First Input Delay, and measures responsiveness across the whole visit rather than only the first tap.
### Does blocking Google-Extended remove my site from Google Search?
No. Google-Extended is a robots.txt control token rather than a separate crawler. Disallowing it limits use of your content for training future Gemini models and for grounding in Gemini Apps and Vertex AI, but Google says it does not affect inclusion or ranking in Google Search. Googlebot continues to crawl and index your site as normal.
### Should I block GPTBot?
It depends on your goals. GPTBot collects content that may be used to train OpenAI models, while OAI-SearchBot powers ChatGPT search results, and OpenAI treats the two settings independently. Blocking GPTBot and allowing OAI-SearchBot keeps your site eligible for ChatGPT search citations while opting out of training. Some brands allow both to maximise presence.
### How do I set up hreflang for India and UAE English pages?
Add link elements on both pages listing en-IN for the Indian URL, en-AE for the UAE URL and an x-default fallback. Each page must list itself and the other, or Google ignores the annotations. Keep canonicals self-referencing on each regional page rather than pointing one country version at the other, and use only one annotation method.
### Can Google index JavaScript websites?
Google can render JavaScript, but rendering happens in a queue and can be delayed or incomplete, and many AI crawlers do not execute JavaScript at all. For pages that need to rank or be cited, use server-side rendering or static generation so headings, copy, links and prices are present in the initial HTML response.
### Do I need special markup to appear in AI Overviews?
No. Google says there are no additional requirements or special schema needed to appear in AI Overviews or AI Mode. A page must be indexed and eligible to show with a snippet. Standard good practice still applies: crawlable content, a good page experience, and structured data that matches the visible text on the page.
## Sources
- [web.dev: Web Vitals](https://web.dev/articles/vitals)
- [web.dev: Largest Contentful Paint (LCP)](https://web.dev/articles/lcp)
- [web.dev: Interaction to Next Paint (INP)](https://web.dev/articles/inp)
- [web.dev: Cumulative Layout Shift (CLS)](https://web.dev/articles/cls)
- [Google Search Central: Understanding Core Web Vitals and Google search results](https://developers.google.com/search/docs/appearance/core-web-vitals)
- [Google Search Central: Tell Google about localized versions of your page](https://developers.google.com/search/docs/specialty/international/localized-versions)
- [Google Search Central: AI features and your website](https://developers.google.com/search/docs/appearance/ai-features)
- [Search Engine Journal: Google updates Gemini and Vertex AI user agent documentation](https://www.searchenginejournal.com/google-updates-gemini-vertex-ai-user-agent-documentation/545409/)
- [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots)
- [Anthropic Help Center: Does Anthropic crawl data from the web?](https://support.claude.com/en/articles/8896518)
- [Perplexity: Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers.md)