The research behind every check

OpenAI, Anthropic and Perplexity each publish the names of the robots that fetch pages for their AI answers, and explain how robots.txt allows or blocks them. A page that blocks them can’t be read by those tools, so they’re less likely to show or link to it.

The same companies, and Google with “Google-Extended”, use separate names for robots that collect text to train AI. They say blocking these doesn’t remove you from their search answers. Blocking them is a fair business choice, so this check never lowers your score.

Google can run a page’s scripts before reading it, but says this takes extra time. A 2024 analysis by Vercel of its own network found that none of the major AI crawlers, including OpenAI’s and Anthropic’s, ran scripts, so text drawn only by scripts was invisible to them. Microsoft Bing also warns that AI systems may not see hidden content.

llms.txt is a 2024 proposal for a short file that points AI tools to a site’s key pages. It isn’t an agreed standard, and Google says no special AI files are needed for its AI features. That’s why we show it for information only, without changing your score.

Google’s documentation on Organization markup says it helps Google understand who is behind a site, with its name, logo and contact details. Google’s guidelines for its human quality raters also ask them to look for clear information about who runs a website.

Google’s guidance on helpful content strongly encourages accurate author names where readers would expect them, and warns that invented authors are deceptive. Its quality rater guidelines treat the author’s experience and expertise as part of judging trust.

Google says a clear, visible date helps it understand when a page was published or updated. Its guidance also warns against changing a date to look fresh when the content hasn’t really changed.

Google’s quality rater guidelines ask raters to find out who is responsible for a website and how to contact them. Too little of this information is a reason for a lower quality rating. Clear links to About and Contact pages make it easy to find.

In a 2024 study of “generative engine optimisation”, researchers from Princeton University and other institutions rewrote web pages in different ways and measured how much AI answers used them. Adding relevant statistics was among the changes that helped most, by up to about 40% in their tests. Their tests used AI systems set up for the study, so real tools may differ. Microsoft Bing also advises backing claims with measurable facts.

The same 2024 study found that citing reliable sources was another of the most effective changes, making pages more visible in AI answers. As with every finding in it, the tests used AI systems set up for the study.

The same 2024 study found that adding quotes from relevant people or experts also made pages noticeably more visible in AI answers. Its tests used AI systems set up for the study, so real tools may behave differently.

Microsoft Bing advises writing headings and text as direct questions with clear answers, because that mirrors how people search, especially in AI answers. Google, by contrast, says its AI features need no special changes, so treat this as one search engine’s advice rather than a universal rule.

Microsoft Bing advises opening with a one- or two-sentence answer that makes sense on its own, so it can be lifted into an AI answer. This matches the “inverted pyramid” long recommended by Nielsen Norman Group: the main point first, details after, because readers often stop early.

A Nielsen Norman Group study found that pages rewritten to be concise, easy to scan (with lists and highlighted words) and objective were 124% more usable. Microsoft Bing also advises lists and tables for steps and comparisons, because they break details into clean pieces that AI can reuse.

The HTML standard has a <main> element for a page’s main content, alongside others for the header, menus and footer. Marking content this way helps screen readers, and lets other tools tell the article apart from menus and adverts.

Google announced in 2014 that it uses HTTPS, the secure version of a web address, as a small ranking signal. Browsers also mark pages without it as “Not secure”, and HTTPS protects what visitors type, such as passwords.

A “noindex” setting tells Google not to show a page in its results. Left on by mistake, it keeps the page out of Google completely, which is why this check counts for so much.

A robots.txt file tells search robots which parts of a site they may read. Google follows it, so a page blocked there can’t be read properly, although Google may still list its address without a description.

A sitemap is a file that lists the pages you want found. Google says it helps it discover pages, especially on new or large sites, but it doesn’t guarantee that they appear in results.

When the same content loads at more than one address, Google picks one to show. A “canonical” tag tells Google which one you prefer. Google treats it as a strong hint rather than a command.

Google says the page title is often used as the headline of your search result, and should be descriptive and concise. Google sets no character limit but cuts long titles short, so the length range we check is our own rule of thumb.

Google may show the meta description as the short text under your result when it describes the page well. Google sets no length limit; the range we check is our own, based on how much usually fits.

Google’s SEO Starter Guide recommends headings that show how a page is organised, and accessibility guidance says the main heading should describe the page. Google doesn’t require exactly one main heading; that part is our own rule for clarity.

W3C’s accessibility guidance asks for headings in order, without skipping levels, so people using screen readers can find their way around. Google says heading order doesn’t matter for its results, so this check is for readers rather than for Google.

Google says it has no preferred word count, and rewards pages that fully answer a visitor’s question. Very short pages rarely do, so we warn below a word count we chose ourselves. It’s a rough sign, not a Google rule.

WCAG, the international accessibility standard, asks for a text description of every meaningful image so blind visitors know what it shows. Google also uses these descriptions (alt text) to understand images.

Google finds most pages by following links. Its guidance says each page you care about should be linked from at least one other page on your site, with link words that describe where they lead.

Google mainly uses the phone version of a page for its results, which it calls mobile-first indexing. Without the “viewport” setting, a phone shows a tiny desktop page that is hard to read and use.

WCAG asks every page to state its language, so screen readers pronounce words correctly and browsers can offer translation. Google says it works out the language from the visible text and ignores this setting, so the check helps visitors rather than rankings.

Time to first byte is how long a server takes to start sending a page. Google’s web.dev guidance calls 0.8 seconds or less good and over 1.8 seconds poor, the limits we use. We measure once from our own server, which isn’t the same as Core Web Vitals from real visitors. A Deloitte study commissioned by Google found that slightly faster mobile pages led to more sales.

Servers can compress a page before sending it, with methods such as gzip or Brotli, and browsers unpack it on arrival. Mozilla’s documentation explains how this works. Smaller pages arrive faster, especially on slow phone connections.

The Open Graph protocol sets the title, text and picture shown when someone shares your link on social networks or in chat apps. It doesn’t affect Google rankings, but it decides how your page looks when it’s shared.

Structured data is hidden labelling that tells search engines what a page is about, such as a business, a product or an article. Google uses it to understand pages and for richer results, but correct labels don’t guarantee them. We only check that the labels can be read without errors.

An old copyright year in the footer is a small sign that nobody looks after the site. No search engine or law asks for a current year. This is our own check, which is why it counts for so little.

Under the UK and EU GDPR, a site that collects personal information must tell people what it collects and why, usually in a privacy notice. The UK regulator, the ICO, explains what this needs. Terms of use aren’t always required by law, but they set clear rules for visitors.

UK and EU rules on cookies say that cookies which aren’t essential, such as analytics and advertising, need a visitor’s agreement first. The ICO’s guide explains this. We look for known tracking and consent tools, but can’t prove what a site really does.

EU law on online services asks businesses to give an email address or another quick way to reach them, and the UK has the same rule. Google’s quality rater guidelines also treat clear contact details as part of judging whether a site can be trusted.

Google’s guide to status codes explains that a link to a missing page (a “404”) leads nowhere, and such pages drop out of search over time. Broken links don’t lower rankings directly, but they stop visitors and search robots reaching your content.

Google asks that missing pages send a real “404 not found” status, not an ordinary page that only says “not found”, which it calls a soft 404. A helpful error page also keeps lost visitors on your site.

If a page loads at both the www and non-www address, or at both http and https, Google has to choose one. Google recommends picking one address and permanently redirecting the others to it.

Links that open a new tab used to let the new page take control of the old one, a trick used for scams. Adding rel="noopener" stops this. Mozilla’s documentation notes that modern browsers now do this by default, so the risk is mainly on older browsers.

Google’s web.dev guide explains that WebP images are usually noticeably smaller than JPEG or PNG at the same quality, and AVIF is newer and often smaller still. Smaller images load faster, especially on phones.

When an image has no width and height set, the browser can’t keep space for it, so text jumps as it loads. Google measures this “layout shift” as one of its Core Web Vitals and recommends setting image sizes to prevent it.

Browsers can wait to load images until a visitor scrolls near them, with loading="lazy". Google’s guide says this saves data and speeds up the first view, but images at the top of the page should load straight away.

When a page uses its own font, some browsers hide the text until the font arrives. Google’s guide recommends “font-display: swap”, so text appears at once in a standard font and switches when the custom one is ready.

Caching lets a browser keep files it has already downloaded, so returning visitors don’t fetch them again. Google’s guide explains how a long “Cache-Control” lifetime does this for files that rarely change.

A “preload” hint tells the browser to fetch a key file, such as the main font or picture, early. Google’s guide says it can speed up loading when used for a few important files. It’s optional, so it never lowers your score.

WCAG asks for a way to skip blocks that repeat on every page, such as the menu. A “skip to content” link lets people who use a keyboard jump straight to the main content.

W3C’s guidance on landmarks (header, navigation, main content and footer) explains that screen readers let people jump between these areas. Pages built from them are much easier to find your way around without seeing them.

WCAG requires that people using a keyboard can always see which link or button is selected. We look for styles that hide this outline without putting something in its place.

WCAG asks for a contrast of at least 4.5 to 1 between normal text and its background, so people with low vision can read it. The WebAIM Million, a yearly test of one million home pages, finds low-contrast text is the most common problem. We don’t measure contrast, because that needs the page drawn on screen, and we say so rather than guess.

Visitors who type your address without https:// should land on the secure version. Mozilla’s documentation explains that a permanent redirect does this, and that the browser setting below (HSTS) closes the gap the redirect leaves.

HSTS is an internet standard (RFC 6797) that tells browsers to always use the secure version of your site, even when someone types or clicks an http:// link. Without it, a visit that starts at http:// can be intercepted before the redirect.

OWASP, a non-profit that publishes web security guidance, lists settings (headers) that a server can send so browsers block common attacks, such as your page being loaded inside another site or harmful scripts running. We check for the main ones it recommends.

Google shows a site’s icon next to its search results and explains how to set one. Browsers also show it in tabs and bookmarks, which helps visitors recognise your site.

GOV.UK, the UK government’s website, keeps sentences to 25 words or fewer, citing research that readers understand much less as sentences get longer. We warn above an average of 20 words, a slightly stricter limit of our own.

Nielsen Norman Group’s research on how people read online found that they scan, and that short paragraphs with one idea each make pages easier to use. The 150-word limit we use is our own.

The same research found that meaningful subheadings help readers scan a page and find what they need. We ask for a subheading at least every 350 words; that number is our own.

The Flesch reading ease score, published by Rudolf Flesch in 1948, rates text by the length of its sentences and words. A score of 60 or more is roughly plain English. It was made for English, so we score English pages only and say other languages aren’t measured.

The GOV.UK style guide says not to write long text in capital letters, because it’s hard to read. We look for runs of six or more words in capitals.

WCAG asks that the purpose of each link is clear from its words, or from its words and the sentence around it. People using screen readers often hear a list of all the links on a page, where “click here” tells them nothing.

WCAG requires every button and link to have a name that assistive technology can read out; an icon without a label is announced only as “button”. The WebAIM Million finds empty links and buttons among the most common problems on home pages.

WCAG asks for a label on every form field. W3C’s forms tutorial explains that grey placeholder text isn’t a replacement, because it disappears as people type and is often hard to see.

WCAG asks that text can be enlarged to twice its size without losing content. Phone settings that block pinch-to-zoom stop people with low vision from doing this.

No standard sets a minimum text size; WCAG asks instead that text can be enlarged. Browsers start at 16 pixels, and our limit of 14 pixels is our own rule of thumb.

WCAG asks for captions on recorded videos with sound, so deaf visitors, and anyone watching with the sound off, can follow them.

WCAG asks that any sound playing by itself for more than three seconds can be paused or turned down. Sound that starts on its own can also drown out a screen reader.

WCAG asks that moving, blinking or scrolling content can be paused, and that animation set off by scrolling can be turned off. Devices let people ask for less motion, and pages can respect that with the “prefers-reduced-motion” setting. Motion can make some people dizzy or sick.

Google’s guidance explains which pop-ups get in the way, such as ones that cover the content as soon as you arrive, especially on phones, and suggests less intrusive banners. We can’t see when a pop-up appears, so this check only warns.

The Coalition for Better Ads, a group of advertisers and publishers, researched which ads people find most annoying, with more than 150,000 people taking part. Its Better Ads Standards list them, such as pop-up ads, video that plays with sound and ads covering most of a phone screen.

Leftover text such as “Lorem ipsum” or “As an AI language model” tells readers nobody checked the page. Google says it judges content by its quality however it was made, and its spam policies cover pages made in bulk with little value. Bing’s guidelines also warn about large amounts of content made without editorial review.

A 2025 study in Science Advances counted words in more than 15 million medical research summaries and found a sudden rise in words such as “delves” and “showcasing” after ChatGPT arrived. Our list of about 40 phrases, and the limit of 2 per 1,000 words, are our own. They’re signs, not proof: a person can write them too.

The same sentence appearing three or more times is a common sign of text that was generated or copied in bulk. This is our own check. Google’s guidance on helpful content asks whether a page adds real value rather than repeating itself.