Google’s John Mueller responded to a submit on Bluesky that lamented all the trouble being wasted on AI agent accessibility as an alternative of specializing in the extra productive exercise of constructing websites accessible for people.
Tactical web optimization
One of many constant facets of the historical past of web optimization is that practitioners are likely to observe tactical traits if it’s obvious that everybody else is doing it. I assume it’s human nature.
Again within the early days there was a man who constructed a listing and bought sufficient excessive PageRank hyperlinks to the house web page to get it to a few PageRank of seven (on a scale of 1-10). SEOs have been virtually pushing one another out of the way in which at hand this man their cash. He bought 1000’s of hyperlinks from his listing, and no person thought to verify if any of the hyperlinks really made a rating distinction. It was simple to verify, however no person did.
The most recent tactic is LLMs.txt and creating markdown pages for AI agent consumption. Cloudflare introduced in February that their infrastructure can mechanically create markdown information, largely as a manner for builders to avoid wasting LLM token consumption. Their announcement said that OAI-SearchBot, OpenAI’s search crawler, was consuming markdown pages.
OpenAI’s official documentation explicitly positions its OAI-SearchBot as crawling web sites and doesn’t point out or advocate markdown information.
“OAI-SearchBot is for search. OAI-SearchBot is used to floor web sites in search ends in ChatGPT’s search options. Websites which can be opted out of OAI-SearchBot is not going to be proven in ChatGPT search solutions, although can nonetheless seem as navigational hyperlinks. To assist guarantee your website seems in search outcomes, we advocate permitting OAI-SearchBot in your website’s robots.txt file and permitting requests from our revealed IP ranges under.”
Sure, it might crawl markdown if it’s pointed to it however that’s not what it’s on the market for.
Stephanie Walter posted:
“Unhappy fact: we’re making the online accessible for AIs, not for individuals.
Some websites now supply a textual content model for LLMs, however nonetheless skip actual accessibility wants like correct heading construction, landmarks that display reader customers want.”
Google’s John Mueller responded:
“A correctly made web site works nicely for AI brokers … and search engines like google and yahoo, and LLMs, and above all, for precise individuals.
For those who’re making an attempt to repair accessibility points by making a separate “agent-friendly” model, you’re simply constructing technical debt. You’ll must redo it a number of occasions. Simply repair it.”
AI Brokers Crawl HTML
It’s a comparatively trivial factor to crawl HTML. Serps have been doing it for over thirty years. Making websites accessible for everybody is smart, particularly because it’s person conduct alerts which have at all times pushed search engine rankings. From hyperlinks to customers looking with a model title, Google has constantly used person alerts for rating functions.
So, from the angle of web optimization, it makes a load of sense to make web sites accessible for everybody.
Featured Picture by Shutterstock/DETHAL