This put up was sponsored by JetOctopus. The opinions expressed on this article are the sponsor’s personal.
How do I optimize my website for ChatGPT and Perplexity, not simply Google?
How do I do know if AI bots are literally crawling my website?
How ought to my technical website positioning technique change for AI Search?
A good portion of your website’s search impressions in 2026 are generated by machines researching on behalf of humans.
These machines don’t care about your key phrase rankings. They care whether or not your:
- HTML hundreds cleanly in beneath 200 milliseconds
- Product element web page is reachable in fewer than 4 clicks
- Content material solutions a particular, nine-word query that has by no means appeared in any key phrase analysis software in your profession.
This isn’t hypothesis. It’s what our server log knowledge throughout lots of of enterprise web sites is displaying us, constantly, since mid-2025.
What’s Really Taking place On Your Website
My colleague, Stan, flagged a sample in a Slack message: question lengths have been rising at charges that didn’t correlate with human habits.
A 161% development price in 10-word queries year-over-year will not be pushed by customers who abruptly acquired extra verbose. It’s pushed by AI brokers decomposing a single consumer immediate into dozens of parallel sub-queries, a course of researchers now name “fan-out.”
Question Size Progress in 2025
The gradient is the inform. Human search habits doesn’t scale this cleanly by phrase rely. Machines do. By October 2025, 7-plus-word queries reached almost 1% of whole question quantity, roughly triple their historic share.
Extra revealing than the amount is the CTR. Whereas impression counts for 10-word queries spiked 161%, click-through price collapsed to 2.26%, down from 8–11% in 2023.
The AI reads your web page, extracts the reply, synthesizes it for the consumer. Your website by no means will get the go to.
We name these “phantom impressions.” They’re actual indicators that your content material is being evaluated inside AI reasoning chains. Should you’re filtering them out of your reporting as a result of they don’t drive visitors, you might be flying blind.
The Three Bots Visiting Your Website & Their Impression On SERP Visibility
Not all AI crawlers are equal, and treating them as a single class is the primary mistake most technical SEOs make.
Training bots crawl broadly and ignore click on depth. A coaching go to means the AI is aware of your content material exists, not that customers will ever see it.
AI search bots drop off rapidly past two or three clicks from the homepage and usually go to every web page solely as soon as a month.
AI consumer bots are initiated when an actual individual asks a query in ChatGPT, Perplexity, or Claude, and the AI researches the reply on their behalf. These are the one visits that translate to precise AI visibility.
| Bot Kind | What Triggers It | Crawl Depth | Impression on AI Visibility |
| Coaching bots | Mannequin schooling cycles | Deep — ignores click on distance | None straight. Consciousness solely. |
| AI search bots | New URL discovery & contemporary content material | Shallow — ~1 go to/month past 2–3 clicks | Essential gatekeeper. If it misses a web page, consumer bots received’t discover it both. |
| AI user bots | Actual consumer question in ChatGPT / Claude / Perplexity | Selective — pushed by pace and construction | Excessive. Closest proxy to an AI impression. |
Your website can obtain heavy crawling from coaching and search bots and nonetheless be utterly absent from AI-generated solutions. Should you’re not segmenting AI bot visitors by sort in your log analysis, you haven’t any thought which third of the iceberg you’re measuring.
Which website positioning Alerts Do LLMs Respect?
Robots.txt is your main lever.
Most main AI platforms (ChatGPT, Claude, Gemini) observe robots.txt directives. Perplexity is a partial exception: PerplexityBot respects robots.txt, however Perplexity-Consumer, the user-triggered bot, doesn’t. Cloudflare confirmed this in an investigation. Most websites haven’t audited their robots.txt with AI entry in thoughts. Do it.
Sitemaps are broadly supported.
ChatGPT, Claude, and PerplexityBot all use XML sitemaps for URL discovery. Hold them correct.
Alerts Finest Saved For website positioning & Rating Efforts
These indicators under don’t seem to influence AI visibility, however are nonetheless key for rating for queries that also set off conventional SERPs.
Canonical tags and noindex directives do nothing for AI bots.
AI crawlers don’t construct a search index, so that they don’t have any use for these meta-signals. Content material hidden from Google utilizing noindex is totally seen to ChatGPT’s crawler.
LLM.txt does nothing.
Our log knowledge exhibits main AI bots don’t learn this file. Don’t make investments time right here.
JavaScript rendering is a crucial blind spot.
Most AI crawlers (ChatGPT, Claude, Perplexity) don’t render JavaScript. In case your product pages load key content material client-side, these brokers learn an empty shell. Server-side rendering is the one structure that works universally. The exception is Google Gemini, which makes use of the identical Net Rendering Service as Googlebot.
How To Make Positive ChatGPT, Perplexity & LLMs Can Attain Your Content material
AI search bots go to deep pages roughly as soon as a month and drop off sharply past three clicks from the homepage. The pages with essentially the most particular, answerable data are sometimes the toughest for brokers to achieve.
The repair: Elevate your Most worthy deep pages via inner linking, guaranteeing they’re reachable inside 4 clicks.
Pages crawled by coaching bots however by no means reached by consumer bots are your highest-priority targets. Pages AI consumer bots go to ceaselessly are telling you what to scale: extra content material protecting the identical matter cluster and depth.
Optimize Content material For Longer, Fan-Out Queries
95% of the queries driving AI citations have zero month-to-month search quantity. They’re artificial sub-queries generated by AI fashions. However they present up in GSC: impressions, no clicks, question lengths you’d by no means goal voluntarily.
How To Discover Fan Out Question Alternatives
To floor fan out queries which can be value chasing, join your GSC API to JetOctopus (to bypass the 1,000-row UI limit) and filter for: question size better than 7 phrases, impressions beneath 50, clicks at 0, during the last 3 months. That’s your Fan-Out Opportunity Matrix, the precise questions AI brokers are asking about your content material.
Immediate Varieties That Fan Out Most
In case your content material isn’t structured to reply listing and comparability queries, with specific rankings, professionals/cons, and side-by-side specs, you’re leaving the very best fan-out floor space unoptimized.
“Product assessment” intent queries surged from 239 in June 2025 to over 40,000 by September 2025. That 16,000% enhance was AI brokers systematically harvesting structured opinion knowledge. In case your product pages lack this depth, you’re invisible to that harvest.
The Technical Audit: The place to Begin
Step 1: Establish AI Consumer Bot Visitors In Logs
Pull uncooked server logs (Apache/Nginx) and export all traces containing these consumer brokers: OAI-SearchBot and ChatGPT-Consumer, PerplexityBot and Perplexity-Consumer, Claude-SearchBot and Claude-Consumer. Then manually group hits by user-agent patterns and endpoints in a spreadsheet. To differentiate coaching bots from consumer bots, you’ll want to keep up your individual classification listing — one which adjustments usually and isn’t standardized.
In JetOctopus Log Analyzer, this segmentation is inbuilt: filter by bot sort (coaching, search, and consumer) in a couple of clicks and instantly see which pages AI consumer bots go to (your AI-visible content material, able to scale) versus pages coaching bots hit however consumer bots by no means attain (your highest-priority repair targets).
Step 2: Audit Technical Accessibility Of Deep Pages
Choose a pattern of deep URLs and examine HTML payload measurement, verify key content material isn’t injected through JavaScript by viewing uncooked HTML, simulate crawl depth by counting clicks from the homepage, and check load time in Chrome DevTools or Lighthouse. Additionally examine whether or not necessary content material sits behind accordions or “View Extra” parts — these require JavaScript execution that AI bots skip totally. For big websites with hundreds of deep pages, this sampling method misses loads. AI brokers don’t click on. If data solely seems after consumer interplay, it doesn’t exist for these crawlers.
Step 3: Clear Up Your Robots.txt
Open your robots.txt and assessment all Disallow and Permit directives for each user-agent line by line. AI bots observe Disallow guidelines, so be sure you’re not unintentionally blocking necessary URLs. Manually check key URLs to verify they aren’t blocked. A 30-minute audit right here can forestall you from blocking crawlers you need in, or exposing content material you’d fairly maintain out.
Step 4: Map Your Phantom Impressions
Export knowledge from GSC Efficiency reviews filtered by impressions with zero clicks. Due to the 1,000-row UI restrict, you’ll want to make use of the GSC API or export in chunks by date and question, then merge datasets in spreadsheets or BigQuery. Additionally think about question frequency: lengthy queries showing every day are possible not fan-outs.
Join your GSC API to JetOctopus to bypass the row restrict and construct your Fan-Out Alternative Matrix mechanically — the precise questions AI brokers are asking about your content material, able to act on.
Step 5: Monitor The Modifications
Arrange a recurring export course of — pull GSC knowledge month-to-month and examine impressions over time, re-run log evaluation scripts and diff bot exercise, monitor Core Net Vitals individually in PageSpeed Insights or CrUX. You’ll find yourself stitching collectively a number of knowledge sources with no unified alerting, making it exhausting to catch regressions early.
JetOctopus Alerts covers precisely this: unified notifications for adjustments in AI bot exercise alongside Googlebot habits, Core Net Vitals, on-page website positioning points, and SERP efficiency drops, so that you catch regressions earlier than they compound.
The New KPI: Technical Accessibility
website positioning in 2026 is restructuring round one constraint: can an AI agent crawl, attain, and extract a truth out of your 50,000th product web page in beneath 200 milliseconds?
If the reply isn’t any, your rankings, backlinks, and content material high quality grow to be irrelevant for a rising share of search interactions. The machines are looking out. The query is how rapidly you possibly can see what’s really occurring.
Begin along with your logs. Every part else follows from there.
| Wish to see precisely how AI bots are interacting along with your website: which pages they attain, which they skip, and the place your fan-out alternatives are hiding? E book a live walkthrough of the JetOctopus platform. We’ll pull your precise log knowledge and present you what your GSC reviews aren’t telling you. |
Picture Credit
Featured Picture: Picture by JetOctopus. Used with permission.