Study: 97% of llms.txt Files Never Read by AI Crawlers

The week's most data-backed story came from Ahrefs, which analyzed server logs and bot traffic across 137,000 domains and found that 97% of llms.txt files received zero requests from AI crawlers. On top of that, AI retrieval bots accounted for just 1% of total crawl traffic across all sites studied. The findings, covered by both Ahrefs directly and Search Engine Journal, represent the most concrete evidence yet that the llms.txt protocol — despite widespread adoption — is delivering essentially no practical value in terms of AI discoverability.

Google responded to the LLMs.txt debate from multiple directions this week. In updated documentation, Google confirmed that LLMs.txt files neither help nor hurt search rankings, and that AI-specific files and Markdown formats are not used in Google Search. Simultaneously, Google's John Mueller stated that LLMs.txt cannot help LLMs differentiate between websites during discovery — limiting any usefulness to a narrow, post-arrival on-site context.

Search Engine Journal's coverage went further, reporting that Google has exposed a fundamental flaw in the protocol: a core assumption driving LLMs.txt adoption — that it helps AI models better access web content during discovery — conflicts with what its original creators actually intended the file to do. In a separate but related clarification, Google also updated its AI SEO guidance to adopt a more permissive tone, stating it's "fine" to use LLMs.txt, while simultaneously confirming it makes no search ranking difference. On the Markdown front, Google's Mueller and Martin Splitt confirmed on the *Off The Record* podcast that HTML remains the SEO standard, and that Markdown strips away elements important for search visibility.

In one of the week's most consequential regulatory developments, the UK's Competition and Markets Authority (CMA) issued formal conduct requirements ordering Google to provide advance notice before making significant ranking changes and to enable search data portability to third-party services. The requirements also cover the fair ranking of organic results including AI Overviews. Covered extensively by both Search Engine Journal and Search Engine Roundtable, the order represents a landmark intervention in search market dynamics.

Barry Schwartz of Search Engine Roundtable noted skepticism that Google will ever fully share its ranking methodology — pointing out this isn't the first time a political body has made such demands — but acknowledged the data portability requirement is more operationally concrete. The development is significant for the SEO industry because, if enforced, it would require Google to give publishers and webmasters structured warning before major algorithmic shifts, fundamentally changing how the industry responds to updates.

Google search ranking volatility that began in the June 8–12 period has continued and intensified into the week of June 17, according to Barry Schwartz at Search Engine Roundtable. Community chatter in SEO forums has remained elevated, and signals point to an active but unconfirmed ranking algorithm update. Google has not acknowledged any update.

SEO professionals have been reporting widespread page deindexing events, while Google maintains it sees nothing unusual in its systems. Search Engine Journal reports that the key challenge for practitioners is distinguishing between three different phenomena: genuine deindexing, standard ranking drops, and Search Console data reporting anomalies — which can all appear similar in the dashboard. The disconnect between community reports and Google's stance remains unresolved.

Google addressed a real-world case in which Search Console reported 51,000 URLs as "Indexed, though blocked by robots.txt" — and clarified that this is not necessarily a problem. The explanation: Google can index a URL it has discovered through links or sitemaps even if it cannot crawl the page itself, using the link signal alone. SEOs troubleshooting index coverage reports in Search Console should factor in this distinction before taking remedial action.

Microsoft has begun rolling out a significant update to Bing Webmaster Tools, adding four new AI performance metrics first previewed at SEO Week in April: Citation Share, Intents, Topics, and Compare. Citation Share shows publishers what proportion of AI-generated responses cite their content, making it one of the first native tools for measuring AI citation performance at scale. Bing is also reportedly considering expanding its country-level reporting to include more individual countries, according to Microsoft's Fabrice Canel.

Multiple analysts this week converged on the same conclusion: traditional search rank and AI citation are fundamentally different metrics that should not be treated as interchangeable. Search Engine Journal contributor Duane Forrester notes that AI models decompose prompts into multiple short retrieval queries before hitting an index, meaning your rank position for a head term is a poor proxy for AI visibility. A separate piece argues that AI prompt tracking should be reframed as a measurement of stability, brand representation, and contextual accuracy — not as another form of rank tracking.

Kevin Indig's Growth Memo adds a strategic layer: the source sets that AI systems draw on differ by topic, meaning off-site authority-building (digital PR, mentions, links) must be topic-targeted rather than broadly distributed to drive AI citation results.

New data from Lily Ray published in Search Engine Journal reveals a counterintuitive dynamic in AI Overviews: Google frequently cites listicles and 'best of' content as sources while actually recommending competitor brands mentioned within those articles. The implication: brands that create 'best of' content featuring themselves alongside competitors may be inadvertently training AI to recommend those competitors. Citations and recommendations are not the same signal in AI-generated results, and optimizing only for citation without considering recommendation logic may backfire.

At Search Central Live Milan, Google representatives shed light on what AI Overview clicks actually look like compared to regular search clicks — confirming they represent a qualitatively different user intent signal. A related Search Engine Journal analysis argues that visitors arriving from Google's AI Mode arrive mid-task with high specificity, expecting instant task completion within roughly 30 seconds. Traditional landing pages designed for browse-oriented visitors are poorly suited to this behavior, requiring SEOs to audit their top AI-referred pages specifically for task-completion efficiency.

Following their US zero-click data release, SparkToro and Similarweb published international zero-click search rates across five additional countries. The UK has the highest rate of zero-click searches in the study, while Germany has the lowest, meaning German searchers are more likely to click through to websites. France earned the label of most "efficient" searchers — completing informational needs with fewer queries before clicking. For SEOs managing international clients, these country-level differences in zero-click behavior have direct implications for traffic forecasting and content strategy by market.

Google Cloud announced the Open Knowledge Format (OKF), a new standard that organizes knowledge as simple directories of Markdown files, enabling AI agents to access organizational knowledge without requiring specialized software. Marie Haynes provides the most detailed breakdown of the announcement, including a working example of an OKF she built, and discusses potential business opportunities arising from the format. The standard is notable because it sits alongside — not inside — Google Search, representing a new layer for how agents consume structured knowledge from websites and organizations.

A coalition of Google, Microsoft, and GitHub has jointly published a new open draft specification called Agentic Resource Discovery, designed to standardize how AI agents locate and verify tools, APIs, and resources online. The spec addresses the growing need for a common protocol as AI agents increasingly perform autonomous browsing and task execution. SEOs and developers building agent-accessible content or APIs should watch this spec closely, as it could influence how websites expose their capabilities to agentic systems in the near future.

Google has begun rolling out Information Agents within AI Mode Search, a feature first announced at Google I/O and now available to Google AI Ultra subscribers. Unlike traditional search, Information Agents proactively seek out information on behalf of users even when they are not actively searching, operating in the background as a persistent discovery layer. The shift toward paid, proactive search agents has significant implications for organic traffic models and how content is discovered without user-initiated queries.

Google this week updated its domain migration documentation on two fronts. First, Search Engine Journal reports that Google has introduced stricter compliance requirements for site owners planning domain moves. Second, and more specifically, Google's updated help documentation now clarifies that all domain variants must be submitted through the Change of Address tool for a migration to work correctly — noting that "domain migrations work best when all variants of a site are migrated properly." SEOs managing site moves should audit their migration checklists immediately to ensure all variants (www, non-www, HTTP, HTTPS variants) are accounted for.

Google's John Mueller settled a recurring international SEO debate this week, confirming that creating a dedicated /us/ folder structure for a multinational site provides no practical SEO advantage. Mueller acknowledged it may simplify analytics and site management, but from a purely ranking perspective, the effort is not justified. This guidance applies to sites whose primary market is the US operating under a generic TLD.

New research from Google proposes detecting AI-generated spam by identifying coordinated publishing networks rather than analyzing individual pieces of content. The finding suggests that Google's spam systems may evolve to flag patterns of synchronized, high-volume AI content publication originating from shared infrastructure — making network-level signals more important than content-level signals for spam detection. SEOs and publishers producing AI content at scale should take note of this methodological shift.

At Search Central Live Milan, Google representatives covered a broad range of practitioner-relevant topics in a rare consolidated session. Key areas discussed included: chunking for AI (how Google breaks content into retrievable segments for AI features), site-wide quality signals (confirming they remain a meaningful ranking factor), how AI Overview clicks differ qualitatively from standard search clicks, the strategic distinction between commodity vs. non-commodity content, and handling of paywalled content and news subscriptions. Google also discussed the new AI settings available in Search Console. The breadth of this session makes it one of the more information-dense Google communications of the month.

Google Ads announced three simultaneous bidding and budgeting changes this week: (1) an expansion of Smart Bidding Exploration to more campaigns, (2) the launch of a Promotion Mode beta, and (3) changes to bidding target optimization for budget-limited campaigns, effective August 17, 2026. The third change has drawn the most concern from PPC practitioners — Google is moving toward more automated, Google-controlled spending for campaigns operating under fixed budget constraints, reducing manual override capability. Google is also automatically enabling conversion-based customer lists for advertisers already using both Enhanced Conversions and Customer Match, with data processing beginning August 18.

Google has removed the pagination feature from its Local Finder interface, replacing it with an infinite scroll experience. The change may affect how far down the local results list users scroll, with potential implications for visibility of businesses ranked beyond the initial view. Local SEOs should monitor whether the behavioral change affects click-through rates for positions outside the top 5–10.

A confirmed bug is preventing Google Business Profile owner and manager access invitation emails from being sent or delivered. Agencies and businesses attempting to delegate GBP access are experiencing failures with no confirmed fix yet in place. Affected users should seek alternative access-sharing workarounds until Google resolves the issue.