Keep Your Structured Data is Accessible with AI Search Optimization

An investigation has uncovered that AI crawlers often overlook JavaScript-injected structured data. To guarantee visibility, adopt server-side rendering or static HTML for optimal results.

AI Search Optimization: Ensure Your Structured Data is Accessible

A recent investigation has uncovered a significant challenge for websites relying on JavaScript to deliver structured data. This data, often formatted in JSON-LD, becomes inaccessible to AI crawlers if it isn’t included in the initial HTML response.

Crawlers like GPTBot (used by ChatGPT), ClaudeBot, and PerplexityBot lack the ability to execute JavaScript. As a result, they fail to detect any structured data added dynamically after the page loads. This creates complications for websites that use tools like Google Tag Manager (GTM) to inject JSON-LD on the client side, as most AI crawlers cannot process dynamically generated content.

Key Findings About JSON-LD & AI Crawlers

Elie Berreby, the founder of SEM King, conducted an analysis of what happens when JSON-LD is added using Google Tag Manager (GTM) without server-side rendering (SSR). His findings shed light on why this type of structured data often remains invisible to AI crawlers:

• Initial HTML Load: When a crawler requests a webpage, the server sends the first version of the HTML. If structured data is injected via JavaScript, it won’t appear in this initial response.
• Client-Side JavaScript Execution: JavaScript runs in the browser, altering the Document Object Model (DOM) for users. At this stage, GTM can add JSON-LD to the DOM.
• Crawlers Without JavaScript Rendering: AI crawlers that cannot execute JavaScript are unable to see changes made to the DOM. Consequently, they miss any JSON-LD added after the page loads.

In summary, structured data added solely through client-side JavaScript is effectively invisible to most AI crawlers.

Why Traditional Search Engines Are Different

Traditional search engine crawlers, such as Googlebot, can interpret JavaScript and process changes made to a webpage after it loads, including JSON-LD data injected by Google Tag Manager (GTM).

In contrast, many AI crawlers cannot execute JavaScript and only access the raw HTML delivered by the server. As a result, they overlook dynamically added content, such as JSON-LD.

Google’s Warning on Overusing JavaScript

This issue aligns with a broader caution from Google regarding the overuse of JavaScript. In a recent podcast, Google’s Search Relations team discussed the growing dependence on JavaScript. While it enables dynamic features, it’s not always ideal for essential SEO elements like structured data.

Martin Splitt, Google’s Search Developer Advocate, explained that websites range from simple pages to complex applications. It’s crucial to balance JavaScript usage with making key content available in the initial HTML.

John Mueller, another Google Search Advocate, echoed this sentiment, noting that developers often rely on JavaScript when simpler solutions, like static HTML, would be more effective.

What To Do Instead

To avoid issues with AI search crawlers, developers and SEO professionals should ensure structured data is accessible to all crawlers. Here are some key strategies:

Server-Side Rendering (SSR): Render pages on the server to include structured data in the initial HTML response. Static HTML: Use schema markup directly in the HTML to minimize reliance on JavaScript. Prerendering: Provide prerendered pages where JavaScript has already been executed, giving crawlers fully rendered HTML.

These approaches align with Google’s recommendation to prioritize HTML-first development and include critical content like structured data in the initial server response.

Why This Matters

AI crawlers are becoming increasingly important and operate under different rules than traditional search engines.

If your site depends on GTM or other client-side JavaScript for structured data, you risk missing opportunities to rank in AI-driven search results. By transitioning to server-side or static solutions, you can future-proof your site and ensure visibility in both traditional and AI-powered searches.