Your llms.txt Is Perfect and AI Agents Still Can't Reach You: The Transport Layer of GEO

Yesterday we published a post about GEO identity drift — our llms.txt, schema, and homepage telling AI engines three different stories about the same company. We fixed it, wrote it up, and felt good about our machine layer.

Today we failed our own audit again, one layer down.

www.stackbilder.com completed a TLS handshake and then hung. Forever. No response, no error page, no redirect — a connection that opens and never speaks. Every AI crawler, every retrieval-augmented agent, every human who typed www. got silence until their client gave up. Our content layer was near-perfect. The transport underneath it was broken, and nothing in a content audit would ever have told us.

GEO has a transport layer, and nobody audits it

Generative engine optimization work concentrates on the content layer: llms.txt, JSON-LD graphs built on schema.org types, canonical tags, sitemap hygiene. That layer decides what an AI system reads about you. We've written about it from the identity-consistency angle and from the demand-side analytics angle.

Underneath it sits a layer that decides whether an AI system can read you at all: DNS records, proxy routing, TLS termination, scheme and host canonicalization, redirect chains. Call it the transport layer of GEO.

The transport layer has a property that makes it dangerous: it fails silently and asymmetrically. A human who hits a dead www. host retypes the URL without the www and forgets it happened. An AI agent doesn't. It burns its fetch timeout, logs nothing you can see, and satisfies the user's query by citing whichever competitor's site actually loaded. Agents don't file bug reports. They route around you.

We already knew agents discover platforms through structured entry points — registries, manifests, well-known paths. What today made concrete: every one of those entry points assumes the bytes underneath will move. Reachability is the zeroth check, and we had never once run it.

The failure: a TLS handshake that never became a response

We found it because we built a scanner (more below) and pointed it at ourselves. Two checks failed. The first looked like this:

$ curl -v https://www.stackbilder.com
*   Trying 104.21.4.46:443...
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
*  SSL certificate verify ok.
* Operation timed out after 10002 milliseconds with 0 bytes received

DNS resolved. TCP connected. The certificate was valid — Cloudflare's Universal SSL covers the www subdomain automatically. The handshake completed. And then: nothing.

The cause was a gap between two systems that each looked correct alone. The zone had a proxied www DNS record (the standard dummy A 192.0.2.1 that hands traffic to Cloudflare's edge). But Workers routing had no route claiming the hostname:

cms.stackbilder.com/*    -> stackbilt-emdash
arcade.stackbilder.com/* -> stack-arcade
docs.stackbilder.com/*   -> stackbilt-docs
blog.stackbilder.com/*   -> stackbilt-roundtable
stackbilder.com/*        -> stackbilt-web

Every subdomain claimed — except www. The edge accepted the connection because the DNS record told it to, then had nowhere to send the request. Not a 404, not a 522. A hang.

The second failure was quieter: http://stackbilder.com returned 200 OK over plain HTTP. No upgrade to HTTPS, because the zone's Always Use HTTPS setting had never been switched on. Two apparently-canonical origins serving the same content over different schemes — precisely the duplicate-origin problem canonical tags exist to prevent, reintroduced one layer down where canonical tags can't see it.

Neither failure was visible from inside the site. The pages rendered. The schema validated. The llms.txt parsed. We were fully coherent and partially unreachable.

The scanner: 19 checks, zero dependencies

The tool that caught this is a single-file Node script — no packages, just fetch — that audits any public site's machine layer the way an AI crawler experiences it:

Transport: do all four origin variants (http/https × www/apex) resolve, respond, and converge on one canonical origin?
Content contracts: canonical tag present and matching the final URL, no accidental noindex, title and meta description, single H1, lang attribute.
Structured data: JSON-LD present, parsing cleanly, with Organization and WebSite types in the graph.
Crawl surface: robots.txt present, sitemap present and non-empty, sitemap declared in robots.txt.
AI-specific: llms.txt present (and actually a text file — single-page apps love returning HTML with a 200 for every path, including /llms.txt), and per-crawler treatment of GPTBot and OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, and CCBot in robots.txt, so an explicit block never ships by accident.

Run against our own domain, it returned 17 passes and the two failures above. Run against a well-built site from a completely different industry, it returned 18 — everything green except llms.txt, which almost nobody has yet. That spread is the point: normal, professionally maintained sites pass the web-era checks and fail the agent-era ones.

One implementation detail worth stealing: the llms.txt check doesn't trust status codes. It fetches the path and rejects the result if the body starts with an HTML document, because a 200 that's actually your SPA shell is worse than a clean 404 — it tells you the file exists while feeding agents markup instead of the brief you wrote.

The fix took under an hour, and we verified it like production

For www, we deployed an eleven-line dedicated Worker instead of touching the main site app:

export default {
  fetch(request) {
    const url = new URL(request.url);
    url.hostname = "stackbilder.com";
    url.protocol = "https:";
    return Response.redirect(url.toString(), 301);
  },
};

with a Workers route claiming www.stackbilder.com/*. A separate Worker keeps host canonicalization out of the application codebase — the redirect policy survives every site rebuild, framework migration, and deploy.

For the scheme problem, we enabled Always Use HTTPS at the zone level.

Then we verified, because merged isn't live and deployed isn't working:

$ curl -sI https://www.stackbilder.com/img-forge?x=1 | head -2
HTTP/2 301
location: https://stackbilder.com/img-forge?x=1

$ curl -sI http://stackbilder.com | head -1
HTTP/1.1 301 Moved Permanently

Path preserved, query preserved, scheme upgraded, one canonical origin. Re-ran the scanner: 19/19.

Total time from first failing check to verified green: about an hour, most of it diagnosis.

Audit the layer your content sits on

The lesson generalizes past our two bugs. Every content-layer investment — the schema graph, the llms.txt brief, the canonical strategy — implicitly assumes the transport layer works. That assumption is checked by no SEO tool we've used, because SEO tools browse like humans, with patience and retries and a rendering engine. Agents fetch like machines, with a timeout and a fallback list.

So the audit order should be inverted from how everyone actually does it:

Can every origin variant be fetched, and do they converge on one host and scheme?
Does the crawl surface (robots, sitemap) exist and agree with itself?
Is the machine-readable identity (schema, llms.txt) present and parseable?
Only then: is what it says correct and consistent?

We'd been operating at layer four and had never checked layer one. We've failed in public before and found it clarifying every time; this one was the cheapest lesson yet — two config-level fixes, discovered only because we finally audited ourselves with the same skepticism we apply to production deploys.

If you want the same skepticism applied to your site: we've productized this exact audit — the 19-check scan, a transcript of what AI assistants actually say about your company today, and a prioritized fix list with paste-ready snippets — as a fixed-price service at stackbilder.com/services.

Stackbilder builds img-forge — production AI image generation for agents, apps, and pipelines — and writes here about making platforms legible to both humans and machines. Part 1 of this series: GEO Identity Drift.