More and more visits to a website don't come from a person looking at it. They come from an AI agent acting for a person: a crawler collecting training or search data, ChatGPT or Claude fetching a page to answer a question, or a browser agent clicking through the site on someone's behalf. I spent a day making this site work well for those visitors. This article walks through every step, with the code, so you can do the same for your site.

The problem: what an agent actually saw

ihork.link is a macOS-style terminal in the browser. Everything you see — the welcome text, the commands, the answers — is drawn by JavaScript. That's fun for people and useless for most agents, because most crawlers and fetch tools don't run JavaScript. Here is everything a non-JS agent could read on my homepage:

Ihor Kruchynskyi Github LinkedIn Email

Four words. No bio, no resume, no projects. An agent asked "who is Ihor Kruchynskyi?" would learn more from LinkedIn than from my own site.

It helps to think of three kinds of agent visitors, because each one needs something different:

To measure progress I used the scanner at isitagentready.com, which checks a site against the current agent standards:

curl -s -X POST https://isitagentready.com/api/scan \
  -H 'Content-Type: application/json' -d '{"url":"https://ihork.link"}'

The first scan put the site at level 1, "Basic Web Presence": no robots.txt, no sitemap, no Markdown, no agent discovery of any kind.

Step 1: robots.txt with explicit AI rules

A robots.txt tells crawlers what they may fetch, per RFC 9309. Mine welcomes AI crawlers and keeps them out of endpoints that are useless to them:

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Google-Extended
User-agent: PerplexityBot
User-agent: CCBot
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
Disallow: /api/
Disallow: /forecast.php

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
Disallow: /api/
Disallow: /forecast.php

Sitemap: https://ihork.link/sitemap.xml

Three details matter here:

• A crawler obeys only the most specific group that names it. GPTBot reads its own group and ignores * entirely, so every named group must repeat the Disallow rules. Forget that and you have silently allowed everything.
• Content Signals (contentsignals.org) declare what the content may be used for: search (indexing), ai-input (grounding answers in real time) and ai-train (training models). Decide your policy and make the signals match your User-agent groups. If you say ai-train=no, also block the training-only crawlers.
• Don't list secret paths. robots.txt is public. Disallow: /admin-backdoor.php is an invitation, not a lock.

Step 2: a sitemap that updates itself

A sitemap lists the canonical URLs of your public pages. I started with a static sitemap.xml, then replaced it with a small PHP script as soon as this blog appeared, so a new article shows up in the sitemap the moment its Markdown file lands on the server:

$urls = [[SITE_URL . '/', ...], [SITE_URL . '/blog/', ...], [SITE_URL . '/weather.html', ...]];
foreach (blogPosts() as $post) {
    $urls[] = [$post['url'], $post['updated']];
}

Step 3: Markdown for agents

This was the biggest win. Agents are happiest with clean text, and Markdown is the format they read best. The idea, also used by Cloudflare's Markdown for Agents, is content negotiation: a request with Accept: text/markdown gets Markdown, browsers keep getting HTML.

Cloudflare can do the conversion at the edge, but it converts your HTML — and my HTML is an empty shell. So I wrote the Markdown version myself (index.md: profile, experience, education, links) and serve it from the origin:

// True when the Accept header prefers text/markdown over text/html
function prefersMarkdown($accept) {
    $q = ['text/markdown' => 0.0, 'text/html' => 0.0];
    foreach (explode(',', strtolower($accept)) as $part) {
        $params = array_map('trim', explode(';', $part));
        $type = array_shift($params);
        if (!isset($q[$type])) continue;
        $weight = 1.0;
        foreach ($params as $param) {
            if (strpos($param, 'q=') === 0) $weight = (float)substr($param, 2);
        }
        $q[$type] = max($q[$type], $weight);
    }
    return $q['text/markdown'] > 0 && $q['text/markdown'] >= $q['text/html'];
}

header('Vary: Accept');
if (prefersMarkdown($_SERVER['HTTP_ACCEPT'] ?? '')) {
    header('Content-Type: text/markdown; charset=utf-8');
    header('Cache-Control: private, max-age=600');
    readfile(__DIR__ . '/index.md');
    exit;
}

Two headers do important work. Vary: Accept tells caches that the same URL has two representations. Cache-Control: private keeps the CDN from caching the Markdown and serving it to the next browser — Cloudflare ignores Vary for most content, so don't rely on it alone.

Next to that, /llms.txt gives a short plain-text summary, and every article on this blog has a Markdown twin: add .md to any post URL, or request it with Accept: text/markdown.

Step 4: an A2A agent

The Agent2Agent protocol (A2A) lets one AI agent talk to another. An agent publishes an Agent Card at /.well-known/agent-card.json describing its skills and endpoint, and other agents send it messages over JSON-RPC.

Before writing any code I had to decide something important. An agent card isn't a description file — it advertises a live service. My terminal's chat endpoint is protected by Cloudflare Turnstile, which agents can't pass, so a card pointing at it would pass the scanner and fail every real agent. I chose to build a small dedicated endpoint with two skills:

• profile returns the Markdown profile instantly, without calling a model.
• ask answers a free-text question about me with the same model and resume as the terminal.

The card, shortened:

{
  "name": "Ihor Kruchynskyi",
  "description": "Answers questions about Ihor Kruchynskyi ...",
  "supportedInterfaces": [
    { "url": "https://chat.ihork.link/a2a.php", "protocolBinding": "JSONRPC", "protocolVersion": "1.0" }
  ],
  "version": "1.0.0",
  "capabilities": { "streaming": false, "pushNotifications": false },
  "defaultInputModes": ["text/plain"],
  "defaultOutputModes": ["text/markdown", "text/plain"],
  "skills": [
    { "id": "profile", "name": "Profile", "description": "...", "tags": ["profile", "resume"] },
    { "id": "ask", "name": "Ask about Ihor", "description": "...", "tags": ["q&a", "career"] }
  ]
}

And a conversation with it:

curl -s https://chat.ihork.link/a2a.php \
  -H 'Content-Type: application/json' -H 'A2A-Version: 1.0' \
  -d '{"jsonrpc":"2.0","id":1,"method":"SendMessage","params":{"message":{
        "messageId":"b1c2d3e4-0000-4000-8000-000000000001","role":"ROLE_USER",
        "parts":[{"text":"What does Ihor work on at Adobe?"}]}}}'

{"jsonrpc":"2.0","id":1,"result":{"message":{
  "messageId":"…","contextId":"…","role":"ROLE_AGENT",
  "parts":[{"text":"At Adobe, Ihor …","mediaType":"text/markdown"}]}}}

Things I learned implementing it:

• Read the spec source, not a summary. A summarized version of the spec told me the result is the Message itself. The protobuf definition (a2a.proto) says SendMessageResponse is a oneof, serialized as {"message": {…}} or {"task": {…}}. Roles are ROLE_USER / ROLE_AGENT, a text part is just {"text": "…"}, and the card needs tags on every skill.
• Versions matter. Clients send an A2A-Version header; no header means 0.3, which used message/send and kind fields. Supporting both takes a few lines and keeps older clients working.
• Return real errors. Streaming, tasks and push notifications aren't supported here, so those methods return the spec's error codes (-32004, -32001, -32003) instead of a generic failure.

Because the endpoint has no human check, it needs its own guardrails: 5 questions per minute and 50 per day per IP, 500 per day in total to protect the model quota (shared with the MCP server below, so a second protocol doesn't double the exposure), no tools (no web search, no image generation), and a system prompt that refuses anything that isn't about me. Ask it to write a Python function and it replies "This agent only answers questions about Ihor Kruchynskyi." An open endpoint with a model behind it is a free LLM for anyone who finds it, unless you scope it.

Step 5: Agent Skills discovery

The Agent Skills Discovery RFC lets a site publish instructions for agents as SKILL.md files — the same format coding agents use for their skills. An index at /.well-known/agent-skills/index.json lists them:

{
  "$schema": "https://schemas.agentskills.io/discovery/0.2.0/schema.json",
  "skills": [{
    "name": "ihork-profile",
    "type": "skill-md",
    "description": "Look up Ihor Kruchynskyi ... through ihork.link's Markdown pages, its A2A agent and its WebMCP browser tools.",
    "url": "/.well-known/agent-skills/ihork-profile/SKILL.md",
    "digest": "sha256:…"
  }]
}

My skill tells an agent the cheapest way to answer questions about me: read the Markdown profile and articles first, connect over MCP or ask the A2A agent only when that isn't enough, and use the WebMCP tools when it's driving a browser. The digest is a SHA-256 of the file, so it must change whenever the skill changes. I generate the index in my build script instead of maintaining it by hand — a stale digest is exactly the kind of bug nobody notices.

Step 6: WebMCP tools in the page

WebMCP is the newest piece: a browser API that lets a page expose tools to an AI agent running in the browser, instead of making the agent guess where to click. Chrome is trialing it (announcement). A tool has a name, a description, a JSON Schema for its input and an execute callback:

navigator.modelContext.registerTool({
  name: "get_profile",
  title: "Ihor's profile",
  description: "Get Ihor Kruchynskyi's profile and resume as Markdown.",
  inputSchema: { type: "object", properties: {} },
  annotations: { readOnlyHint: true },
  async execute() {
    const res = await fetch("/index.md");
    return { content: [{ type: "text", text: await res.text() }] };
  },
}, { signal: controller.signal });

The site registers five tools: get_profile, get_projects, get_blog_posts, run_terminal_command and ask_about_ihor. Two design choices I'd recommend:

• Reuse the real UI. run_terminal_command types the command into the visible terminal and returns what it printed, and it only accepts the built-in commands. The person watching sees exactly what the agent did, and there is no second code path to maintain.
• Don't make agents wait for a human click. My first version sent ask_about_ihor through the terminal's Turnstile check. When I tested it as a browser agent, the check wanted a click and the tool just hung. Now, while the visitor isn't verified, the question goes to the MCP server's ask_about_ihor instead — no Turnstile, but also no tools and the shared agent rate limit — and the answer still appears in the terminal. Verified visitors keep the full chat. The rule: an agent-facing tool should either work or fail fast with a way forward.

The signal option unregisters the tools when the page goes away. I abort it on pagehide and register again if the page comes back from the back/forward cache.

Step 7: an MCP server

The Model Context Protocol (MCP) is how AI assistants and coding agents connect to tools and data. A site can advertise an MCP server with a server card at /.well-known/mcp/server-card.json (SEP-1649). Same rule as with A2A: a card is a promise, so I built the server first — one PHP file, https://chat.ihork.link/mcp.php, no SDK.

It exposes five tools and the matching resources:

The interesting part was the protocol version. The newest revision, 2026-07-28, made MCP stateless: there is no initialize handshake any more. Every request carries its protocol version and client capabilities in _meta, and the HTTP headers mirror the body so gateways can route without parsing JSON:

curl -s https://chat.ihork.link/mcp.php \
  -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' \
  -H 'MCP-Protocol-Version: 2026-07-28' -H 'Mcp-Method: tools/call' -H 'Mcp-Name: get_profile' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"get_profile","arguments":{},
       "_meta":{"io.modelcontextprotocol/protocolVersion":"2026-07-28",
                "io.modelcontextprotocol/clientCapabilities":{}}}}'

The server must check that MCP-Protocol-Version, Mcp-Method and Mcp-Name match the body and answer 400 with error -32020 when they don't; an unknown version gets -32022 with the list of supported versions, and every result carries "resultType": "complete". A new server/discover method replaces the handshake.

Most MCP clients in the wild still speak 2025-11-25 or older, which start with initialize. The spec calls a server that handles both a dual-era server, and it's cheap to be one: if the request has the new _meta fields, apply the stateless rules; if it's initialize, answer the handshake and treat the rest the old way.

$meta = is_array($params['_meta'] ?? null) ? $params['_meta'] : [];
$modern = isset($meta['io.modelcontextprotocol/protocolVersion'])
    || in_array(mcpHeader('MCP-Protocol-Version'), ['2026-07-28'], true);

Two details that make the server nicer to use from a real assistant:

• Return follow-up state where the model can see it. ask_about_ihor returns a conversationId for follow-up questions both in structuredContent and at the end of the text, because many clients only show the text to the model.
• One budget, two doors. The MCP tool and the A2A agent call the same PHP function, agentAsk(), and share one rate limit. Adding a protocol shouldn't double what the model can cost you.

Step 8: an API catalog and Link headers

With a Markdown profile, a blog, an MCP server and an A2A agent, there are now several entry points. Two standards tie them together.

The API catalog (RFC 9727) lives at /.well-known/api-catalog and lists a publisher's APIs as a Linkset: each API is an anchor with a service-desc (a machine-readable description, here the MCP and A2A cards) and a service-doc (documentation, here the SKILL.md):

{
  "linkset": [
    {
      "anchor": "https://chat.ihork.link/mcp.php",
      "service-desc": [{ "href": "https://ihork.link/.well-known/mcp/server-card.json", "type": "application/json" }],
      "service-doc": [{ "href": "https://ihork.link/.well-known/agent-skills/ihork-profile/SKILL.md", "type": "text/markdown" }]
    },
    {
      "anchor": "https://ihork.link/.well-known/api-catalog",
      "item": [
        { "href": "https://chat.ihork.link/mcp.php" },
        { "href": "https://chat.ihork.link/a2a.php" },
        { "href": "https://ihork.link/blog/index.json" }
      ]
    }
  ]
}

It has to be served as application/linkset+json with an RFC 9727 profile, and a HEAD request must return a Link header pointing to itself. The file has no extension, so the web server needs to be told. In lighttpd:

$HTTP["url"] == "/.well-known/api-catalog" {
  mimetype.assign = ( "" => "application/linkset+json; profile=\"https://www.rfc-editor.org/info/rfc9727\"" )
  setenv.add-response-header = ( "Link" => "</.well-known/api-catalog>; rel=\"api-catalog\"" )
}

Link headers (RFC 8288) put the same pointers on the homepage response itself, so an agent that fetches / — even with HEAD — learns where everything is without guessing well-known paths:

Link: </.well-known/api-catalog>; rel="api-catalog",
      </.well-known/mcp/server-card.json>; rel="service-desc"; type="application/json",
      </.well-known/agent-card.json>; rel="service-desc"; type="application/json",
      </.well-known/agent-skills/ihork-profile/SKILL.md>; rel="service-doc"; type="text/markdown",
      </llms.txt>; rel="describedby"; type="text/plain",
      </index.md>; rel="alternate"; type="text/markdown"

Every article sends one too, pointing at its Markdown version. The result is a discovery chain an agent can follow from any page: Link header → API catalog → cards → endpoints.

Checking the result

After these steps the scanner reports a pass for robots.txt, the sitemap, Link headers, AI crawler rules, Markdown negotiation, the API catalog, the MCP server card, the A2A Agent Card, the Agent Skills index and WebMCP — it found all five tools by loading the page in a browser.

One check kept failing for a while for a boring reason: Cloudflare had cached the old robots.txt. Text files are cached at the edge by default, so the new rules and Content Signals were invisible until I purged the cache. If a check fails right after a change, fetch the file with a cache-busting query string before you start debugging.

Lessons

• Start with what agents see. curl your homepage and strip the tags. If it's four words, nothing else matters until you fix that.
• Markdown is the highest-value, lowest-effort step. One content-negotiated Markdown file made the site readable for every agent.
• Don't advertise what you don't serve. An agent card, a skill or a tool that leads nowhere is worse than none.
• Every open door needs a budget. Rate limits, a narrow scope and no tools turned an open LLM endpoint into a small, predictable one — and one shared budget means a new protocol doesn't add new cost.
• Specs move fast; support the old and the new. MCP dropped its whole handshake in 2026-07-28 while most clients still use it. Reading the spec's own text for the latest version, then accepting both eras, is cheap and keeps every client working.
• Keep one source of truth. My resume now lives in the model's prompt, the Markdown profile and llms.txt. When one changes, all three must.

There's still more on the list, like DNS-based agent discovery; OAuth metadata only matters once an API needs sign-in. And this blog itself follows the same rules — every article is a Markdown file on the server, rendered to HTML for you and served as-is to agents at /blog/<slug>.md.