CodeMud

AI Web - Capability Advertisement and Usage

Recently, I have done some experiments to integrate AI and WEB browsers. However, I found some problems, especially scraping WEB content of tabs. This document discusses an idea of how websites could advertise their capabilities to AI agents and how AI models could discover and use these capabilities. It would enable websites to expose structured, safe functionality to AI systems.

The Problem

Today, AI agents interact with websites by reading the entire DOM (expensive, noisy), guessing what actions are possible, using generic browser automation (fragile, unsafe), and no standard way to know what a site can do

We need a standardized way for websites to say "here's what you can do with me" and for AI to discover and use these capabilities safely.

Core Concepts

Capability Advertisement

Websites would declare what AI agents can do through tool declarations. Think of tools as API endpoints that AI can call, but with:

Self-describing schemas (what parameters they need)
Permission requirements (what user consent is needed)
Clear descriptions (so AI knows when to use them)

Discovery

AI agents would query available tools without reading the entire DOM or guessing.

Execution

Once discovered, AI would call tools with proper validation, user consent, and error handling. The AI is a JavaScript program running in the browser - it can implement a model locally or connect to a server through XHR, WebSocket, or other mechanisms.

Benefits

For Website Owners

Control: Explicitly define what AI can do
Safety: AI uses your APIs, not fragile DOM manipulation
Performance: AI doesn't need to parse entire page
Analytics: Track AI-driven interactions

For IT Departments

Policies: Apply custom browser policies for AI behavior and access control
Governance: Centralized control over AI interactions across the organization
Compliance: Ensure AI usage meets regulatory and company requirements

For AI Agents

Efficiency: No need to parse full DOM
Reliability: Tools have schemas and validation
Discovery: Know what's possible without guessing
Error handling: Structured errors instead of failures

For Users

Transparency: See exactly what AI is doing
Control: Approve sensitive actions
Privacy: Tools declare data usage; AI only accesses what it needs, not scraping entire pages
Audit: Complete history of AI actions

Advertisement Methods

Method 1: Declarative HTML

Websites would include capability manifests in HTML using a function-like schema:

<script type="application/ai-capability+json">
{
  "description": "Weather forecast and current conditions service",
  "tools": [
    {
      "name": "getWeather",
      "description": "Get current weather conditions for a location",
      "parameters": [
        {
          "name": "location",
          "type": "string",
          "description": "City name or zip code",
          "required": true
        }
      ],
      "returns": {
        "type": "object",
        "description": "Weather information"
      },
      "endpoint": "/api/weather",
      "permissions": {
        "requiresConsent": false
      }
    }
  ]
}
</script>

When to use: Static sites, SEO-friendly declaration, when tools don't change often.

Method 2: JavaScript API

Websites would use JavaScript to programmatically declare capabilities with the same schema:

// Declare all tools for the site at once
navigator.ai.declareTools({
  description: "Weather forecast and current conditions service",
  tools: [
    {
      name: "getWeather",
      description: "Get current weather conditions for a location",
      parameters: [
        {
          name: "location",
          type: "string",
          description: "City name or zip code",
          required: true
        }
      ],
      returns: {
        type: "object",
        description: "Weather information"
      },
      execute: async function(params) {
        const response = await fetch(`/api/weather?location=${params.location}`);
        return await response.json();
      },
      permissions: {
        requiresConsent: false,
        requiresAuth: false
      }
    }
  ]
});

When to use: Dynamic sites, SPAs, when tools depend on user state.

Discovery Process

From AI Agent's Perspective

// 1. Agent discovers available tools organized by site
const toolsByOrigin = await navigator.ai.discoverTools();

// Result - tools organized by origin:
{
  "https://weather.com": {
    origin: "https://weather.com",
    description: "Weather forecast and current conditions service",
    tools: [
      {
        id: "weather.com:getWeather:v1",  // Unique tool ID
        name: "getWeather",
        description: "Get current weather conditions for a location",
        parameters: [
          {
            name: "location",
            type: "string",
            description: "City name or zip code",
            required: true
          }
        ],
        returns: {
          type: "object",
          description: "Weather information"
        },
        permissions: { requiresConsent: false },
        availability: "available"
      }
    ]
  }
}

// 2. Agent can inspect a specific tool
const weatherOrigin = toolsByOrigin["https://weather.com"];
const weatherTool = weatherOrigin.tools.find(t => t.name === 'getWeather');
console.log(weatherTool.id);  // "weather.com:getWeather:v1"
console.log(weatherTool.description);
// "Get current weather conditions for a location"

Tool Execution

Basic Execution

// AI discovers tools first
const toolsByOrigin = await navigator.ai.discoverTools();

// Get the tool ID from discovery result (treat as opaque data)
const weatherOrigin = toolsByOrigin["https://weather.com"];
const weatherTool = weatherOrigin.tools.find(t => t.name === 'getWeather');

// Execute using the ID from discovery
const result = await navigator.ai.executeTool(weatherTool.id, {
  location: "San Francisco"
});

// Result:
{
  success: true,
  data: {
    location: "San Francisco, CA",
    temperature: 68,
    conditions: "Partly Cloudy",
    humidity: 65,
    windSpeed: 12
  }
}

CodeMud

Other articles

Commit Messages Matter: Comparing Claude Code and Codex

The Test Case: A Real GTK Bug Fix

Codex vs Claude Code: A Developer's First Impressions

The Speed Factor

Simplified DOM: Making Web Pages Digestible for AI Models

The New Era of Knowledge Distillation: From Programming to Demonstration

Why We Need Web Standards for AI (Before It's Too Late)

ClaudeBugzilla AI Infrastructure Setup

A Day in Bug Triage: How It Works