AI Page Agent

Learning objectives

• Explain the observe-decide-act-result loop that browser-agent frameworks use to operate a UI.
• Compare text/DOM-based grounding (no screenshots) against vision-based agent approaches.
• Evaluate the build-vs-buy tradeoff between a custom in-page agent and frameworks like Stagehand, browser-use, or Nanobrowser.
• Identify where a human-in-the-loop guardrail belongs when an agent can take destructive actions.
• Run the same command through a rule-based parser and a real on-device model, and explain concretely why one succeeds where the other fails.

Run modes

• Supported modes: Browser
• Demo type: Interactive browser demo
• Primary launch surface: index.html

Before you start

Starts immediately in browser with no installs, no API keys, and classroom-safe defaults. Rule-based mode is the default and never downloads anything.

On-device AI is opt-in: selecting it only checks for Chrome's built-in AI first (no download); if that's unavailable, it falls back to WebLLM, which downloads a small (roughly 300-500MB) model the first time over WebGPU. Nothing downloads until you explicitly choose that mode.

Rule-based (default)

Regex and keyword matching against the command text. Zero setup, deterministic, and instantly reproducible — but brittle. Natural phrasings outside its fixed vocabulary (e.g. "the customer paying twice needs this fixed") return "no confident match" even though a human reading it knows exactly what to do.

Chrome built-in AI

Uses Chrome's on-device Prompt API (LanguageModel/window.ai.languageModel), backed by Gemini Nano. No download managed by this page — Chrome handles that. Availability varies by Chrome version and may require enabling a flag; the demo detects this and reports status honestly rather than pretending it always works.

WebLLM fallback

If Chrome's built-in AI isn't available, the demo checks for WebGPU and, if present, downloads a small model (Qwen2.5-0.5B-Instruct) via WebLLM and runs it locally. Works in any WebGPU-capable browser, not just Chrome — at the cost of a real download and slower first response.

Stagehand (Browserbase)

Node SDK, CDP-based, code + natural language hybrid. Needs a Browserbase account and an LLM API key.

Live-demo fit: Poor — needs a backend and paid keys.

browser-use

Python + Playwright. Needs Python 3.11+ and an LLM API key.

Live-demo fit: Poor — Python backend, not static-hostable.

Nanobrowser

Chrome extension, multi-agent (Planner + Navigator). Needs an extension install and an LLM API key (or local Ollama).

Live-demo fit: Partial — real production pattern, but needs an install.

Core context

A chatbot that answers questions about your product is a different (and much lower-risk) product decision than an agent that takes actions inside your product's own UI. This demo makes that second category concrete.

Concepts covered

What students should note

The rule-based parser will fail on phrasings a real LLM handles easily — try "the customer paying twice needs this fixed" in both modes and compare. That gap is the honest tradeoff of zero-setup, zero-cost teaching demos versus what a production agent actually needs. On-device AI narrows that gap without needing an API key, but isn't guaranteed to be available in every browser.

List of steps

• Click an example command chip, or type your own (e.g. "Resolve ticket 101").
• Watch the Agent Trace panel's four steps: Observe, Decide, Act, Result.
• Try a destructive command (resolve/escalate) with the guardrail on, then confirm or cancel it.
• Turn the guardrail off and re-run the same command to see it execute immediately.
• Try a command the rule-based parser can't handle (e.g. "the customer paying twice needs this fixed"), note the "no confident match" result, then switch the interpreter to On-device AI and try the exact same command.
• Check the "On-device AI status" panel — if it says "Not supported," that's a real, honest capability check, not a bug; discuss what that means for shipping AI features to a real user base with mixed browsers.

Controls explained

• `Command input` is the only control a user needs — the ticket-level buttons and dropdowns still work by hand, for comparison.
• `Command interpreter` switches the "Decide" step between the rule-based parser and on-device AI. Switching to on-device AI triggers a one-time capability check (and possibly a model download) — nothing loads until you choose it.
• `Safety guardrail` toggles whether destructive actions (resolve, escalate) pause for human confirmation, regardless of which interpreter decided the action.
• `Run Agent` executes the observe-decide-act-result loop once per click.

Buttons / actions

• `Confirm Action` / `Cancel` only appear when the guardrail intercepts a destructive action.
• `Reset Demo` restores the ticket queue, action counter, guardrail, and interpreter to their starting state. An already-loaded on-device model stays in memory — it won't re-download.

Outputs explained

• `Observe` shows how many actionable elements the agent found and one example, proving it read the live DOM, not a fixed script.
• `Decide` shows the matched action and target element, or an honest "no confident match" when the parser fails.
• `Act` / `Result` show the actual DOM mutation performed and the resulting ticket state.

Discussion and reflection

• Which commands did the agent handle well, and which did it miss? What would a real LLM need to fix that gap?
• Where in your own product would an agent like this be useful — and where would the guardrail be non-negotiable?
• Given the framework comparison above, would you build a custom in-page agent, or adopt Stagehand/browser-use/Nanobrowser? What does that decision cost in infrastructure and risk?