guide/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

A FastAPI-based guided demo platform that automates browser interactions with Raindrop using Playwright. The app executes data-driven actions (stored in `ActionRegistry`) on behalf of personas that target configured browser hosts (CDP or headless). All configuration is externalized via YAML files and environment overrides.

**Entry Point:** `python -m guide` (runs `src/guide/main.py` → `guide.app.main:app`)
**Python Version:** 3.12+
**Key Dependencies:** FastAPI, Playwright, Pydantic v2, PyYAML, httpx

## Essential Commands

```bash
# Install dependencies
uv sync

# Type checking (required before commits)
basedpyright src

# Compile sanity check
python -m compileall src/guide

# Run development server (default: localhost:8000)
python -m guide
# or with custom host/port:
HOST=127.0.0.1 PORT=9000 python -m guide

# View API docs
# Navigate to http://localhost:8000/docs

# Key endpoints:
# GET  /healthz                           # liveness check
# GET  /actions                           # list action metadata
# POST /actions/{id}/execute              # execute action; returns ActionEnvelope with correlation_id
# GET  /config/browser-hosts              # view current default + host map
```

## Code Structure

**Root module:** `src/guide/app/`

- **`actions/`** — Demo action implementations with auto-discovery via `@register_action` decorator. Auto-wired via `ActionRegistry` with dependency injection. Submodules: `base.py` (DemoAction, CompositeAction, ActionRegistry), `registry.py` (auto-discovery), `playbooks.py` (multi-step workflows like OnboardingFlow, FullDemoFlow), `auth/` (LoginAsPersonaAction), `intake/` (CreateIntakeAction), `sourcing/` (AddSupplierAction).
- **`auth/`** — Pluggable MFA/auth helpers. `mfa.py` defines `MfaCodeProvider` interface; `DummyMfaCodeProvider` raises `NotImplementedError` (implement for production). `session.py` provides `ensure_persona()` for login flows.
- **`browser/`** — `BrowserPool` (persistent browser instances per host) + `BrowserClient` (context-managed page access). Handles both CDP attach and headless launch. `pool.py` manages lifecycle; `client.py` wraps for async context manager pattern. `diagnostics.py` captures screenshots, HTML, console logs for error debugging.
- **`core/`** — App bootstrap: `config.py` (AppSettings with Pydantic v2, env prefix `RAINDROP_DEMO_`, YAML + JSON override cascade), `logging.py` (structured logging with request-scoped context variables).
- **`errors/`** — `GuideError` hierarchy (ConfigError, BrowserConnectionError, PersonaError, AuthError, MfaError, ActionExecutionError, GraphQLTransportError, GraphQLOperationError); routers normalize to HTTP responses with debug info.
- **`raindrop/`** — GraphQL client + operations. `graphql.py` (httpx-based HTTP client), `operations/` (intake.py, sourcing.py with query/mutation definitions), `generated/` (ariadne-codegen auto-generated Pydantic models), `queries/` (GraphQL query/mutation files).
- **`strings/`** — Flattened registry (Phase 1 refactoring complete). `registry.py` enforces domain-keyed lookups; missing keys raise immediately. No inline UI strings in actions. Submodules: `selectors/` (CSS, data-testid, aria), `labels/` (UI element names), `demo_texts/` (copy snippets).
- **`models/`** — Domain and persona models using Pydantic v2. `domain/models.py` (ActionRequest, ActionContext, ActionResult, ActionMetadata, ActionEnvelope, DebugInfo, BrowserHostDTO, BrowserHostsResponse). `personas/models.py` (DemoPersona with auto-coerced enums PersonaRole, LoginMethod; unified from Phase 2A refactoring). `personas/store.py` (PersonaStore in-memory registry from AppSettings).
- **`utils/`** — Shared helpers. Keep <300 LoC per file; avoid circular imports. Modules: `ids.py` (UUID/correlation ID), `env.py` (environment utilities), `retry.py` (backoff + jitter), `timing.py` (rate-limiting).
- **`api/`** — FastAPI routers in `routes/`. `health.py` (GET /healthz), `actions.py` (GET /actions, POST /actions/{id}/execute), `config.py` (GET /config/browser-hosts). Map requests → `ActionRegistry` → `BrowserClient` → `ActionEnvelope` responses with error capture.

**Config files (git-tracked):**
- `config/hosts.yaml` — Browser host targets (id, kind: cdp|headless, host, port, browser type).
- `config/personas.yaml` — Personas (id, role, email, login_method, browser_host_id).

**Config overrides (runtime only, never commit):**
- `RAINDROP_DEMO_BROWSER_HOSTS_JSON` — JSON array overrides `hosts.yaml`.
- `RAINDROP_DEMO_PERSONAS_JSON` — JSON array overrides `personas.yaml`.
- `RAINDROP_DEMO_RAINDROP_BASE_URL` — Override default `https://app.raindrop.com`.

## Architecture Patterns

### App Initialization (main.py → create_app)

1. Load `AppSettings` (env + YAML + JSON overrides).
2. Build `PersonaStore` from config.
3. Build `ActionRegistry` with auto-discovered actions (via `@register_action` decorator + `pkgutil.walk_packages()`). DI context includes persona store, Raindrop URL, etc. Auto-instantiates actions with parameter matching.
4. Create `BrowserPool` (manages persistent browser instances per host, allocates fresh contexts/pages per request).
5. Create `BrowserClient` (wraps BrowserPool with async context manager).
6. Stash instances on `app.state` for dependency injection in routers.
7. Register error handlers (GuideError → HTTP + debug info; unhandled → 500 + logging).

### Action Execution Pipeline

- Request: `POST /actions/{action_id}/execute` with `ActionRequest` (persona_id, host_id, params).
- Router resolves persona + host from config → validates persona exists.
- Router generates `correlation_id` (UUID) and `ActionContext` (includes persona, host, params, correlation_id, shared_state dict for composite actions).
- `ActionRegistry.get(action_id)` retrieves action (auto-discovered, dependency-injected).
- `BrowserClient.open_page(host_id)` → allocates fresh context + page from BrowserPool. Reuses persistent browser instance for host; creates new page/context for isolation.
- `Action.run(page, context)` executes logic; may call `ensure_persona()` (login flow) before starting. For composite actions, passes shared_state dict to child actions.
- On error: captures debug info (screenshot, HTML, console logs) and returns with DebugInfo attached.
- Response: `ActionEnvelope` (status, correlation_id, result/error, debug_info).

### Browser Host Resolution & Pooling

- **BrowserPool Architecture:**
  - One persistent browser instance per host (lazy-initialized on first request).
  - Fresh context + page allocated per request for isolation (no cross-request state).
  - Proper cleanup via async context manager pattern.
  - Handles timeouts, connection errors, context cleanup.

- **Host Kind Resolution:**
  - `kind: cdp` — connect to running Raindrop instance via Chrome DevTools Protocol (requires `host` + `port`). Errors surface as `BrowserConnectionError`.
  - `kind: headless` — launch Playwright browser (chromium/firefox/webkit); set `browser` field in config.
  - Always use `async with BrowserClient.open_page(host_id) as page:` to ensure proper cleanup (context manager unwinding in BrowserPool).

### GraphQL & Data Layer

- `raindrop/graphql.py` — HTTP client (httpx, 10s timeout).
- `raindrop/operations/` — query/mutation definitions + Pydantic-validated response models.
- `raindrop/generated/` — Auto-generated Pydantic models (via ariadne-codegen) from Raindrop GraphQL schema. Type-safe; auto-synced with schema.
- `raindrop/queries/` — GraphQL query/mutation .graphql files (single source of truth for operations).
- Validate all responses with Pydantic models; schema mismatches → `GuideError`.
- Never embed tokens/URLs; pull from `AppSettings` (env-driven).
- Transport errors → `GraphQLTransportError`; operation errors → `GraphQLOperationError` (includes `details` from server).

### Selector & String Management (strings/)

- **Flattened Registry (Phase 1 refactoring complete):** All selectors, labels, copy in `strings/` submodules; no 3-layer wrappers.
- Direct field access: `app_strings.intake.field` (was `app_strings.intake.selectors.field`). Registry enforces domain-keyed lookups; missing keys raise immediately.
- Organize into: `selectors/` (CSS, data-testid, aria attributes), `labels/` (UI element names), `demo_texts/` (copy/text snippets).
- Selectors should be reusable and labeled; avoid brittle text selectors—prefer `data-testid` or aria labels.
- No inline UI strings in action code; all strings centralized in `strings/` for easy maintenance and i18n readiness.

## Development Workflow

1. **Edit code** (actions, browser logic, GraphQL ops, etc.).
2. **Run type check:** `basedpyright src` (catches generic types, missing annotations).
3. **Sanity compile:** `python -m compileall src/guide` (syntax check).
4. **Smoke test:** `python -m guide` then hit `/docs` or manual test via curl.
5. **Review error handling:** ensure `GuideError` subclasses are raised, not generic exceptions.
6. **Commit** with scoped, descriptive message (e.g., `feat: add auth login action`, `chore: tighten typing`).

## Type & Linting Standards

- **Python 3.12+:** Use PEP 604 unions (`str | None`), built-in generics (`list[str]`, `dict[str, JSONValue]`).
- **Ban `Any` and `# type: ignore`:** Use type guards or Protocol instead.
- **Pydantic v2:** Explicit types, model_validate for parsing, model_copy for immutable updates.
- **Type checker:** Pyright (via basedpyright).
- **Docstrings:** Imperative style, document public APIs, include usage examples.

## Error Handling & Logging

- Always raise `GuideError` subclasses (not generic `Exception`); routers translate to HTTP responses.
- Log via `core/logging` (structured, levelled). Include persona/action IDs and host targets for traceability.
- For browser flows, use Playwright traces (enabled by default in `BrowserClient`); disable only intentionally.
- Validate external inputs early; surface schema/connection issues as `GuideError`.

## Testing & Quality Gates

- **Minimum gate:** `basedpyright src` + `python -m compileall src/guide` before merge.
- **Test Coverage:** 28 passing tests across unit and integration suites (in `tests/` directory).
- **Test Structure:**
  - `tests/unit/` — Unit tests for strings registry, models (persona/host validation), action registration.
  - `tests/integration/` — Integration tests for BrowserClient, BrowserPool, and browser lifecycle.
  - `conftest.py` — Shared fixtures: mock_browser_hosts, mock_personas, app_settings, persona_store, action_registry, action_context, mock_page, mock_browser_client, mock_browser_pool.
  - Pytest configured with asyncio_mode=auto for async test support.
- Mock Playwright/GraphQL in tests; avoid real network/CDP calls.
- Require deterministic fixtures; document any env vars needed in test module docstring.
- No loops or conditionals in tests; use `@pytest.mark.parametrize` for multiple cases.

## MFA & Auth

- Default `DummyMfaCodeProvider` raises `NotImplementedError`.
- For real runs, implement a provider and wire it in `core/config.py` or `auth/` modules.
- `ensure_persona()` in actions calls the provider; stub or override for demo/test execution.

## Performance & Footprint

- Keep browser sessions short-lived; close contexts to avoid handle leaks.
- Cache expensive GraphQL lookups (per-request OK, global only if safe).
- Don't widen dependencies without justification; stick to project pins in `pyproject.toml`.
- Promptly close Playwright contexts/browser handles (wrapped in contextmanager; keep action code lean).

## Refactoring Status (100% Complete)

The project has completed **8 major refactoring phases** achieving full architectural modernization:

| Phase | Focus | Status |
|-------|-------|--------|
| Phase 1 | Strings Registry Flattening | ✅ Complete |
| Phase 2A | Persona Model Unification | ✅ Complete |
| Phase 2B | BrowserHostDTO Separation | ✅ Complete |
| Phase 3 | Action Registration Standardization | ✅ Complete |
| Phase 4 | GraphQL Code Generation (ariadne-codegen) | ✅ Complete |
| Phase 5 | Browser Context Isolation | ✅ Complete |
| Phase 6-8 | Error Handling & Logging Enhancements | ✅ Complete |

**Quality Metrics:**
- Zero type errors (basedpyright)
- All linting passed
- 28 tests passing
- Zero code redundancy
- ~2,898 lines of production code

## Git & PR Hygiene

- Scoped, descriptive commits (e.g., `feat: add sourcing action`, `fix: handle missing persona host`).
- PRs should state changes, commands run, new config entries (hosts/personas).
- Link related issues; include screenshots/logs for UI or API behavior changes.
- Never commit credentials, MFA tokens, or sensitive config; use env overrides.

## Action Registration & Dependency Injection

### Registering a New Action

Use the `@register_action` decorator to auto-discover and register actions:

```python
from actions.base import DemoAction, register_action

@register_action
class MyAction(DemoAction):
    id = "my-action"
    description = "Does something cool"
    category = "demo"

    # Optional: Declare dependencies (auto-injected by ActionRegistry)
    def __init__(self, persona_store: PersonaStore):
        self.persona_store = persona_store

    async def run(self, page: Page, context: ActionContext) -> ActionResult:
        # Implementation
        ...
```

**Auto-Discovery:** `ActionRegistry` uses `pkgutil.walk_packages()` to discover all modules in `actions/` and collect all `@register_action` decorated classes. No manual registration needed.

**Dependency Injection:** Parameters in `__init__` are matched by name against DI context dict. Example: `persona_store` parameter → resolved from DI context during action instantiation.

### Multi-Step Workflows (CompositeAction)

For workflows spanning multiple steps with shared state:

```python
@register_action
class MyWorkflow(CompositeAction):
    id = "my-workflow"
    description = "Multi-step workflow"
    category = "demo"

    child_actions = ("step1-action", "step2-action", "step3-action")

    async def on_step_complete(self, step_id: str, result: ActionResult) -> None:
        # Optional: Update shared_state after each step
        # Accessed in child actions via context.shared_state dict
        pass
```

## Quick Checklist (New Feature)

- [ ] Add action in `actions/` submodule (or submodule directory like `actions/intake/`); use `@register_action` decorator.
- [ ] Add action-specific logic; keep it thin and testable. Use `strings/` for all selectors/copy.
- [ ] Ensure persona/host exist in `config/hosts.yaml` + `config/personas.yaml` (or use env overrides).
- [ ] If action needs GraphQL, add query/mutation to `raindrop/operations/` + `.graphql` files in `raindrop/queries/`.
- [ ] If action needs UI strings, add to `strings/` submodules (selectors, labels, demo_texts).
- [ ] Run `basedpyright src` + `python -m compileall src/guide` (type check + syntax check).
- [ ] Run `pytest tests/` to ensure no regressions (28 tests must pass).
- [ ] Test via `python -m guide` + navigate to `http://localhost:8000/docs` to test endpoint.
- [ ] If auth flow required, implement/mock MFA provider or use `DummyMfaCodeProvider` for testing.
- [ ] Review error handling; raise `GuideError` subclasses, not generic exceptions.
- [ ] Commit with descriptive message (e.g., `feat: add my-action`, `test: add my-action tests`).