noteflow/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

NoteFlow is an intelligent meeting notetaker: local-first audio capture + navigable recall + evidence-linked summaries. It is a client-server system built around a gRPC API for bidirectional audio streaming and transcription. The repository includes:

• A Python backend (src/noteflow/) hosting the gRPC server, domain logic, and infrastructure adapters.
• A Tauri + React desktop client (client/) that uses Rust for IPC and a React UI (Vite).

The gRPC schema is the shared contract between backend and client; keep proto changes in sync across Python, Rust, and TypeScript.

Quick Orientation (Start Here)

• Backend server entry point: python -m noteflow.grpc.server (implementation in src/noteflow/grpc/service.py).
• Tauri/React client: cd client && npm run dev (web), npm run tauri dev (desktop).
• Tauri IPC bridge: TypeScript calls in client/src/lib/tauri.ts map to Rust commands in client/src-tauri/src/commands/.
• Protobuf schema and generated Python stubs live in src/noteflow/grpc/proto/.

Build and Development Commands

# Install (editable with dev dependencies)
python -m pip install -e ".[dev]"

# Run gRPC server
python -m noteflow.grpc.server --help

# Run Tauri + React client UI
cd client
npm install
npm run dev
# Desktop Tauri dev (requires Rust toolchain)
npm run tauri dev
cd -

# Tests
pytest                           # Full suite
pytest -m "not integration"      # Skip external-service tests
pytest tests/domain/             # Run specific test directory
pytest -k "test_segment"         # Run by pattern

# Linting and type checking
ruff check .                     # Lint
ruff check --fix .               # Autofix
mypy src/noteflow                # Strict type checks
basedpyright                     # Additional type checks

# Client lint/format (from client/)
npm run lint                      # Biome
npm run lint:fix                  # Biome autofix
npm run format                    # Prettier
npm run format:check              # Prettier check
npm run lint:rs                   # Clippy (Rust)
npm run format:rs                 # rustfmt (Rust)

# Docker development
docker compose up -d postgres    # PostgreSQL with health checks
python scripts/dev_watch_server.py  # Auto-reload server (watches src/)

Docker Development

# Start PostgreSQL (with pgvector)
docker compose up -d postgres

# Dev container (VS Code) - full GUI environment
# .devcontainer/ includes PortAudio, GTK, pystray, pynput support
code .  # Open in VS Code, select "Reopen in Container"

# Development server with auto-reload
python scripts/dev_watch_server.py  # Uses watchfiles, monitors src/ and alembic.ini

Dev container features: dbus-x11, GTK-3, libgl1 for system tray and hotkey support.

Architecture

src/noteflow/
├── domain/           # Entities (meeting, segment, annotation, summary, triggers, webhooks, integrations) + ports
├── application/      # Use-cases/services (MeetingService, RecoveryService, ExportService, SummarizationService, TriggerService, WebhookService)
├── infrastructure/   # Implementations
│   ├── audio/        # sounddevice capture, ring buffer, VU levels, playback, buffered writer
│   ├── asr/          # faster-whisper engine, VAD segmenter, streaming
│   ├── diarization/  # Speaker diarization (streaming: diart, offline: pyannote.audio)
│   ├── summarization/# Multi-provider summarization (CloudProvider, OllamaProvider) + citation verification
│   ├── triggers/     # Auto-start signal providers (calendar, audio activity, foreground app)
│   ├── persistence/  # SQLAlchemy + asyncpg + pgvector, Alembic migrations
│   ├── security/     # keyring keystore, AES-GCM encryption
│   ├── export/       # Markdown/HTML/PDF export
│   ├── webhooks/     # Webhook executor with retry logic and HMAC signing
│   └── converters/   # ORM ↔ domain entity converters (including webhook converters)
├── grpc/             # Proto definitions, server, client, meeting store, modular mixins
└── config/           # Pydantic settings (NOTEFLOW_ env vars) + feature flags

Frontend (Tauri + React) lives outside the Python package:

client/
├── src/              # React UI, state, and view components
├── src-tauri/        # Rust/Tauri shell, IPC commands, gRPC client
├── e2e/              # Playwright tests
├── package.json      # Vite + test/lint scripts
└── vite.config.ts    # Vite configuration

Key patterns:

• Hexagonal architecture: domain → application → infrastructure
• Repository pattern with Unit of Work (SQLAlchemyUnitOfWork)
• gRPC bidirectional streaming for audio → transcript flow
• Protocol-based DI (see domain/ports/ and infrastructure protocols.py files)
• Modular gRPC mixins for separation of concerns (see below)

Domain Package Structure

domain/
├── entities/         # Core domain entities
│   ├── meeting.py    # Meeting, MeetingId, MeetingState
│   ├── segment.py    # Segment, WordTiming
│   ├── summary.py    # Summary, KeyPoint, ActionItem
│   ├── annotation.py # Annotation
│   └── integration.py# Integration, IntegrationType, IntegrationStatus
├── webhooks/         # Webhook domain
│   └── events.py     # WebhookEventType, WebhookConfig, WebhookDelivery, payload classes
├── ports/            # Repository protocols
│   ├── repositories.py # All repository protocols (MeetingRepository, WebhookRepository, etc.)
│   └── unit_of_work.py # UnitOfWork protocol with supports_* capability checks
└── utils/            # Domain utilities
    └── time.py       # utc_now() helper

gRPC Mixin Architecture

The gRPC server uses modular mixins for maintainability:

grpc/_mixins/
├── streaming.py      # ASR streaming, audio processing, partial buffers
├── diarization.py    # Speaker diarization jobs (background refinement, job TTL)
├── summarization.py  # Summary generation (separates LLM inference from DB transactions)
├── meeting.py        # Meeting lifecycle (create, get, list, delete, stop)
├── annotation.py     # Segment annotations CRUD
├── export.py         # Markdown/HTML/PDF document export
├── converters.py     # Protobuf ↔ domain entity converters
├── errors.py         # gRPC error helpers (abort_not_found, abort_invalid_argument)
└── protocols.py      # ServicerHost protocol for mixin composition

Each mixin operates on ServicerHost protocol, enabling clean composition in NoteFlowServicer.

Service Injection Pattern

Services are injected through a three-tier pattern:

1. ServicerHost Protocol (protocols.py) — declares required service attributes
2. NoteFlowServicer (service.py) — accepts services via __init__ and stores as instance attributes
3. NoteFlowServer (server.py) — creates/initializes services and passes to servicer

Example: _webhook_service, _summarization_service, _ner_service all follow this pattern.

Client Architecture (Tauri + React)

• React components are in client/src/components/, shared UI types in client/src/types/, and Zustand state in client/src/store/.
• Tauri IPC calls live in client/src/lib/tauri.ts and map to Rust handlers in client/src-tauri/src/commands/.
• Rust application entry points are client/src-tauri/src/main.rs and client/src-tauri/src/lib.rs; shared runtime state is in client/src-tauri/src/state/.

Database

PostgreSQL with pgvector extension. Async SQLAlchemy with asyncpg driver.

# Alembic migrations
alembic upgrade head
alembic revision --autogenerate -m "description"

Connection via NOTEFLOW_DATABASE_URL env var or settings.

Testing Conventions

• Test files: test_*.py, functions: test_*
• Markers: @pytest.mark.slow (model loading), @pytest.mark.integration (external services)
• Integration tests use testcontainers for PostgreSQL
• Asyncio auto-mode enabled
• React unit tests use Vitest; e2e tests use Playwright in client/e2e/.

Quality Gates

After any non-trivial changes, run the quality and test smell suite:

pytest tests/quality/                # Test smell detection (23 checks)

This suite enforces:

• No assertion roulette (multiple assertions without messages)
• No conditional test logic (loops/conditionals with assertions)
• No cross-file fixture duplicates (consolidate to conftest.py)
• No unittest-style assertions (use plain assert)
• Proper fixture typing and scope
• No pytest.raises without match=
• And 17 other test quality checks

Fixtures like crypto, meetings_dir, and mock_uow are provided by tests/conftest.py — do not redefine them in test files.

Proto/gRPC

Proto definitions: src/noteflow/grpc/proto/noteflow.proto Generated files excluded from lint: *_pb2.py, *_pb2_grpc.py

Regenerate after proto changes:

python -m grpc_tools.protoc -I src/noteflow/grpc/proto \
  --python_out=src/noteflow/grpc/proto \
  --grpc_python_out=src/noteflow/grpc/proto \
  src/noteflow/grpc/proto/noteflow.proto

Sync points (high risk of breakage):

• Rust gRPC types are generated at build time by client/src-tauri/build.rs. Keep Rust DTOs aligned with proto changes.
• Frontend enums/DTOs in client/src/types/ mirror proto enums and backend domain types; update these when proto enums change.
• When adding or renaming RPCs, update: server mixins, src/noteflow/grpc/client.py, Tauri command handlers, and client/src/lib/tauri.ts.

Common Pitfalls & Change Checklist

Proto / API evolution

• Update the schema in src/noteflow/grpc/proto/noteflow.proto first; treat it as the source of truth.
• Regenerate Python stubs (*_pb2.py, *_pb2_grpc.py) and verify imports still align in src/noteflow/grpc/.
• Ensure the gRPC server mixins in src/noteflow/grpc/_mixins/ implement new/changed RPCs.
• Update src/noteflow/grpc/client.py (Python client wrapper) to match the RPC signature and response types.
• Update Tauri/Rust command handlers (client/src-tauri/src/commands/) and any Rust gRPC types used by commands.
• Update TypeScript wrappers in client/src/lib/tauri.ts and shared DTOs/enums in client/src/types/.
• Add/adjust tests on both sides (backend unit/integration + client unit tests) when changing payload shapes.

Database schema & migrations

• Schema changes belong in src/noteflow/infrastructure/persistence/ plus an Alembic migration in src/noteflow/infrastructure/persistence/migrations/.
• Use alembic revision --autogenerate only after updating models; review the migration for correctness.
• Keep NOTEFLOW_DATABASE_URL in mind when running integration tests; default behavior may fall back to memory storage.
• Update repository/UnitOfWork implementations if new tables or relations are introduced.
• If you add fields used by export/summarization, ensure converters in infrastructure/converters/ are updated too.

Client sync points (Rust + TS)

• Tauri IPC surfaces (Rust commands) must match the TypeScript calls in client/src/lib/tauri.ts.
• Rust gRPC types are generated by client/src-tauri/build.rs; verify the proto path if you move proto files.
• Frontend enums in client/src/types/ mirror backend/proto enums; keep them aligned to avoid silent UI bugs.

Code Style

• Python 3.12+, 100-char line length
• Strict mypy (allow type: ignore[code] only with comment explaining why)
• Ruff for linting (E, W, F, I, B, C4, UP, SIM, RUF)
• Module soft limit 500 LoC, hard limit 750 LoC
• Frontend formatting uses Prettier (single quotes, 100 char width); linting uses Biome.
• Rust formatting uses rustfmt; linting uses clippy via the client scripts.

Feature Flags

Optional features controlled via NOTEFLOW_FEATURE_* environment variables:

Flag	Default	Controls	Prerequisites
NOTEFLOW_FEATURE_TEMPLATES_ENABLED	true	AI summarization templates	—
NOTEFLOW_FEATURE_PDF_EXPORT_ENABLED	true	PDF export format	WeasyPrint installed
NOTEFLOW_FEATURE_NER_ENABLED	false	Named entity extraction	spaCy model downloaded
NOTEFLOW_FEATURE_CALENDAR_ENABLED	false	Calendar sync	OAuth credentials configured
NOTEFLOW_FEATURE_WEBHOOKS_ENABLED	true	Webhook notifications	—

Access via get_settings().features.<flag_name>. Features with external dependencies default to false.

Spikes (De-risking Experiments)

spikes/ contains validated platform experiments with FINDINGS.md:

• spike_02_audio_capture/ - sounddevice + PortAudio
• spike_03_asr_latency/ - faster-whisper benchmarks (0.05x real-time)
• spike_04_encryption/ - keyring + AES-GCM (826 MB/s throughput)

Key Subsystems

Speaker Diarization

• Streaming: diart for real-time speaker detection during recording
• Offline: pyannote.audio for post-meeting refinement (higher quality)
• gRPC: RefineSpeakerDiarization (background job), GetDiarizationJobStatus (polling), RenameSpeaker

Summarization

• Providers: CloudProvider (Anthropic/OpenAI), OllamaProvider (local), MockProvider (testing)
• Templates: Configurable tone (professional/casual/technical), format (bullet_points/narrative/structured), verbosity (minimal/balanced/detailed)
• Citation verification: Links summary claims to transcript evidence
• Consent: Cloud providers require explicit user consent (stored in user_preferences)

Export

• Formats: Markdown, HTML, PDF (via WeasyPrint)
• Content: Transcript with timestamps, speaker labels, summary with key points and action items
• gRPC: ExportTranscript RPC with ExportFormat enum
• PDF styling: Embedded CSS for professional document layout

Named Entity Recognition (NER)

Automatic extraction of people, companies, products, locations from transcripts.

• Engine: spaCy with transformer models (en_core_web_sm or en_core_web_trf)
• Categories: person, company, product, technical, acronym, location, date, other
• Segment tracking: Entities link back to source segment_ids for navigation
• Confidence scores: Model confidence for each extracted entity
• Pinning: Users can pin (confirm) entities for future reference
• gRPC: ExtractEntities RPC with optional force_refresh
• Caching: Entities persisted in named_entities table, cached until refresh

Trigger Detection

• Signals: Calendar proximity, audio activity, foreground app detection
• Actions: IGNORE, NOTIFY, AUTO_START with confidence thresholds
• Client integration: Background polling with dialog prompts (start/snooze/dismiss)

Webhooks

Automated HTTP notifications for meeting lifecycle events.

• Events: meeting.completed, summary.generated, recording.started, recording.stopped
• Delivery: Exponential backoff retries (configurable max_retries, default 3)
• Security: HMAC-SHA256 signing via X-NoteFlow-Signature header when secret configured
• Headers: X-NoteFlow-Event (event type), X-NoteFlow-Delivery (unique delivery ID)
• Fire-and-forget: Webhook failures never block primary RPC operations
• Persistence: webhook_configs stores URL/events/secret, webhook_deliveries logs delivery attempts

Key files:

• domain/webhooks/events.py — WebhookEventType, WebhookConfig, WebhookDelivery, payload dataclasses
• infrastructure/webhooks/executor.py — HTTP client with retry logic
• application/services/webhook_service.py — orchestrates delivery to registered webhooks

Integrations

OAuth-based external service connections (calendar providers, etc.).

• Types: calendar (Google, Outlook)
• Status tracking: pending, connected, error, disconnected
• Secure storage: OAuth tokens stored in integration_secrets table
• Sync history: integration_sync_runs tracks each sync operation

ORM models in persistence/models/integrations/:

• IntegrationModel — provider config and status
• IntegrationSecretModel — encrypted OAuth tokens
• CalendarEventModel — cached calendar events
• MeetingCalendarLinkModel — links meetings to calendar events

Shared Utilities & Factories

Factories

Location	Function	Purpose
infrastructure/summarization/factory.py	create_summarization_service()	Auto-configured service with provider detection
infrastructure/persistence/database.py	create_async_engine()	SQLAlchemy async engine from settings
infrastructure/persistence/database.py	create_async_session_factory()	Session factory from DB URL
config/settings.py	get_settings()	Cached Settings from env vars
config/settings.py	get_trigger_settings()	Cached TriggerSettings from env vars

Converters

Location	Class/Function	Purpose
infrastructure/converters/orm_converters.py	OrmConverter	ORM ↔ domain entities (Meeting, Segment, Summary, etc.)
infrastructure/converters/asr_converters.py	AsrConverter	ASR DTOs → domain WordTiming
infrastructure/converters/webhook_converters.py	WebhookConverter	ORM ↔ domain (WebhookConfig, WebhookDelivery)
grpc/_mixins/converters.py	meeting_to_proto(), segment_to_proto_update()	Domain → protobuf messages
grpc/_mixins/converters.py	create_segment_from_asr()	ASR result → Segment with word timings

Repository Base (persistence/repositories/_base.py)

Method	Purpose
_execute_scalar()	Single result query (or None)
_execute_scalars()	All scalar results from query
_add_and_flush()	Add model and flush to DB
_delete_and_flush()	Delete model and flush

Security Helpers (infrastructure/security/keystore.py)

Function	Purpose
_decode_and_validate_key()	Validate base64 key, check size
_generate_key()	Generate 256-bit key as (bytes, base64_str)

Export Helpers (infrastructure/export/_formatting.py)

Function	Purpose
format_timestamp()	Seconds → MM:SS or HH:MM:SS
format_datetime()	Datetime → display string

Summarization (infrastructure/summarization/)

Location	Function	Purpose
_parsing.py	build_transcript_prompt()	Transcript with segment markers for LLM
_parsing.py	parse_llm_response()	JSON → Summary entity
citation_verifier.py	verify_citations()	Validate segment_ids exist

Diarization (infrastructure/diarization/assigner.py)

Function	Purpose
assign_speaker()	Speaker for time range from turns
assign_speakers_batch()	Batch speaker assignment

Triggers (infrastructure/triggers/calendar.py)

Function	Purpose
parse_calendar_events()	Parse events from config/env

Webhooks (infrastructure/webhooks/executor.py)

Class/Method	Purpose
WebhookExecutor	HTTP delivery with retry logic and HMAC signing
WebhookExecutor.deliver()	Deliver payload to webhook URL with exponential backoff
WebhookExecutor._build_headers()	Build headers including X-NoteFlow-Signature

Recovery Service (application/services/recovery_service.py)

Method	Purpose
recover_all()	Orchestrate meeting + job recovery
RecoveryResult	Dataclass with recovery counts

Webhook Service (application/services/webhook_service.py)

Method	Purpose
register_webhook()	Register a webhook configuration
trigger_meeting_completed()	Fire webhooks on meeting completion
trigger_summary_generated()	Fire webhooks on summary generation
trigger_recording_started/stopped()	Fire webhooks on recording lifecycle

Unit of Work Repositories

The UnitOfWork protocol provides access to all repositories:

Property	Repository	Supports In-Memory
meetings	MeetingRepository	Yes
segments	SegmentRepository	Yes
summaries	SummaryRepository	Yes
annotations	AnnotationRepository	Yes
diarization_jobs	DiarizationJobRepository	Yes
preferences	PreferencesRepository	Yes
entities	EntityRepository	Yes
integrations	IntegrationRepository	DB only
webhooks	WebhookRepository	Yes

Check capability with supports_* properties (e.g., uow.supports_webhooks).

Known Issues

See docs/triage.md for tracked technical debt. See docs/sprints/ for feature implementation plans and status.

Resolved:

• Server-side state volatility → Diarization jobs persisted to DB
• Hardcoded directory paths → asset_path column added to meetings
• Synchronous blocking in async gRPC → run_in_executor for diarization
• Summarization consent not persisted → Stored in user_preferences table
• VU meter update throttling → 20fps throttle implemented
• Webhook infrastructure missing → Full webhook subsystem with executor, service, and repository
• Integration/OAuth token storage → IntegrationSecretModel for secure token storage

MCP Tools Reference

Firecrawl (Web Scraping & Search)

Tool	Use Case
firecrawl_scrape	Extract content from a single URL (markdown, HTML, screenshots)
firecrawl_search	Web search with optional content extraction; supports operators (site:, inurl:, -exclude)
firecrawl_map	Discover all URLs on a website before scraping
firecrawl_crawl	Multi-page crawl with depth/limit controls (use sparingly—token-heavy)
firecrawl_extract	LLM-powered structured data extraction with JSON schema

Workflow: Search first without scrapeOptions, then scrape relevant URLs individually.

Sequential Thinking (Reasoning)

Tool	Use Case
sequentialthinking	Break down complex problems into numbered thought steps; supports revision, branching, and hypothesis verification

When to use: Multi-step analysis, design decisions, debugging with course-correction, architecture planning.

Context7 (Library Documentation)

Tool	Use Case
resolve-library-id	Convert package name → Context7 library ID (required first step)
get-library-docs	Fetch up-to-date docs; mode='code' for API/examples, mode='info' for concepts

Workflow: resolve-library-id("fastapi") → get-library-docs("/tiangolo/fastapi", topic="dependencies").

Serena (Semantic Code Tools)

Navigation (prefer over grep/read for code):

Tool	Use Case
get_symbols_overview	List top-level symbols in a file (classes, functions, variables)
find_symbol	Search by name path pattern; use depth for children, include_body for source
find_referencing_symbols	Find all references to a symbol across the codebase
search_for_pattern	Regex search across files (for non-symbol searches)

Editing (atomic symbol-level changes):

Tool	Use Case
replace_symbol_body	Replace entire symbol definition
insert_before_symbol / insert_after_symbol	Add code adjacent to a symbol
rename_symbol	Rename across entire codebase

Memory (persistent cross-session knowledge):

Tool	Use Case
list_memories	Show available project memories
read_memory / write_memory	Retrieve or store project knowledge

Guardrails (call before major actions):

Tool	Use Case
think_about_collected_information	Validate research completeness after search sequences
think_about_task_adherence	Verify alignment before editing code
think_about_whether_you_are_done	Confirm task completion

Principle: Use symbolic tools over file reads; retrieve symbol bodies only when needed for understanding or editing.