# Centralized Logging Migration - Agent-Driven Execution Plan

> **Sprint Reference**: `docs/sprints/sprint_logging_centralization.md`
> **Technical Debt**: `docs/triage.md`
> **Quality Gates**: `docs/sprints/QUALITY_STANDARDS.md`

---

## Executive Summary

This plan orchestrates the migration from stdlib `logging` to `structlog` using specialized agents for discovery, validation, and implementation. Each phase is designed for parallel execution where possible.

---

## ⚠️ CRITICAL: Quality Enforcement Rules

### ABSOLUTE PROHIBITIONS

1. **NEVER modify quality test thresholds** - If violations exceed thresholds, FIX THE CODE, not the tests
2. **NEVER claim errors are "preexisting"** without baseline proof - Capture baselines BEFORE any changes
3. **NEVER batch quality checks** - Run `make quality-py` after EVERY file modification
4. **NEVER skip quality gates** to "fix later" - All code must pass before proceeding

### MANDATORY QUALITY CHECKPOINTS

**After EVERY code change** (not "at the end of a phase"):

```bash
# Run after EACH file edit - no exceptions
make quality-py
```

This runs:
- `ruff check .` — Linting (ALL code, not just tests)
- `basedpyright` — Type checking (ALL code, not just tests)
- `pytest tests/quality/ -q` — Code smell detection (ALL code)

### BASELINE CAPTURE (Required Before Phase 2)

```bash
# Capture baseline BEFORE any migration work
make quality-py 2>&1 | tee /tmp/quality_baseline_$(date +%Y%m%d_%H%M%S).log

# Record current threshold violations
pytest tests/quality/ -v --tb=no | grep -E "(PASSED|FAILED|violations)" > /tmp/threshold_baseline.txt
```

Any NEW violations introduced during migration are **agent responsibility** and must be fixed immediately.

### CODE QUALITY STANDARDS (Apply to ALL Code)

These apply to **infrastructure modules, services, gRPC handlers, processors** — not just tests:

| Rule | Applies To | Enforcement |
|------|-----------|-------------|
| No `# type: ignore` | ALL Python code | `basedpyright` |
| No `Any` type | ALL Python code | `basedpyright` |
| Union syntax `str \| None` | ALL Python code | `ruff UP` |
| Module < 500 lines (soft) | ALL modules | `tests/quality/test_code_smells.py` |
| Module < 750 lines (hard) | ALL modules | `tests/quality/test_code_smells.py` |
| Function < 75 lines | ALL functions | `tests/quality/test_code_smells.py` |
| Complexity < 15 | ALL functions | `tests/quality/test_code_smells.py` |
| Parameters ≤ 7 | ALL functions | `tests/quality/test_code_smells.py` |
| No magic numbers > 100 | ALL code | `tests/quality/test_magic_values.py` |
| No hardcoded paths | ALL code | `tests/quality/test_magic_values.py` |
| No repeated string literals | ALL code | `tests/quality/test_magic_values.py` |
| No stale TODO/FIXME | ALL code | `tests/quality/test_stale_code.py` |
| No commented-out code | ALL code | `tests/quality/test_stale_code.py` |

---

## Phase 0: Pre-Flight Validation

### Agent Task: Dependency Audit

| Agent | Purpose | Deliverable |
|-------|---------|-------------|
| `Explore` | Verify structlog compatibility with existing Rich usage | Compatibility report |
| `Explore` | Locate all `logging.basicConfig()` calls | File list with line numbers |
| `Explore` | Find LogBuffer integration points | Integration map |

**Commands to validate:**
```bash
# Verify Rich is installed
python -c "import rich; print(rich.__version__)"

# Dry-run structlog install
uv pip install --dry-run "structlog>=24.0"
```

---

## Phase 1: Discovery & Target Mapping

### 1.1 Agent: Locate All Logging Usage

**Objective**: Build comprehensive map of all 71+ files using stdlib logging.

**Agent Type**: `Explore` (thorough mode)

**Queries**:
1. "Find all files with `import logging` in src/noteflow/"
2. "Find all `logging.getLogger(__name__)` patterns"
3. "Find all `logger.info/debug/warning/error/exception` calls with their argument patterns"
4. "Identify %-style formatting vs f-string usage in log calls"

**Expected Output Structure**:
```yaml
discovery:
  files_with_logging: 71
  patterns:
    module_logger: 68  # logger = logging.getLogger(__name__)
    basic_config: 2    # logging.basicConfig()
    percent_style: 45  # logger.info("msg %s", arg)
    fstring_style: 23  # logger.info(f"msg {arg}")
    exception_calls: 12 # logger.exception()
  by_layer:
    application: 12
    infrastructure: 35
    grpc: 24
```

### 1.2 Agent: Map Critical Logging Gaps (from triage.md)

**Agent Type**: `Explore` (thorough mode)

**Objective**: Validate each issue in triage.md still exists and capture exact locations.

**Target Categories**:

| Category | File Pattern | Agent Query |
|----------|--------------|-------------|
| Network/External | `*_provider.py`, `*_adapter.py` | "Find async HTTP calls without timing logs" |
| Blocking Ops | `*_engine.py`, `*_service.py` | "Find `run_in_executor` calls without duration logging" |
| Silent Failures | `repositories/*.py` | "Find try/except blocks that return None without logging" |
| State Transitions | `_mixins/*.py` | "Find state assignments without transition logs" |
| DB Operations | `repositories/*.py`, `unit_of_work.py` | "Find commit/rollback without logging" |

### 1.3 Agent: Context Variable Analysis

**Agent Type**: `feature-dev:code-explorer`

**Objective**: Trace `get_logging_context()` usage for processor design.

**Tasks**:
1. Find all `request_id_var`, `user_id_var`, `workspace_id_var` usages
2. Map where context is SET vs where it's READ
3. Identify any gaps in context propagation

---

## Phase 2: Infrastructure Implementation

### 2.1 Create Core Configuration Module

**File**: `src/noteflow/infrastructure/logging/config.py`

**Agent Type**: `feature-dev:code-architect`

**Design Constraints** (from QUALITY_STANDARDS.md):
- No `Any` types
- No `# type: ignore` without justification
- All public functions must have return type annotations
- Docstrings written imperatively

**Implementation Spec**:
```python
"""Centralized logging configuration with dual output.

Configures structlog with Rich console + JSON file output.
"""

from __future__ import annotations

from dataclasses import dataclass
from pathlib import Path
from typing import TYPE_CHECKING

import structlog

if TYPE_CHECKING:
    from collections.abc import Sequence

    from structlog.typing import Processor


@dataclass(frozen=True, slots=True)
class LoggingConfig:
    """Configuration for centralized logging."""

    level: str = "INFO"
    json_file: Path | None = None
    enable_console: bool = True
    enable_json_console: bool = False
    enable_log_buffer: bool = True
    enable_otel_context: bool = True
    enable_noteflow_context: bool = True
    console_colors: bool = True


def configure_logging(
    config: LoggingConfig | None = None,
    *,
    level: str = "INFO",
    json_file: Path | None = None,
) -> None:
    """Configure centralized logging with dual output.

    Call once at application startup.

    Args:
        config: Full configuration object, or use keyword args.
        level: Log level (DEBUG, INFO, WARNING, ERROR).
        json_file: Optional path for JSON log file.
    """
    ...
```

### 2.2 Create Custom Processors Module

**File**: `src/noteflow/infrastructure/logging/processors.py`

**Agent Type**: `feature-dev:code-architect`

**Processors to Implement**:

| Processor | Source | Purpose |
|-----------|--------|---------|
| `add_noteflow_context` | New | Inject request_id, user_id, workspace_id |
| `add_otel_trace_context` | Adapt from `log_buffer.py:132-156` | Inject trace_id, span_id |
| `log_buffer_processor` | New | Feed events to existing LogBuffer |

**Quality Requirements**:
- Each processor must be a pure function
- Must handle missing context gracefully (no exceptions)
- Must include type annotations for all parameters

### 2.3 Create Handlers Module

**File**: `src/noteflow/infrastructure/logging/handlers.py`

**Agent Type**: `feature-dev:code-architect`

**Responsibilities**:
- Configure Rich ConsoleRenderer for TTY
- Configure JSONRenderer for file/OTEL
- Wire both to stdlib logging handlers

### 2.4 Adapt LogBuffer

**File**: `src/noteflow/infrastructure/logging/log_buffer.py`

**Agent Type**: `feature-dev:code-reviewer` (review current implementation first)

**Changes Required**:
1. Create structlog processor that feeds LogBuffer
2. Convert `LogEntry` creation to use structlog event_dict
3. Preserve existing `_get_current_trace_context()` logic

---

## Phase 3: Entry Point Integration

### 3.1 Agent: Locate Entry Points

**Agent Type**: `Explore`

**Query**: "Find all main() functions and startup initialization in src/noteflow/"

**Expected Entry Points**:
- `src/noteflow/grpc/server.py` - Main server
- `src/noteflow/cli/retention.py` - CLI tool
- `src/noteflow/cli/models.py` - CLI tool

### 3.2 Integration Tasks

| File | Change | Validation |
|------|--------|------------|
| `grpc/server.py` | Add `configure_logging()` call before server start | Server logs in both formats |
| `cli/retention.py` | Remove `basicConfig()`, add `configure_logging()` | CLI logs correctly |
| `cli/models.py` | Remove `basicConfig()`, add `configure_logging()` | CLI logs correctly |

---

## Phase 4: Automated Migration Script

### 4.1 Migration Script Design

**File**: `scripts/migrate_logging.py`

**Agent Type**: `feature-dev:code-architect`

**Transformations**:

```python
# Transform 1: Import statement
# Before: import logging
# After:  import structlog

# Transform 2: Logger creation
# Before: logger = logging.getLogger(__name__)
# After:  logger = structlog.get_logger()

# Transform 3: %-style formatting
# Before: logger.info("Processing %s for %s", item_id, user_id)
# After:  logger.info("processing", item_id=item_id, user_id=user_id)

# Transform 4: Exception logging
# Before: logger.exception("Failed to process")
# After:  logger.exception("processing_failed")
```

**Quality Requirements**:
- Must preserve semantic meaning
- Must handle all patterns found in Phase 1
- Must be idempotent (safe to run multiple times)
- Must generate report of changes

### 4.2 Validation Agent

**Agent Type**: `agent-code-quality`

**Post-Migration Checks**:
1. Run `ruff check` on migrated files
2. Run `basedpyright` on migrated files
3. Verify no `import logging` remains (except stdlib bridge)
4. Verify all `logger.` calls use keyword arguments

---

## Phase 5: Batch Migration Execution

### 5.1 Batch 1: Application Services (12 files)

**Agent Type**: `agent-python-executor`

**Files** (to be confirmed by discovery agent):
```
src/noteflow/application/services/
├── meeting_service.py
├── recovery_service.py
├── export_service.py
├── summarization_service.py
├── trigger_service.py
├── webhook_service.py
├── calendar_service.py
├── retention_service.py
├── ner_service.py
└── ...
```

**⚠️ Execution Strategy (PER-FILE, NOT PER-BATCH)**:
```bash
# For EACH file in the batch:

# 1. Migrate ONE file
python scripts/migrate_logging.py src/noteflow/application/services/meeting_service.py

# 2. IMMEDIATELY run quality check
make quality-py

# 3. If NEW violations introduced:
#    - FIX THEM NOW
#    - Re-run make quality-py
#    - Do NOT proceed until clean

# 4. Only then migrate next file
python scripts/migrate_logging.py src/noteflow/application/services/recovery_service.py
make quality-py
# ... repeat for each file

# 5. After ALL files in batch pass individually:
pytest tests/application/ -v
```

**PROHIBITED**: Running migration script on entire batch then checking quality once

### 5.2 Batch 2: Infrastructure (35 files)

**Agent Type**: `agent-python-executor`

**Subdirectories**:
- `audio/` - capture, writer, playback
- `asr/` - engine, segmenter
- `diarization/` - engine, session
- `summarization/` - providers, parsing
- `persistence/` - database, repositories, unit_of_work
- `triggers/` - calendar, audio, app
- `webhooks/` - executor
- `calendar/` - adapters, oauth
- `ner/` - engine
- `export/` - markdown, html, pdf
- `security/` - keystore
- `observability/` - otel

**⚠️ Same per-file workflow as Batch 1:**
```bash
# For EACH of the 35 files:
# 1. Migrate ONE file
# 2. make quality-py
# 3. Fix any NEW violations
# 4. Proceed only when clean
```

### 5.3 Batch 3: gRPC Layer (24 files)

**Agent Type**: `agent-python-executor`

**Components**:
- `server.py`, `service.py`, `client.py`
- `_mixins/` - all mixins
- `interceptors/` - identity interceptor
- `_client_mixins/` - client mixins

**⚠️ Same per-file workflow as Batch 1:**
```bash
# For EACH of the 24 files:
# 1. Migrate ONE file
# 2. make quality-py
# 3. Fix any NEW violations
# 4. Proceed only when clean
```

### Migration Abort Conditions

**STOP IMMEDIATELY if any of these occur:**

1. **Threshold modification detected** - Any change to `tests/quality/*.py` threshold values
2. **Cumulative violations > 5** - Too many unfixed violations accumulating
3. **Type errors without fix** - `basedpyright` errors not immediately addressed
4. **Baseline not captured** - Starting migration without `/tmp/quality_baseline.log`

**Recovery**: Revert all changes since last known-good state, re-capture baseline, restart

---

## Phase 6: Test Updates & Validation

### 6.1 Agent: Update Test Fixtures

**Agent Type**: `agent-testing-architect`

**Tasks**:
1. Create `tests/infrastructure/logging/conftest.py` with shared fixtures
2. Update any tests asserting on log output
3. Add integration tests for dual output

**Fixture Requirements** (per QUALITY_STANDARDS.md):
```python
@pytest.fixture
def logging_config() -> LoggingConfig:
    """Provide test logging configuration."""
    return LoggingConfig(
        level="DEBUG",
        enable_console=False,  # Suppress console in tests
        enable_log_buffer=True,
    )
```

### 6.2 Agent: Write Unit Tests

**Agent Type**: `agent-testing-architect`

**Test Files to Create**:
- `tests/infrastructure/logging/test_config.py`
- `tests/infrastructure/logging/test_processors.py`
- `tests/infrastructure/logging/test_handlers.py`

**Test Requirements** (per QUALITY_STANDARDS.md):
- No loops in tests
- No conditionals in tests
- Use `pytest.mark.parametrize` for multiple cases
- Use `pytest.param` with descriptive IDs
- All fixtures must have type hints

### 6.3 Quality Gate Execution

**Commands**:
```bash
# Run quality checks
pytest tests/quality/ -v

# Run new logging tests
pytest tests/infrastructure/logging/ -v

# Run full test suite
pytest -m "not integration" -v

# Type checking
basedpyright src/noteflow/infrastructure/logging/
```

---

## Phase 7: Documentation & Cleanup

### 7.1 Documentation Updates

| File | Change |
|------|--------|
| `CLAUDE.md` | Add logging configuration section |
| `docs/guides/logging.md` | Create usage guide (NEW) |
| `docs/triage.md` | Mark resolved issues |

### 7.2 Cleanup Tasks

**Agent Type**: `Explore`

**Verification Queries**:
1. "Confirm no `logging.basicConfig()` calls remain"
2. "Confirm no `logging.getLogger(__name__)` patterns remain"
3. "Confirm all files use `structlog.get_logger()`"

---

## Execution Order & Dependencies

```mermaid
graph TD
    P0[Phase 0: Pre-Flight] --> P1[Phase 1: Discovery]
    P1 --> P2[Phase 2: Infrastructure]
    P2 --> P3[Phase 3: Entry Points]
    P3 --> P4[Phase 4: Migration Script]
    P4 --> P5A[Phase 5.1: App Services]
    P5A --> P5B[Phase 5.2: Infrastructure]
    P5B --> P5C[Phase 5.3: gRPC]
    P5C --> P6[Phase 6: Tests]
    P6 --> P7[Phase 7: Docs]
```

**Parallelization Opportunities**:
- Phase 2 modules (config.py, processors.py, handlers.py) can be developed in parallel
- Batch migrations can be parallelized across different directories
- Test writing can happen in parallel with Phase 5 batches

---

## Agent Orchestration Protocol

### MANDATORY: Quality Gate After Every Edit

**Every agent MUST run this after each file modification:**

```bash
# IMMEDIATE - after every file edit
make quality-py

# If ANY failure:
# 1. FIX THE VIOLATION IMMEDIATELY
# 2. Do NOT proceed to next file
# 3. Do NOT claim "preexisting" without baseline proof
```

### Agent Workflow Pattern

```
┌─────────────────────────────────────────────────────────────┐
│  FOR EACH FILE MODIFICATION:                                │
│                                                             │
│  1. Read current file state                                 │
│  2. Make edit                                               │
│  3. Run: make quality-py                                    │
│  4. IF FAIL → Fix immediately, go to step 3                 │
│  5. IF PASS → Proceed to next edit                          │
│                                                             │
│  NEVER: Skip step 3-4                                       │
│  NEVER: Batch multiple edits before quality check           │
│  NEVER: Change threshold values in tests/quality/           │
└─────────────────────────────────────────────────────────────┘
```

### Discovery Phase
```bash
# Capture baseline FIRST
make quality-py 2>&1 | tee /tmp/quality_baseline.log

# Launch exploration agents in parallel
# Agent: Explore (thorough)
# Queries:
#   - "Find all files with 'import logging' in src/noteflow/"
#   - "Map logging patterns by file type"
#   - "Find silent error handlers returning None"
```

### Implementation Phase (Per-File Quality Gates)

```bash
# For EACH new file created:

# Step 1: Create file
# Step 2: IMMEDIATELY run quality check
make quality-py

# Step 3: If violations, fix before creating next file
# Step 4: Only proceed when clean

# Example sequence for config.py:
# - Write config.py
# - make quality-py  ← MUST PASS
# - Write processors.py
# - make quality-py  ← MUST PASS
# - Write handlers.py
# - make quality-py  ← MUST PASS
```

### Migration Phase (Per-File Quality Gates)

```bash
# For EACH migrated file:

# Step 1: Migrate single file
# Step 2: IMMEDIATELY run quality check
make quality-py

# Step 3: If new violations introduced, fix before next file
# Compare against baseline to identify NEW vs preexisting

# Example: Migrating meeting_service.py
# - Edit meeting_service.py (logging → structlog)
# - make quality-py
# - If new violations: FIX THEM
# - Only then proceed to next service file
```

### Continuous Validation Commands

```bash
# Run continuously during development
watch -n 30 'make quality-py'

# Or after each save (if using editor hooks)
# VSCode: tasks.json with "runOn": "save"
```

---

## Risk Mitigation

| Risk | Mitigation | Agent Responsibility |
|------|------------|---------------------|
| Migration breaks existing log parsing | Maintain JSON schema compatibility | `agent-code-quality` |
| Rich console conflicts | Test CLI integration early | `Explore` |
| OTEL context not propagating | Integration tests with mock tracer | `agent-testing-architect` |
| Performance regression | Benchmark before/after | `agent-feasibility` |

---

## Success Criteria

- [ ] All 71 files migrated to structlog
- [ ] Zero `logging.basicConfig()` calls remain
- [ ] Zero `logging.getLogger(__name__)` patterns remain
- [ ] Dual output working (Rich console + JSON)
- [ ] Context variables auto-injected
- [ ] OTEL trace/span IDs appear when tracing enabled
- [ ] LogBuffer receives structured events
- [ ] All quality checks pass (`pytest tests/quality/`)
- [ ] All type checks pass (`basedpyright`)
- [ ] Documentation updated

---

## Appendix A: Quality Compliance Checklist

Per `docs/sprints/QUALITY_STANDARDS.md`:

### 🚫 PROHIBITED ACTIONS (Violation = Immediate Rollback)

- [ ] **NEVER** modify threshold values in `tests/quality/*.py`
- [ ] **NEVER** add `# type: ignore` without explicit user approval
- [ ] **NEVER** use `Any` type
- [ ] **NEVER** skip `make quality-py` after a file edit
- [ ] **NEVER** blame "preexisting issues" without baseline comparison

### ALL CODE Requirements (Not Just Tests)

These apply to `config.py`, `processors.py`, `handlers.py`, AND all migrated files:

| Requirement | Check Command | Applies To |
|-------------|---------------|------------|
| No `# type: ignore` | `basedpyright` | ALL `.py` files |
| No `Any` types | `basedpyright` | ALL `.py` files |
| Union syntax `X \| None` | `ruff check` | ALL `.py` files |
| Module < 500 lines | `pytest tests/quality/` | ALL modules |
| Function < 75 lines | `pytest tests/quality/` | ALL functions |
| Complexity < 15 | `pytest tests/quality/` | ALL functions |
| Parameters ≤ 7 | `pytest tests/quality/` | ALL functions |
| No magic numbers | `pytest tests/quality/` | ALL code |
| No hardcoded paths | `pytest tests/quality/` | ALL code |
| Docstrings imperative | Manual review | ALL public APIs |

### Test-Specific Requirements

These apply ONLY to test files in `tests/`:

- [ ] No loops around assertions
- [ ] No conditionals around assertions
- [ ] `pytest.mark.parametrize` for multiple cases
- [ ] `pytest.raises` with `match=` parameter
- [ ] All fixtures have type hints
- [ ] Fixtures in conftest.py (not duplicated)

### Per-Edit Verification Workflow

```bash
# After EVERY edit (not batched):
make quality-py

# Expected output for clean code:
# === Ruff (Python Lint) ===
# All checks passed!
# === Basedpyright ===
# 0 errors, 0 warnings, 0 informations
# === Python Test Quality ===
# XX passed in X.XXs

# If ANY failure: FIX IMMEDIATELY before next edit
```

---

## Appendix B: Threshold Values (READ-ONLY Reference)

**⚠️ These values are READ-ONLY. Agents MUST NOT modify them.**

From `tests/quality/test_code_smells.py`:
```python
# DO NOT CHANGE THESE VALUES
MODULE_SOFT_LIMIT = 500
MODULE_HARD_LIMIT = 750
FUNCTION_LINE_LIMIT = 75
COMPLEXITY_LIMIT = 15
PARAMETER_LIMIT = 7
```

From `tests/quality/test_magic_values.py`:
```python
# DO NOT CHANGE THESE VALUES
MAX_MAGIC_NUMBERS = 10
MAX_REPEATED_STRINGS = 30
MAX_HARDCODED_PATHS = 0
```

If your code exceeds these limits, **refactor the code**, not the thresholds.
