# Sprint: Centralized Logging Infrastructure

| Attribute | Value |
|-----------|-------|
| **Size** | L (Large) |
| **Phase** | Infrastructure |
| **Prerequisites** | None |
| **Owner** | TBD |

---

## Open Issues

| Issue | Blocking? | Resolution Path |
|-------|-----------|-----------------|
| LogBuffer integration | No | Adapt LogBuffer to consume structlog events |
| CLI modules use Rich Console | No | Ensure no conflicts with structlog Rich renderer |

---

## Validation Status

| Component | Exists | Notes |
|-----------|--------|-------|
| `LogBuffer` / `LogBufferHandler` | Yes | Needs adaptation for structlog |
| `get_logging_context()` | Yes | Context vars for request_id, user_id, workspace_id |
| OTEL trace context capture | Yes | `_get_current_trace_context()` in log_buffer.py |
| Rich dependency | Yes | `rich>=14.2.0` in pyproject.toml |
| structlog dependency | No | Must add to pyproject.toml |

---

## Objective

Centralize NoteFlow's logging infrastructure using **structlog** with dual output: **Rich console rendering** for development and **JSON output** for observability/OTEL integration. Migrate all 71 existing files from stdlib `logging` to structlog while preserving existing context propagation and OTEL trace correlation.

---

## Key Decisions

| Decision | Choice | Rationale |
|----------|--------|-----------|
| **Library** | structlog + Rich | Structured logging with context binding; Rich console renderer included |
| **Output Strategy** | Dual simultaneous | JSON to file/collector AND Rich to console always |
| **Context Handling** | Auto-inject + override | Leverage existing context vars; allow per-call extras |
| **Migration Scope** | Full migration | Convert all 71 files to `structlog.get_logger()` |
| **stdlib Bridge** | Yes | Use `structlog.stdlib` for seamless integration |

---

## What Already Exists

### Reusable Assets

| Asset | Location | Reuse Strategy |
|-------|----------|----------------|
| Context variables | `infrastructure/logging/structured.py` | Inject via structlog processor |
| LogEntry dataclass | `infrastructure/logging/log_buffer.py` | Adapt as structlog processor output |
| LogBuffer | `infrastructure/logging/log_buffer.py` | Create structlog processor that feeds LogBuffer |
| OTEL trace extraction | `infrastructure/logging/log_buffer.py:132-156` | Convert to structlog processor |
| Observability setup | `infrastructure/observability/otel.py` | Integrate with structlog OTEL processor |

### Current Logging Patterns

```python
# Pattern 1: Module-level logger (71 files)
import logging
logger = logging.getLogger(__name__)
logger.info("message", extra={...})

# Pattern 2: %-style formatting (widespread)
logger.warning("Failed to process %s: %s", item_id, error)

# Pattern 3: Exception logging
logger.exception("Operation failed")

# Pattern 4: CLI modules (2 files)
logging.basicConfig(level=logging.INFO)
```

---

## Scope

### Task Breakdown

| Task | Effort | Description |
|------|--------|-------------|
| **T1: Core Configuration** | M | Create `configure_logging()` with dual output |
| **T2: Structlog Processors** | M | Build processor chain (context, OTEL, timestamps) |
| **T3: Rich Renderer Integration** | S | Configure structlog's ConsoleRenderer with Rich |
| **T4: JSON Renderer** | S | Configure JSONRenderer for observability |
| **T5: LogBuffer Processor** | M | Create processor that feeds existing LogBuffer |
| **T6: Context Injection Processor** | S | Processor using `get_logging_context()` |
| **T7: OTEL Span Processor** | S | Extract trace_id/span_id from current span |
| **T8: Entry Point Updates** | S | Update `grpc/server.py`, CLI entry points |
| **T9: Migration Script** | M | AST-based migration of 71 files |
| **T10: File Migration (Batch 1)** | L | Migrate application/services (12 files) |
| **T11: File Migration (Batch 2)** | L | Migrate infrastructure/* (35 files) |
| **T12: File Migration (Batch 3)** | L | Migrate grpc/* (24 files) |
| **T13: Test Updates** | M | Update test fixtures and assertions |
| **T14: Documentation** | S | Update CLAUDE.md and add logging guide |

**Total Effort:** XL (spans multiple sessions)

---

## Architecture

### Processor Chain

```
┌─────────────────────────────────────────────────────────────────────────┐
│                         Structlog Processor Chain                        │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│  1. filter_by_level          ─► Skip if level too low                   │
│  2. add_logger_name          ─► Add logger name to event                │
│  3. add_log_level            ─► Add level string                        │
│  4. PositionalArgumentsFormatter ─► Handle %-style formatting           │
│  5. TimeStamper(fmt="iso")   ─► ISO 8601 timestamp                      │
│  6. add_noteflow_context     ─► request_id, user_id, workspace_id       │
│  7. add_otel_trace_context   ─► trace_id, span_id, parent_span_id       │
│  8. CallsiteParameterAdder   ─► filename, func_name, lineno             │
│  9. StackInfoRenderer        ─► Stack traces if requested               │
│  10. format_exc_info         ─► Exception formatting                    │
│  11. UnicodeDecoder          ─► Decode bytes to str                     │
│                                                                          │
│  ┌─────────────────────┐    ┌─────────────────────┐                     │
│  │   Rich Console      │    │   JSON File/OTLP    │                     │
│  │   (dev.ConsoleRenderer)│ │   (JSONRenderer)    │                     │
│  └──────────┬──────────┘    └──────────┬──────────┘                     │
│             │                          │                                 │
│             ▼                          ▼                                 │
│        StreamHandler              FileHandler / LogBuffer               │
│        (stderr, TTY)              (noteflow.log / in-memory)            │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘
```

### Module Structure

```
infrastructure/logging/
├── __init__.py           # Public API exports
├── config.py             # NEW: configure_logging(), LoggingConfig
├── processors.py         # NEW: Custom processors (context, OTEL, LogBuffer)
├── handlers.py           # NEW: Dual-output handler configuration
├── structured.py         # KEEP: Context variables (minimal changes)
└── log_buffer.py         # ADAPT: LogBuffer processor integration
```

---

## Domain Model

### LoggingConfig (New)

```python
@dataclass(frozen=True, slots=True)
class LoggingConfig:
    """Configuration for centralized logging."""

    level: str = "INFO"
    json_file: Path | None = None  # None = no file output
    enable_console: bool = True
    enable_json_console: bool = False  # Force JSON even on TTY
    enable_log_buffer: bool = True
    enable_otel_context: bool = True
    enable_noteflow_context: bool = True
    console_colors: bool = True  # Auto-detect TTY if None
```

### Custom Processors (New)

```python
def add_noteflow_context(
    logger: WrappedLogger,
    method_name: str,
    event_dict: EventDict,
) -> EventDict:
    """Inject request_id, user_id, workspace_id from context vars."""
    ctx = get_logging_context()
    for key, value in ctx.items():
        if value is not None and key not in event_dict:
            event_dict[key] = value
    return event_dict


def add_otel_trace_context(
    logger: WrappedLogger,
    method_name: str,
    event_dict: EventDict,
) -> EventDict:
    """Inject OpenTelemetry trace/span IDs if available."""
    try:
        from opentelemetry import trace

        span = trace.get_current_span()
        if span.is_recording():
            ctx = span.get_span_context()
            event_dict["trace_id"] = format(ctx.trace_id, "032x")
            event_dict["span_id"] = format(ctx.span_id, "016x")
            parent = getattr(span, "parent", None)
            if parent:
                event_dict["parent_span_id"] = format(parent.span_id, "016x")
    except ImportError:
        pass
    return event_dict


def log_buffer_processor(
    logger: WrappedLogger,
    method_name: str,
    event_dict: EventDict,
) -> EventDict:
    """Feed structured event to LogBuffer for UI streaming."""
    buffer = get_log_buffer()
    buffer.append(
        LogEntry(
            timestamp=event_dict.get("timestamp", datetime.now(UTC)),
            level=event_dict.get("level", "info"),
            source=event_dict.get("logger", ""),
            message=event_dict.get("event", ""),
            details={k: str(v) for k, v in event_dict.items()
                     if k not in ("timestamp", "level", "logger", "event")},
            trace_id=event_dict.get("trace_id"),
            span_id=event_dict.get("span_id"),
        )
    )
    return event_dict
```

---

## Configuration API

### Primary Entry Point

```python
# infrastructure/logging/config.py

def configure_logging(
    config: LoggingConfig | None = None,
    *,
    level: str = "INFO",
    json_file: Path | None = None,
) -> None:
    """Configure centralized logging with dual output.

    Call once at application startup (e.g., in grpc/server.py main()).

    Args:
        config: Full configuration object, or use keyword args.
        level: Log level (DEBUG, INFO, WARNING, ERROR).
        json_file: Optional path for JSON log file.
    """
    if config is None:
        config = LoggingConfig(level=level, json_file=json_file)

    shared_processors = _build_processor_chain(config)

    # Configure structlog
    structlog.configure(
        processors=shared_processors + [
            structlog.stdlib.ProcessorFormatter.wrap_for_formatter,
        ],
        wrapper_class=structlog.stdlib.BoundLogger,
        logger_factory=structlog.stdlib.LoggerFactory(),
        cache_logger_on_first_use=True,
    )

    # Configure stdlib logging handlers
    _configure_handlers(config, shared_processors)
```

### Usage After Migration

```python
# Before (current)
import logging
logger = logging.getLogger(__name__)
logger.info("Processing meeting %s", meeting_id)

# After (migrated)
import structlog
logger = structlog.get_logger()
logger.info("processing_meeting", meeting_id=meeting_id)

# Or with bound context
logger = structlog.get_logger().bind(meeting_id=meeting_id)
logger.info("processing_started")
logger.info("processing_completed", segments=42)
```

---

## Migration Strategy

### Phase 1: Infrastructure (T1-T8)

1. Add `structlog>=24.0` to `pyproject.toml`
2. Create `infrastructure/logging/config.py` with `configure_logging()`
3. Create `infrastructure/logging/processors.py` with custom processors
4. Create `infrastructure/logging/handlers.py` for handler setup
5. Update entry points to call `configure_logging()`

### Phase 2: Automated Migration (T9)

Create AST-based migration script:

```python
# scripts/migrate_logging.py

"""Migrate stdlib logging to structlog.

Transforms:
  import logging
  logger = logging.getLogger(__name__)
  logger.info("message %s", arg)

To:
  import structlog
  logger = structlog.get_logger()
  logger.info("message", arg=arg)
"""
```

### Phase 3: Batch Migration (T10-T12)

| Batch | Files | Priority | Notes |
|-------|-------|----------|-------|
| **Batch 1** | `application/services/*` (12) | High | Core business logic |
| **Batch 2** | `infrastructure/*` (35) | Medium | Infrastructure adapters |
| **Batch 3** | `grpc/*` (24) | High | API layer, interceptors |

### Rollback Strategy

- Keep stdlib logging configured as structlog backend
- If issues arise, revert `configure_logging()` to stdlib-only mode
- Migration is reversible via git; no database changes

---

## Deliverables

### New Files

- [ ] `src/noteflow/infrastructure/logging/config.py`
- [ ] `src/noteflow/infrastructure/logging/processors.py`
- [ ] `src/noteflow/infrastructure/logging/handlers.py`
- [ ] `scripts/migrate_logging.py`
- [ ] `docs/guides/logging.md`

### Modified Files

- [ ] `pyproject.toml` — add structlog dependency
- [ ] `src/noteflow/infrastructure/logging/__init__.py` — export new API
- [ ] `src/noteflow/infrastructure/logging/log_buffer.py` — adapt for structlog
- [ ] `src/noteflow/grpc/server.py` — call `configure_logging()`
- [ ] `src/noteflow/cli/retention.py` — remove `basicConfig`, use structlog
- [ ] `src/noteflow/cli/models.py` — remove `basicConfig`, use structlog
- [ ] 71 files with `logging.getLogger()` — migrate to structlog

### Tests

- [ ] `tests/infrastructure/logging/test_config.py`
- [ ] `tests/infrastructure/logging/test_processors.py`
- [ ] `tests/infrastructure/logging/test_handlers.py`
- [ ] Update existing tests that assert on log output

---

## Test Strategy

### Unit Tests

```python
# tests/infrastructure/logging/test_processors.py

@pytest.fixture
def mock_context_vars(monkeypatch):
    """Set up context variables for testing."""
    monkeypatch.setattr("noteflow.infrastructure.logging.structured.request_id_var",
                        ContextVar("request_id", default="test-req-123"))
    # ...

def test_add_noteflow_context_injects_request_id(mock_context_vars):
    """Verify context vars are injected into event dict."""
    event_dict = {"event": "test"}
    result = add_noteflow_context(None, "info", event_dict)
    assert result["request_id"] == "test-req-123"

def test_add_otel_trace_context_graceful_without_otel():
    """Verify processor works when OpenTelemetry not installed."""
    event_dict = {"event": "test"}
    result = add_otel_trace_context(None, "info", event_dict)
    assert "trace_id" not in result  # Graceful degradation

@pytest.mark.parametrize("level,expected", [
    ("DEBUG", True),
    ("INFO", True),
    ("WARNING", True),
    ("ERROR", True),
])
def test_configure_logging_accepts_all_levels(level, expected):
    """Verify all log levels are accepted."""
    config = LoggingConfig(level=level)
    configure_logging(config)
    # Assert no exception raised
```

### Integration Tests

```python
# tests/infrastructure/logging/test_integration.py

@pytest.mark.integration
def test_dual_output_produces_both_formats(tmp_path, capsys):
    """Verify console and JSON outputs are produced simultaneously."""
    json_file = tmp_path / "test.log"
    configure_logging(LoggingConfig(
        level="INFO",
        json_file=json_file,
        enable_console=True,
    ))

    logger = structlog.get_logger("test")
    logger.info("test_event", key="value")

    # Verify console output (Rich formatted)
    captured = capsys.readouterr()
    assert "test_event" in captured.err

    # Verify JSON file output
    with open(json_file) as f:
        log_line = json.loads(f.readline())
    assert log_line["event"] == "test_event"
    assert log_line["key"] == "value"
```

---

## Quality Gates

### Exit Criteria

- [ ] All 71 files migrated to structlog
- [ ] Dual output working (Rich console + JSON)
- [ ] Context variables auto-injected (request_id, user_id, workspace_id)
- [ ] OTEL trace/span IDs appear in logs when tracing enabled
- [ ] LogBuffer receives structured events for UI streaming
- [ ] No `logging.basicConfig()` calls remain
- [ ] All tests pass
- [ ] `ruff check` and `basedpyright` pass
- [ ] Documentation updated

### Performance Requirements

- Log emission overhead < 10μs per call
- No blocking I/O in hot paths (async file writes)
- Memory-bounded LogBuffer (existing 1000-entry limit)

---

## Dependencies

### New Dependencies

```toml
# pyproject.toml
[project]
dependencies = [
    # ... existing ...
    "structlog>=24.0",
]
```

### Compatibility Notes

- structlog 24.0+ required for `ProcessorFormatter` improvements
- Rich 14.2.0 already installed (compatible)
- OpenTelemetry integration optional (graceful degradation)

---

## Risks & Mitigations

| Risk | Impact | Mitigation |
|------|--------|------------|
| Migration breaks existing log parsing | Medium | Maintain JSON schema compatibility |
| Performance regression | Low | Benchmark before/after; structlog is fast |
| Rich console conflicts with existing CLI usage | Low | CLI modules already use Rich; test integration |
| OTEL context not propagating | Medium | Integration tests with mock tracer |

---

## References

- [structlog Documentation](https://www.structlog.org/)
- [structlog + Rich Integration](https://www.structlog.org/en/stable/console-output.html)
- [structlog + OTEL](https://www.structlog.org/en/stable/frameworks.html#opentelemetry)
- [Existing LogBuffer Implementation](../src/noteflow/infrastructure/logging/log_buffer.py)
