# Sprint 4 gRPC Layer Validation Report

## Executive Summary

Sprint 4's proposed approach to implement `EntitiesMixin` for Named Entity Recognition aligns perfectly with NoteFlow's established gRPC architecture. The codebase already has:
- Proto definitions for entity extraction (`ExtractEntitiesRequest/Response` lines 593-630)
- ORM model (`NamedEntityModel`) and migration
- No NER service yet (but pattern is well-established)

**Key Finding**: The mixin pattern is mature and battle-tested across 7 existing mixins. Adding `EntitiesMixin` follows proven conventions with minimal surprises.

---

## 1. EXISTING MIXINS ARCHITECTURE

### Mixins Implemented
Located in `src/noteflow/grpc/_mixins/`:
1. **StreamingMixin** - Audio bidirectional streaming + ASR
2. **DiarizationMixin** - Speaker identification 
3. **DiarizationJobMixin** - Background diarization job tracking
4. **MeetingMixin** - Meeting CRUD (create, get, list, delete, stop)
5. **SummarizationMixin** - Summary generation
6. **AnnotationMixin** - Annotation management (CRUD)
7. **ExportMixin** - Transcript export (Markdown/HTML/PDF)

All exported from `src/noteflow/grpc/_mixins/__init__.py`.

### NoteFlowServicer Composition
```python
class NoteFlowServicer(
    StreamingMixin,
    DiarizationMixin,
    DiarizationJobMixin,
    MeetingMixin,
    SummarizationMixin,
    AnnotationMixin,
    ExportMixin,
    noteflow_pb2_grpc.NoteFlowServiceServicer,
):
```
**Pattern**: Multiple inheritance ordering matters—last mixin checked wins on method conflicts.

---

## 2. MIXIN STRUCTURE PATTERN

### Standard Mixin Template (from AnnotationMixin)

**Class Definition**:
```python
class AnnotationMixin:
    """Docstring explaining what it does.
    
    Requires host to implement ServicerHost protocol.
    Works with both database and memory backends via RepositoryProvider.
    """
```

**Key Characteristics**:
- No `__init__` required (state lives in NoteFlowServicer)
- Type hint `self: ServicerHost` to access host attributes
- All service dependencies injected through protocol
- Works with both database and in-memory backends

### Method Signature Pattern
```python
async def RPCMethod(
    self: ServicerHost,
    request: noteflow_pb2.SomeRequest,
    context: grpc.aio.ServicerContext,
) -> noteflow_pb2.SomeResponse:
    """RPC implementation."""
    # Get meeting ID and validate
    meeting_id = await parse_meeting_id_or_abort(request.meeting_id, context)
    
    # Use unified repository provider
    async with self._create_repository_provider() as repo:
        # Load data
        item = await repo.entities.get(item_id)
        if item is None:
            await abort_not_found(context, "Entity", request.id)
        
        # Process
        result = await repo.entities.update(item)
        await repo.commit()
        return converted_to_proto(result)
```

---

## 3. SERVICEHOST PROTOCOL (src/noteflow/grpc/_mixins/protocols.py)

The `ServicerHost` protocol defines the contract mixins expect from NoteFlowServicer:

### Configuration Attributes
- `_session_factory: async_sessionmaker[AsyncSession] | None` 
- `_memory_store: MeetingStore | None`
- `_meetings_dir: Path`
- `_crypto: AesGcmCryptoBox`

### Service Engines
- `_asr_engine: FasterWhisperEngine | None`
- `_diarization_engine: DiarizationEngine | None`
- `_summarization_service: object | None` ← **Generic type for flexibility**

**For EntitiesMixin**: Would need to add:
```python
_ner_service: NerService | None  # Type will be added to protocol
```

### Key Methods Mixins Use
```python
def _use_database(self) -> bool
def _get_memory_store(self) -> MeetingStore
def _create_uow(self) -> SqlAlchemyUnitOfWork
def _create_repository_provider(self) -> UnitOfWork  # Handles both DB and memory
def _next_segment_id(self, meeting_id: str, fallback: int = 0) -> int
def _init_streaming_state(self, meeting_id: str, next_segment_id: int) -> None
def _cleanup_streaming_state(self, meeting_id: str) -> None
def _ensure_meeting_dek(self, meeting: Meeting) -> tuple[bytes, bytes, bool]
def _start_meeting_if_needed(self, meeting: Meeting) -> tuple[bool, str | None]
def _open_meeting_audio_writer(...) -> None
def _close_audio_writer(self, meeting_id: str) -> None
```

---

## 4. PROTO DEFINITIONS (noteflow.proto, lines 45-46, 593-630)

**Already defined for Sprint 4**:

### Service RPC
```protobuf
rpc ExtractEntities(ExtractEntitiesRequest) returns (ExtractEntitiesResponse);
```

### Request Message
```protobuf
message ExtractEntitiesRequest {
  string meeting_id = 1;
  bool force_refresh = 2;
}
```

### Response Messages
```protobuf
message ExtractedEntity {
  string id = 1;
  string text = 2;
  string category = 3;  // person, company, product, technical, acronym, location, date, other
  repeated int32 segment_ids = 4;
  float confidence = 5;
  bool is_pinned = 6;
}

message ExtractEntitiesResponse {
  repeated ExtractedEntity entities = 1;
  int32 total_count = 2;
  bool cached = 3;
}
```

**Status**: Proto defs complete, no regeneration needed unless modifying messages.

---

## 5. CONVERTER PATTERN (src/noteflow/grpc/_mixins/converters.py)

### Pattern Overview
All converters are **standalone functions** in a single module, not class methods:

```python
def parse_meeting_id_or_abort(meeting_id_str: str, context: object) -> MeetingId
def parse_annotation_id(annotation_id_str: str) -> AnnotationId
def word_to_proto(word: WordTiming) -> noteflow_pb2.WordTiming
def segment_to_final_segment_proto(segment: Segment) -> noteflow_pb2.FinalSegment
def meeting_to_proto(meeting: Meeting, ...) -> noteflow_pb2.Meeting
def summary_to_proto(summary: Summary) -> noteflow_pb2.Summary
def annotation_to_proto(annotation: Annotation) -> noteflow_pb2.Annotation
def create_segment_from_asr(meeting_id: MeetingId, ...) -> Segment
def proto_to_export_format(proto_format: int) -> ExportFormat
```

### For EntitiesMixin, Need to Add:
```python
def entity_to_proto(entity: Entity) -> noteflow_pb2.ExtractedEntity
def proto_to_entity(request: noteflow_pb2.ExtractEntitiesRequest, ...) -> Entity | None
def proto_entity_list_to_domain(protos: list[...]) -> list[Entity]
```

### Key Principles
1. **Stateless functions** - no dependencies on mixins
2. **Handle both directions** - proto→domain and domain→proto
3. **Consolidate patterns** - e.g., `parse_meeting_id_or_abort` replaces 11+ duplicated patterns
4. **Use domain converters** - import from infrastructure when needed (e.g., `AsrConverter`)

---

## 6. DEPENDENCY INJECTION PATTERN

### How Services Get Injected

**Example: SummarizationMixin**
```python
class SummarizationMixin:
    _summarization_service: SummarizationService | None
    
    async def GenerateSummary(self: ServicerHost, ...):
        # Access through self
        summary = await self._summarize_or_placeholder(meeting_id, segments, style_prompt)
```

**Pattern for NerService**:
1. Add field to NoteFlowServicer `__init__` parameter:
```python
def __init__(
    self,
    ...,
    ner_service: NerService | None = None,
) -> None:
    self._ner_service = ner_service
```

2. Add to ServicerHost protocol:
```python
class ServicerHost(Protocol):
    _ner_service: NerService | None
```

3. Use in EntitiesMixin:
```python
async def ExtractEntities(self: ServicerHost, request, context):
    # Service may be None; handle gracefully or abort
    if self._ner_service is None:
        await abort_service_unavailable(context, "NER")
```

### Repository Provider Pattern
All mixins use `self._create_repository_provider()` to abstract DB vs. in-memory:
```python
async with self._create_repository_provider() as repo:
    entities = await repo.entities.get_by_meeting(meeting_id)  # Need to implement
```

---

## 7. HOW TO ADD ENTITESMIXIN

### Step-by-Step Integration Points

**A. Create `src/noteflow/grpc/_mixins/entities.py`**
```python
class EntitiesMixin:
    """Mixin providing named entity extraction functionality."""
    
    async def ExtractEntities(
        self: ServicerHost,
        request: noteflow_pb2.ExtractEntitiesRequest,
        context: grpc.aio.ServicerContext,
    ) -> noteflow_pb2.ExtractEntitiesResponse:
        # Implementation
```

**B. Update `src/noteflow/grpc/_mixins/__init__.py`**
```python
from .entities import EntitiesMixin
__all__ = [..., "EntitiesMixin"]
```

**C. Update `src/noteflow/grpc/service.py`**
```python
from ._mixins import (
    ...,
    EntitiesMixin,  # Add
)

class NoteFlowServicer(
    ...,
    EntitiesMixin,  # Add before noteflow_pb2_grpc.NoteFlowServiceServicer
    noteflow_pb2_grpc.NoteFlowServiceServicer,
):
    def __init__(
        self,
        ...,
        ner_service: NerService | None = None,
    ):
        ...,
        self._ner_service = ner_service
```

**D. Update `src/noteflow/grpc/_mixins/protocols.py`**
```python
class ServicerHost(Protocol):
    ...
    _ner_service: object | None  # or import NerService if in TYPE_CHECKING
```

**E. Update `src/noteflow/grpc/_mixins/converters.py`**
Add entity conversion functions:
```python
def entity_to_proto(entity: Entity) -> noteflow_pb2.ExtractedEntity:
    """Convert domain Entity to protobuf."""
    return noteflow_pb2.ExtractedEntity(
        id=str(entity.id),
        text=entity.text,
        category=entity.category,
        segment_ids=entity.segment_ids,
        confidence=entity.confidence,
        is_pinned=entity.is_pinned,
    )
```

---

## 8. REPOSITORY INTEGRATION

### Current Repositories Available
Located in `src/noteflow/infrastructure/persistence/repositories/`:
- `annotation_repo.py` → `SqlAlchemyAnnotationRepository`
- `diarization_job_repo.py` → `SqlAlchemyDiarizationJobRepository`
- `meeting_repo.py` → `SqlAlchemyMeetingRepository`
- `preferences_repo.py` → `SqlAlchemyPreferencesRepository`
- `segment_repo.py` → `SqlAlchemySegmentRepository`
- `summary_repo.py` → `SqlAlchemySummaryRepository`

**Missing**: `entity_repo.py` (Sprint 4 task)

### UnitOfWork Integration
Current `SqlAlchemyUnitOfWork` (line 87-127):
```python
@property
def annotations(self) -> SqlAlchemyAnnotationRepository: ...
@property
def meetings(self) -> SqlAlchemyMeetingRepository: ...
```

**Needed for Sprint 4**:
```python
@property
def entities(self) -> SqlAlchemyEntityRepository: ...

# And in __aenter__/__aexit__:
self._entities_repo = SqlAlchemyEntityRepository(self._session)
```

### Domain vs. In-Memory
NoteFlowServicer uses `_create_repository_provider()` which returns:
- `SqlAlchemyUnitOfWork` if database configured
- `MemoryUnitOfWork` if in-memory fallback

EntitiesMixin must handle both seamlessly (like AnnotationMixin does).

---

## 9. ERROR HANDLING PATTERN

All mixins use helper functions from `src/noteflow/grpc/_mixins/errors.py`:

```python
async def abort_not_found(context, entity_type, entity_id) -> None
async def abort_invalid_argument(context, message) -> None
async def abort_database_required(context, feature_name) -> None
```

**Pattern in EntitiesMixin**:
```python
async def ExtractEntities(self: ServicerHost, request, context):
    meeting_id = await parse_meeting_id_or_abort(request.meeting_id, context)
    async with self._create_repository_provider() as repo:
        meeting = await repo.meetings.get(meeting_id)
        if meeting is None:
            await abort_not_found(context, "Meeting", request.meeting_id)
        
        entities = await repo.entities.get_by_meeting(meeting_id)
        # Handle not found, validation errors, etc.
```

---

## 10. CACHING & REFRESH PATTERN

From proto: `force_refresh` flag in `ExtractEntitiesRequest`.

**SummarizationMixin pattern** (similar):
```python
existing = await repo.summaries.get_by_meeting(meeting_id)
if existing and not request.force_regenerate:
    return summary_to_proto(existing)

# Otherwise, generate fresh
```

**EntitiesMixin should follow**:
```python
async def ExtractEntities(self: ServicerHost, request, context):
    meeting_id = await parse_meeting_id_or_abort(request.meeting_id, context)
    
    async with self._create_repository_provider() as repo:
        meeting = await repo.meetings.get(meeting_id)
        if meeting is None:
            await abort_not_found(context, "Meeting", request.meeting_id)
        
        # Check cache first
        cached_entities = await repo.entities.get_by_meeting(meeting_id)
        if cached_entities and not request.force_refresh:
            return noteflow_pb2.ExtractEntitiesResponse(
                entities=[entity_to_proto(e) for e in cached_entities],
                total_count=len(cached_entities),
                cached=True,
            )
    
    # Load segments, run NER service (outside repo context if slow)
    segments = await repo.segments.get_by_meeting(meeting_id)
    entities = await self._extract_entities(meeting_id, segments)
    
    # Persist in fresh transaction
    async with self._create_repository_provider() as repo:
        saved = await repo.entities.save_batch(entities)
        await repo.commit()
    
    return noteflow_pb2.ExtractEntitiesResponse(
        entities=[entity_to_proto(e) for e in saved],
        total_count=len(saved),
        cached=False,
    )
```

---

## 11. KEY INSIGHTS & RISKS

### Strengths of Current Pattern
✅ **Modular**: Each mixin is self-contained, easy to test
✅ **Composable**: NoteFlowServicer inherits from all mixins without complexity
✅ **Database-agnostic**: `_create_repository_provider()` abstracts DB vs. memory
✅ **Proven**: 7 existing mixins provide battle-tested templates
✅ **Asyncio-native**: All methods are async, no blocking calls
✅ **Type-safe**: ServicerHost protocol ensures mixin contracts

### Potential Issues Sprint 4 Should Watch For

1. **Missing Domain Entity**: No `Entity` class in `src/noteflow/domain/entities/` yet
   - Proto has `ExtractedEntity` (protobuf message)
   - ORM has `NamedEntityModel` (persistence)
   - Need to bridge: `Entity` domain class

2. **Missing Repository**: `entity_repo.py` doesn't exist
   - Must implement CRUD methods used in mixin
   - Must handle both database and in-memory backends

3. **NER Service Not Started**: Service layer (`NerService`) needs implementation
   - How to get segments for extraction?
   - What model/provider does NER use?
   - Is it async? Does it block?

4. **Proto Regeneration**: After any proto changes:
```bash
python -m grpc_tools.protoc -I src/noteflow/grpc/proto \
  --python_out=src/noteflow/grpc/proto \
  --grpc_python_out=src/noteflow/grpc/proto \
  src/noteflow/grpc/proto/noteflow.proto
```
Proto files are already generated, so check if they match lines 45-46 and 593-630.

5. **Memory Store Fallback**: Annotation repository supports in-memory; entity repo must too
   - See `src/noteflow/infrastructure/persistence/memory.py` for fallback pattern

---

## 12. PROTO SYNC CHECKLIST FOR SPRINT 4

- [x] `ExtractEntities` RPC defined (line 46)
- [x] `ExtractEntitiesRequest` message (lines 593-599)
- [x] `ExtractedEntity` message (lines 601-619)
- [x] `ExtractEntitiesResponse` message (lines 621-630)
- [ ] Regenerate Python stubs if proto modified
- [ ] Verify `noteflow_pb2.ExtractedEntity` exists in generated code
- [ ] Verify `noteflow_pb2_grpc.NoteFlowService` includes `ExtractEntities` method stub

---

## Summary Table

| Component | Status | File | Notes |
|-----------|--------|------|-------|
| **Mixin Class** | READY | `_mixins/entities.py` (new) | Follow AnnotationMixin pattern |
| **Service RPC** | ✅ EXISTS | `proto/noteflow.proto:46` | Already defined |
| **Request/Response** | ✅ EXISTS | `proto/noteflow.proto:593-630` | Complete messages |
| **ORM Model** | ✅ EXISTS | `persistence/models/entities/named_entity.py` | `NamedEntityModel` ready |
| **Migration** | ✅ EXISTS | `persistence/migrations/.../add_named_entities_table.py` | Table created |
| **Domain Entity** | ❌ MISSING | `domain/entities/` | Need `Entity` class |
| **Repository** | ❌ MISSING | `persistence/repositories/entity_repo.py` | Need CRUD implementation |
| **NER Service** | ❌ MISSING | `application/services/` | Need extraction logic |
| **Converter Functions** | PARTIAL | `_mixins/converters.py` | Need `entity_to_proto()` |
| **UnitOfWork** | READY | `persistence/unit_of_work.py` | Just add `entities` property |
| **ServicerHost** | READY | `_mixins/protocols.py` | Just add `_ner_service` field |

---

## VALIDATION CONCLUSION

**Status**: ✅ **SPRINT 4 APPROACH IS SOUND**

The proposed mixin-based architecture for EntitiesMixin:
1. Follows established patterns from 7 existing mixins
2. Leverages existing proto definitions
3. Has ORM and migration infrastructure ready
4. Fits cleanly into NoteFlowServicer composition

**Next Steps**:
1. Create domain `Entity` class (mirrors proto `ExtractedEntity`)
2. Implement `SqlAlchemyEntityRepository` with CRUD operations
3. Create `NerService` application service
4. Implement `EntitiesMixin.ExtractEntities()` RPC handler
5. Add converter functions `entity_to_proto()` etc.
6. Add `entities` property to `SqlAlchemyUnitOfWork`
7. Add `_ner_service` to `ServicerHost` protocol

No architectural surprises or deviations from established patterns.
