# Sprint 20: Artifacts v2 + RAG Migration

> **Size**: XL | **Owner**: Backend + Client | **Prerequisites**: Sprint 19 (shared ports)
> **Phase**: 5 - Platform Evolution
> **Structure**: Split into Sprint 20a (RAG Core) and Sprint 20b (Connectors + Q&A)

---

## Open Issues & Prerequisites

> ⚠️ **Review Date**: 2025-12-31 — Refined with phased approach

### Blocking Issues

| ID | Issue | Status | Resolution |
|----|-------|--------|------------|
| **B1** | Sprint 19 defines EmbeddingProvider/VectorStore ports | ⏳ In Progress | Parallel development: Sprint 19 defines ports, Sprint 20 implements Qdrant |
| **B2** | Qdrant Docker service not configured | ⏳ Pending | Add to compose.yaml with `rag` profile (Sprint 20a) |

### Design Gaps Addressed

| ID | Gap | Resolution |
|----|-----|------------|
| G1 | Embedding dimension mismatch risk | Store model+dimension in collection metadata; re-embed on mismatch |
| G2 | Connector priority undefined | All connectors (local, GDrive, OneDrive) equal priority in 20b |
| G3 | Migration downtime risk | Dual-read feature flag; gradual rollout |
| G4 | Active project defaults | Extend Sprint 18 active project for search/Q&A defaults |

### Prerequisite Verification

| Prerequisite | Status | Notes |
|--------------|--------|-------|
| StartIntegrationSync RPC | ✅ Verified | `noteflow.proto:73`, `grpc/_mixins/sync.py:47-105` |
| Google OAuth (PKCE) | ✅ Verified | `oauth_manager.py` with refresh, revoke |
| CalendarEventModel | ✅ Verified | `models/integrations/integration.py:188-255` |
| SegmentModel.embedding (pgvector) | ✅ Verified | `models/core/meeting.py:202-205` |
| search_semantic() pattern | ✅ Verified | `segment_repo.py:130-170` |
| Project entity + active project | ✅ Verified | Sprint 18 complete |

**Parallel with Sprint 19**: VectorStorePort and EmbeddingProviderPort defined in Sprint 19; Qdrant implementation in Sprint 20a.

---

## Key Decision: Qdrant over pgvector

| Aspect | pgvector (Sprint 19) | Qdrant (Sprint 20) |
|--------|----------------------|-------------------|
| **Scaling** | Limited by PostgreSQL | Horizontal scaling |
| **Hybrid search** | Manual implementation | Built-in BM25 + vector |
| **Filtering** | SQL joins | Native payload filters |
| **Project isolation** | WHERE clauses | Collection per project |
| **Multi-modal** | Vector only | Vector + sparse + keyword |
| **Ops complexity** | Zero (part of DB) | Additional service |

**Decision**: Migrate from pgvector to Qdrant for better scaling, hybrid search, and project isolation.

---

## Sprint 20a: RAG Core + Qdrant Migration

> **Size**: L | **Duration**: 2-3 weeks

### Objectives

1. **Qdrant Infrastructure**: Docker service, client wrapper, collection management
2. **Indexing Pipeline**: Auto-index meetings, summaries, artifacts on completion
3. **Migration**: Dual-read flag, pgvector data export, Qdrant import
4. **Retrieval Service**: Search with filters, project scoping

### Scope

| Task | Effort | Notes |
|------|--------|-------|
| **Infrastructure** | | |
| Qdrant Docker service (compose.yaml) | S | Profile: `rag`, healthcheck |
| `QdrantVectorStore` implementation | M | Implements VectorStorePort from Sprint 19 |
| Collection-per-project schema | M | Auto-create on project creation |
| Collection metadata (model, dimension) | S | Detect and handle mismatches |
| **Indexing Pipeline** | | |
| Meeting transcript indexing | M | Trigger on meeting completion |
| Summary indexing | S | Trigger on summary generation |
| Artifact indexing | M | Trigger on artifact ready status |
| Re-indexing CLI command | S | `noteflow rag reindex --project-id` |
| **Migration** | | |
| pgvector data migration script | M | Export existing segment embeddings |
| Dual-read feature flag | S | `NOTEFLOW_FEATURE_RAG_BACKEND` |
| Dimension mismatch handling | S | Re-embed if provider changed |
| **Retrieval** | | |
| `RetrievalService` | M | Unified search across chunk types |
| Search RPCs (`SearchProject`) | M | With filters, pagination |

**Total Effort**: L (2-3 weeks)

---

## Sprint 20b: External Connectors + Q&A UI

> **Size**: L | **Duration**: 2-3 weeks | **Prerequisites**: Sprint 20a

### Objectives

1. **External Connectors**: Sync artifacts from local directories and cloud drives
2. **Q&A Interface**: Project-scoped question-answering with citations
3. **Active Project Defaults**: Default to active project when `project_id` omitted

### Scope

| Task | Effort | Notes |
|------|--------|-------|
| **External Connectors** | | |
| Local directory watcher | M | File change detection, incremental sync |
| Google Drive connector | L | OAuth scopes, delta API, extend OAuthManager |
| OneDrive connector (MS Graph) | L | OAuth scopes, delta API |
| Incremental re-embedding | M | Hash-based change detection |
| Delta sync with change detection | M | Unified sync status tracking |
| Sync status UI | S | Progress, errors, last sync |
| **Q&A Service** | | |
| `QAService` | M | LLM-powered Q&A with citations |
| `AskQuestion` RPC | M | With context limit, confidence |
| Active project defaults | S | Optional project_id → active project |
| **Client** | | |
| Q&A Panel component | L | Chat interface, citations, source navigation |
| Search bar with filters | M | Global project search |
| Connector settings UI | M | Add/remove sources, sync triggers |

**Total Effort**: L (2-3 weeks)

---

## Shared Abstractions (Defined in Sprint 19)

Sprint 19 defines the port interfaces; Sprint 20 implements Qdrant:

```python
# src/noteflow/domain/ports/embedding.py (Sprint 19)

class EmbeddingProvider(Protocol):
    """Protocol for generating text embeddings."""

    @property
    def dimension(self) -> int: ...

    @property
    def model_name(self) -> str: ...

    async def embed_text(self, text: str) -> list[float]: ...
    async def embed_batch(self, texts: list[str]) -> list[list[float]]: ...
```

```python
# src/noteflow/domain/ports/vector_store.py (Sprint 19)

class VectorStorePort(Protocol):
    """Port for vector storage operations."""

    async def create_collection(self, project_id: UUID) -> None: ...
    async def delete_collection(self, project_id: UUID) -> None: ...
    async def upsert(self, project_id: UUID, chunks: list[Chunk]) -> None: ...
    async def search(
        self,
        project_id: UUID,
        query_embedding: list[float],
        limit: int = 10,
        filters: dict | None = None,
    ) -> list[SearchResult]: ...
    async def delete(self, project_id: UUID, chunk_ids: list[str]) -> None: ...
```

---

## Qdrant Architecture

### Collection Schema

```python
# One collection per project
collection_name = f"noteflow_project_{project_id}"

# Vector configuration (matches embedding provider)
vector_config = VectorParams(
    size=embedding_provider.dimension,  # e.g., 1536 for OpenAI, 768 for Ollama
    distance=Distance.COSINE,
)

# Collection metadata for dimension safety
collection_metadata = {
    "embedding_model": embedding_provider.model_name,
    "embedding_dimension": embedding_provider.dimension,
}

# Payload schema
payload_schema = {
    "chunk_type": "transcript" | "summary" | "artifact",
    "workspace_id": "uuid",
    "meeting_id": "uuid",
    "segment_id": int | None,
    "artifact_id": "uuid" | None,
    "text": str,  # Original text for display
    "source_title": str | None,
    "timestamp_start": float | None,
    "timestamp_end": float | None,
    "speaker": str | None,
    "created_at": "iso8601",
}
```

### Docker Compose

```yaml
# compose.yaml additions
services:
  qdrant:
    image: qdrant/qdrant:latest
    ports:
      - "6333:6333"   # REST API
      - "6334:6334"   # gRPC
    volumes:
      - qdrant_data:/qdrant/storage
    environment:
      - QDRANT__SERVICE__GRPC_PORT=6334
    profiles:
      - rag
    healthcheck:
      test: ["CMD", "wget", "-q", "--spider", "http://localhost:6333/healthz"]
      interval: 10s
      timeout: 5s
      retries: 3

volumes:
  qdrant_data:
```

---

## Domain Model

### RAG Entities

```python
# src/noteflow/domain/rag/entities.py

from dataclasses import dataclass, field
from datetime import datetime
from enum import Enum
from uuid import UUID

from noteflow.domain.utils.time import utc_now


class ChunkType(Enum):
    TRANSCRIPT = "transcript"
    SUMMARY = "summary"
    ARTIFACT = "artifact"


@dataclass(frozen=True)
class Chunk:
    """A chunk of text with embedding metadata."""

    id: str
    project_id: UUID
    workspace_id: UUID
    chunk_type: ChunkType
    text: str
    source_title: str | None = None
    embedding: list[float] | None = None

    # Source references
    meeting_id: UUID | None = None
    segment_id: int | None = None
    artifact_id: UUID | None = None

    # Temporal metadata
    timestamp_start: float | None = None
    timestamp_end: float | None = None
    speaker: str | None = None
    created_at: datetime = field(default_factory=utc_now)


@dataclass
class SearchResult:
    """A search result with relevance score."""

    chunk: Chunk
    score: float
    highlights: list[str] = field(default_factory=list)


@dataclass
class QAResponse:
    """Response from Q&A service."""

    answer: str
    sources: list[SearchResult]
    confidence: float
    tokens_used: int
```

---

## Proto Additions

```protobuf
// noteflow.proto additions

// === Sprint 20a: Search ===

message SearchProjectRequest {
  optional string project_id = 1; // default to active project if omitted
  string query = 2;
  int32 limit = 3;
  optional SearchFilters filters = 4;
}

enum ChunkType {
  CHUNK_TYPE_UNSPECIFIED = 0;
  CHUNK_TYPE_TRANSCRIPT = 1;
  CHUNK_TYPE_SUMMARY = 2;
  CHUNK_TYPE_ARTIFACT = 3;
}

message SearchFilters {
  repeated ChunkType chunk_types = 1;
  optional string meeting_id = 2;
  optional string artifact_id = 3;
  optional google.protobuf.Timestamp after = 4;
  optional google.protobuf.Timestamp before = 5;
}

message SearchResult {
  string chunk_id = 1;
  string text = 2;
  float score = 3;
  ChunkType chunk_type = 4;
  optional string meeting_id = 5;
  optional int32 segment_id = 6;
  optional string artifact_id = 7;
  optional float timestamp_start = 8;
  optional float timestamp_end = 9;
  optional string speaker = 10;
  optional string source_title = 11;
}

message SearchProjectResponse {
  repeated SearchResult results = 1;
}

// === Sprint 20b: Q&A ===

message AskQuestionRequest {
  optional string project_id = 1; // default to active project if omitted
  string question = 2;
  int32 context_limit = 3;
}

message AskQuestionResponse {
  string answer = 1;
  repeated SearchResult sources = 2;
  float confidence = 3;
  int32 tokens_used = 4;
}

// Service additions
service NoteFlow {
  // ... existing RPCs ...

  // Sprint 20a: RAG search
  rpc SearchProject(SearchProjectRequest) returns (SearchProjectResponse);

  // Sprint 20b: Q&A
  rpc AskQuestion(AskQuestionRequest) returns (AskQuestionResponse);
}
```

---

## Migration from pgvector

### Migration Script

```python
# scripts/migrate_to_qdrant.py

async def migrate_embeddings_to_qdrant(
    db_session: AsyncSession,
    qdrant: QdrantVectorStore,
    embedding_provider: EmbeddingProvider,
) -> MigrationResult:
    """Migrate existing pgvector embeddings to Qdrant.

    Handles dimension mismatches by re-embedding when necessary.
    """
    result = MigrationResult()

    # 1. Get all projects
    projects = await db_session.execute(select(ProjectModel))

    for project in projects.scalars():
        # 2. Create collection
        await qdrant.create_collection(project.id)

        # 3. Get all segments with embeddings for this project
        segments = await db_session.execute(
            select(SegmentModel)
            .join(MeetingModel)
            .where(MeetingModel.project_id == project.id)
            .where(SegmentModel.embedding.isnot(None))
        )

        # 4. Check dimension compatibility
        chunks = []
        for segment in segments.scalars():
            # If dimensions match, copy embedding; otherwise re-embed
            if len(segment.embedding) == embedding_provider.dimension:
                embedding = segment.embedding
            else:
                embedding = await embedding_provider.embed_text(segment.text)
                result.reembedded_count += 1

            chunk = Chunk(
                id=f"{segment.meeting_id}_{segment.segment_id}",
                project_id=project.id,
                workspace_id=segment.meeting.workspace_id,
                chunk_type=ChunkType.TRANSCRIPT,
                text=segment.text,
                embedding=embedding,
                source_title=segment.meeting.title,
                meeting_id=segment.meeting_id,
                segment_id=segment.segment_id,
                timestamp_start=segment.start_time,
                timestamp_end=segment.end_time,
                speaker=segment.speaker,
            )
            chunks.append(chunk)

        if chunks:
            await qdrant.upsert(project.id, chunks)
            result.migrated_count += len(chunks)
            logger.info(f"Migrated {len(chunks)} chunks for project {project.id}")

    return result


@dataclass
class MigrationResult:
    migrated_count: int = 0
    reembedded_count: int = 0
    errors: list[str] = field(default_factory=list)
```

### Feature Flag

```python
# src/noteflow/config/settings.py

class FeatureFlags(BaseSettings):
    # ... existing flags ...
    rag_backend: Literal["pgvector", "qdrant"] = "pgvector"
    rag_dual_read: bool = False  # Read from both during migration
```

---

## External Connectors (Sprint 20b)

### ArtifactSource Entity

```python
# src/noteflow/domain/entities/artifact_source.py

class SourceType(Enum):
    LOCAL_DIRECTORY = "local_directory"
    GOOGLE_DRIVE = "google_drive"
    ONEDRIVE = "onedrive"


@dataclass
class ArtifactSource:
    """Configuration for an external artifact source."""

    id: UUID
    project_id: UUID
    workspace_id: UUID
    source_type: SourceType
    name: str
    config: dict[str, Any]  # Type-specific config
    status: SourceStatus = SourceStatus.ACTIVE
    last_sync_at: datetime | None = None
    last_sync_status: str | None = None
    created_at: datetime = field(default_factory=utc_now)


class SourceStatus(Enum):
    ACTIVE = "active"
    PAUSED = "paused"
    ERROR = "error"
    SYNCING = "syncing"
```

### Connector Protocol

```python
# src/noteflow/domain/ports/artifact_source.py

class ArtifactSourceConnector(Protocol):
    """Protocol for external artifact source connectors."""

    async def list_files(self, source: ArtifactSource) -> AsyncIterator[RemoteFile]:
        """List all files in the source."""
        ...

    async def get_file_content(self, source: ArtifactSource, file_id: str) -> bytes:
        """Download file content."""
        ...

    async def get_delta(
        self, source: ArtifactSource, since: datetime | None
    ) -> AsyncIterator[DeltaChange]:
        """Get changes since last sync (for incremental sync)."""
        ...
```

---

## Deliverables

### Sprint 20a: RAG Core

**Backend - Infrastructure**:
- [ ] `compose.yaml` — Add Qdrant service with `rag` profile
- [ ] `src/noteflow/infrastructure/rag/qdrant_client.py` — QdrantVectorStore
- [ ] `src/noteflow/infrastructure/rag/indexing.py` — IndexingService
- [ ] `src/noteflow/infrastructure/rag/retrieval.py` — RetrievalService

**Backend - Domain**:
- [ ] `src/noteflow/domain/rag/entities.py` — Chunk, SearchResult
- [ ] `src/noteflow/domain/ports/vector_store.py` — VectorStorePort (if not in Sprint 19)

**Backend - Migration**:
- [ ] `scripts/migrate_to_qdrant.py` — pgvector → Qdrant migration
- [ ] Feature flag: `NOTEFLOW_FEATURE_RAG_BACKEND`

**Backend - API**:
- [ ] `src/noteflow/grpc/proto/noteflow.proto` — SearchProject RPC
- [ ] `src/noteflow/grpc/_mixins/search.py` — SearchMixin

**CLI**:
- [ ] `src/noteflow/cli/rag.py` — `noteflow rag reindex` command

### Sprint 20b: Connectors + Q&A

**Backend - Connectors**:
- [ ] `src/noteflow/domain/entities/artifact_source.py` — ArtifactSource entity
- [ ] `src/noteflow/infrastructure/artifacts/sources/directory.py` — Local directory
- [ ] `src/noteflow/infrastructure/artifacts/sources/gdrive.py` — Google Drive
- [ ] `src/noteflow/infrastructure/artifacts/sources/onedrive.py` — OneDrive

**Backend - Q&A**:
- [ ] `src/noteflow/application/services/qa_service.py` — QAService
- [ ] `src/noteflow/grpc/proto/noteflow.proto` — AskQuestion RPC
- [ ] `src/noteflow/grpc/_mixins/qa.py` — QAMixin

**Client (Rust)**:
- [ ] `client/src-tauri/src/commands/search.rs` — Search commands
- [ ] `client/src-tauri/src/commands/qa.rs` — Q&A commands
- [ ] `client/src-tauri/src/commands/sources.rs` — Connector commands

**Client (React)**:
- [ ] `client/src/components/qa/QAPanel.tsx` — Q&A interface
- [ ] `client/src/components/search/SearchBar.tsx` — Project search
- [ ] `client/src/components/sources/SourceSettings.tsx` — Connector UI
- [ ] `client/src/hooks/use-qa.ts` — Q&A hook
- [ ] `client/src/hooks/use-search.ts` — Search hook

---

## Test Strategy

### Fixtures to extend or create

- `tests/conftest.py`: add `workspace_id`, `project_id`, `active_project_id`, `sample_chunk`, `embedding_provider_stub`
- `tests/infrastructure/rag/conftest.py`: `qdrant_client_stub`, `vector_store_port`, `sample_chunks`
- `tests/grpc/conftest.py`: add `mock_search_service`, `mock_qa_service`, search result fixtures
- `tests/integration/conftest.py`: optional `qdrant_url` fixture; skip when Qdrant unavailable

### Parameterized tests

- Embedding providers and dimensions: OpenAI (1536) vs Ollama (768)
- Chunk types: transcript, summary, artifact
- Search filters: `chunk_types`, `meeting_id`, `artifact_id`, `after/before` timestamps
- RAG backends: pgvector vs Qdrant (feature flags), including dual-read
- Connector types: local, gdrive, onedrive (parameterize sync behavior)

### Core test cases (behavior + robustness)

**Sprint 20a**:
- **QdrantVectorStore**: create collection, upsert, search with filters, delete; handles network errors
- **Dimension safety**: collection size matches embedding provider; re-embed on mismatch
- **IndexingService**: emits workspace_id + source_title; summary uses `render_summary`; stable chunk IDs
- **Search RPC**: `project_id` optional; defaults to active project; limit + ordering enforced
- **Migration**: migrates pgvector segments; re-embeds when dimensions differ; handles empty projects

**Sprint 20b**:
- **Connectors**: delta sync, idempotent re-runs, deletion detection, auth errors propagate cleanly
- **Q&A**: empty results return low-confidence response; citations include chunk_type + source_title
- **UI**: Q&A panel renders sources; search filters apply correctly

---

## Quality Gates

### Sprint 20a

- [ ] `pytest tests/infrastructure/rag/test_qdrant_client.py` passes
- [ ] `pytest tests/infrastructure/rag/test_indexing.py` passes
- [ ] `pytest tests/grpc/test_search.py` passes
- [ ] Migration script handles all edge cases
- [ ] Dual-read mode verified with integration tests
- [ ] Meets `docs/sprints/QUALITY_STANDARDS.md`

### Sprint 20b

- [ ] `pytest tests/infrastructure/artifacts/sources/` passes
- [ ] `pytest tests/application/test_qa_service.py` passes
- [ ] `pytest tests/grpc/test_qa.py` passes
- [ ] `npm run test` frontend tests pass
- [ ] All connectors have auth error handling tests
- [ ] Q&A citations verified for all chunk types

---

## Dependency Diagram

```
Sprint 18 (Projects v1) ─────────────────────────────────────────┐
         │                                                        │
         ▼                                                        │
Sprint 19 (Artifacts v1) ──────────────────────────┐              │
  • EmbeddingProvider port                         │              │
  • VectorStorePort interface                      │              │
  • Artifact entity + chunking                     │              │
  • pgvector implementation                        │              │
         │                                         │              │
         │ (shared abstractions)                   │              │
         ▼                                         ▼              ▼
Sprint 20a (RAG Core) ─────────────────────► Sprint 20b (Connectors + Q&A)
  • Qdrant implementation                     • External connectors
  • Indexing pipeline                         • QAService
  • Migration script                          • Client UI
  • SearchProject RPC                         • AskQuestion RPC
```

---

## Risk Mitigation

| Risk | Mitigation |
|------|------------|
| Qdrant service unavailable | Feature flag fallback to pgvector; health checks |
| Embedding dimension mismatch | Store dimension in collection metadata; re-embed on mismatch |
| OAuth token expiry during sync | Extend OAuthManager refresh logic; retry with backoff |
| Large file sync timeouts | Chunked downloads; resume capability; progress tracking |
| Q&A hallucination | Confidence scoring; source verification; citation display |
| Migration data loss | Dual-read period; backup before migration; rollback script |

---

## Post-Sprint

- [ ] Dropbox connector
- [ ] Notion connector
- [ ] Confluence connector
- [ ] Hybrid search (BM25 + vector)
- [ ] RAG evaluation metrics
- [ ] Streaming Q&A responses
- [ ] Multi-turn conversation context
