# SPRINT-GAP-004: Diarization Job Lifecycle Issues

| Attribute | Value |
|-----------|-------|
| **Sprint** | GAP-004 |
| **Size** | S (Small) |
| **Owner** | TBD |
| **Phase** | Hardening |
| **Prerequisites** | None |

## Open Issues

- [x] Define maximum poll duration aligned to server timeout (300s = 5 minutes)
- [x] Determine transient error detection heuristics (uses `err.classify().retryable`)
- [x] Decide on job recovery strategy after app restart (implemented - see Job Recovery section)

## Validation Status

| Component | Exists | Status |
|-----------|--------|--------|
| Job status polling | Yes | ✅ Complete - Rust has retries + max duration; React hook has retries + max duration |
| Job cancellation | Yes | ✅ OK (response includes status; hook uses it) |
| DB persistence | Yes | ✅ Required - diarization now requires database support |
| Server recovery | Yes | ✅ Marks orphaned RUNNING/QUEUED jobs as FAILED on startup |
| Client recovery | Yes | ✅ GetActiveDiarizationJobs RPC + hook auto-recovery |

## Objective

Improve diarization job lifecycle management to handle transient failures gracefully, surface cancellation state consistently, and ensure job state survives application restarts.

## Key Decisions

| Decision | Choice | Rationale |
|----------|--------|-----------|
| Poll failure handling | Retry with backoff | React hook already backs off; Rust progress poll should too |
| Cancellation | Use response status | Cancel response already includes final status |
| Memory fallback | **Removed** - require DB | Diarization requires persistent state for recovery |
| Max poll duration | Align with server timeout (5 minutes) | Avoid premature stop vs backend timeout |
| Server recovery | Mark orphaned jobs as FAILED | Jobs left RUNNING/QUEUED from previous server indicate crash |
| Client recovery | GetActiveDiarizationJobs RPC | Query server for in-flight jobs after reconnection |

## What Already Exists

### Backend (`src/noteflow/grpc/_mixins/diarization/_jobs.py`)
- Job creation with DB or memory fallback
- Background task execution with timeout
- Status updates (QUEUED → RUNNING → COMPLETED/FAILED/CANCELLED)
- Job timeout enforced at 5 minutes (`DIARIZATION_TIMEOUT_SECONDS = 300`)
- Cancellation via task cancellation

### Client
- Rust Tauri command polls every 2 seconds for progress events (`client/src-tauri/src/commands/diarization.rs`)
- React hook polls with backoff + retry (`client/src/hooks/use-diarization.ts`)
- `cancel_diarization_job()` sends cancel request; response includes final status
- Error emission on failures for refine/get-status in Rust

## Identified Issues

### 1. Polling Loop Breaks on Any Error (Medium)

**Location**: `client/src-tauri/src/commands/diarization.rs:128-133`

```rust
let status = match state.grpc_client.get_diarization_job_status(&job_id).await {
    Ok(status) => status,
    Err(err) => {
        emit_error(&app, "diarization_error", &err);
        break;  // Stops polling on ANY error
    }
};
```

**Problem**: Any error terminates the poll loop:
- Transient network error → polling stops
- User sees "error" but job may still be running
- No distinction between fatal and recoverable errors

**Impact**: Progress events stop; users may miss completion toast even if the job finishes.

### 2. In-Memory Fallback Doesn't Persist (Low)

**Location**: `src/noteflow/grpc/_mixins/diarization/_jobs.py:129-135`

```python
if repo.supports_diarization_jobs:
    await repo.diarization_jobs.create(job)
    await repo.commit()
else:
    self.diarization_jobs[job_id] = job  # Lost on restart
```

**Problem**: When DB not available:
- Jobs stored in memory only
- Server restart loses all job state
- No warning to user

### 3. No Maximum Poll Duration (Low)

**Location**: `client/src-tauri/src/commands/diarization.rs:124-144`, `client/src/hooks/use-diarization.ts:104-177`

```rust
tauri::async_runtime::spawn(async move {
    let mut interval = tokio::time::interval(Duration::from_secs(2));
    loop {
        interval.tick().await;
        // ... no attempt counter
        if matches!(status.status, ...) {
            break;
        }
    }
});
```

**Problem**: Polling continues indefinitely on the happy path:
- No cap aligned to the 5-minute server timeout
- Polling persists if job stalls without transitioning
- Resource waste if job disappears and no error is raised

## Scope

### Task Breakdown

| Task | Effort | Description |
|------|--------|-------------|
| Add retry logic to Rust progress polling | S | Retry transient errors with backoff |
| Add max poll duration | S | Align client poll duration with server timeout |
| Warn on memory fallback | S | Emit warning when using in-memory |
| Distinguish error types | S | Identify transient vs fatal errors |

### Files to Modify

**Client:**
- `client/src-tauri/src/commands/diarization.rs`
- `client/src/hooks/use-diarization.ts`

**Backend:**
- `src/noteflow/grpc/_mixins/diarization/_jobs.py`

## Error Classification for Polling

```rust
fn is_transient_error(err: &Error) -> bool {
    // Network errors, timeouts, temporary unavailable
    matches!(err.code(),
        ErrorCode::Unavailable |
        ErrorCode::DeadlineExceeded |
        ErrorCode::ResourceExhausted
    )
}

fn is_fatal_error(err: &Error) -> bool {
    // Job gone, invalid state
    matches!(err.code(),
        ErrorCode::NotFound |
        ErrorCode::InvalidArgument |
        ErrorCode::PermissionDenied
    )
}
```

## Migration Strategy

### Phase 1: Add Resilience (Low Risk)
- Retry on transient errors in Rust progress polling (up to 3 times)
- Add max poll duration aligned to server timeout
- No behavior change for success path

### Phase 2: Add Warnings (Low Risk)
- Emit warning event for memory fallback
- UI can show notification
- No breaking changes

## Deliverables

### Backend
- [x] ~~Log warning when using in-memory job storage~~ → **Removed memory fallback, require DB**
- [x] Emit metric for storage mode - now always "database" since memory fallback removed
- [x] `mark_running_as_failed()` - recover orphaned jobs on server startup
- [x] `get_all_active()` repository method - query RUNNING/QUEUED jobs
- [x] `GetActiveDiarizationJobs` RPC - expose active jobs to clients

### Client
- [x] Retry transient polling errors in Rust progress poll (3 attempts with backoff) (`diarization.rs:190-202`)
- [x] Maximum poll duration aligned to server timeout (`diarization.rs:176-185`, `use-diarization.ts:116-136`)
- [x] `get_active_diarization_jobs()` Rust client method
- [x] `getActiveDiarizationJobs()` TypeScript adapter method
- [x] `recover()` function in useDiarization hook
- [x] `autoRecover` option for automatic recovery on mount

### Tests
- [x] Unit test: transient error retry (`use-diarization.test.ts` - retry on network failure)
- [x] Unit test: max duration reached (`use-diarization.test.ts` - stops polling after max duration)
- [x] Python test: database requirement (`test_diarization_lifecycle.py`)
- [x] Integration test: server restart during poll (`test_grpc_servicer_database.py`)

## Test Strategy

### Fixtures
- Mock server that fails intermittently
- Mock server that cancels slowly
- In-memory mode server

### Test Cases

| Case | Input | Expected |
|------|-------|----------|
| Transient error during poll | Network blip | Retry up to 3 times, continue polling |
| Fatal error during poll | Job not found | Stop polling, emit error |
| Max duration reached | Zombie job | Stop polling, emit timeout error |
| Cancel job | Active job | Returns CANCELLED status |
| Memory fallback | No DB | Warning emitted, job proceeds |

## Quality Gates

- [x] Polling survives transient errors (Rust: 3 retries with backoff; TypeScript: 3 retries)
- [x] Polling stops after max duration (300s aligned with server timeout)
- [x] Cancel returns confirmed state (unchanged - already working)
- [x] Memory fallback warning visible (structlog warning with storage_mode extra)
- [x] Tests cover error scenarios (6 Python tests, 26 TypeScript tests)
- [x] No zombie poll loops in production (max duration + proper cleanup)

## Job Recovery

### Server-Side Recovery

On server startup, the server calls `mark_running_as_failed()` to handle orphaned jobs:

```python
# In server.py startup
await self.servicer.mark_running_as_failed()
```

This marks all RUNNING and QUEUED jobs as FAILED with error message "ERR_SERVER_RESTART". This is the correct behavior because:
- A RUNNING job from a previous server instance cannot resume (state is lost)
- Clients polling these jobs need to know they failed
- The user can retry diarization if needed

### Client-Side Recovery

The client uses `GetActiveDiarizationJobs` RPC to recover polling state:

```typescript
// Option 1: Auto-recovery on mount
const { state, start, cancel, recover } = useDiarization({
  autoRecover: true,  // Automatically queries server on mount
});

// Option 2: Manual recovery (e.g., after reconnection)
const { recover } = useDiarization();
useEffect(() => {
  if (isConnected) {
    recover();
  }
}, [isConnected, recover]);
```

The recovery flow:
1. Client queries server for active (QUEUED/RUNNING) jobs
2. For each active job, resume polling
3. Emit toast notification that polling has resumed

### Database Requirement

**Important**: Diarization now requires database support. If the database is unavailable:
- `RefineSpeakerDiarization` returns error with FAILED_PRECONDITION status
- Error message indicates database is required
- No in-memory fallback

This ensures job state is always persisted and can be recovered after restarts.

## Completion

**Status**: ✅ Complete

**Date**: 2026-01-02

**Summary**: Sprint GAP-004 addresses diarization job lifecycle issues by adding:
1. Retry logic with exponential backoff for transient polling errors (Rust + TypeScript)
2. Maximum poll duration of 5 minutes aligned with server timeout
3. **Server-side job recovery** - marks orphaned RUNNING/QUEUED jobs as FAILED on startup
4. **Client-side job recovery** - GetActiveDiarizationJobs RPC + hook auto-recovery
5. **Database requirement** - removed memory fallback, diarization requires DB
6. Comprehensive test coverage for the above scenarios
