TUI Feeds

This guide explains how the terminal dashboard surfaces collection activity and status signals so new backends can plug in without duplicating UI logic.

Activity Feed

Primary surface: #activity_feed widget inside DashboardScreen (ingest_pipeline/cli/tui/screens/dashboard.py).
Data source: self.collections, populated by refresh_collections() after gathering payloads from Weaviate and OpenWebUI via describe_collections().
Selection logic: _generate_activity_text() formats the three most recent CollectionInfo entries and appends an aggregate line when additional collections exist.
Empty state: Presents the call-to-action “🚀 No collections found…” encouraging the user to launch an ingestion run.
Icons: _get_content_type_icon() maps collection names containing web, doc, or repo to 🌐/📖/📦 respectively, and falls back to 📄. Update this helper when introducing new naming conventions.

When it refreshes

refresh_collections() loads data for each connected backend and caches it in self.collections.
_update_activity_feed() is triggered from update_metrics() immediately after metrics cards recompute.
The Static widget updates with a newline-delimited summary, keeping the dashboard reactive without rerendering the entire layout.

To surface a new backend, extend either list_weaviate_collections() or list_openwebui_collections() with the additional source (or introduce a new list helper) and ensure the resulting dictionaries match the CollectionInfo contract.

Status Ticker

Widget: #status_text Static component under the metrics card cluster.
Lifecycle: refresh_collections() pushes human-readable messages as each backend initializes, succeeds, or fails, ending with a ready state.
Problem reporting: Failures bubble into rich notifications via self.notify and remain visible in the ticker until the next refresh attempt.
System health badge: _update_status_card() converts backend counts into 🟢/🟡/🔴 badges so operators can judge connectivity at a glance.

When adding a backend integration, hook into the progress text updates inside refresh_collections() so the ticker narrates each stage consistently.

Notifications & Progress

Toast notifications: All feed-relevant exceptions use self.notify with severity hints, keeping the activity feed focused on successful runs.
Ingestion progress: IngestionScreen.perform_ingestion() (same module) drives the animated progress bar and sends celebratory/failure messages that complement the dashboard feed once control returns to the main screen.

Extending the Feed System

Return a fully populated CollectionInfo (name, type, backend label, status, last_updated, size_mb, count).
Call update_metrics() after mutating self.collections so both metrics cards and the activity feed stay in sync.
Adjust _get_content_type_icon() or _format_collection_item() if the new source warrants distinct labeling.
Update end-to-end tests or manual runbooks to verify the ticker, notifications, and activity feed stay coherent after integration.

Implementation Status (September 17, 2025)

Component	Responsibility	Location
Activity feed rendering	`_update_activity_feed`, `_generate_activity_text`, `_format_collection_item`	`ingest_pipeline/cli/tui/screens/dashboard.py`
Backend loaders	`list_weaviate_collections`, `list_openwebui_collections`	`ingest_pipeline/cli/tui/screens/dashboard.py`
Status ticker & health badge	`_update_status_card`, `refresh_collections` progress updates	`ingest_pipeline/cli/tui/screens/dashboard.py`
Ingestion progress hand-off	`perform_ingestion` success/error notifications	`ingest_pipeline/cli/tui/screens/ingestion.py`

Multi-Storage Ingestion Refactor Plan

0. Guardrails and Baseline

Activate the virtual environment (source .venv/bin/activate) before running any tooling.
Capture current lint, type, and test status (uv run basedpyright, uv run ruff check, uv run pytest) to compare after the refactor.
Record the existing ingestion modal behaviour (screenshots or a short textual run --dev ingest_pipeline/cli/tui demo) to verify UX parity later.

1. Storage Layer Enhancements

Graduate MultiStorageAdapter into ingest_pipeline/storage/ so it can be reused outside the TUI package.
Extend BaseStorage with a descriptive display_name property that downstream UIs can show without hard-coding labels.
Harden the adapter: aggregate per-backend failures, short-circuit close() safely, and surface a structured result containing success_ids and failed_targets.
Add StorageManager.build_multi_adapter(backends: Sequence[StorageBackend]) that returns an initialised adapter (invokes initialize() on each child) and memoises singletons for reuse inside the session.

2. Application Wiring

Refactor CollectionManagementApp to accept a StorageManager plus optional cached clients, removing direct constructor parameters for Weaviate/OpenWebUI.
Update all screens (dashboard.py, documents.py, search.py, dialogs) to pull storages through the shared manager instead of owning bespoke references.
Expose a capability flag (e.g., StorageCapabilities.REPLICATION) so the dashboard can badge backends that support multi-target ingestion.

Replace the single-backend select with a checkbox group generated from StorageManager.get_available_backends(); preserve keyboard shortcuts (1, 2, 3, plus ctrl+shift+<n> for toggling if feasible).
Default the selection to the collection’s current backend but allow "Select All"/"Clear" convenience buttons.
Persist the latest selection inside a lightweight config file (for example ~/.config/rag-manager/tui.json) to improve repeated runs.

4. Flow Integration

Update IngestionScreen.perform_ingestion() to build the multi-adapter, pass it to ingest_documents_task, and capture per-backend success/failure counts for feed reporting.
Teach ingest_pipeline/flows/ingestion.py helpers to recognise the adapter (inspect for fanout_targets) and log progress per backend, while keeping Firecrawl→R2R flow single-target until replication lands there.
Ensure partial failures propagate as IngestionStatus.PARTIAL with an error message enumerating the failing targets.

5. Feeds, Ticker, and Notifications

Extend _generate_activity_text() to append the backend list (e.g., → weaviate + open_webui) when a multi-target run finishes.
Add per-backend status lines to the progress ticker so operators know which replication stage is executing.
Emit granular toast notifications: success summary plus warning toasts for any backend that failed to store documents.

6. Validation

Add unit coverage for MultiStorageAdapter (full success, partial failure, close semantics) under ingest_pipeline/tests/storage/.
Create a focused TUI smoke test that opens the ingestion modal, toggles multiple checkboxes, and asserts the resulting progress copy.
Re-run uv run basedpyright, uv run ruff check, and the targeted pytest suite before and after changes; address new diagnostics immediately.
Optionally script a headless textual run that simulates ingestion across two mock storages to guard against regressions.

7. Documentation and Rollout

Update this document and README.md with refreshed screenshots/GIFs demonstrating multi-backend ingestion.
Draft release notes covering required configuration (API keys for every backend) and outline rollback instructions (git tag + revert steps).
Brief support/playbook owners on interpreting the enriched feed/ticker signals so incidents can be triaged quickly.

7.7 KiB Raw Blame History Unescape Escape