vasceannie/lightrag
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Add project intelligence files for AI agent collaboration

Browse Source 
- Add .clinerules with technical patterns
- Create Agments.md for Codex agent guidance
- Ensures consistent behavior across all team members
This commit is contained in:
yangdx
2025-10-09 16:35:38 +08:00
parent 0f15fdc3e2
commit 577b9e6882
3 changed files with 315 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

207
.clinerules/01-basic.md Normal file
View File

@@ -0,0 +1,207 @@
# LightRAG Project Intelligence (.clinerules)
## Project Overview
LightRAG is a mature, production-ready Retrieval-Augmented Generation (RAG) system with comprehensive knowledge graph capabilities. The system has evolved from experimental to production-ready status with extensive functionality across all major components.
## Current System State (August 15, 2025)
- **Status**: Production Ready - Stable and Mature
- **Configuration**: Gemini 2.5 Flash + BAAI/bge-m3 embeddings via custom endpoints
- **Storage**: Default in-memory with file persistence (JsonKVStorage, NetworkXStorage, NanoVectorDBStorage)
- **Language**: Chinese for summaries
- **Workspace**: `space1` for data isolation
- **Authentication**: JWT-based with admin/user accounts
## Critical Implementation Patterns
### 1. Embedding Format Compatibility (CRITICAL)
**Pattern**: Always handle both base64 and raw array embedding formats
**Location**: `lightrag/llm/openai.py` - `openai_embed` function
**Issue**: Custom OpenAI-compatible endpoints return embeddings as raw arrays, not base64 strings
**Solution**:
```python
np.array(dp.embedding, dtype=np.float32) if isinstance(dp.embedding, list)
else np.frombuffer(base64.b64decode(dp.embedding), dtype=np.float32)
```
**Impact**: Document processing fails completely without this dual format support
### 2. Async Pattern Consistency (CRITICAL)
**Pattern**: Always await coroutines before calling methods on the result
**Common Error**: `coroutine.method()` instead of `(await coroutine).method()`
**Locations**: MongoDB implementations, Neo4j operations
**Example**: `await self._data.list_indexes()` then `await cursor.to_list()`
### 3. Storage Layer Data Compatibility (CRITICAL)
**Pattern**: Always filter deprecated/incompatible fields during deserialization
**Common Fields to Remove**: `content`, `_id` (MongoDB), database-specific fields
**Implementation**: `data.pop('field_name', None)` before creating dataclass objects
**Locations**: All storage implementations (JSON, Redis, MongoDB, PostgreSQL)
### 4. Lock Key Generation (CRITICAL)
**Pattern**: Always sort relationship pairs for consistent lock keys
**Implementation**: `sorted_key_parts = sorted([src, tgt])` then `f"{sorted_key_parts[0]}-{sorted_key_parts[1]}"`
**Impact**: Prevents deadlocks in concurrent relationship processing
### 5. Event Loop Management (CRITICAL)
**Pattern**: Handle event loop mismatches during shutdown gracefully
**Implementation**: Timeout + specific RuntimeError handling for "attached to a different loop"
**Location**: Neo4j storage finalization
**Impact**: Prevents application shutdown failures
## Architecture Patterns
### 1. Dependency Injection
**Pattern**: Pass configuration through object constructors, not direct imports
**Example**: OllamaAPI receives configuration through LightRAG object
**Benefit**: Better testability and modularity
### 2. Memory Bank Documentation
**Pattern**: Maintain comprehensive memory bank for development continuity
**Structure**: Core files (projectbrief.md, activeContext.md, progress.md, etc.)
**Purpose**: Essential for context preservation across development sessions
### 3. Configuration Management
**Pattern**: Centralize defaults in constants.py, use environment variables for runtime config
**Implementation**: Default values in constants, override via .env file
**Benefit**: Consistent configuration across components
## Development Workflow Patterns
### 1. Frontend Development (CRITICAL)
**Package Manager**: **ALWAYS USE BUN** - Never use npm or yarn unless Bun is unavailable
**Commands**:
- `bun install` - Install dependencies
- `bun run dev` - Start development server
- `bun run build` - Build for production
- `bun run lint` - Run linting
- `bun test` - Run tests
- `bun run preview` - Preview production build
**Pattern**: All frontend operations must use Bun commands
**Fallback**: Only use npm/yarn if Bun installation fails
**Testing**: Use `bun test` for all frontend testing
### 2. Bug Fix Approach
1. **Identify root cause** - Don't just fix symptoms
2. **Implement robust solution** - Handle edge cases and format variations
3. **Maintain backward compatibility** - Preserve existing functionality
4. **Add comprehensive error handling** - Graceful degradation
5. **Document the fix** - Update memory bank with technical details
### 3. Feature Implementation
1. **Follow existing patterns** - Maintain architectural consistency
2. **Use dependency injection** - Avoid direct imports between modules
3. **Implement comprehensive error handling** - Handle all failure modes
4. **Add proper logging** - Debug and warning messages
5. **Update documentation** - Memory bank and code comments
6. **Comment Language** - Use English for comments and documentation
### 4. Performance Optimization
1. **Profile before optimizing** - Identify actual bottlenecks
2. **Maintain algorithmic correctness** - Don't sacrifice functionality for speed
3. **Use appropriate data structures** - Match structure to access patterns
4. **Implement caching strategically** - Cache expensive operations
5. **Monitor memory usage** - Prevent memory leaks
## Technology Stack Intelligence
### 1. LLM Integration
- **Primary**: Gemini 2.5 Flash via custom endpoint
- **Embedding**: BAAI/bge-m3 via custom endpoint
- **Reranking**: BAAI/bge-reranker-v2-m3
- **Pattern**: Always handle multiple provider formats
### 2. Storage Backends
- **Default**: In-memory with file persistence
- **Production Options**: PostgreSQL, MongoDB, Redis, Neo4j
- **Pattern**: Abstract storage interface with multiple implementations
### 3. API Architecture
- **Framework**: FastAPI with Gunicorn for production
- **Authentication**: JWT-based with role support
- **Compatibility**: Ollama-compatible endpoints for easy integration
### 4. Frontend
- **Framework**: React with TypeScript
- **Package Manager**: **BUN (REQUIRED)** - Always use Bun for all frontend operations
- **Build Tool**: Vite with Bun runtime
- **Visualization**: Sigma.js for graph rendering
- **State Management**: React hooks with context
- **Internationalization**: i18next for multi-language support
## Common Pitfalls and Solutions
### 1. Embedding Format Issues
**Pitfall**: Assuming all endpoints return base64-encoded embeddings
**Solution**: Always check format and handle both base64 and raw arrays
### 2. Async/Await Patterns
**Pitfall**: Calling methods on coroutines instead of awaited results
**Solution**: Always await coroutines before accessing their methods
### 3. Data Model Evolution
**Pitfall**: Breaking changes when removing fields from dataclasses
**Solution**: Filter deprecated fields during deserialization, don't break storage
### 4. Concurrency Issues
**Pitfall**: Inconsistent lock key generation causing deadlocks
**Solution**: Always sort keys for deterministic lock ordering
### 5. Event Loop Management
**Pitfall**: Event loop mismatches during shutdown
**Solution**: Implement timeout and specific error handling for loop issues
## Performance Considerations
### 1. Query Context Building
- **Algorithm**: Linear gradient weighted polling for fair resource allocation
- **Optimization**: Round-robin merging to eliminate mode bias
- **Pattern**: Smart chunk selection based on cross-entity occurrence
### 2. Graph Operations
- **Optimization**: Batch operations where possible
- **Pattern**: Use appropriate indexing for large datasets
- **Consideration**: Memory usage with large graphs
### 3. LLM Request Management
- **Pattern**: Priority-based queue for request ordering
- **Optimization**: Connection pooling and retry mechanisms
- **Consideration**: Rate limiting and cost management
## Security Patterns
### 1. Authentication
- **Implementation**: JWT tokens with role-based access
- **Pattern**: Stateless authentication with configurable expiration
- **Security**: Proper token validation and refresh mechanisms
### 2. API Security
- **Pattern**: Input validation and sanitization
- **Implementation**: FastAPI dependency injection for auth
- **Consideration**: Rate limiting and abuse prevention
## Maintenance Guidelines
### 1. Memory Bank Updates
- **Trigger**: After significant changes or bug fixes
- **Pattern**: Update activeContext.md and progress.md
- **Purpose**: Maintain development continuity
### 2. Configuration Management
- **Pattern**: Environment-based configuration with sensible defaults
- **Implementation**: .env files with example templates
- **Consideration**: Security for production deployments
### 3. Error Handling
- **Pattern**: Comprehensive logging with appropriate levels
- **Implementation**: Graceful degradation where possible
- **Consideration**: User-friendly error messages
## Project Evolution Notes
The project has evolved from experimental to production-ready status. Key milestones:
- **Early 2025**: Basic RAG implementation
- **Mid 2025**: Multiple storage backends and LLM providers
- **July 2025**: Major query optimization and algorithm improvements
- **August 2025**: Production-ready stable state
The system now supports enterprise-level deployments with comprehensive functionality across all components.

1
.gitignore vendored
View File

@@ -71,4 +71,3 @@ test_*
# Cline files
memory-bank
memory-bank/
.clinerules

108
Agments.md Normal file
View File

@@ -0,0 +1,108 @@
# Project Guide for AI Agents
This Agments.md file provides operational guidance for AI assistants collaborating on the LightRAG codebase. Use it to understand the repository layout, preferred tooling, and expectations for adding or modifying functionality.
## Core Purpose
LightRAG is an advanced Retrieval-Augmented Generation (RAG) framework designed to enhance information retrieval and generation through graph-based knowledge representation. The project aims to provide a more intelligent and efficient way to process and retrieve information from documents by leveraging both graph structures and vector embeddings.
## Project Structure for Navigation
- `/lightrag`: Core Python package (ingestion, querying, storage abstractions, utilities). Key modules include `lightrag/lightrag.py` orchestration, `operate.py` pipeline helpers, `kg/` storage backends, `llm/` bindings, and `utils*.py`.
- `/lightrag/api`: FastAPI with Gunicorn for production. FastAPI service for LightRAG , auth, WebUI assets live in `lightrag_server.py`. Routers live in `routers/`, shared helpers in `utils_api.py`. Gunicorn startup logic lives in `run_with_gunicorn.py`.
- `/lightrag_webui`: React 19 + TypeScript + Tailwind front-end built with Vite/Bun. Uses component folders under `src/` and configuration via `env.*.sample`.
- `/inputs`, `/rag_storage`, `/dickens`, `/temp`: data directories. Treat contents as mutable working data; avoid committing generated artefacts.
- `/tests` and root-level `test_*.py`: Integration and smoke-test scripts (graph storage, API endpoints, behaviour regressions). Many expect specific environment variables or services.
- `/docs`, `/k8s-deploy`, `docker-compose.yml`: Deployment notes, Kubernetes manifests, and container orchestration helpers.
- Configuration templates: `.env.example`, `config.ini.example`, `lightrag.service.example`. Copy and adapt for local runs without committing secrets.
## Environment Setup and Tooling
- Python 3.10 is required. Recommended bootstrap:
```bash
# Development installation
python -m venv .venv
source .venv/bin/activate
pip install -e .
pip install -e .[api]
# Start API server
lightrag-server
# Production deployment
lightrag-gunicorn --workers 3
```
- Duplicate `.env.example` to `.env` and adjust storage, LLM, and reranker bindings. Mirror `config.ini.example` when customising pipeline defaults.
- Storage backends (PostgreSQL, Redis, Neo4j, Milvus, etc.) are selected via `LIGHTRAG_*` environment variables. Ensure connection URLs and credentials are in place before running ingestion or tests.
- CLI entry points: `python -m lightrag` for package usage, `lightrag-server` (or `uvicorn lightrag.api.lightrag_server:app --reload`) for the API, `lightrag-gunicorn` for production gunicorn runs.
- Front-end work: install dependencies with `bun install` (preferred) or `npm install`, then use `bunx --bun vite` commands defined in `package.json`.
## Frontend Development
* **Package Manager**: **ALWAYS USE BUN** - Never use npm or yarn unless Bun is unavailable
**Commands**:
- `bun install` - Install dependencies
- `bun run dev` - Start development server
- `bun run build` - Build for production
- `bun run lint` - Run linting
- `bun test` - Run tests
- `bun run preview` - Preview production build
* **Pattern**: All frontend operations must use Bun commands
* **Testing**: Use `bun test` for all frontend testing
## Coding Conventions
- Embrace type hints, dataclasses, and asynchronous patterns already present in `lightrag/lightrag.py` and storage implementations. Keep long-running jobs within `asyncio` flows and reuse helpers from `lightrag.operate`.
- Honour abstraction boundaries: new storage providers should inherit from the relevant base classes in `lightrag.base`; reusable logic belongs in `utils.py`/`utils_graph.py`.
- Use `lightrag.utils.logger` (not bare `print`) and let environment toggles (`VERBOSE`, `LOG_LEVEL`) control verbosity.
- Respect configuration defaults in `lightrag/constants.py`, extending with care and synchronising related documentation when behaviour changes.
- API additions should live under `lightrag/api/routers`, leverage dependency injections from `utils_api.py`, and return structured responses consistent with existing handlers.
- Front-end code should remain in TypeScript, rely on functional React components with hooks, and follow Tailwind utility style. Co-locate component-specific styles; reserve custom CSS for cases Tailwind cannot cover.
- Storage Backends
- **Default**: In-memory with file persistence
- **Production Options**: PostgreSQL, MongoDB, Redis, Neo4j
- **Pattern**: Abstract storage interface with multiple implementations
* Lock Key Generation Consistency
- **Critical Pattern**: Always sort parameters for lock key generation to prevent deadlocks
- **Example**: `sorted_key_parts = sorted([src, tgt])` before creating lock key
- **Why**: Prevents different lock keys for same relationship pair processed in different orders
- **Apply to**: Any function that uses locks with multiple parameters
* Priority Queue Implementation
* **Pattern**: Use priority-based task queuing for LLM requests
* **Benefits**: Critical operations get higher priority
* **Implementation**: Lower priority values = higher priority
## Testing and Quality Gates
- Run Python tests with `python -m pytest tests` for the FastAPI suite, and execute targeted scripts (for example `python tests/test_graph_storage.py`, `python test_lock_fix.py`) when touching related functionality. Many scripts require running backing services; check `.env` for prerequisites.
- Perform linting via `ruff check .` (configured in `pyproject.toml`) and address warnings. For formatting, match the existing style rather than introducing new tools.
- Front-end validation: `bun test`, `bunx --bun vite build`, and `bunx --bun vite lint`. The `*-no-bun` scripts exist if Bun is unavailable.
- When touching deployment assets, ensure `docker-compose config` or relevant `kubectl` dry-runs succeed before submitting changes.
## Runtime and Operational Notes
- Knowledge ingestion expects documents inside `inputs/` and writes intermediate state to `rag_storage/`. Keep these directories gitignored; never check in private data or large artefacts.
- Use `operate.py` helpers (e.g., `chunking_by_token_size`, `extract_entities`) to keep ingestion behaviour consistent. If extending the pipeline, document new steps in `docs/` and update any affected CLI usage.
- The API and core package rely on `.env`/`config.ini` being co-located with the current working directory. Scripts such as `tests/test_graph_storage.py` dynamically read these files; ensure they are in sync.
## Contribution Checklist
1. Run `pre-commit run --all-files` before sumitting PR.
2. Describe the change, affected modules, and operational impact in your PR. Mention any new environment knobs or storage dependencies.
3. Link related issues or discussions when available.
4. Confirm all applicable checks pass (`ruff`, pytest suite, targeted integration scripts, front-end build/tests when touched).
5. Capture screenshots or GIFs for front-end or API changes that affect user-visible behaviour.
6. Keep each PR focused on a single concern and update documentation (`README.md`, `docs/`, `.env.example`) when behaviour or configuration changes.
Follow this playbook to keep LightRAG contributions predictable, testable, and production-ready.