vasceannie/lightrag
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
6,166 Commits 1 Branch 0 Tags
main
Commit Graph

6166 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
yangdx
be744a28a7 Update Postgres tests for keyset pagination and API changes 
- Use check_table_exists DB method
- Update mocks for keyset pagination
- Enforce error on dimension mismatch
- Remove deprecated module patches
- Verify workspace migration isolation
 2025-12-21 18:37:28 +08:00
yangdx
5fef7e4042 Skip legacy vector table init in Postgres and fix migration checks 
* Pass model_name in API embedding setup
* Skip legacy vector tables in check_tables
* Verify legacy tables exist before legacy migrating
* Exclude legacy vector tables from index check
* Add model_name to embedding_func of LightRAG Server (Kick start data migration for vector table with model and dimension suffix)
 2025-12-21 17:24:05 +08:00
yangdx
8ef86c4898 Refactor PG vector storage and add index creation 
* Move helper functions to static methods
* Move check table exists functions to PostgreSQLDB
* Create ID and workspace indexes in DDL
 2025-12-21 16:25:58 +08:00
yangdx
2228a75dd0 Fix NumPy ambiguity and array support in Postgres 
- Fix NumPy boolean ambiguity error
- Use `is not None` for vector check
- Support NumPy arrays for dimensions
- Handle array-like vector data
 2025-12-21 16:02:04 +08:00
yangdx
ff19a67feb Add model_suffix argument to Qdrant tests 
- Pass suffix to dimension tests
- Add explicit suffix to safety tests
- Test empty suffix scenario
- Update collection init calls
 2025-12-21 02:16:47 +08:00
yangdx
77ed23a7ab Fix markdown table formatting in README files 2025-12-21 01:47:36 +08:00
yangdx
caed4fb9b6 Add model_name attribute to embedding wrappers 
- Add `model_name` to embedding decorators
- Update `EmbeddingFunc` class definition
- Set default models for LLM providers
- Refactor wrapper docstrings in utils
- Update README usage examples
 2025-12-21 01:32:27 +08:00
yangdx
9c52e32a14 Fix legacy collection name in Qdrant warning log 2025-12-21 01:31:10 +08:00
yangdx
9381dee6e5 Elevate manual deletion log to warning level 2025-12-20 23:54:42 +08:00
yangdx
7618de44df Refine Qdrant legacy collection lookup with model suffix support 
- Add model_suffix to legacy lookup
- Update collection search priorities
- Pass suffix to migration setup
- Store model_suffix in instance
- Adjust candidate generation logic
 2025-12-20 16:18:41 +08:00
yangdx
c65d606784 Correct comments regarding __post_init__ invocation sources 2025-12-20 16:09:21 +08:00
yangdx
098751773b Refine migration warning messages for PG and Qdrant 2025-12-20 13:24:37 +08:00
yangdx
9726431f50 Improve vector storage logging and migration warnings 
- Update missing model suffix warnings
- Clarify migration conflict messages
- Apply changes to PG and Qdrant
 2025-12-20 13:17:51 +08:00
yangdx
e596512ba3 Fix __post_init__ usage in Mongo and Qdrant storage implementations 
* Remove manual `__post_init__` in `__init__`
* Add `super().__post_init__` in vector DBs
* Ensure base validation runs correctly
* Cleanup Mongo and Qdrant init logic
 2025-12-20 12:53:19 +08:00
yangdx
0ac35bfee4 Prevent mutation of shared EmbeddingFunc instances 
- Import replace from dataclasses
- Use replace() for embedding func
- Safely wrap priority async func
 2025-12-20 12:29:43 +08:00
yangdx
1aa4a3a385 Fix PostgreSQL index lookup failure for long table names 
*   Implement safe index name generation
*   Hash table names if index exceeds 63B
*   Fix index detection for long models
*   Define PG identifier limit constant
*   Add tests for index name safety
 2025-12-20 05:40:59 +08:00
yangdx
e12dfdb9e0 Bootstrap vector extension before pool creation 
- Create standalone bootstrap connection
- Enable vector extension early
- Fix startup failure on fresh DBs
- Ensure vector type exists for pool
 2025-12-20 05:14:16 +08:00
yangdx
c81e9c9ef6 Register pgvector codec in pool init for consistent vector handling 
- Add init callback to asyncpg pool
- Register vector codec globally
- Remove redundant manual registrations
- Fix non-deterministic codec issues
 2025-12-20 04:39:37 +08:00
yangdx
85e8e33573 Fix string vector parsing in PG workspace migration 
- Check if content_vector is string
- Parse pgvector string to numpy array
- Fix type mismatch during batch write
 2025-12-20 04:31:49 +08:00
yangdx
911585fd22 Refactor Qdrant deletion logic for safety and scalability 
- Query entity by payload ID for delete
- Paginate relation deletion loop
- Optimize query payloads and vectors
- Support deleting >1k relations
 2025-12-20 04:10:05 +08:00
yangdx
2073f957d7 Add validation for PostgreSQL table name length 2025-12-20 03:54:24 +08:00
yangdx
864131a622 Enforce embedding_func validation in BaseVectorStorage 
- Add validation in BaseVectorStorage
- Call super().__post_init__ in subclasses
- Simplify collection suffix logic
 2025-12-20 03:49:31 +08:00
yangdx
81a0d632ca feat: add Qdrant legacy data prep tool for migration tests 
- Copy workspace data to legacy format
- Remove workspace_id from payloads
- Support dry-run and batch operations
- Verify data before creating targets
- Aid in testing migration logic
 2025-12-20 03:04:43 +08:00
yangdx
3456818b0b Wrap inner embedding func to preserve attributes 
- Apply wrapper to inner `.func` only
- Preserve `EmbeddingFunc` attributes
- Ensure `model_name` remains accessible
- Fix collection suffix generation
 2025-12-20 01:28:26 +08:00
yangdx
dfe628ad0b Use keyset pagination for PostgreSQL migration 
- Switch to keyset pagination for migration
- Ensure stable ordering via ID column
- Prevent row skipping or duplication
- Update tests for new query pattern
- Minor doc comment fix in Qdrant
 2025-12-20 00:32:40 +08:00
yangdx
93ea50c4aa Restrict Qdrant legacy scroll filter to specific workspace 
- Switch filter from `should` to `must`
- Remove `IsNullCondition` for workspace
- Enforce strict workspace ID matching
 2025-12-19 23:41:14 +08:00
yangdx
c1ed2e31f8 Handle diverse vector types in Postgres storage 
- Support list and tuple return types
- Parse JSON strings for vectors
- Handle numpy arrays via tolist
- Fix vector decoding errors
 2025-12-19 23:16:01 +08:00
yangdx
73c3c4135c Drop Python 3.13 from tests and reformat code 2025-12-19 18:15:12 +08:00
yangdx
27863a6019 Suppress empty warning for legacy tables in PostgreSQL if legacy and new table name match 2025-12-19 18:12:06 +08:00
yangdx
4ac5ec4c2f Improve Qdrant workspace detection via payload sampling 
- Detect unindexed workspace_id via sample
- Prevent cross-workspace data leakage
- Fix empty workspace warning logic
- Update migration tests for sampling
 2025-12-19 18:00:17 +08:00
yangdx
e77a506467 Add workspace filtering to Qdrant legacy migration 
* Detect workspace field in legacy schema
* Filter scroll by active workspace
* Recount records for accurate tracking
* Prevent cross-workspace data leakage
* Update payload workspace assignment
 2025-12-19 16:45:15 +08:00
yangdx
1c083c6699 Remove redundant pytest.mark.asyncio decorators 
- Remove explicit asyncio markers
- Clean up unused imports in tests
 2025-12-19 16:00:37 +08:00
yangdx
e9003f3f13 Move shared lock validation to factory functions and fix test formatting 
- Enforce init in lock factory functions
- Simplify UnifiedLock class logic
- Update lock safety tests
- Fix line wrapping in test files
 2025-12-19 15:58:02 +08:00
yangdx
a3b33bbc3c Remove E2E tests and update migration unit tests 
- Delete E2E workflows and test files
- Remove multi-model demo example
- Update Postgres migration unit tests
- Enforce workspace requirement in tests
- Fix dimension mismatch test mocks
 2025-12-19 15:20:32 +08:00
yangdx
343ccac929 Add 'd' suffix to dimensions in migration error message 2025-12-19 14:06:34 +08:00
yangdx
37e4d94d19 Add vector dimension validation and storage safety checks 
*   Validate Faiss index dimensions
*   Check Mongo vector index dimensions
*   Add migration warnings for Postgres
*   Add migration warnings for Qdrant
 2025-12-19 13:58:31 +08:00
yangdx
ada5f10be7 Optimize Postgres batch operations and refine workspace migration logic 
- Use executemany for efficient upserts
- Optimize data migration with batching
- Refine multi-workspace migration logic
- Add pgvector dependency
- Update DDL templates for dynamic dims
 2025-12-19 12:05:22 +08:00
yangdx
0ae60d36bc Improve Qdrant migration checks and verification logic 
- Check workspace data before migrating
- Update Qdrant migration tests
 2025-12-19 12:03:38 +08:00
yangdx
bf618fc976 Refactor Qdrant setup and migration logic 
- Validate dimensions before migration
- Require namespace and workspace args
- Raise error on vector size mismatch
- Simplify collection initialization flow
- Update tests for strict checks
 2025-12-19 10:45:18 +08:00
dependabot[bot]
377ea22629 Bump the build-tools group in /lightrag_webui with 2 updates 
Bumps the build-tools group in /lightrag_webui with 2 updates: [@eslint/js](https://github.com/eslint/eslint/tree/HEAD/packages/js) and [eslint](https://github.com/eslint/eslint).


Updates `@eslint/js` from 9.39.1 to 9.39.2
- [Release notes](https://github.com/eslint/eslint/releases)
- [Commits](https://github.com/eslint/eslint/commits/v9.39.2/packages/js)

Updates `eslint` from 9.39.1 to 9.39.2
- [Release notes](https://github.com/eslint/eslint/releases)
- [Commits](https://github.com/eslint/eslint/compare/v9.39.1...v9.39.2)

---
updated-dependencies:
- dependency-name: "@eslint/js"
  dependency-version: 9.39.2
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: build-tools
- dependency-name: eslint
  dependency-version: 9.39.2
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: build-tools
...

Signed-off-by: dependabot[bot] <support@github.com>
 2025-12-18 18:06:52 +00:00
dependabot[bot]
f3b0a99958 Bump the ui-components group in /lightrag_webui with 3 updates 
Bumps the ui-components group in /lightrag_webui with 3 updates: [lucide-react](https://github.com/lucide-icons/lucide/tree/HEAD/packages/lucide-react), [@tailwindcss/vite](https://github.com/tailwindlabs/tailwindcss/tree/HEAD/packages/@tailwindcss-vite) and [tailwindcss](https://github.com/tailwindlabs/tailwindcss/tree/HEAD/packages/tailwindcss).


Updates `lucide-react` from 0.555.0 to 0.560.0
- [Release notes](https://github.com/lucide-icons/lucide/releases)
- [Commits](https://github.com/lucide-icons/lucide/commits/0.560.0/packages/lucide-react)

Updates `@tailwindcss/vite` from 4.1.17 to 4.1.18
- [Release notes](https://github.com/tailwindlabs/tailwindcss/releases)
- [Changelog](https://github.com/tailwindlabs/tailwindcss/blob/main/CHANGELOG.md)
- [Commits](https://github.com/tailwindlabs/tailwindcss/commits/v4.1.18/packages/@tailwindcss-vite)

Updates `tailwindcss` from 4.1.17 to 4.1.18
- [Release notes](https://github.com/tailwindlabs/tailwindcss/releases)
- [Changelog](https://github.com/tailwindlabs/tailwindcss/blob/main/CHANGELOG.md)
- [Commits](https://github.com/tailwindlabs/tailwindcss/commits/v4.1.18/packages/tailwindcss)

---
updated-dependencies:
- dependency-name: lucide-react
  dependency-version: 0.560.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: ui-components
- dependency-name: "@tailwindcss/vite"
  dependency-version: 4.1.18
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: ui-components
- dependency-name: tailwindcss
  dependency-version: 4.1.18
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: ui-components
...

Signed-off-by: dependabot[bot] <support@github.com>
 2025-12-18 18:06:28 +00:00
dependabot[bot]
e95860ea29 Bump the react group in /lightrag_webui with 2 updates 
Bumps the react group in /lightrag_webui with 2 updates: [react](https://github.com/facebook/react/tree/HEAD/packages/react) and [react-dom](https://github.com/facebook/react/tree/HEAD/packages/react-dom).


Updates `react` from 19.2.1 to 19.2.3
- [Release notes](https://github.com/facebook/react/releases)
- [Changelog](https://github.com/facebook/react/blob/main/CHANGELOG.md)
- [Commits](https://github.com/facebook/react/commits/v19.2.3/packages/react)

Updates `react-dom` from 19.2.1 to 19.2.3
- [Release notes](https://github.com/facebook/react/releases)
- [Changelog](https://github.com/facebook/react/blob/main/CHANGELOG.md)
- [Commits](https://github.com/facebook/react/commits/v19.2.3/packages/react-dom)

---
updated-dependencies:
- dependency-name: react
  dependency-version: 19.2.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: react
- dependency-name: react-dom
  dependency-version: 19.2.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: react
...

Signed-off-by: dependabot[bot] <support@github.com>
 2025-12-18 18:06:01 +00:00
yangdx
6a9e368382 Rename QdrantMigrationError to DataMigrationError for generalization 2025-12-16 15:56:10 +08:00
yangdx
1b62ec9af5 refactor(Qdrant): simplify suffix generation and improve migration logic 
- Move suffix generation logic to BaseVectorStorage._generate_collection_suffix()
- Remove EmbeddingFunc.get_model_identifier() and unused abstract methods
- Qdrant: raise error on dimension mismatch, disable auto-deletion of legacy collections
- Update tests accordingly

BREAKING CHANGE: Qdrant dimension mismatch raises error; legacy collections require manual cleanup
 2025-12-16 12:33:17 +08:00
dependabot[bot]
a8bc957610 Bump the github-actions group with 3 updates 
Bumps the github-actions group with 3 updates: [actions/cache](https://github.com/actions/cache), [actions/upload-artifact](https://github.com/actions/upload-artifact) and [actions/download-artifact](https://github.com/actions/download-artifact).


Updates `actions/cache` from 4 to 5
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/v4...v5)

Updates `actions/upload-artifact` from 5 to 6
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v5...v6)

Updates `actions/download-artifact` from 6 to 7
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](https://github.com/actions/download-artifact/compare/v6...v7)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
- dependency-name: actions/upload-artifact
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
- dependency-name: actions/download-artifact
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
 2025-12-14 18:09:18 +00:00
yangdx
19ab979a9c Merge branch 'main' into feature/vector-model-isolation 2025-12-12 10:28:59 +08:00
yangdx
9562a974d2 Bump API version to 0260 2025-12-12 10:13:34 +08:00
yangdx
f254ccae8e Merge branch 'fix/missing-langue-in-keywords-extraction-prompt' 2025-12-12 10:11:56 +08:00
yangdx
90984c6061 Merge branch 'fix/openai-prompt-caching-optimization-2355' 2025-12-12 06:13:49 +08:00
yangdx
834778eb01 Reorganize entity extraction prompt for better clarity 
- Move instructions before data section
- Update task description wording
 2025-12-12 06:12:47 +08:00