vasceannie/rag-manager
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

init

Browse Source
This commit is contained in:
Travis Vasceannie
2025-09-15 12:35:42 -04:00
commit 94ddcfeff6
94 changed files with 9583 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

248
docs/elysia.md Normal file
View File

@@ -0,0 +1,248 @@
38 async def output_resources(): │
│ │
│ /Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/elysia/api/services/use │
│ r.py:223 in check_all_trees_timeout │
│ │
│ 220 │ │ Check all trees in all TreeManagers across all users and remove any │
│ not been active in the last tree_timeout. │
│ 221 │ │ """ │
│ 222 │ │ for user_id in self.users: │
│ ❱ 223 │ │ │ self.users[user_id]["tree_manager"].check_all_trees_timeout() │
│ 224 │ │
│ 225 │ def check_user_timeout(self, user_id: str): │
│ 226 │ │ """ │
╰───────────────────────────────────────────────────────────────────────────────────╯
KeyError: 'tree_manager'
[10:08:31] ERROR Job "check_timeouts (trigger: interval[0:00:29], next run at: 2025-09-15 10:09:00 base.py:195
EDT)" raised an exception
╭──────────────────────── Traceback (most recent call last) ────────────────────────╮
│ /Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/apscheduler/executors/b │
│ ase.py:181 in run_coroutine_job │
│ │
│ 178 │ │ │
│ 179 │ │ logger.info('Running job "%s" (scheduled at %s)', job, run_time) │
│ 180 │ │ try: │
│ ❱ 181 │ │ │ retval = await job.func(*job.args, **job.kwargs) │
│ 182 │ │ except BaseException: │
│ 183 │ │ │ exc, tb = sys.exc_info()[1:] │
│ 184 │ │ │ formatted_tb = "".join(format_tb(tb)) │
│ │
│ /Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/elysia/api/app.py:35 in │
│ check_timeouts │
│ │
│ 32 │
│ 33 async def check_timeouts(): │
│ 34 │ user_manager = get_user_manager() │
│ ❱ 35 │ await user_manager.check_all_trees_timeout() │
│ 36 │
│ 37 │
│ 38 async def output_resources(): │
│ │
│ /Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/elysia/api/services/use │
│ r.py:223 in check_all_trees_timeout │
│ │
│ 220 │ │ Check all trees in all TreeManagers across all users and remove any │
│ not been active in the last tree_timeout. │
│ 221 │ │ """ │
│ 222 │ │ for user_id in self.users: │
│ ❱ 223 │ │ │ self.users[user_id]["tree_manager"].check_all_trees_timeout() │
│ 224 │ │
│ 225 │ def check_user_timeout(self, user_id: str): │
│ 226 │ │ """ │
╰───────────────────────────────────────────────────────────────────────────────────╯
KeyError: 'tree_manager'
[10:26:25] WARNING Run time of job "check_restart_clients (trigger: interval[0:00:31], next run at: base.py:176
2025-09-15 10:26:33 EDT)" was missed by 0:00:23.029499
WARNING Run time of job "check_timeouts (trigger: interval[0:00:29], next run at: 2025-09-15 base.py:176
10:26:53 EDT)" was missed by 0:00:01.030848
WARNING Run time of job "output_resources (trigger: interval[0:18:23], next run at: base.py:176
2025-09-15 10:33:44 EDT)" was missed by 0:11:04.063842
[10:41:41] WARNING Run time of job "check_restart_clients (trigger: interval[0:00:31], next run at: base.py:176
2025-09-15 10:42:03 EDT)" was missed by 0:00:09.036380
WARNING Run time of job "check_timeouts (trigger: interval[0:00:29], next run at: 2025-09-15 base.py:176
10:41:52 EDT)" was missed by 0:00:18.037363
WARNING Run time of job "output_resources (trigger: interval[0:18:23], next run at: base.py:176
2025-09-15 10:52:07 EDT)" was missed by 0:07:57.071763
[10:51:25] WARNING Run time of job "check_timeouts (trigger: interval[0:00:29], next run at: 2025-09-15 base.py:176
10:51:32 EDT)" was missed by 0:00:21.808772
WARNING Run time of job "check_restart_clients (trigger: interval[0:00:31], next run at: base.py:176
2025-09-15 10:51:52 EDT)" was missed by 0:00:03.810823
[10:51:32] ERROR Job "check_timeouts (trigger: interval[0:00:29], next run at: 2025-09-15 10:52:01 base.py:195
EDT)" raised an exception
╭──────────────────────── Traceback (most recent call last) ────────────────────────╮
│ /Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/apscheduler/executors/b │
│ ase.py:181 in run_coroutine_job │
│ │
│ 178 │ │ │
│ 179 │ │ logger.info('Running job "%s" (scheduled at %s)', job, run_time) │
│ 180 │ │ try: │
│ ❱ 181 │ │ │ retval = await job.func(*job.args, **job.kwargs) │
│ 182 │ │ except BaseException: │
│ 183 │ │ │ exc, tb = sys.exc_info()[1:] │
│ 184 │ │ │ formatted_tb = "".join(format_tb(tb)) │
│ │
│ /Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/elysia/api/app.py:35 in │
│ check_timeouts │
│ │
│ 32 │
│ 33 async def check_timeouts(): │
│ 34 │ user_manager = get_user_manager() │
│ ❱ 35 │ await user_manager.check_all_trees_timeout() │
│ 36 │
│ 37 │
│ 38 async def output_resources(): │
│ │
│ /Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/elysia/api/services/use │
│ r.py:223 in check_all_trees_timeout │
│ │
│ 220 │ │ Check all trees in all TreeManagers across all users and remove any │
│ not been active in the last tree_timeout. │
│ 221 │ │ """ │
│ 222 │ │ for user_id in self.users: │
│ ❱ 223 │ │ │ self.users[user_id]["tree_manager"].check_all_trees_timeout() │
│ 224 │ │
│ 225 │ def check_user_timeout(self, user_id: str): │
│ 226 │ │ """ │
╰───────────────────────────────────────────────────────────────────────────────────╯
KeyError: 'tree_manager'
[10:51:43] ERROR Unexpected error: 'client_manager' error_handlers.py:32
INFO: 127.0.0.1:50043 - "GET /feedback/metadata/b6c0f65db8197395b453a7777a5e4c44 HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
await super().__call__(scope, receive, send)
File "/Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
await self.middleware_stack(scope, receive, send)
File "/Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__
raise exc
File "/Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
await self.app(scope, receive, _send)
File "/Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
await self.app(scope, receive, send)
File "/Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/starlette/routing.py", line 716, in __call__
await self.middleware_stack(scope, receive, send)
File "/Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/starlette/routing.py", line 736, in app
await route.handle(scope, receive, send)
File "/Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/starlette/routing.py", line 290, in handle
await self.app(scope, receive, send)
File "/Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/starlette/routing.py", line 78, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/starlette/routing.py", line 75, in app
response = await f(request)
^^^^^^^^^^^^^^^^
File "/Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 302, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 213, in run_endpoint_function
return await dependant.call(**values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/elysia/api/routes/feedback.py", line 81, in run_feedback_metadata
client_manager: ClientManager = user["client_manager"]
~~~~^^^^^^^^^^^^^^^^^^
KeyError: 'client_manager'
ERROR HTTP error occurred: Not Found error_handlers.py:14
INFO: 127.0.0.1:50045 - "GET /icon.svg?d6c34577c7161f78 HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:50045 - "GET /user/config/models HTTP/1.1" 200 OK
INFO: 127.0.0.1:50054 - "GET /user/config/models HTTP/1.1" 200 OK
[10:52:01] ERROR Job "check_timeouts (trigger: interval[0:00:29], next run at: 2025-09-15 10:52:30 base.py:195
EDT)" raised an exception
╭──────────────────────── Traceback (most recent call last) ────────────────────────╮
│ /Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/apscheduler/executors/b │
│ ase.py:181 in run_coroutine_job │
│ │
│ 178 │ │ │
│ 179 │ │ logger.info('Running job "%s" (scheduled at %s)', job, run_time) │
│ 180 │ │ try: │
│ ❱ 181 │ │ │ retval = await job.func(*job.args, **job.kwargs) │
│ 182 │ │ except BaseException: │
│ 183 │ │ │ exc, tb = sys.exc_info()[1:] │
│ 184 │ │ │ formatted_tb = "".join(format_tb(tb)) │
│ │
│ /Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/elysia/api/app.py:35 in │
│ check_timeouts │
│ │
│ 32 │
│ 33 async def check_timeouts(): │
│ 34 │ user_manager = get_user_manager() │
│ ❱ 35 │ await user_manager.check_all_trees_timeout() │
│ 36 │
│ 37 │
│ 38 async def output_resources(): │
│ │
│ /Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/elysia/api/services/use │
│ r.py:223 in check_all_trees_timeout │
│ │
│ 220 │ │ Check all trees in all TreeManagers across all users and remove any │
│ not been active in the last tree_timeout. │
│ 221 │ │ """ │
│ 222 │ │ for user_id in self.users: │
│ ❱ 223 │ │ │ self.users[user_id]["tree_manager"].check_all_trees_timeout() │
│ 224 │ │
│ 225 │ def check_user_timeout(self, user_id: str): │
│ 226 │ │ """ │
╰───────────────────────────────────────────────────────────────────────────────────╯
KeyError: 'tree_manager'
^X[10:52:07] ERROR Job "output_resources (trigger: interval[0:18:23], next run at: 2025-09-15 11:10:30 base.py:195
EDT)" raised an exception
╭──────────────────────── Traceback (most recent call last) ────────────────────────╮
│ /Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/apscheduler/executors/b │
│ ase.py:181 in run_coroutine_job │
│ │
│ 178 │ │ │
│ 179 │ │ logger.info('Running job "%s" (scheduled at %s)', job, run_time) │
│ 180 │ │ try: │
│ ❱ 181 │ │ │ retval = await job.func(*job.args, **job.kwargs) │
│ 182 │ │ except BaseException: │
│ 183 │ │ │ exc, tb = sys.exc_info()[1:] │
│ 184 │ │ │ formatted_tb = "".join(format_tb(tb)) │
│ │
│ /Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/elysia/api/app.py:40 in │
│ output_resources │
│ │
│ 37 │
│ 38 async def output_resources(): │
│ 39 │ user_manager = get_user_manager() │
│ ❱ 40 │ await print_resources(user_manager, save_to_file=True) │
│ 41 │
│ 42 │
│ 43 async def check_restart_clients(): │
│ │
│ /Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/elysia/api/utils/resour │
│ ces.py:59 in print_resources │
│ │
│ 56 │ user_manager: UserManager | None = None, save_to_file: bool = False │
│ 57 ): │
│ 58 │ if user_manager is not None: │
│ ❱ 59 │ │ avg_user_memory, avg_tree_memory = await get_average_user_memory(us │
│ 60 │ │ # avg_user_requests = await get_average_user_requests(user_manager) │
│ 61 │ │ # num_users_db = await get_number_local_users_db(user_manager) │
│ 62 │
│ │
│ /Users/trav/dev/elysia/.venv/lib/python3.12/site-packages/elysia/api/utils/resour │
│ ces.py:37 in get_average_user_memory │
│ │
│ 34 │ avg_tree_memory = 0 │
│ 35 │ for user in user_manager.users.values(): │
│ 36 │ │ user_memory = 0 │
│ ❱ 37 │ │ for tree in user["tree_manager"].trees.values(): │
│ 38 │ │ │ user_memory += tree["tree"].detailed_memory_usage()["total"] / │
│ 39 │ │ │
│ 40 │ │ if len(user["tree_manager"].trees) > 0: │
╰───────────────────────────────────────────────────────────────────────────────────╯
KeyError: 'tree_manager'

108
docs/tagging.md Normal file
View File

@@ -0,0 +1,108 @@
Here are clear written examples of **metadata tagging** in both Open WebUI and Weaviate, showing how you can associate tags and structured metadata with knowledge objects for RAG and semantic search.
***
### Example: Metadata Tagging in Open WebUI
You send a document to the Open WebUI API endpoint, attaching metadata and tags in the content field as a JSON string:
```json
POST http://localhost/api/v1/documents/create
Content-Type: application/json
{
"name": "policy_doc_2022",
"title": "2022 Policy Handbook",
"collection_name": "company_handbooks",
"filename": "policy_2022.pdf",
"content": "{\"tags\": [\"policy\", \"2022\", \"hr\"], \"source_url\": \"https://example.com/policy_2022.pdf\", \"author\": \"Jane Doe\"}"
}
```
- The `"tags"` field is a list of labels for classification (policy, 2022, hr).
- The `"source_url"` and `"author"` fields provide additional metadata useful for retrieval, audit, and filtering.[1][2]
For pipeline-based ingestion, you might design a function to extract and append metadata before vectorization:
```python
metadata = {
"tags": ["policy", "2022"],
"source_url": document_url,
"author": document_author
}
embed_with_metadata(chunk, metadata)
```
This metadata becomes part of your retrieval context in RAG workflows.[1]
***
### Example: Metadata Tagging in Weaviate
In Weaviate, metadata and tags are defined directly in the schema and attached to each object when added:
**Schema definition:**
```json
{
"class": "Document",
"properties": [
{"name": "title", "dataType": ["text"]},
{"name": "tags", "dataType": ["text[]"]},
{"name": "source_url", "dataType": ["text"]},
{"name": "author", "dataType": ["text"]}
]
}
```
**Object creation example:**
```python
client.data_object.create(
data_object={
"title": "2022 Policy Handbook",
"tags": ["policy", "2022", "hr"],
"source_url": "https://example.com/policy_2022.pdf",
"author": "Jane Doe"
},
class_name="Document"
)
```
- The `"tags"` field is a text array, ideal for semantic filtering and faceting.
- Other fields store provenance metadata, supporting advanced queries and data governance.[3][4][5]
**Query with metadata filtering:**
```python
result = (
client.query
.get("Document", ["title", "tags", "author"])
.with_filter({"path": ["tags"], "operator": "ContainsAny", "value": ["policy", "hr"]})
.do()
)
```
This retrieves documents classified with either "policy" or "hr" tags.[4][3]
***
Both platforms support **metadata tagging** for documents, which enables powerful RAG scenarios, detailed filtering, and context-rich retrievals.[5][2][3][4][1]
[1](https://www.reddit.com/r/OpenWebUI/comments/1hmmg9a/how_to_handle_metadata_during_vectorization/)
[2](https://github.com/open-webui/open-webui/discussions/4692)
[3](https://stackoverflow.com/questions/75006703/query-large-list-of-metadate-in-weaviate)
[4](https://weaviate.io/blog/enterprise-workflow-langchain-weaviate)
[5](https://docs.weaviate.io/academy/py/zero_to_mvp/schema_and_imports/schema)
[6](https://docs.weaviate.io/weaviate/api/graphql/additional-properties)
[7](https://weaviate.io/blog/sycamore-and-weaviate)
[8](https://docs.llamaindex.ai/en/stable/examples/vector_stores/WeaviateIndex_auto_retriever/)
[9](https://forum.weaviate.io/t/recommendations-for-metadata-or-knowledge-graphs/960)
[10](https://weaviate.io/blog/agent-workflow-automation-n8n-weaviate)
[11](https://github.com/open-webui/open-webui/discussions/9804)
[12](https://docs.quarkiverse.io/quarkus-langchain4j/dev/rag-weaviate.html)
[13](https://github.com/weaviate/weaviate-examples)
[14](https://docs.openwebui.com/getting-started/api-endpoints/)
[15](https://weaviate.io/blog/hybrid-search-for-web-developers)
[16](https://dev.to/stephenc222/how-to-use-weaviate-to-store-and-query-vector-embeddings-4b9b)
[17](https://helpdesk.egnyte.com/hc/en-us/articles/360035813612-Using-Metadata-in-the-WebUI)
[18](https://docs.datadoghq.com/integrations/weaviate/)
[19](https://docs.openwebui.com/features/)
[20](https://documentation.suse.com/suse-ai/1.0/html/openwebui-configuring/index.html)
[21](https://docs.openwebui.com/getting-started/env-configuration/)