10 Commits

Author SHA1 Message Date
Athul
1897ee621e UN-2679 [FEAT] Add support for external usage tracking plugins (#1496)
* feat(UN-2679): Add support for external usage tracking plugins

- Pass request metadata through workflow execution chain
- Add execution context passing for plugin systems
- Support caching of request metadata during async execution

This enables the workflow execution system to pass context data
to optional plugins. All changes are backward compatible.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* refactor(UN-2679): Update API usage tracking with review feedback

- Pass request object to plugin instead of exposing headers in OSS
- Improve error handling and logging for resilience
- Remove internal implementation comments
- Make header caching non-fatal with proper error messages

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* UN-2679 Pass request.META instead of full request to plugin

Improve security by passing only request.META dictionary to the plugin
instead of the entire request object. This limits what the plugin can access
and provides a cleaner interface.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* UN-2679 Optimize to pass only request.META instead of full request

Since the request parameter was introduced in this PR for API hub usage tracking,
optimize it to only pass request.META which is all that's needed. This provides:
- Better interface clarity (parameter name shows what's needed)
- Improved security (no full request object passed around)
- Cleaner separation of concerns

Changes:
- api_deployment_views.py: Pass request.META instead of full request
- deployment_helper.py: Accept request_meta parameter instead of request

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* UN-2679 Clean up API hub usage tracking and switch to request.headers

Major improvements to API hub usage tracking:
- Switch from request.META to request.headers following doc-splitter pattern
- Simplify Celery task to call store_usage() directly with no header conversion
- Eliminate double header format conversion for better performance

Changes:
- api_deployment_views.py: Pass dict(request.headers) instead of request.META
- deployment_helper.py: Accept request_headers parameter
- file_execution_tasks.py: Direct store_usage() call, no double conversion

Performance: Eliminates unnecessary header format conversion
Security: Only passes required headers, not full request object

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Refactoring

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* improvements

* Updated usage_metrics to get page_count aloen

* Updated usage_metrics to page_count

* updated logic for fiel based usage tracking

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Refactoring as per git comments

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Removign unwanted code

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2025-08-26 15:53:49 +05:30
vishnuszipstack
6f23ce9aca UN-2417 [FEAT] HITL TTL (#1441)
* queue utils and destinatino changes for hitl ttl

* moved hitl setting ttl getting logic to plugins

* sonar issue fix

* pr comment fixes

* pr comment fixes

* pr comment fixes

* added utils

* pr comment fix
2025-08-08 10:22:19 +05:30
Jaseem Jas
ba1df894d2 Python 3.9 to 3.12 (#1231)
* python version updated from 3.9 into 3.12

* x2text-service updated with uv and python version 3.12

* x2text-service docker file updated

* Unstract packages updated with uv

* Runner updated with uv

* Promptservice updated with uv

* Platform service updated with uv

* backend service updated with uv

* root pyproject.toml file updated

* sdk version updated in services

* unstract package modules updated based on sdk version:

* docker file update

* pdm lock workflow modified to support uv

* Docs updated based on uv support

* lock automation updated

* snowflake module version updated into 3.14.0

* tox updated to support UV

* tox updated to support UV

* tox updated with pytest

* tox updated with pytest-md-report

* tox updated with module requirements

* python migration from 3.9 to 3.12

* tox updated with module requirements

* runner updated

* Commit uv.lock changes

* runner updated

* Commit uv.lock changes

* pytest.ini added

* x2text-service docker file updated

* pytest.ini removed

* environment updated to test

* docformatter commented on pre-commit

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* some pre-commit issues ignored

* some pre-commit issues ignored

* some pre-commit issues ignored

* some pre-commit issues ignored

* some pre-commit issues ignored

* pre-commit updates

* un used import removed from platfrom service controller

* tox issue fixed

* tox issue fixed

* docker files updated

* backend dockerfile updated

* open installation issue fixed

* Tools docker file updated with base python version 3.12

* python version updated into min 3.12 in pyproject.toml

* linting issue fixed

* uv version upgraded into 0.6.14

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* migrations excluded from ruff

* added PoethePoet task runner

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* feat: Added poe tasks for services (#1248)

* Added poe tasks for services

* reverted FE change made by mistake

* updated tool-sidecar to uv and python to 3.12.9

* minor updates in pyproject descreption

* feat: platform-service logging improvements (#1255)

feat: Used flask util from core to improve logging in platform-service, added core as a dependency to platform-service:

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: Platform-service build issue and numpy issue with Python 3.12 (#1258)

* fix: Platform-service build and numpy issue with Py 3.12

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: Removed backend dockerfile install statements for numpy

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* minor: Handled scenario when cost is not calculated due to no usage

* minor: Corrected content shown for workflow input

* fix: Minor fixes, used gthread for prompt-service, runner

* Commit uv.lock changes

* Removed unused line in tool dockerfile

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Chandrasekharan M <chandrasekharan@zipstack.com>
Co-authored-by: Chandrasekharan M <117059509+chandrasekharan-zipstack@users.noreply.github.com>
Co-authored-by: ali-zipstack <muhammad.ali@zipstack.com>
2025-04-24 16:07:02 +05:30
Deepak K
ba48b87454 Prompt service v2 refactor (#1149)
* Prompt service v2 initial changes

Signed-off-by: Deepak <89829542+Deepak-Kesavan@users.noreply.github.com>

* Commit pdm.lock changes

* Prompt service v2 additional changes

Signed-off-by: Deepak <89829542+Deepak-Kesavan@users.noreply.github.com>

* Lock file update

Signed-off-by: Deepak <89829542+Deepak-Kesavan@users.noreply.github.com>

* Commit pdm.lock changes

* Minor fixes

Signed-off-by: Deepak <89829542+Deepak-Kesavan@users.noreply.github.com>

* Minor fixes

Signed-off-by: Deepak <89829542+Deepak-Kesavan@users.noreply.github.com>

* Removed unwanted code

Signed-off-by: Deepak <89829542+Deepak-Kesavan@users.noreply.github.com>

* Update prompt-service/src/unstract/prompt_service_v2/helper/usage_helper.py

Co-authored-by: Chandrasekharan M <117059509+chandrasekharan-zipstack@users.noreply.github.com>
Signed-off-by: Deepak K <89829542+Deepak-Kesavan@users.noreply.github.com>

* Minor fixes

Signed-off-by: Deepak <89829542+Deepak-Kesavan@users.noreply.github.com>

* Code and logger improvements

Signed-off-by: Deepak <89829542+Deepak-Kesavan@users.noreply.github.com>

* Code and logger improvements

Signed-off-by: Deepak <89829542+Deepak-Kesavan@users.noreply.github.com>

* Pass request id from backend to prompt service

Signed-off-by: Deepak <89829542+Deepak-Kesavan@users.noreply.github.com>

* Minor fix

Signed-off-by: Deepak <89829542+Deepak-Kesavan@users.noreply.github.com>

* Removed unwanted code

Signed-off-by: Deepak <89829542+Deepak-Kesavan@users.noreply.github.com>

* Updated .gitignore

Signed-off-by: Deepak <89829542+Deepak-Kesavan@users.noreply.github.com>

* Sync changes to v2 folder

Signed-off-by: Deepak <89829542+Deepak-Kesavan@users.noreply.github.com>

* Code improvements

Signed-off-by: Deepak <89829542+Deepak-Kesavan@users.noreply.github.com>

* Minor fix

Signed-off-by: Deepak <89829542+Deepak-Kesavan@users.noreply.github.com>

* Fixes in code

Signed-off-by: Deepak <89829542+Deepak-Kesavan@users.noreply.github.com>

* Changed to flask logger

Signed-off-by: Deepak <89829542+Deepak-Kesavan@users.noreply.github.com>

* Removed file postfix

Signed-off-by: Deepak <89829542+Deepak-Kesavan@users.noreply.github.com>

* Minor fix

Signed-off-by: Deepak <89829542+Deepak-Kesavan@users.noreply.github.com>

* Handled minor issues

Signed-off-by: Deepak <89829542+Deepak-Kesavan@users.noreply.github.com>

* Handled minor issues

Signed-off-by: Deepak <89829542+Deepak-Kesavan@users.noreply.github.com>

* Reused flask utils from unstract core

Signed-off-by: Deepak <89829542+Deepak-Kesavan@users.noreply.github.com>

* [FIX] Refactored indexing, extraction and retrieval (#1172)

* REFACTOR: Indexing API for prompt service

* Refactor

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Indexing changes

* Integration Tests for API

* Skipping VectorDB for zero chunks

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Addressing review comments

* Adding helper functions

* Typo fixes

* Invoking index utils

* Invoking index utils

* Fixing nits

* Fixing sonar issues

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* integration fixes

* back merging answer prompt 

Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added dev comments

* Added dev comments

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added dev comments

* Added dev comments

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Deepak K <89829542+Deepak-Kesavan@users.noreply.github.com>
Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com>
Co-authored-by: Deepak K <89829542+Deepak-Kesavan@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Commit pdm.lock changes

* Additional fixes

Signed-off-by: Deepak <89829542+Deepak-Kesavan@users.noreply.github.com>

* Fixed imports

Signed-off-by: Deepak <89829542+Deepak-Kesavan@users.noreply.github.com>

* Commit pdm.lock changes

---------

Signed-off-by: Deepak <89829542+Deepak-Kesavan@users.noreply.github.com>
Signed-off-by: Deepak K <89829542+Deepak-Kesavan@users.noreply.github.com>
Signed-off-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com>
Co-authored-by: Chandrasekharan M <117059509+chandrasekharan-zipstack@users.noreply.github.com>
Co-authored-by: harini-venkataraman <115449948+harini-venkataraman@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2025-04-02 10:57:24 +05:30
vishnuszipstack
e365e6ca8a api deployment dto registry to load dto from plugins (#1156) 2025-03-10 10:33:22 +05:30
vishnuszipstack
6a2165f4bf moved workflow manager workflowutils to plugins (#1169) 2025-03-06 13:02:44 +05:30
Kirtiman Mishra
6871fe1d8d fix: flake8 black file changes (#259)
refactoring files based on flake8 and black changes
2024-04-15 15:17:08 +05:30
Ritwik G
43dce88dcf Added precommit badge in readme (#201)
* Added precommit badge in readme

Signed-off-by: Ritwik G <100672805+ritwik-g@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added sonar cloud badges

Signed-off-by: Ritwik G <100672805+ritwik-g@users.noreply.github.com>

* CLA badge added

Signed-off-by: Ritwik G <100672805+ritwik-g@users.noreply.github.com>

* Commented out `language: system` in pre-commit config

Signed-off-by: Ritwik G <100672805+ritwik-g@users.noreply.github.com>

* Update .pre-commit-config.yaml

Signed-off-by: Ritwik G <100672805+ritwik-g@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update .pre-commit-config.yaml

Signed-off-by: Ritwik G <100672805+ritwik-g@users.noreply.github.com>

* Commented out failing checks

Signed-off-by: Ritwik G <100672805+ritwik-g@users.noreply.github.com>

---------

Signed-off-by: Ritwik G <100672805+ritwik-g@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2024-03-27 16:38:59 +05:30
Deepak
6744598e8b Added processor directory 2024-02-28 20:23:08 +05:30
jaseemjaskp
26ebb17d47 Initial commit on Unstract 2024-02-25 16:19:36 +05:30