* Update tox.ini to document libmagic1 system dependency requirement
* [CI] Consolidate test reports into single PR comment
This change reduces noise in PR comments by combining multiple test
reports (runner and sdk1) into a single consolidated comment.
Changes:
- Add combine-test-reports.sh script to merge test reports
- Update CI workflow to combine reports before posting
- Replace separate runner/sdk1 comment steps with single combined step
- Summary section shows test counts (passed/failed/total) collapsed by default
- Full detailed reports available in collapsible sections
- Simplify job summary to use combined report
Benefits:
- Single PR comment instead of multiple separate comments
- Cleaner PR comment section with less clutter
- Easy-to-read summary with detailed inspection on demand
- Maintains all existing test information
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix test count extraction in combine-test-reports script
The previous version was using text pattern matching which incorrectly
extracted test counts. This fix properly parses the pytest-md-report
markdown table format by:
- Finding the table header row to determine column positions
- Locating the TOTAL row (handles both TOTAL and **TOTAL** formatting)
- Extracting values from the passed, failed, and SUBTOTAL columns
- Using proper table parsing instead of pattern matching
This resolves the issue where the summary showed incorrect counts
(23 for both runner and sdk1 instead of the actual 11 and 66).
Fixes: Test count summary in PR comments now shows correct values
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix LaTeX formatting in pytest-md-report test count extraction
pytest-md-report wraps all table values in LaTeX formatting:
$$\textcolor{...}{\tt{VALUE}}$$
The previous fix attempted to parse the table but didn't handle
the LaTeX formatting, causing it to extract 0 for all counts.
Changes:
- Add strip_latex() function to extract values from LaTeX wrappers
- Update grep pattern to match TOTAL row specifically (not SUBTOTAL in header)
- Apply LaTeX stripping to all extracted values before parsing
This fix was tested locally with tox-generated reports and correctly
shows: Runner=11 passed, SDK1=66 passed
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* UN-2793 [FEAT] Add retry logic with exponential backoff to SDK1
Implemented automatic retry logic for platform and prompt service calls
with configurable exponential backoff, comprehensive test coverage, and
CI integration.
Features:
- Exponential backoff with jitter for transient failures
- Configurable via environment variables (MAX_RETRIES, MAX_TIME, BASE_DELAY, etc.)
- Retries ConnectionError, Timeout, HTTPError (502/503/504), OSError
- 67 tests with 100% pass rate
- CI integration with test reporting
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* [SECURITY] Use full commit SHA for sticky-pull-request-comment action
Replace tag reference with full commit SHA for better security:
- marocchino/sticky-pull-request-comment@v2 → @7737449 (v2.9.4)
This prevents potential supply chain attacks where tags could be moved
to point to malicious code. Commit SHAs are immutable.
Fixes SonarQube security hotspot for external GitHub action.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* [FIX] Allow retryable HTTP errors (502/503/504) to propagate for retry
Fixed HTTPError handling in _get_adapter_configuration to check status
codes and re-raise retryable errors (502, 503, 504) so the retry
decorator can handle them. Non-retryable errors are still converted
to SdkError as before.
Changes:
- Check HTTPError status code before converting to SdkError
- Re-raise HTTPError for 502/503/504 to allow retry decorator to retry
- Added parametrized test for all retryable status codes (502, 503, 504)
- All 12 platform tests pass
This fixes a bug where 502/503/504 errors were not being retried
because they were converted to SdkError before the retry decorator
could see them.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* [FIX] Use pytest.approx() for floating point comparisons in tests
Replaced direct equality comparisons (==) with pytest.approx() for
floating point values to avoid precision issues and satisfy SonarQube
code quality check (python:S1244).
Changes in test_retry_utils.py:
- test_exponential_backoff_without_jitter: Use pytest.approx() for 1.0, 2.0, 4.0, 8.0
- test_max_delay_cap: Use pytest.approx() for 5.0
This is the proper way to compare floating point values in tests,
accounting for floating point precision limitations.
All 4 TestCalculateDelay tests pass.
Fixes SonarQube: python:S1244 - Do not perform equality checks with
floating point values.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* minor: Addressed code smells, ruff fixes
* misc: Fixed tox config for sdk1 tests
* misc: Ruff issues fixed
* misc: tox tests fixed
* prompt service lock file for venv
* updated lock files for backend and prompt-service
* UN-2793 [FEAT] Update to unstract-sdk v0.78.0 with retry logic support (#1567)
[FEAT] Update unstract-sdk to v0.78.0 across all services and tools
- Updated unstract-sdk dependency from v0.77.3 to v0.78.0 in all pyproject.toml files
- Main repository, backend, workers, platform-service, prompt-service
- filesystem and tool-registry modules
- Updated tool requirements.txt files (structure, classifier, text_extractor)
- Bumped tool versions in properties.json:
- Structure tool: 0.0.88 → 0.0.89
- Classifier tool: 0.0.68 → 0.0.69
- Text extractor tool: 0.0.64 → 0.0.65
- Updated tool versions in backend/sample.env and public_tools.json
- Regenerated all uv.lock files with new SDK version
This update brings in the retry logic with exponential backoff from unstract-sdk v0.78.0
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude <noreply@anthropic.com>
---------
Signed-off-by: Chandrasekharan M <117059509+chandrasekharan-zipstack@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
[FEAT] Add path ignore patterns to CI workflow
Skip CI runs for changes to docker/, frontend/, and docs/ directories
to optimize CI resource usage and reduce unnecessary test runs.
* feat(ci): add latest tag along with specific tag for Docker image builds
- Modified docker-tools-build-push.yaml workflow to tag images with both specific tag and latest
- Ensures latest tag always points to most recently built image
- Maintains backward compatibility with existing versioning
- Follows Docker best practices for image tagging
Fixes UN-2678
* feat(ci): exclude latest tag for rc and snapshot builds
- Add conditional logic to determine Docker tags based on input tag
- Skip latest tag when tag contains 'rc' or 'snapshot' keywords
- Prevents release candidates and snapshot builds from being tagged as latest
- Maintains latest tag only for stable releases
Relates to UN-2678
* feat(ci): replace conditional logic with user-controlled checkbox for latest tag
- Add 'add_latest_tag' boolean input to workflow with default true
- Replace hardcoded rc/snapshot detection with explicit user control
- Provides better flexibility for users to decide when to tag as latest
- Maintains backward compatibility with default checked behavior
Improves UN-2678
Fix JSON parsing error when PR titles contain double quotes by using toJSON() function to properly escape release body content.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* UN-2544 [FIX] Fix build notifications and add individual service tracking
- Fixed incorrect success/failure detection in build-summary job
- Added individual service status tracking using artifacts
- Removed problematic timestamp fields from Slack notifications
- Enhanced GitHub workflow summary with detailed service status table
- Added cache information to build summary
- Shows specific failed services in both GitHub summary and Slack notifications
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* UN-2544 [REFINE] Remove caching details and optimize notification content
- Removed caching information from GitHub summary table and artifacts
- Simplified service status table to focus on success/failure only
- Created shared content generator for consistent GitHub and Slack messaging
- Optimized Slack notification to show concise build details
- Maintained artifact-based individual service tracking for accuracy
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* UN-2544 [MISC] Consolidate build notifications into single summary
- Remove individual Slack notifications for each service build
- Add build-summary job that runs after all builds complete
- Generate GitHub workflow summary with build status and service list
- Send single consolidated Slack notification with overall build results
- Reduces notification spam from 6 messages to 1 summary message
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* UN-2544 [MISC] Add Slack notifications for releases and Docker builds
- Added release-notification.yaml workflow to send Slack notifications when releases are created
- Enhanced production-build.yaml with success/failure Slack notifications for Docker builds
- Notifications include relevant details like service name, tag, repository, and workflow run URLs
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* UN-2544 [SECURITY] Switch to official Slack GitHub action
- Replace third-party 8398a7/action-slack@v3 with official slackapi/slack-github-action@v2.1.0
- Update payload structure to match official action format
- Improve security by using verified Slack-maintained action
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* [MISC] Fixed build workflow involving multiple tags
* [MISC] Added option to build workflow to accept latest tag
* [MISC] Avoid push of image to test build workflow changes
* [MISC] Uncommented push option of bake that was added for testing
* [misc] Updated workflows to use github's cache for docker builds
* minor: Updated workflows to not push images for testing
* [MISC] Updated workflow build to use correct Dockerfile for services
* minor: Updated workflows to push images again post testing
* [MISC] Refactored build push workflows to use, marked push to false for testing
* [MISC] Fixed production build workflow issue related to VERSION env and context resolution
* [MISC] Refactored build workflow to simplify tag resolution
* [MISC] Moved commented line in build workflow
* [MISC] Uncomment push option in workflow runs that was added for testing
* [MISC] Updated build definition to add tools in a separate profile, removed related fix added to run-platform.sh, updated Github tool build workflow to also use bake
* [MISC] Updated Dockerfile paths and context for tools to make workflow build easily
* Revert "[MISC] Updated build definition to add tools in a separate profile, removed related fix added to run-platform.sh, updated Github tool build workflow to also use bake"
This reverts commit 9e7fe1cbe123dfd1c1466e1082b0b385b278d4bb.
* python version updated from 3.9 into 3.12
* x2text-service updated with uv and python version 3.12
* x2text-service docker file updated
* Unstract packages updated with uv
* Runner updated with uv
* Promptservice updated with uv
* Platform service updated with uv
* backend service updated with uv
* root pyproject.toml file updated
* sdk version updated in services
* unstract package modules updated based on sdk version:
* docker file update
* pdm lock workflow modified to support uv
* Docs updated based on uv support
* lock automation updated
* snowflake module version updated into 3.14.0
* tox updated to support UV
* tox updated to support UV
* tox updated with pytest
* tox updated with pytest-md-report
* tox updated with module requirements
* python migration from 3.9 to 3.12
* tox updated with module requirements
* runner updated
* Commit uv.lock changes
* runner updated
* Commit uv.lock changes
* pytest.ini added
* x2text-service docker file updated
* pytest.ini removed
* environment updated to test
* docformatter commented on pre-commit
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* some pre-commit issues ignored
* some pre-commit issues ignored
* some pre-commit issues ignored
* some pre-commit issues ignored
* some pre-commit issues ignored
* pre-commit updates
* un used import removed from platfrom service controller
* tox issue fixed
* tox issue fixed
* docker files updated
* backend dockerfile updated
* open installation issue fixed
* Tools docker file updated with base python version 3.12
* python version updated into min 3.12 in pyproject.toml
* linting issue fixed
* uv version upgraded into 0.6.14
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* migrations excluded from ruff
* added PoethePoet task runner
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* feat: Added poe tasks for services (#1248)
* Added poe tasks for services
* reverted FE change made by mistake
* updated tool-sidecar to uv and python to 3.12.9
* minor updates in pyproject descreption
* feat: platform-service logging improvements (#1255)
feat: Used flask util from core to improve logging in platform-service, added core as a dependency to platform-service:
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: Platform-service build issue and numpy issue with Python 3.12 (#1258)
* fix: Platform-service build and numpy issue with Py 3.12
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: Removed backend dockerfile install statements for numpy
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* minor: Handled scenario when cost is not calculated due to no usage
* minor: Corrected content shown for workflow input
* fix: Minor fixes, used gthread for prompt-service, runner
* Commit uv.lock changes
* Removed unused line in tool dockerfile
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Chandrasekharan M <chandrasekharan@zipstack.com>
Co-authored-by: Chandrasekharan M <117059509+chandrasekharan-zipstack@users.noreply.github.com>
Co-authored-by: ali-zipstack <muhammad.ali@zipstack.com>
* reolved file skipping while API-deloyment execution
* sidecar implementation
* update docker build yaml
* tool version updates
* remove docker tool-sidecar build
* minor updates on runner
* update docker-tool-build workflow with sidecar
* update docker-tool-build workflow with sidecar
* update docker-tool-build workflow with sidecar
* Commit pdm.lock changes
* removed tool-sidecar from workflow build
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* validation on imagename exist and minor changes in docker file
* format fixes
* tool dockerfile cmd to entrypoint
* updated tool version in public tools
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* * Add issue template config
* Improve readme readability
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Hari John Kuriakose <ejhari@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Lock file script changes to fail fast
* Single quoted dir names in lock file script's print statements
* Updated delimiter for sed in lock file script to support / in dir path
* Fix to lock file automation workflow - actionlint-docker related involving shellcheck
---------
Co-authored-by: Ritwik G <100672805+ritwik-g@users.noreply.github.com>
* Added script to update lock files, update pdm-lock-automation.yaml workflow
* Support PDM locking in parallel in the script
* Support PDM locking in parallel in the script - missed changes
---------
Co-authored-by: Ritwik G <100672805+ritwik-g@users.noreply.github.com>
* Allow mounting tool registry in OSS to load private tools too
* Minor README update for tool registry related env
* Minor log update to include tool name and version loaded
* Added unit test setup for worker
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Removed coverage upload
* Fixed flake8 issue and corrected the comment file path
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Changed comment input
* Updated workflow file
* Updated workflow file
* Updated workflow file
* Added markdown report
* Added updated tox.ini
* Hardcoded file name in workflow
* Replaced commenting with summary
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Added sticky comment
* Removed other comment
* Added build on merge to main also
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Kirtiman Mishra <110175055+kirtimanmishrazipstack@users.noreply.github.com>
* Added migration to delete incorrect index keys, updated index key generation invocation in prompt service
* Minor PR template fix
* Removed file hashing logic in backend (this was moved to SDK)
* SDK bumped to 0.24.0 and updated tools
* SDK bumped to 0.24.0 and updated tools
* Tool registry updates for OSS
* Updated structure tool to 0.0.18
* Minor code smell fix
* Reverted classifier icon SVG - minor ID change (search defs8113)
---------
Co-authored-by: Neha <115609453+nehabagdia@users.noreply.github.com>
* Add tweaks to run script
* Update readme
* Fix user ownership of build context
* Add db migrations and env changes to pr template
---------
Co-authored-by: Neha <115609453+nehabagdia@users.noreply.github.com>
