- Introduced `inferMimeType` utility to improve MIME type detection for uploaded files, including support for HEIC and HEIF formats.
- Updated DragDropModal to utilize the new inference logic for validating file types, ensuring compatibility with various document upload providers.
- Added comprehensive tests for `inferMimeType` to cover various scenarios, including handling of unknown extensions and preserving browser-provided types.
* 🔧 fix: Gemini as Custom Endpoint Auth. Error for OAI-compatible API
* refactor: Google Compatibility in OpenAI Config
- Added a test to ensure `googleSearch` is filtered out when `web_search` is only present in `modelOptions`, not in `addParams` or `defaultParams`.
- Updated `transformToOpenAIConfig` to preserve `googleSearch` tools if `web_search` is explicitly enabled via `addParams` or `defaultParams`.
- Refactored the filtering logic for Google-specific tools to accommodate the new behavior.
* refactor: implement sanitizeFileForTransmit and sanitizeMessageForTransmit functions for smaller payload to client transmission
* refactor: enhance sanitizeMessageForTransmit to preserve empty files array and avoid mutating original message
* refactor: update sanitizeMessageForTransmit to ensure immutability of files array and improve test clarity
* refactor: Token Limit Processing with Enhanced Efficiency
- Added a new test suite for `processTextWithTokenLimit`, ensuring comprehensive coverage of various scenarios including under, at, and exceeding token limits.
- Refactored the `processTextWithTokenLimit` function to utilize a ratio-based estimation method, significantly reducing the number of token counting function calls compared to the previous binary search approach.
- Improved handling of edge cases and variable token density, ensuring accurate truncation and performance across diverse text inputs.
- Included direct comparisons with the old implementation to validate correctness and efficiency improvements.
* refactor: Remove Tokenizer Route and Related References
- Deleted the tokenizer route from the server and removed its references from the routes index and server files, streamlining the API structure.
- This change simplifies the routing configuration by eliminating unused endpoints.
* refactor: Migrate countTokens Utility to API Module
- Removed the local countTokens utility and integrated it into the @librechat/api module for centralized access.
- Updated various files to reference the new countTokens import from the API module, ensuring consistent usage across the application.
- Cleaned up unused references and imports related to the previous countTokens implementation.
* refactor: Centralize escapeRegExp Utility in API Module
- Moved the escapeRegExp function from local utility files to the @librechat/api module for consistent usage across the application.
- Updated imports in various files to reference the new centralized escapeRegExp function, ensuring cleaner code and reducing redundancy.
- Removed duplicate implementations of escapeRegExp from multiple files, streamlining the codebase.
* refactor: Enhance Token Counting Flexibility in Text Processing
- Updated the `processTextWithTokenLimit` function to accept both synchronous and asynchronous token counting functions, improving its versatility.
- Introduced a new `TokenCountFn` type to define the token counting function signature.
- Added comprehensive tests to validate the behavior of `processTextWithTokenLimit` with both sync and async token counting functions, ensuring consistent results.
- Implemented a wrapper to track call counts for the `countTokens` function, optimizing performance and reducing unnecessary calls.
- Enhanced existing tests to compare the performance of the new implementation against the old one, demonstrating significant improvements in efficiency.
* chore: documentation for Truncation Safety Buffer in Token Processing
- Added a safety buffer multiplier to the character position estimates during text truncation to prevent overshooting token limits.
- Updated the `processTextWithTokenLimit` function to utilize the new `TRUNCATION_SAFETY_BUFFER` constant, enhancing the accuracy of token limit processing.
- Improved documentation to clarify the rationale behind the buffer and its impact on performance and efficiency in token counting.
- Updated `useChatFunctions` to derive `iconURL` from conversation data before parsing
- Modified `parseCompactConvo` to explicitly omit `iconURL` from parsed conversations, reinforcing security measures around URL handling.
* 🖥️ feat: Add Proxy Support for Tavily API Tool
- Integrated ProxyAgent from undici to enable proxy support for API requests in TavilySearch and TavilySearchResults.
- Updated fetch options to conditionally include the proxy configuration based on the environment variable, enhancing flexibility for network requests.
* ci: TavilySearchResults with Proxy Support Tests
- Added tests to verify the integration of ProxyAgent for API requests in TavilySearchResults.
- Implemented conditional logic to check for the PROXY environment variable, ensuring correct usage of ProxyAgent based on its presence.
- Updated test setup to clear mocks before each test for improved isolation and reliability.
* refactor: Add back user variable descriptions for MCP under input and not as Tooltips
- Integrated DOMPurify to sanitize HTML content in user variable descriptions, ensuring safe rendering of links and formatting.
- Updated the AuthField component to display sanitized descriptions, enhancing security and user experience.
- Removed TooltipAnchor in favor of direct label rendering for improved clarity.
* 📦 chore: Update `dompurify` to v3.3.0 in package dependencies
- Added `dompurify` version 3.3.0 to `package.json` and `package-lock.json` for improved HTML sanitization.
- Updated existing references to `dompurify` to ensure consistency across the project.
* refactor: Update tooltip styles for sanitized description in AuthField component
* 🗨️ fix: Safe Validation for Prompt Updates
- Added `safeValidatePromptGroupUpdate` function to validate and sanitize prompt group update requests, ensuring only allowed fields are processed and sensitive fields are stripped.
- Updated the `patchPromptGroup` route to utilize the new validation function, returning appropriate error messages for invalid requests.
- Introduced comprehensive tests for the validation logic, covering various scenarios including allowed and disallowed fields, enhancing overall request integrity and security.
- Created a new schema file for prompt group updates, defining validation rules and types for better maintainability.
* 🔒 feat: Add JSON parse error handling middleware
* 🔒 fix: `iconURL` in conversation parsing
- Updated the `buildEndpointOption` middleware to derive `iconURL` from model specs when not provided by the client, improving security by preventing malicious URLs.
- Modified the `parseCompactConvo` function to strip `iconURL` from conversation inputs, ensuring it is only set server-side.
- Added comprehensive tests to validate the stripping of `iconURL` across various endpoint types, enhancing overall input sanitization.
* ✨ feat: Add ESLint rule for unused variables
- Introduced a new ESLint rule to warn about unused variables, allowing for better code quality and maintainability.
- Configured the rule to ignore variables and arguments that start with an underscore, accommodating common coding practices.
* 🔧 chore: Update @librechat/agents to version 3.0.35
* ✨ feat: Add DeepSeek Model Pricing and Token Handling
- Introduced pricing and token limits for 'deepseek-chat' and 'deepseek-reasoner' models, including prompt and completion rates.
- Enhanced tests to validate pricing and token limits for DeepSeek models, ensuring correct handling of model variations and provider prefixes.
- Updated cache multipliers for DeepSeek models to reflect new pricing structure.
- Improved max output token handling for DeepSeek models, ensuring consistency across different endpoints.
* 🔧 fix: Handle null content parts in message processing
- Added checks to filter out null content parts in various message handling functions, ensuring robustness against undefined values.
- Updated the `extractMessageContent`, `useContentHandler`, `useEventHandlers`, and `useStepHandler` hooks to prevent errors caused by null parts.
- Enhanced the `getAllContentText` utility to only include valid content types, improving overall message integrity.
* 🔧 fix: Enhance error handling in event and SSE handlers
- Wrapped critical sections in try-catch blocks within `useEventHandlers` and `useSSE` hooks to improve error management and prevent application crashes.
- Added console error logging for better debugging and tracking of issues during message processing and conversation aborting.
- Ensured that UI states like `setIsSubmitting` and `setShowStopButton` are correctly updated in case of errors, maintaining a consistent user experience.
* 🔧 fix: Filter out null and empty content in message export
- Enhanced the `useExportConversation` hook to filter out null content parts and empty strings during message processing, ensuring only valid content is included in the export.
- This change improves the integrity of exported conversations by preventing unnecessary empty entries in the output.
