- Introduced `inferMimeType` utility to improve MIME type detection for uploaded files, including support for HEIC and HEIF formats.
- Updated DragDropModal to utilize the new inference logic for validating file types, ensuring compatibility with various document upload providers.
- Added comprehensive tests for `inferMimeType` to cover various scenarios, including handling of unknown extensions and preserving browser-provided types.
* 🔧 fix: Gemini as Custom Endpoint Auth. Error for OAI-compatible API
* refactor: Google Compatibility in OpenAI Config
- Added a test to ensure `googleSearch` is filtered out when `web_search` is only present in `modelOptions`, not in `addParams` or `defaultParams`.
- Updated `transformToOpenAIConfig` to preserve `googleSearch` tools if `web_search` is explicitly enabled via `addParams` or `defaultParams`.
- Refactored the filtering logic for Google-specific tools to accommodate the new behavior.
* refactor: implement sanitizeFileForTransmit and sanitizeMessageForTransmit functions for smaller payload to client transmission
* refactor: enhance sanitizeMessageForTransmit to preserve empty files array and avoid mutating original message
* refactor: update sanitizeMessageForTransmit to ensure immutability of files array and improve test clarity
* refactor: Token Limit Processing with Enhanced Efficiency
- Added a new test suite for `processTextWithTokenLimit`, ensuring comprehensive coverage of various scenarios including under, at, and exceeding token limits.
- Refactored the `processTextWithTokenLimit` function to utilize a ratio-based estimation method, significantly reducing the number of token counting function calls compared to the previous binary search approach.
- Improved handling of edge cases and variable token density, ensuring accurate truncation and performance across diverse text inputs.
- Included direct comparisons with the old implementation to validate correctness and efficiency improvements.
* refactor: Remove Tokenizer Route and Related References
- Deleted the tokenizer route from the server and removed its references from the routes index and server files, streamlining the API structure.
- This change simplifies the routing configuration by eliminating unused endpoints.
* refactor: Migrate countTokens Utility to API Module
- Removed the local countTokens utility and integrated it into the @librechat/api module for centralized access.
- Updated various files to reference the new countTokens import from the API module, ensuring consistent usage across the application.
- Cleaned up unused references and imports related to the previous countTokens implementation.
* refactor: Centralize escapeRegExp Utility in API Module
- Moved the escapeRegExp function from local utility files to the @librechat/api module for consistent usage across the application.
- Updated imports in various files to reference the new centralized escapeRegExp function, ensuring cleaner code and reducing redundancy.
- Removed duplicate implementations of escapeRegExp from multiple files, streamlining the codebase.
* refactor: Enhance Token Counting Flexibility in Text Processing
- Updated the `processTextWithTokenLimit` function to accept both synchronous and asynchronous token counting functions, improving its versatility.
- Introduced a new `TokenCountFn` type to define the token counting function signature.
- Added comprehensive tests to validate the behavior of `processTextWithTokenLimit` with both sync and async token counting functions, ensuring consistent results.
- Implemented a wrapper to track call counts for the `countTokens` function, optimizing performance and reducing unnecessary calls.
- Enhanced existing tests to compare the performance of the new implementation against the old one, demonstrating significant improvements in efficiency.
* chore: documentation for Truncation Safety Buffer in Token Processing
- Added a safety buffer multiplier to the character position estimates during text truncation to prevent overshooting token limits.
- Updated the `processTextWithTokenLimit` function to utilize the new `TRUNCATION_SAFETY_BUFFER` constant, enhancing the accuracy of token limit processing.
- Improved documentation to clarify the rationale behind the buffer and its impact on performance and efficiency in token counting.
- Updated `useChatFunctions` to derive `iconURL` from conversation data before parsing
- Modified `parseCompactConvo` to explicitly omit `iconURL` from parsed conversations, reinforcing security measures around URL handling.
* 🖥️ feat: Add Proxy Support for Tavily API Tool
- Integrated ProxyAgent from undici to enable proxy support for API requests in TavilySearch and TavilySearchResults.
- Updated fetch options to conditionally include the proxy configuration based on the environment variable, enhancing flexibility for network requests.
* ci: TavilySearchResults with Proxy Support Tests
- Added tests to verify the integration of ProxyAgent for API requests in TavilySearchResults.
- Implemented conditional logic to check for the PROXY environment variable, ensuring correct usage of ProxyAgent based on its presence.
- Updated test setup to clear mocks before each test for improved isolation and reliability.
