ef9d9b12763568c8fbed999fa1cda29d183bf08d
7 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Danny Avila
|
842b68fc32 |
🏗️ fix: Agents Token Spend Race Conditions, Add Auto-refill Tx, Add Relevant Tests (#6480)
* 🏗️ refactor: Improve spendTokens logic to handle zero completion tokens and enhance test coverage * 🏗️ test: Add tests to ensure balance does not go below zero when spending tokens * 🏗️ fix: Ensure proper continuation in AgentClient when handling errors * fix: spend token race conditions * 🏗️ test: Add test for handling multiple concurrent transactions with high balance * fix: Handle Omni models prompt prefix handling for user messages with array content in OpenAIClient * refactor: Update checkBalance import paths to use new balanceMethods module * refactor: Update checkBalance imports and implement updateBalance function for atomic balance updates * fix: import from replace method * feat: Add createAutoRefillTransaction method to handle non-balance updating transactions * refactor: Move auto-refill logic to balanceMethods and enhance checkBalance functionality * feat: Implement logging for auto-refill transactions in balance checks * refactor: Remove logRefill calls from multiple client and handler files * refactor: Move balance checking and auto-refill logic to balanceMethods for improved structure * refactor: Simplify balance check calls by removing unnecessary balanceRecord assignments * fix: Prevent negative rawAmount in spendTokens when promptTokens is zero * fix: Update balanceMethods to use Balance model for findOneAndUpdate * chore: import order * refactor: remove unused txMethods file to streamline codebase * feat: enhance updateBalance and createAutoRefillTransaction methods to support additional parameters for improved balance management |
||
Ruben Talstra
|
3a62a2633d |
💵 feat: Add Automatic Balance Refill (#6452)
* 🚀 feat: Add automatic refill settings to balance schema * 🚀 feat: Refactor balance feature to use global interface configuration * 🚀 feat: Implement auto-refill functionality for balance management * 🚀 feat: Enhance auto-refill logic and configuration for balance management * 🚀 chore: Bump version to 0.7.74 in package.json and package-lock.json * 🚀 chore: Bump version to 0.0.5 in package.json and package-lock.json * 🚀 docs: Update comment for balance settings in librechat.example.yaml * chore: space in `.env.example` * 🚀 feat: Implement balance configuration loading and refactor related components * 🚀 test: Refactor tests to use custom config for balance feature * 🚀 fix: Update balance response handling in Transaction.js to use Balance model * 🚀 test: Update AppService tests to include balance configuration in mock setup * 🚀 test: Enhance AppService tests with complete balance configuration scenarios * 🚀 refactor: Rename balanceConfig to balance and update related tests for clarity * 🚀 refactor: Remove loadDefaultBalance and update balance handling in AppService * 🚀 test: Update AppService tests to reflect new balance structure and defaults * 🚀 test: Mock getCustomConfig in BaseClient tests to control balance configuration * 🚀 test: Add get method to mockCache in OpenAIClient tests for improved cache handling * 🚀 test: Mock getCustomConfig in OpenAIClient tests to control balance configuration * 🚀 test: Remove mock for getCustomConfig in OpenAIClient tests to streamline configuration handling * 🚀 fix: Update balance configuration reference in config.js for consistency * refactor: Add getBalanceConfig function to retrieve balance configuration * chore: Comment out example balance settings in librechat.example.yaml * refactor: Replace getCustomConfig with getBalanceConfig for balance handling * fix: tests * refactor: Replace getBalanceConfig call with balance from request locals * refactor: Update balance handling to use environment variables for configuration * refactor: Replace getBalanceConfig calls with balance from request locals * refactor: Simplify balance configuration logic in getBalanceConfig --------- Co-authored-by: Danny Avila <danny@librechat.ai> |
||
Danny Avila
|
ff59a2e41d | fix: Avoid Throwing Errors for Unsupported Token Count Endpoints 🪙 (#1356) | ||
|
Danny Avila
|
ea1dd59ef4 |
refactor(api): Central Logging 📜 (#1348)
* WIP: initial logging changes add several transports in ~/config/winston omit messages in logs, truncate long strings add short blurb in dotenv for debug logging GoogleClient: using logger OpenAIClient: using logger, handleOpenAIErrors Adding typedef for payload message bumped winston and using winston-daily-rotate-file moved config for server paths to ~/config dir Added `DEBUG_LOGGING=true` to .env.example * WIP: Refactor logging statements in code * WIP: Refactor logging statements and import configurations * WIP: Refactor logging statements and import configurations * refactor: broadcast Redis initialization message with `info` not `debug` * refactor: complete Refactor logging statements and import configurations * chore: delete unused tools * fix: circular dependencies due to accessing logger * refactor(handleText): handle booleans and write tests * refactor: redact sensitive values, better formatting * chore: improve log formatting, avoid passing strings to 2nd arg * fix(ci): fix jest tests due to logger changes * refactor(getAvailablePluginsController): cache plugins as they are static and avoids async addOpenAPISpecs call every time * chore: update docs * chore: update docs * chore: create separate meiliSync logger, clean up logs to avoid being unnecessarily verbose * chore: spread objects where they are commonly logged to allow string truncation * chore: improve error log formatting |
||
|
Danny Avila
|
df1dfa7d46 |
refactor: Use librechat-data-provider app-wide 🔄 (#1326)
* chore: bump vite, vitejs/plugin-react, mark client package as esm, move react-query as a peer dep in data-provider * chore: import changes due to new data-provider export strategy, also fix type imports where applicable * chore: export react-query services as separate to avoid react dependencies in /api/ * chore: suppress sourcemap warnings and polyfill node:path which is used by filenamify TODO: replace filenamify with an alternative and REMOVE polyfill * chore: /api/ changes to support `librechat-data-provider` * refactor: rewrite Dockerfile.multi in light of /api/ changes to support `librechat-data-provider` * chore: remove volume mapping to node_modules directories in default compose file * chore: remove schemas from /api/ as is no longer needed with use of `librechat-data-provider` * fix(ci): jest `librechat-data-provider/react-query` module resolution |
||
|
Danny Avila
|
583e978a82 |
feat(Google): Support all Text/Chat Models, Response streaming, PaLM -> Google 🤖 (#1316)
* feat: update PaLM icons * feat: add additional google models * POC: formatting inputs for Vertex AI streaming * refactor: move endpoints services outside of /routes dir to /services/Endpoints * refactor: shorten schemas import * refactor: rename PALM to GOOGLE * feat: make Google editable endpoint * feat: reusable Ask and Edit controllers based off Anthropic * chore: organize imports/logic * fix(parseConvo): include examples in googleSchema * fix: google only allows odd number of messages to be sent * fix: pass proxy to AnthropicClient * refactor: change `google` altName to `Google` * refactor: update getModelMaxTokens and related functions to handle maxTokensMap with nested endpoint model key/values * refactor: google Icon and response sender changes (Codey and Google logo instead of PaLM in all cases) * feat: google support for maxTokensMap * feat: google updated endpoints with Ask/Edit controllers, buildOptions, and initializeClient * feat(GoogleClient): now builds prompt for text models and supports real streaming from Vertex AI through langchain * chore(GoogleClient): remove comments, left before for reference in git history * docs: update google instructions (WIP) * docs(apis_and_tokens.md): add images to google instructions * docs: remove typo apis_and_tokens.md * Update apis_and_tokens.md * feat(Google): use default settings map, fully support context for both text and chat models, fully support examples for chat models * chore: update more PaLM references to Google * chore: move playwright out of workflows to avoid failing tests |
||
|
Danny Avila
|
365c39c405 |
feat: Accurate Token Usage Tracking & Optional Balance (#1018)
* refactor(Chains/llms): allow passing callbacks * refactor(BaseClient): accurately count completion tokens as generation only * refactor(OpenAIClient): remove unused getTokenCountForResponse, pass streaming var and callbacks in initializeLLM * wip: summary prompt tokens * refactor(summarizeMessages): new cut-off strategy that generates a better summary by adding context from beginning, truncating the middle, and providing the end wip: draft out relevant providers and variables for token tracing * refactor(createLLM): make streaming prop false by default * chore: remove use of getTokenCountForResponse * refactor(agents): use BufferMemory as ConversationSummaryBufferMemory token usage not easy to trace * chore: remove passing of streaming prop, also console log useful vars for tracing * feat: formatFromLangChain helper function to count tokens for ChatModelStart * refactor(initializeLLM): add role for LLM tracing * chore(formatFromLangChain): update JSDoc * feat(formatMessages): formats langChain messages into OpenAI payload format * chore: install openai-chat-tokens * refactor(formatMessage): optimize conditional langChain logic fix(formatFromLangChain): fix destructuring * feat: accurate prompt tokens for ChatModelStart before generation * refactor(handleChatModelStart): move to callbacks dir, use factory function * refactor(initializeLLM): rename 'role' to 'context' * feat(Balance/Transaction): new schema/models for tracking token spend refactor(Key): factor out model export to separate file * refactor(initializeClient): add req,res objects to client options * feat: add-balance script to add to an existing users' token balance refactor(Transaction): use multiplier map/function, return balance update * refactor(Tx): update enum for tokenType, return 1 for multiplier if no map match * refactor(Tx): add fair fallback value multiplier incase the config result is undefined * refactor(Balance): rename 'tokens' to 'tokenCredits' * feat: balance check, add tx.js for new tx-related methods and tests * chore(summaryPrompts): update prompt token count * refactor(callbacks): pass req, res wip: check balance * refactor(Tx): make convoId a String type, fix(calculateTokenValue) * refactor(BaseClient): add conversationId as client prop when assigned * feat(RunManager): track LLM runs with manager, track token spend from LLM, refactor(OpenAIClient): use RunManager to create callbacks, pass user prop to langchain api calls * feat(spendTokens): helper to spend prompt/completion tokens * feat(checkBalance): add helper to check, log, deny request if balance doesn't have enough funds refactor(Balance): static check method to return object instead of boolean now wip(OpenAIClient): implement use of checkBalance * refactor(initializeLLM): add token buffer to assure summary isn't generated when subsequent payload is too large refactor(OpenAIClient): add checkBalance refactor(createStartHandler): add checkBalance * chore: remove prompt and completion token logging from route handler * chore(spendTokens): add JSDoc * feat(logTokenCost): record transactions for basic api calls * chore(ask/edit): invoke getResponseSender only once per API call * refactor(ask/edit): pass promptTokens to getIds and include in abort data * refactor(getIds -> getReqData): rename function * refactor(Tx): increase value if incomplete message * feat: record tokenUsage when message is aborted * refactor: subtract tokens when payload includes function_call * refactor: add namespace for token_balance * fix(spendTokens): only execute if corresponding token type amounts are defined * refactor(checkBalance): throws Error if not enough token credits * refactor(runTitleChain): pass and use signal, spread object props in create helpers, and use 'call' instead of 'run' * fix(abortMiddleware): circular dependency, and default to empty string for completionTokens * fix: properly cancel title requests when there isn't enough tokens to generate * feat(predictNewSummary): custom chain for summaries to allow signal passing refactor(summaryBuffer): use new custom chain * feat(RunManager): add getRunByConversationId method, refactor: remove run and throw llm error on handleLLMError * refactor(createStartHandler): if summary, add error details to runs * fix(OpenAIClient): support aborting from summarization & showing error to user refactor(summarizeMessages): remove unnecessary operations counting summaryPromptTokens and note for alternative, pass signal to summaryBuffer * refactor(logTokenCost -> recordTokenUsage): rename * refactor(checkBalance): include promptTokens in errorMessage * refactor(checkBalance/spendTokens): move to models dir * fix(createLanguageChain): correctly pass config * refactor(initializeLLM/title): add tokenBuffer of 150 for balance check * refactor(openAPIPlugin): pass signal and memory, filter functions by the one being called * refactor(createStartHandler): add error to run if context is plugins as well * refactor(RunManager/handleLLMError): throw error immediately if plugins, don't remove run * refactor(PluginsClient): pass memory and signal to tools, cleanup error handling logic * chore: use absolute equality for addTitle condition * refactor(checkBalance): move checkBalance to execute after userMessage and tokenCounts are saved, also make conditional * style: icon changes to match official * fix(BaseClient): getTokenCountForResponse -> getTokenCount * fix(formatLangChainMessages): add kwargs as fallback prop from lc_kwargs, update JSDoc * refactor(Tx.create): does not update balance if CHECK_BALANCE is not enabled * fix(e2e/cleanUp): cleanup new collections, import all model methods from index * fix(config/add-balance): add uncaughtException listener * fix: circular dependency * refactor(initializeLLM/checkBalance): append new generations to errorMessage if cost exceeds balance * fix(handleResponseMessage): only record token usage in this method if not error and completion is not skipped * fix(createStartHandler): correct condition for generations * chore: bump postcss due to moderate severity vulnerability * chore: bump zod due to low severity vulnerability * chore: bump openai & data-provider version * feat(types): OpenAI Message types * chore: update bun lockfile * refactor(CodeBlock): add error block formatting * refactor(utils/Plugin): factor out formatJSON and cn to separate files (json.ts and cn.ts), add extractJSON * chore(logViolation): delete user_id after error is logged * refactor(getMessageError -> Error): change to React.FC, add token_balance handling, use extractJSON to determine JSON instead of regex * fix(DALL-E): use latest openai SDK * chore: reorganize imports, fix type issue * feat(server): add balance route * fix(api/models): add auth * feat(data-provider): /api/balance query * feat: show balance if checking is enabled, refetch on final message or error * chore: update docs, .env.example with token_usage info, add balance script command * fix(Balance): fallback to empty obj for balance query * style: slight adjustment of balance element * docs(token_usage): add PR notes |