fix: updated logger and model caching minor bugfix #release

[thecodacus]

Optimize Model List Fetching and Improve Provider Management

Overview

This PR optimizes the LLM provider management by implementing lazy loading and caching of dynamic models. Instead of fetching models from all providers on every LLM call, we now only fetch models from the selected provider. Additionally, we've added a caching mechanism for model lists and improved logging throughout the system.

Key Changes

1. Optimized Model List Fetching

• Implemented lazy loading for dynamic models - only fetch from selected provider
• Added caching mechanism for provider model lists
• Moved model list management logic to LLMManager class

2. Universal Logging System

• Enhanced logging with structured scopes using Chalk
• Added cross-platform colored console output that works in terminals
• Implemented consistent debug levels and scoped loggers
• Unified logging format across different environments

Technical Details

Model Fetching Optimization

Key implementation changes (simplified):

// Before
const MODEL_LIST = await getModelList({ apiKeys, providerSettings, serverEnv });
const modelDetails = MODEL_LIST.find((m) => m.name === currentModel);

// After
const modelsList = [
  ...(provider.staticModels || []),
  ...(await LLMManager.getInstance().getModelListFromProvider(provider, {
    apiKeys,
    providerSettings,
    serverEnv
  }))
];

Caching Implementation

class BaseProvider {
  cachedDynamicModels?: {
    cacheId: string;
    models: ModelInfo[];
  };

  getModelsFromCache(options: ProviderOptions): ModelInfo[] | null {
    if (!this.cachedDynamicModels) return null;
    
    const cacheKey = this.getDynamicModelsCacheKey(options);
    if (cacheKey !== this.cachedDynamicModels.cacheId) {
      this.cachedDynamicModels = undefined;
      return null;
    }
    
    return this.cachedDynamicModels.models;
  }
}

Enhanced Cross-Platform Logging

// Universal logging with Chalk for consistent terminal output
const chalk = new Chalk({ level: 3 });

function formatText(text: string, color: string, bg: string) {
  return chalk.bgHex(bg)(chalk.hex(color)(text));
}

// Usage in logger
const labelText = formatText(` ${level.toUpperCase()} `, textColor, bgColor);
console.log(`${labelText}`, allMessages);

// Scoped logger usage example
const logger = createScopedLogger('stream-text');
logger.info(`Sending llm call to ${provider.name}`); // Outputs with consistent colors

Migration Impact

• No breaking changes in the API
• Cached models are automatically invalidated when provider settings change
• Existing code paths continue to work with fallback mechanisms

Testing

• Verified model caching behavior across multiple requests
• Tested cache invalidation scenarios
• Confirmed logging output in different environments
• Validated fallback behavior for unknown models

Future Improvements

• Add cache expiration mechanism
• Implement background refresh for cached models
• Add metrics for cache hit/miss rates
• Consider alternative color schemes for different terminal themes
• Add log rotation and persistence options
• Consider implementing a shared cache for multi-instance deployments

[dustinwloring1988]

worked for static models but the dynmic models tried to call claude-3-5-sonnet-latest when using ollama, then it tried the one I had selected but never got a response. I can look into this more later one today.

[thecodacus]

the model selector is sometimes not updating the model correctly when selected for dropdown, can you check how can we replicate this issue ?

[chrismahoney]

I just tested this with the following:

• Qwen2.5 Coder 32B: Generate a todo app
• Llama 3.3:70B: Same thing
• Qwen2.5 Coder 14b: Generate Snake in python

This all worked fine, which would not be the case if it was loading all of these models at once based on my hardware. I ran into issues trying to use the starter templates but I believe that's unrelated. I'm happy with this update.

[HlebKrah]

When i add screenshot to deepseek-chat model i got this error:

Cannot read properties of undefined (reading 'toolCalls')
    at Object.flush (file:///Users/apple/Projects/dash-ai/node_modules/ai/core/generate-text/stream-text.ts:569:33)
    at invokePromiseCallback (node:internal/webstreams/util:162:10)
    at Object.<anonymous> (node:internal/webstreams/util:167:23)
    at transformStreamDefaultSinkCloseAlgorithm (node:internal/webstreams/transformstream:621:43)
    at node:internal/webstreams/transformstream:379:11
    at writableStreamDefaultControllerProcessClose (node:internal/webstreams/writablestream:1162:28)
    at writableStreamDefaultControllerAdvanceQueueIfNeeded (node:internal/webstreams/writablestream:1242:5)
    at writableStreamDefaultControllerClose (node:internal/webstreams/writablestream:1209:3)
    at writableStreamClose (node:internal/webstreams/writablestream:722:3)
    at writableStreamDefaultWriterClose (node:internal/webstreams/writablestream:1091:10)
    at writableStreamDefaultWriterCloseWithErrorPropagation (node:internal/webstreams/writablestream:1083:10)
    at node:internal/webstreams/readablestream:1558:15
    at complete (node:internal/webstreams/readablestream:1437:9)
    at processTicksAndRejections (node:internal/process/task_queues:105:5

With claude sonnet everything is fine

[dustinwloring1988]

@HlebKrah I dont not think deepseek supports images