Commit Graph

4 Commits

Author SHA1 Message Date
Nick Nyanjui
fe347e4ea5 fix(stt): dispatch live-API-only Gemini models over the Live WebSocket transport (#4006)
Addresses #4006 by letting Gemini STT models that only exist on the Live
API transcribe instead of failing.

transcribeGemini sends audio to :generateContent and that is the only Gemini
path. A model that is realtime-only (exposed by the Live API's
bidiGenerateContent WebSocket) therefore fails outright, even though the account
can transcribe it.

open-sse/handlers/geminiLiveStt.js owns the WebSocket lifecycle: opens
:bidiGenerateContent, sends setup frame, waits for setupComplete, streams
audio as realtimeInput media chunks, and settles on turnComplete.
Dispatch is driven by transport marker 'gemini-live'. Adds custom model transport
persistence and selection on the dashboard.
2026-09-26 12:00:39 +07:00
omar-nahhas
fe547f4dc0 feat(providers): self-hosted OpenAI-compatible STT, TTS and embedding providers
Add Self-hosted STT/TTS/Embedding providers that read baseUrl per connection
instead of a fixed registry endpoint, so 9Router can point at whisper.cpp,
faster-whisper, Kokoro-FastAPI, llama-server, vLLM, Infinity, and similar
OpenAI-compatible local servers.

Self-hosted Embedding refuses to run without a baseUrl rather than falling
back to api.openai.com like openaiCompatNode does, since that fallback would
silently send input text and the API key to OpenAI under a provider named
"Self-hosted". Also fixes embeddingsCore to catch adapter build errors as a
400 instead of letting them escape uncaught, and bounds the upstream fetch
with FETCH_CONNECT_TIMEOUT_MS to avoid hanging forever on a dead endpoint.

Self-hosted TTS treats a bare model value as the model rather than the voice,
since the generic OpenAI TTS convention (bare = voice) is backwards for a
provider where the model is the variable part.
2026-08-05 13:38:13 +07:00
decolua
87fe069e9e refactor(open-sse): remove reverse coupling open-sse -> src (E2)
Move clineAuth into open-sse/shared (src re-exports back). Add standalone
open-sse/shared/machineId for codex session hashing (no @/lib/dataDir).
sttCore receives sttConfig via param instead of importing AI_PROVIDERS.
No behavior change; gate: no regression (26 known-fails unchanged).

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-13 16:35:07 +07:00
decolua
d4bc42e1f5 feat: add STT support, Gemini TTS, and expand usage tracking
- Speech-to-Text: full pipeline with sttCore handler, /v1/audio/transcriptions
  endpoint, sttConfig for OpenAI, Gemini, Groq, Deepgram, AssemblyAI,
  HuggingFace, NVIDIA Parakeet; new 9router-stt skill
- Gemini TTS: add gemini provider with 30 prebuilt voices and TTS_PROVIDER_CONFIG
- Usage: implement GLM (intl/cn) and MiniMax (intl/cn) quota fetchers; refactor
  Gemini CLI usage to use retrieveUserQuota with per-model buckets
- Disabled models: lowdb-backed disabledModelsDb + /api/models/disabled route
- Header search: reusable Zustand store (headerSearchStore) wired into Header
- CLI tools: add Claude Cowork tool card and cowork-settings API
- Providers: introduce mediaPriority sorting in getProvidersByKind, add
  Kimi K2.6, reorder hermes, drop qwen STT kind
- UI: expand media-providers/[kind]/[id] page (+314), enhance OAuthModal,
  ModelSelectModal, ProviderTopology, ProxyPools, ProviderLimits
- Assets: refresh provider PNGs (alicode, byteplus, cloudflare-ai, nvidia,
  ollama, vertex, volcengine-ark) and add aws-polly, fal-ai, jina-ai, recraft,
  runwayml, stability-ai, topaz, black-forest-labs
2026-05-05 10:32:59 +07:00