Skip to main content

CompletionService Call Pipeline

CompletionService is the core facade for LLM calls. It orchestrates frontend completion requests into a complete API call pipeline. This document details its six-step pipeline, streaming handling, thinking budget calculation, retry mechanism, and tool call loop.


File Locations​

FilePath
CompletionServicepackages/desktop/app/main/services/capabilities/llm/completion/CompletionService.ts
DirectApiHandlerpackages/desktop/app/main/services/capabilities/llm/completion/DirectApiHandler.ts
StreamHandlerpackages/desktop/app/main/services/capabilities/llm/completion/StreamHandler.ts
ToolHandlerpackages/desktop/app/main/services/capabilities/llm/completion/ToolHandler.ts
TransformerHandlerpackages/desktop/app/main/services/capabilities/llm/completion/TransformerHandler.ts
ThinkingResolverpackages/desktop/app/main/services/capabilities/llm/completion/ThinkingResolver.ts
URL Builderpackages/desktop/app/main/services/capabilities/llm/completion/url-builder.ts
Header Builderpackages/desktop/app/main/services/capabilities/llm/completion/header-builder.ts
Message Converterpackages/desktop/app/main/services/capabilities/llm/completion/message-converter.ts
Typespackages/desktop/app/main/services/capabilities/llm/completion/types.ts
NativeSearchInjectorpackages/desktop/app/main/services/capabilities/llm/completion/NativeSearchInjector.ts
ProviderSearchInjectorpackages/desktop/app/main/services/capabilities/llm/completion/ProviderSearchInjector.ts

Architectural Context​

graph TB
subgraph CompletionService ["CompletionService (Facade)"]
direction TB
Complete[complete]
Stream[completeStream]
WithTransformers[completeWithTransformers]
StreamTransformers[completeStreamWithTransformers]
WithTools[streamWithTools]
TestModel[testModel]
end

subgraph PipelineSteps ["Pipeline Steps"]
direction TB
S1["(1) Route resolution<br/>resolveRoutedModel"]
S2["(2) Provider lookup<br/>getProvider + enabled check"]
S3["(3) API Key resolution<br/>codingPlan → pool → legacy"]
S4["(4) API format resolution<br/>resolveApiFormat"]
S5["(5) Handler dispatch<br/>callDirectHandler / callStreamHandler"]
S6["(6) Retry + success report"]
end

subgraph Handlers
DAH[DirectApiHandler<br/>non-streaming]
SH[StreamHandler<br/>SSE streaming]
TH[ToolHandler<br/>tool loop]
THR[TransformerHandler<br/>transformer chain]
end

subgraph AuxiliaryServices ["Auxiliary Services"]
TR2[ThinkingResolver]
NSI[NativeSearchInjector]
PSI[ProviderSearchInjector]
MC[Message Converter]
UB[URL Builder]
HB[Header Builder]
end

Complete --> S1 --> S2 --> S3 --> S4 --> S5 --> S6
S5 --> DAH
S5 --> SH

WithTools --> TH
WithTransformers --> THR
StreamTransformers --> THR

DAH --> UB
DAH --> HB
DAH --> MC
SH --> UB
SH --> HB
SH --> MC
SH --> TR2
TH --> UB
TH --> HB

Data Structures​

Request and Response Types​

// Completion request options
interface CompletionOptions {
providerId: string; // Provider ID
model: string; // Model ID (v89+: bare SDK id, no `<backend>:` prefix or `[1m]` suffix)
messages: SimpleChatMessage[]; // Conversation messages
maxTokens?: number; // Max tokens to generate
temperature?: number; // Temperature
stream?: boolean; // Whether streaming
thinkLevel?: ThinkLevel; // Thinking level: 'none' | 'low' | 'medium' | 'high'
nativeSearchAugmentation?: NativeSearchAugmentation; // SDK native search augmentation
sessionId?: string; // Session ID (API Key Pool affinity)
/**
* 1M-context flag (v89+). When true and the model is on the 1M-capable whitelist
* (`claude-opus-4-7` / `claude-opus-4-6` / `claude-sonnet-4-6`),
* `TransformerHandler` calls `injectExtendedContextBeta()` at the transformer chain
* exit, merging `'context-1m-2025-08-07'` into the outbound request's
* `anthropic-beta` HTTP header
* (NOT a body field; `/v1/messages` rejects unknown body fields).
*/
useExtendedContext?: boolean;
}

// Completion result
interface CompletionResult {
success: boolean;
message?: SimpleChatMessage; // Generated message
error?: string; // Error message
usage?: {
promptTokens: number;
completionTokens: number;
totalTokens: number;
};
finishReason?: string; // 'stop' | 'tool_use' | 'max_tokens' etc.
}

// Streaming callbacks
interface StreamCallbacks {
onStart?: (messageId: string) => void;
onDelta?: (content: string) => void;
onReasoning?: (reasoning: string) => void;
onAudio?: (audio: SimpleChatAudio) => void;
onVideo?: (video: SimpleChatVideo) => void;
onBlock?: (block: MessageBlock) => void; // Content blocks: thinking/text/tool_use/tool_result
onDone?: (message, usage?, metrics?) => void;
onError?: (error: string) => void;
}

// API format
type ApiFormat = 'openai' | 'anthropic' | 'google' | 'azure-openai' | 'openai-response';

Algorithms and Logic​

Six-Step Request Pipeline​

Step 1: Route Resolution​

routedInfo = llmConfig.resolveRoutedModel(providerId, model)
actualProviderId = routedInfo?.actualProviderId || providerId
actualModel = routedInfo?.actualModelId || model

Route resolution handles Chat → Code and Code → Chat model routing. If model matches a routing rule, it is replaced with the actual provider and model.

Step 2: Provider Lookup​

provider = getProvider(actualProviderId)
if (!provider) → return error "Provider not found"
if (!provider.enabled) → return error "Provider is disabled"

The lookup goes through LLMConfigService's Provider Index as an O(1) lookup.

Step 3: API Key Resolution​

resolveApiKeyForRequest(provider, providerId, sessionId):
// Priority 1: Coding Plan override
if provider.codingPlan?.enabled && provider.codingPlan.apiKey:
return resolveApiKey(codingPlan.apiKey)

// Priority 2: API Key Pool (session-affinity weighted round-robin)
if apiKeyPool available:
poolKey = sessionId
? apiKeyPool.getKeyForSession(providerId, sessionId)
: apiKeyPool.getKey(providerId)
if poolKey: return poolKey

// Priority 3: Legacy single key
return resolveApiKey(provider.api_key)

resolveApiKey() handles environment variable expansion ($ENV_VAR → process.env.ENV_VAR).

Step 4: API Format Resolution​

resolveApiFormat(provider):
// Priority 1: apiFormat field (v3 preferred)
if provider.apiFormat: return provider.apiFormat

// Priority 2: chatApiFormat field (legacy v3)
if provider.chatApiFormat: return provider.chatApiFormat

// Priority 3: implicit apiType conversion
if provider.apiType === 'claudecode' || 'anthropic': return 'anthropic'
if provider.apiType === 'google': return 'google'

// Default: OpenAI format
return 'openai'

Step 5: Handler Dispatch​

Select the appropriate handler based on apiFormat:

apiFormatNon-streaming HandlerStreaming Handler
openaicallOpenAICompletionstreamOpenAICompletion
openai-responsecallOpenAIResponseCompletionstreamOpenAIResponseCompletion
anthropiccallAnthropicCompletionstreamAnthropicCompletion
googlecallGeminiCompletionstreamGeminiCompletion
azure-openaicallOpenAICompletionstreamOpenAICompletion

Step 6: Retry + Success Report​

result = callHandler(...)
if result failed && apiKeyPool available && sessionId present:
status = extractHttpStatus(result.error)
if status in [429, 529, 401, 403]:
newKey = apiKeyPool.reportError(providerId, sessionId, status)
if newKey:
result = callHandler(..., newKey) // retry once with new key

if result succeeded && apiKeyPool available:
apiKeyPool.reportSuccess(sessionId) // reset cooldown counter

Streaming​

SSE Stream Parsing​

All streaming handlers use the streamSSEResponse() utility, based on the standard SSE (Server-Sent Events) protocol:

sequenceDiagram
participant CS as CompletionService
participant SH as StreamHandler
participant API as Provider API

CS->>SH: callStreamHandler(format, provider, key, options)
SH->>API: POST request (stream: true)
API-->>SH: SSE stream

loop Each SSE event
SH->>SH: Parse event data
alt content delta
SH->>CS: callbacks.onDelta(content)
else reasoning delta
SH->>CS: callbacks.onReasoning(reasoning)
else block event
SH->>CS: callbacks.onBlock(block)
else [DONE]
SH->>CS: callbacks.onDone(message, usage, metrics)
else error
SH->>CS: callbacks.onError(error)
end
end

Stream Retry (Pool Mode)​

flowchart TD
Start[completeStream] --> HasPool{apiKeyPool available?}
HasPool -->|No| DirectCall[Call callStreamHandler directly]
HasPool -->|Yes| InterceptCall[Call with intercepting callbacks]

InterceptCall --> StreamDone{Stream completed successfully?}
StreamDone -->|Yes| ReportSuccess[reportSuccess]
StreamDone -->|No| CheckStatus{Is 429/529/401/403?}

CheckStatus -->|Yes| GetNewKey[reportError → get new key]
CheckStatus -->|No| PropagatError[Propagate error to frontend]

GetNewKey --> HasNewKey{New key available?}
HasNewKey -->|Yes| RetryStream[Retry stream with new key]
HasNewKey -->|No| PropagatError

Key design points for stream retry:

  • 429/529 errors are intercepted by wrapping callbacks.onError
  • A retryState object reference is used to track error state across closures
  • On retry, onStart is not triggered again (it was already fired once)

Anthropic Thinking Budget Calculation​

flowchart TD
Start[resolveThinkingBudget] --> CheckLevel{thinkLevel === 'none'?}
CheckLevel -->|Yes| NoThinking[Return raw maxTokens<br/>no thinking config]
CheckLevel -->|No| CheckModel{isReasoningModel?}
CheckModel -->|No| NoThinking
CheckModel -->|Yes| CalcBudget[calculateThinkingBudget<br/>model, thinkLevel, maxTokens]

CalcBudget --> CheckFormat{API format is Anthropic?}
CheckFormat -->|Yes| AdjustTokens[adjustedMaxTokens = getClaudeMaxTokens<br/>maxTokens - thinkingBudget]
CheckFormat -->|No| KeepTokens[adjustedMaxTokens = maxTokens]

AdjustTokens --> BuildConfig[buildAnthropicThinking<br/>generate thinking config object]
BuildConfig --> Return[Return adjustedMaxTokens + thinkingConfig]
KeepTokens --> Return

Anthropic special handling: For Claude models, max_tokens includes thinking tokens, so the following is required:

  1. Calculate the thinking budget thinkingBudget
  2. Subtract the thinking budget from max_tokens to get adjustedMaxTokens
  3. Generate a thinking config object to include in the request body

OpenAI Reasoning Models​

For OpenAI o-series models (o1, o3, etc.), use the reasoning_effort parameter instead of adjusting the token budget:

if thinkLevel !== 'none':
effort = getOpenAIReasoningEffort(thinkLevel)
// 'low' | 'medium' | 'high'
request.reasoning_effort = effort

max_tokens Resolution Priority​

resolveEffectiveMaxTokens(providerId, modelId, sessionMaxTokens):
// 1. Session-level setting (highest priority, user-set manually, no cap applied)
if sessionMaxTokens > 0: return sessionMaxTokens

// 2. Global model parameters (admin-set, no cap applied)
globalParams = llmConfig.getGlobalModelParameters()
if globalParams.maxTokens.enabled && value > 0: return value

// -------- Values below are auto-resolved and capped at MAX_TOKENS_CAP=65536 --------

// 3. maxTokens from model config
modelConfig = provider.modelConfigs.find(id === modelId)
if modelConfig.maxTokens > 0: return min(value, 65536)

// 4. maxTokens from model group
modelGroup = provider.modelGroups.find(models.id === modelId)
if model.maxTokens > 0: return min(value, 65536)

// 5. Model discovery cache
discovered = llmConfig.getDiscoveredModelMaxTokens(providerId, modelId)
if discovered > 0: return min(value, 65536)

// 6. undefined (let the API use its default)
return undefined

// For providers requiring max_tokens (e.g. Anthropic):
getRequiredMaxTokens():
resolved = resolveEffectiveMaxTokens(...)
return resolved ?? DEFAULT_MAX_TOKENS // typically 4096

Tool Call Loop (ToolHandler)​

sequenceDiagram
participant TH as ToolHandler
participant LLM as LLM API
participant MCP as MCP Service

TH->>TH: MAX_ITERATIONS = globalParams.toolMaxTurns ?? 5
TH->>TH: iteration = 0

loop iteration < MAX_ITERATIONS
TH->>TH: iteration++
TH->>TH: buildToolRequest(format, messages, model, options)
TH->>LLM: POST request with tools
LLM-->>TH: SSE stream response

TH->>TH: extractToolCalls(response)

alt No tool calls
TH->>TH: break (LLM finished answering)
else Has tool calls
loop Each tool call
TH->>TH: callbacks.onToolCall(toolCall)
TH->>MCP: executeToolCalls(toolCalls, mcpService)
MCP-->>TH: tool results
TH->>TH: callbacks.onToolResult(id, result)
end
TH->>TH: Append tool calls and results to messages
TH->>TH: buildIterationBlocks(tool call blocks)
end
end

TH->>TH: callbacks.onDone(finalContent, usage)

Key behaviors:

ParameterDefaultDescription
MAX_ITERATIONSglobalParams.toolMaxTurns ?? 5Maximum iteration count
Tool formatAuto-detected from apiFormatTool definitions in OpenAI/Anthropic/Gemini format
Termination conditionNo tool calls, or limit reachedEnds naturally when LLM stops requesting tools

Tool calls support three formats, auto-detected by logToolFormat():

  • OpenAI format: { type: 'function', function: { name, parameters } }
  • Anthropic format: { name, input_schema }
  • Gemini format: { functionDeclarations: [...] }

Vision Fallback​

When messages contain images but the model does not support vision, an auxiliary vision model is used automatically:

applyVisionFallback(options):
if no images in messages: return
if model supports vision: return

visionModel = llmConfig.resolveEffectiveModels().vision
if no vision model:
// Strip images
for msg in messages:
msg.images = undefined
return

// Use VisionDescriptionService to describe images
for msg in messages with images:
description = visionService.describeImages(images, msg.content, visionModel)
msg.content += "\n\n[Image Description]\n" + description
msg.images = undefined

Error Interception Pattern​

extractHttpStatus() extracts the HTTP status code from an error message string:

extractHttpStatus(error: string):
match = error.match(/\((\d{3})\):/)
return match ? parseInt(match[1]) : null

// Example: "API error (429): Rate limit exceeded" → 429

IPC Integration Table​

IPC ChannelDirectionRouterDescription
completion:completeR → MCompletionRouterNon-streaming completion
completion:getModelsR → MCompletionRouterGet available models
completion:testModelR → MCompletionRouterTest model connection
Streaming completionR → MChatStreamHandlerSSE stream delivered via IPC messages
RegenerateR → MRegenerateHandlerRegenerate a reply

Extension Points​

Adding a New API Format​

  1. Add the new value to the ApiFormat type in types.ts
  2. Add a URL builder function in url-builder.ts
  3. Add header-building logic in header-builder.ts
  4. Add message format conversion in message-converter.ts
  5. Add a callXxxCompletion function in DirectApiHandler.ts
  6. Add a streamXxxCompletion function in StreamHandler.ts
  7. Add the case to the switch in CompletionService.callDirectHandler() and callStreamHandler()

Custom Retry Strategy​

Currently only one retry is attempted. For more complex retry behavior (e.g., multiple retries, varying wait times), modify the retry logic in complete() and completeStream().

Adding New Search Injection​

  • SDK native search (e.g., Anthropic): inject via NativeSearchInjector.applyAugmentation()
  • Provider-specific search (e.g., model-param / builtin-tool): inject via ProviderSearchInjector

FileRelationship
capabilities/llm/config-service/LLMConfigService.tsProvides provider lookup and route resolution
capabilities/llm/completion/ApiKeyPoolService.tsKey selection and load balancing
infra/utils/sse-parser.tsSSE stream parsing utilities
shared/completion-types.tsShared types such as SimpleChatMessage
shared/thinking-config.tsThinking budget calculation and reasoning model detection
shared/llm-config.tsLLMProvider type definition
capabilities/tools/mcp-users/McpService.tsTool execution (called by ToolHandler)
capabilities/llm/api-converter/openai-to-anthropic.tsOpenAI → Anthropic format conversion
routers/CompletionRouter.tsIPC entry point