Custom Endpoints

The custom endpoint feature lets you connect locally deployed LLM inference services (such as Ollama or LM Studio) or enterprise private-deployment APIs to Elftia. Most local inference frameworks provide an OpenAI-compatible API, making configuration straightforward.

When to Use

• You want to run open-source models locally (Llama, Qwen, Mistral, etc.) for fully offline use
• Your company has deployed a private LLM service on an internal network
• You are using Azure OpenAI Service rather than the official OpenAI API
• You need to connect to a self-hosted inference cluster (such as vLLM or TGI)
• You want to use LM Studio's graphical local inference environment

Ollama

Ollama is the most popular local model runner, supporting one-click download and execution of a wide variety of open-source models.

Prerequisites

• Ollama installed (download)
• At least one model downloaded (e.g. ollama pull llama3.1)

Steps

1. Make sure Ollama is running
   • Windows/macOS: Ollama runs as a background service by default after installation
   • Linux: run ollama serve to start the service
   • Default listening address: http://localhost:11434

2. In Elftia, open Settings → Provider Management

3. Find the default Ollama provider (if it doesn't exist, click Add Provider → select the Ollama preset)

4. Confirm the configuration:

Field	Value
API Format	openai
Base URL	http://localhost:11434/v1
API Key	(leave empty — Ollama requires no authentication by default)

5. Add the models you have downloaded:
   • Click Add Model
   • Enter the model name as it appears in Ollama (matching the output of ollama list)
   • Examples: llama3.1, qwen2.5:14b, codellama:34b

6. Enable the Enabled toggle and save

7. The Ollama models will now appear in the model selection in the chat UI

Common Ollama Models

Model name	Parameters	Use case	Download command
llama3.1	8B	General chat	ollama pull llama3.1
llama3.1:70b	70B	High-quality chat	ollama pull llama3.1:70b
qwen2.5:14b	14B	Chinese/English chat	ollama pull qwen2.5:14b
codellama:34b	34B	Code generation	ollama pull codellama:34b
deepseek-coder-v2	16B	Code assistance	ollama pull deepseek-coder-v2
mistral	7B	Lightweight chat	ollama pull mistral
llava	7B	Vision understanding	ollama pull llava

Remote Ollama

If Ollama is running on another machine on your local network:

1. On the machine running Ollama, set the environment variable OLLAMA_HOST=0.0.0.0:11434
2. Restart the Ollama service
3. In Elftia, change the Base URL to http://<remote-IP>:11434/v1

LM Studio

LM Studio provides a graphical interface for local model management and inference, with a built-in OpenAI-compatible API server.

Prerequisites

• LM Studio installed (download)
• At least one model downloaded and loaded

Steps

1. Start the local server in LM Studio:
   • Open LM Studio
   • Switch to the Local Server tab
   • Load a model
   • Click Start Server
   • Note the server address (default http://localhost:1234)

2. In Elftia, open Settings → Provider Management

3. Click Add Provider → select Custom Provider

4. Fill in the configuration:

Field	Value
Name	LM Studio
API Format	openai
Base URL	http://localhost:1234/v1/chat/completions
API Key	lm-studio (LM Studio does not validate the key, but if the field is required, enter any value)

5. Add models:

   • Enter the model name as it appears loaded in LM Studio
   • The model name must match the model identifier shown in LM Studio

6. Test the connection, enable, and save

vLLM / Text Generation Inference

vLLM and TGI are high-performance LLM inference engines widely used in production deployments. Both provide an OpenAI-compatible API.

vLLM Configuration

1. Start the vLLM service (example):

   python -m vllm.entrypoints.openai.api_server \
     --model meta-llama/Llama-3.1-8B-Instruct \
     --port 8000

2. In Elftia, add a custom provider:

Field	Value
Name	vLLM Local
API Format	openai
Base URL	http://localhost:8000/v1/chat/completions
API Key	(leave empty or enter the --api-key value from vLLM)

3. Add models: use the --model name specified when starting vLLM

TGI Configuration

1. After starting the TGI service, use its OpenAI-compatible endpoint

2. In Elftia, add a custom provider:

Field	Value
Name	TGI
API Format	openai
Base URL	http://localhost:8080/v1/chat/completions
API Key	(leave empty or enter the corresponding value)

Azure OpenAI

Azure OpenAI Service uses the same models as OpenAI but hosts them on the Azure platform, offering enterprise-grade SLAs and data compliance guarantees.

Prerequisites

• An Azure subscription and access to Azure OpenAI Service
• An Azure OpenAI resource created in the Azure Portal
• At least one model deployed (with a deployment name)

Steps

1. Obtain the following information from the Azure Portal:
   • Endpoint: in the format https://<resource-name>.openai.azure.com
   • API Key: available on the "Keys and Endpoint" page
   • Deployment name: available on the "Model Deployments" page
   • API Version: 2024-08-01-preview is recommended

2. In Elftia, open Settings → Provider Management

3. Click Add Provider → select the Azure OpenAI preset

4. Fill in the configuration:

Field	Value	Example
API Format	azure-openai	--
Base URL	Azure endpoint	https://my-resource.openai.azure.com
API Key	Azure API key	xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
API Version	API version number	2024-08-01-preview

5. Add models:

   • Use the Azure deployment name as the model ID (not the model name)
   • For example, if you deployed gpt-4o as my-gpt4o, enter my-gpt4o as the model ID

6. Test the connection, enable, and save

Azure vs. Standard OpenAI

Feature	OpenAI	Azure OpenAI
Base URL	https://api.openai.com/v1/...	https://<resource>.openai.azure.com
Authentication	Authorization: Bearer sk-...	api-key: ...
Model specification	Uses model name (e.g. gpt-4o)	Uses deployment name
API version	Not required	Required (URL parameter)
API Format	openai	azure-openai

Configuration Reference

The following configuration items apply to custom endpoint scenarios:

Setting	Type	Default	Description
Name	String	--	Custom display name
API Format	Enum	openai	Local services typically use openai; Azure uses azure-openai
Base URL	URL	--	API address of the inference service
API Key	String	(empty)	Usually not required for local services; required for Azure
API Version	String	(empty)	Required for Azure OpenAI only, e.g. 2024-08-01-preview
Model list	Array	(empty)	Manually enter the names of models available in the service
Transformer	Array	(empty)	Custom endpoints usually do not require an additional transformer

Behavior Notes

Special Considerations for Local Services

• No API key validation: Most local services (Ollama, LM Studio) do not require an API key. If Elftia's API key field is required, enter any placeholder value
• Hot model loading: Some services support dynamic model loading. After adding a new model to the service, simply add the corresponding model name in Elftia
• Concurrency limit: Local services typically handle only one request at a time; set the concurrency limit to 1
• First-response latency: Local models may need a few seconds to load into GPU memory; the first request will have noticeable latency

Proxy and Networking

• Local services (localhost / 127.0.0.1) do not go through the global proxy
• LAN services may need to be added to the proxy exclusion list, depending on your proxy configuration

Model Capabilities

Capability flags for local models (vision, function calling, etc.) must be set manually in Elftia. If a model supports vision but the flag is not set, Elftia will not send images to that model.

Troubleshooting

Issue	Possible cause	Solution
Connection refused	Local service is not running	Confirm Ollama/LM Studio/vLLM is running
Connection timeout	Wrong port or firewall blocking	Check the port number; confirm the firewall allows the connection
Model not found	Model name does not match the server	Use ollama list or the server's model list to confirm the name
Ollama remote connection fails	Ollama only listens on localhost by default	Set OLLAMA_HOST=0.0.0.0:11434 and restart
LM Studio returns empty response	Model not loaded into memory	Confirm the model is loaded and shows "Ready" in LM Studio
Azure returns 404	Incorrect deployment name	Confirm you are using the deployment name, not the model name, as the model ID
Azure returns 401	Incorrect API version or key	Check the API Version field and API Key
Garbled characters in response	Model does not support the language	Switch to a model that supports the target language (e.g. Qwen, ChatGLM)
Insufficient GPU memory	Model is too large	Choose a smaller quantized version (e.g. llama3.1:8b-q4_0)
Request hangs with no response	Local service is loading the model	Wait for the model to finish loading; the first request may take 10–30 seconds

Related Pages

• LLM Providers Overview - Understand the provider system and API formats
• Adding a Provider - Complete guide for adding a provider
• API Key Pools - Multi-key management for cloud providers
• Model Parameters - Adjust generation parameters such as temperature