How-to guide
Use your own model
Run the assistant on Hugging Face, Ollama, vLLM or any OpenAI style API by changing settings.
The assistant talks to one language model through one setting. Pick a hosted provider, a server you run, or a model on your own machine. Personal values are masked before any request leaves Yaqeen, whichever you pick. The setting that chooses the provider is ASSISTANT_LLM_PROVIDER: huggingface, endpoint, openai, openai_compatible, vllm, tgi, ollama or litellm.
Use a hosted provider#
A hosted provider runs the model for you. Personal values are masked before the request leaves Yaqeen, and your key stays in Yaqeen's settings. Plain http is refused for anything that is not a local address, so a hosted address must start with https://.
Hugging Face (open models, many providers behind one address)
- Make a Hugging Face account and create an access token at
https://huggingface.co/settings/tokens. Give it the permission to make calls to Inference Providers. Treat it like a password and never commit it. - Pick a model on the Hub that an inference provider serves, and note its id, for example
<org>/<model>. Add a provider after a colon to choose who runs it (<org>/<model>:<provider>), or:fastestor:cheapest. Set these settings:
ASSISTANT_LLM_PROVIDER=huggingface ASSISTANT_LLM_API_KEY=<your Hugging Face token> ASSISTANT_LLM_MODEL=<org>/<model>:<provider>- Restart Yaqeen and ask the assistant something. If the reply is the plain "unavailable" message, read the log line for that call: it names the provider and the error class, and never the request text.
- Check the model can call tools. If it answers in text but never looks anything up, set
ASSISTANT_LLM_TOOLS=falseor pick a model that supports tools.
Hugging Face bills usage through its providers. Read the current pricing and your usage page before you rely on it.
A dedicated Hugging Face Inference Endpoint
Deploy the model from the Hub as an endpoint, then use its own address:
ASSISTANT_LLM_PROVIDER=endpoint
ASSISTANT_LLM_BASE_URL=https://<your endpoint address>/v1
ASSISTANT_LLM_API_KEY=<your Hugging Face token>
ASSISTANT_LLM_MODEL=<name the endpoint expects>The model name depends on the container the endpoint runs, so read it from the endpoint's page.
OpenAI, or any other OpenAI style API
ASSISTANT_LLM_PROVIDER=openai
ASSISTANT_LLM_API_KEY=<your key>
ASSISTANT_LLM_MODEL=<model name>For any other service that speaks the same format, use ASSISTANT_LLM_PROVIDER=openai_compatible and add ASSISTANT_LLM_BASE_URL=https://<its address>/v1. For a service that does not speak that format, use LiteLLM below.
Sending personal data to a provider outside the Kingdom can count as a transfer under the Saudi PDPL. Masking lowers what leaves. It does not settle the legal question, so ask counsel before you rely on it.
Run vLLM on your own GPU server#
vLLM serves open models fast on a GPU. It is the usual choice when a team hosts a model itself.
Install it on the GPU machine, following
https://docs.vllm.ai, then start a server for your model:vllm serve <org>/<model> --enable-auto-tool-choice --tool-call-parser <parser> --port 8001Without
--enable-auto-tool-choiceand a parser that matches the model's tool format, the server cannot return tool calls. The vLLM tool calling page lists the parser for each model family.- Use port 8001 or another free one, so it does not clash with other services on the machine.
Set these settings:
ASSISTANT_LLM_PROVIDER=vllm ASSISTANT_LLM_BASE_URL=http://<server address>:8001/v1 ASSISTANT_LLM_MODEL=<org>/<model>Add
ASSISTANT_LLM_API_KEYif you started vLLM with--api-key. Plain http is accepted only for local and private addresses, so put the server on a private network or put TLS in front of it.- Text Generation Inference works the same way with
ASSISTANT_LLM_PROVIDER=tgi. Its tool support depends on the version, so test it.
Run a model on your own machine with Ollama#
Ollama runs open models on your computer and serves them on http://localhost:11434. Nothing leaves your network and no key is needed. These steps follow Ollama's own docs, so check each one against the version you install.
- Install Ollama.
- macOS: download the app from
https://ollama.com/download, or runbrew install ollama. - Linux:
curl -fsSL https://ollama.com/install.sh | sh. - Windows: run the installer from the same download page.
- macOS: download the app from
- Start the server. The macOS and Windows apps start it for you. On Linux, or with Homebrew, run
ollama servein its own terminal. Check it answers:curl http://localhost:11434/api/tagsreturns a list, empty at first. - Pull a model that can call tools. The assistant needs tool calls to look things up, and not every model has them. On
https://ollama.com/searchfilter by thetoolslabel, then pull one that fits your memory, for exampleollama pull llama3.1:8b. Runollama listto see the exact name. Check the model answers over the OpenAI style route Yaqeen uses:
curl http://localhost:11434/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "llama3.1:8b", "messages": [{"role": "user", "content": "Say hello"}]}'Point Yaqeen at it and restart:
ASSISTANT_LLM_PROVIDER=ollama ASSISTANT_LLM_MODEL=llama3.1:8b ASSISTANT_LLM_MAX_TOOLS=8ASSISTANT_LLM_MAX_TOOLS=8matters on a small model: the full set of tool definitions is long, and Ollama's default context window is small enough to cut them off. If your Ollama version supports it, start the server with a larger window, for exampleOLLAMA_CONTEXT_LENGTH=8192 ollama serve.- If Yaqeen runs in Docker,
localhostis the container, not your computer. SetASSISTANT_LLM_BASE_URL=http://host.docker.internal:11434/v1. Plain http is accepted for this name because it is a local address.
When something is wrong:
| Symptom | Likely cause and fix |
|---|---|
| Connection refused | The server is not running. Start the app or run ollama serve. |
| Model not found | The name must match ollama list exactly, including the tag. |
| The assistant answers but never looks anything up | The model cannot call tools. Pick one with the tools label, or set ASSISTANT_LLM_TOOLS=false and accept text only. |
| The first reply is slow | Ollama is loading the model into memory. Later replies are faster. |
| Tool calls look cut off or wrong | The context window is too small. Lower ASSISTANT_LLM_MAX_TOOLS or raise the window. |
Yaqeen logs the provider, whether tools were sent, token counts and time for every call, and the error class if a call fails. It never logs the request text.
Small open models and tools#
"OpenAI compatible" describes the wire format, not the behaviour. Open models are weaker than hosted ones at calling several tools at once (Berkeley Function Calling Leaderboard), and a server may not parse a model's tool format at all. Three settings help:
ASSISTANT_LLM_TOOLS=falsesends no tool definitions. The model then only writes text.ASSISTANT_LLM_MAX_TOOLS=<n>offers only thentools whose name and summary share the most words with the request. The full set of definitions is a lot for a small model. Default 0 offers all.- Reasoning models that put
<think>...</think>in the reply are cleaned before the text is used. Tool arguments that arrive as an object instead of a string are accepted. Arguments that are not valid JSON become empty, the tool refuses them, and the model sees the refusal and can try again inside the same limits as any other call.