Ollama

Ollama runs open LLMs on a GPU in your cluster, so that agents can use models that never leave your premises. It's installed on one of your GPU nodes, in the inference namespace, at:

http://ollama.inference.svc.cluster.local:11434

An AIFabric LLM with provider: ollama and no endpoint uses this URL automatically.

Check That It's Running

kubectl -n inference get pods -l app=ollama

The pod should be Running. To check that the agents' namespace can reach the server:

kubectl -n trirematics run curl-test --rm -it --restart=Never --image=curlimages/curl -- curl -s http://ollama.inference.svc.cluster.local:11434

The answer is Ollama is running.

Manage Models

Ollama starts with no models. Pull at least one that supports tool calling, since the agents use tools. On ollama.com/library, these models carry the tools tag.

kubectl -n inference exec deploy/ollama -- ollama pull gpt-oss:20b

To list the pulled models, and remove one you no longer need:

kubectl -n inference exec deploy/ollama -- ollama list
kubectl -n inference exec deploy/ollama -- ollama rm <model>

Models take tens of GB each. They're stored on a 300 GiB volume in the cluster, and kept across restarts and upgrades. To free space, remove the models you no longer need.

The first request to a model is slow, because Ollama reads the whole model into GPU memory. After that, the model stays loaded until Ollama restarts, or until another model needs the GPU memory. To load a model ahead of the first request:

kubectl -n inference exec deploy/ollama -- ollama run gpt-oss:20b ""

To see the loaded models, and whether each one fits on the GPU:

kubectl -n inference exec deploy/ollama -- ollama ps

PROCESSOR should read 100% GPU. A CPU/GPU split means the model doesn't fit in the GPU's memory, and part of it runs on the CPU, much more slowly. Choose a smaller model or quantization. The CONTEXT column shows the context size in use (see Context Length).

Ollama handles one request at a time per model, so requests from several agents wait in a queue.

Try a Model

Forward the port in one terminal:

kubectl -n inference port-forward svc/ollama 11434

Then send a prompt from another terminal:

curl -s http://localhost:11434/api/generate -d '{
  "model": "gpt-oss:20b",
  "prompt": "What color is the sky?",
  "stream": false
}' | jq -r .response

Use It in an AIFabric

Declare an LLM with provider: ollama, and have each agent reference it by name. The model must exactly match a name in ollama list, tag included. You don't need an endpoint: the orama-operator fills in the URL above.

spec:
  llms:
    - name: local-model
      provider: ollama
      model: gpt-oss:20b
  agents:
    - name: supervisor
      type: internal
      internal:
        model: <composition-model>/<mode>
        llm: local-model

Apply it with:

brc install aifabric myfabric.yaml

To use an Ollama server somewhere else, set endpoint.baseUrl on the LLM.

Context Length

The context is how much of a conversation the model sees at once. Agents don't choose it, so Ollama sets it from the GPU's memory:

GPU memory	Context
Under 23 GiB	4096 tokens
23 to 47 GiB	32768 tokens
47 GiB or more	262144 tokens, capped at the model's own maximum

Agents need more than 4096 tokens. When a conversation doesn't fit, Ollama drops its oldest messages, and the agent loses track of earlier steps. On a GPU under 23 GiB, raise the context to 32768 tokens:

kubectl -n inference set env deployment/ollama OLLAMA_CONTEXT_LENGTH=32768

Ollama restarts, and models load again on their next request.

⚠ Important

OLLAMA_CONTEXT_LENGTH sets a fixed value, not a minimum, and a larger context uses more GPU memory. Leave it unset on GPUs with 23 GiB or more.

Security

⚠ Important

The Ollama API has no authentication. Any pod in the cluster can call it, including to pull, delete or run models. Never expose it outside the cluster through a NodePort, a LoadBalancer or an Ingress.

For testing from your machine, use kubectl -n inference port-forward svc/ollama 11434.

Troubleshooting

Symptom	Check
An agent gets "model not found"	The model in the AIFabric doesn't exactly match a name in ollama list, tag included.
An agent forgets earlier steps or tool results	The context is too small. Check CONTEXT in ollama ps, and see Context Length.
Answers are very slow	Check PROCESSOR in ollama ps. A CPU/GPU split means the model doesn't fit in the GPU's memory.
ollama pull fails with no space left on device	The model volume is full. Remove the models you no longer need with ollama rm.
An agent can't connect	Run the curl-test from Check That It's Running.
The pod isn't Running	Contact BubbleRAN support with the output of kubectl -n inference describe pod -l app=ollama.