Inference Services

Agents need LLMs. Besides calling a cloud provider directly, they can use two services installed in your cluster, in the inference namespace:

• Ollama runs open models, such as gpt-oss, on a GPU in your cluster. The models run on your own hardware, with no external API and no per-token cost.
• OmniRoute is a gateway for cloud LLM providers, especially their free tiers. It puts them all behind one OpenAI-compatible endpoint. You manage provider keys and routing in its dashboard instead of in every AIFabric. With the model set to auto, it picks a provider for each request and falls back to another when one fails.

The two are independent, and agents call each one directly. Don't route Ollama through OmniRoute:

agents ──► Ollama    ──► models on your GPU
agents ──► OmniRoute ──► cloud providers

Which One Do I Need?

You want to	Use
Run models on your own GPU	Ollama
Use cloud providers, especially free tiers, through one endpoint	OmniRoute
Use both local and cloud models	Both, each one called directly by the agents

Getting Started

1. Check which services run in your cluster:

   kubectl -n inference get pods

   Each service has one pod, which should be Running.

2. Set up the services you need:

   • Ollama: pull a model.
   • OmniRoute: log in to the dashboard, then add your providers and an API key.

3. Point your AIFabric at the service. Each guide's "Use It in an AIFabric" section (Ollama, OmniRoute) shows the llms entry to use.

⚠ Important

Don't delete the inference namespace. It removes both services, the models pulled into Ollama, the configuration you made in OmniRoute's dashboard, and its API keys.