For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Configure kagent to use any provider that serves the OpenAI API, including a self-hosted gateway in front of your own models.

Create the ModelConfig

The following example uses Cohere, which serves an OpenAI-compatible endpoint.

1. Save the API key from your provider as an environment variable.

   export PROVIDER_API_KEY=<your_api_key>

2. Create a Kubernetes Secret that stores the API key. Create the Secret in the same namespace as the ModelConfig, such as kagent.

   kubectl create secret generic kagent-my-provider -n kagent --from-literal PROVIDER_API_KEY=$PROVIDER_API_KEY

3. Create a ModelConfig that points at your provider’s endpoint.

   kubectl apply -f - <<EOF
   apiVersion: api.kagent.dev/v1alpha3
   kind: ModelConfig
   metadata:
     name: my-provider-model-config
     namespace: kagent
   spec:
     apiKeySecret: kagent-my-provider
     apiKeySecretKey: PROVIDER_API_KEY
     model: command-a-03-2025
     provider: OpenAI
     openAI:
       baseUrl: https://api.cohere.ai/compatibility/v1
   EOF

   Field	Description
   apiKeySecret	The name of the Kubernetes Secret that stores the API key, in the same namespace as this ModelConfig.
   apiKeySecretKey	The key within that Secret that stores the API key.
   model	The model identifier that your provider routes on. Consult your provider’s documentation.
   provider	The provider to use, OpenAI.
   openAI.baseUrl	Your provider’s OpenAI-compatible endpoint. Providers often serve this at a dedicated path, such as /compatibility/v1.

For the rest of the settings that the openAI block accepts, see OpenAI. Not every compatible provider honors every setting, so check your provider’s documentation before setting one. For every openAI field, including its type, default, and validation rules, see the API reference.

Self-hosted vLLM behind a gateway

To serve your own model, run a vLLM server behind an OpenAI-compatible gateway, such as Bifrost or LiteLLM. Requests then travel from kagent through the gateway to vLLM, and you configure the gateway as the provider endpoint.

Launch vLLM with automatic tool choice, and set model to the identifier that the gateway routes on.

Enable tool calling in vLLM

kagent sends a tools array with tool_choice: "auto" on every request. By default, kagent’s runtime registers a built-in ask_user tool on every agent, so kagent sends that array even when you do not bind any tools yourself. To prevent every agent turn from failing, launch vLLM with automatic tool choice enabled.

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --enable-auto-tool-choice \
  --tool-call-parser hermes

The correct --tool-call-parser depends on your model family. Qwen2.5 uses hermes. Llama 3.1 uses llama3_json. Parser names change across vLLM releases, so check the vLLM tool calling docs for the current name for your model.

Use the gateway’s model identifier

Set model to the identifier that your gateway routes on, which is often provider-prefixed and can differ from the bare model name that vLLM serves internally. Point openAI.baseUrl at the gateway. LiteLLM defaults to port 4000. Bifrost defaults to port 8080.

spec:
  apiKeySecret: kagent-my-provider
  apiKeySecretKey: PROVIDER_API_KEY
  model: vllm/Qwen/Qwen2.5-7B-Instruct
  provider: OpenAI
  openAI:
    baseUrl: http://litellm.kagent.svc.cluster.local:4000/v1

Troubleshooting a 400 from the provider

When every agent message fails with a generic provider API error (status 400), the most common cause is a vLLM server started without --enable-auto-tool-choice and a matching --tool-call-parser. Restart vLLM with both flags and try again.

TLS

A provider on your own network might present a certificate that the agent does not already trust. The tls block defines one field, disableVerify, which turns off certificate verification entirely when you set it to true. The field defaults to false.

The following ModelConfig turns off verification for a provider on an internal network.

spec:
  apiKeySecret: kagent-my-provider
  apiKeySecretKey: PROVIDER_API_KEY
  model: command-a-03-2025
  provider: OpenAI
  openAI:
    baseUrl: https://llm.internal.example.com/v1
  tls:
    disableVerify: true

Use the ModelConfig

Reference the ModelConfig by name from an AgentTemplate in the same namespace.

spec:
  modelConfig:
    name: my-provider-model-config

Next steps