Support pluggable model providers: Ollama / local models, Amazon Bedrock, and OpenAI-compatible endpoints

[ash663]

Summary

Today the agent model is not selectable. agent.provider is effectively fixed to acp, and the model arrives through the kiro-cli (KiroACP) backend. There is no way to point KiroCrew at a local model, at Amazon Bedrock, or at any OpenAI-compatible endpoint.

Requesting a provider abstraction for the agent model, with these backends:

1. Ollama / local models — http://localhost:11434, or any OpenAI-compatible base URL (llama.cpp server, LM Studio, vLLM, LiteLLM proxy)
2. Amazon Bedrock — native support, using the standard AWS credential chain and an explicit --profile / region
3. Generic OpenAI-compatible — a base_url + api_key pair, which covers OpenRouter, Together, Groq, and self-hosted proxies in one adapter

Current state (KiroCrew 2026-08)

agent.provider           = "acp"      # not user-selectable
agent.model              = "auto"
memory.embedding_provider = "llama_cpp"

Local inference exists in the product, but only for embeddings: llama-cpp-python is bundled and runs Qwen3-Embedding-0.6B in-process. Notably, Ollama used to be a supported embeddings backend and was removed — per configuration.md, every legacy value "including "ollama" and "none" is coerced to "llama_cpp"". So the local-model plumbing is half-present; it just doesn't extend to the agent.

Motivation

• Data residency / privacy. A local model keeps prompts, file contents, and page/DM text on the machine. This matters more for KiroCrew than for a plain coding assistant, because the agent reads real personal surfaces (browser pages, iMessage, calendar, local files).
• Bedrock is the obvious enterprise path. KiroCrew is an AWS-adjacent product and users already have Bedrock quota, IAM, VPC endpoints, model guardrails, and consolidated billing. Routing agent traffic through Bedrock lets an org adopt KiroCrew under controls it already owns.
• Air-gapped and offline use. Currently impossible for the agent, even though the embedding path is fully offline.
• Cost and model choice. Long autonomous runs (cron jobs, monitors, sub-agent fan-out) are exactly where a cheap local or self-hosted model is worth using for the routine steps, reserving a frontier model for hard ones. agent.tips_model / judge_model already hint at per-role model selection; extending that to providers is a natural follow-on.

Proposed shape

{
  "agent": {
    "provider": "ollama",              // acp | ollama | bedrock | openai_compatible
    "model": "qwen3:32b",
    "base_url": "http://localhost:11434",
    "bedrock": { "profile": "default", "region": "us-east-1" }
  }
}

Ideally the existing per-role keys (tips_model, judge_model, sub-agent models) can each target a different provider, so cheap roles go local while the main session stays on a frontier model.

Notes

• Tool-calling fidelity is the main risk: KiroCrew leans hard on tool use, and smaller local models are inconsistent at it. A capability probe at configure-time (does this model reliably emit tool calls?) with a clear warning would be better than silent degradation mid-run.
• A LiteLLM-style single adapter would cover Ollama, OpenRouter, and most hosted APIs with one implementation, at the cost of a dependency.

Secondary: stale documentation

getting-started.md still lists:

(Optional) Ollama for local vector-memory embeddings

This contradicts configuration.md, which states embeddings are always in-process via bundled llama-cpp and that "ollama" is coerced away. The line should be removed or rewritten.

[openSourcerer9000]

I thought I was crazy scanning the whole readme for how to change url endpoint.

Looks cool, may revisit once this is resolved