MCP server tools with Ollama

Ollama's OpenAI-compatible API

Ollama exposes its API at http://localhost:11434/v1 with OpenAI-compatible endpoints. The openai Python package works out-of-the-box by pointing base_url at Ollama:

import openai

# Ollama's OpenAI-compatible client
ollama_client = openai.AsyncOpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # Ollama doesn't require a real API key
)

# Test the connection
async def test_ollama():
    response = await ollama_client.chat.completions.create(
        model="llama3.1:8b",
        messages=[{"role": "user", "content": "Hello from MCP!"}],
    )
    print(response.choices[0].message.content)

Ollama also exposes its own native API at http://localhost:11434/api with a slightly different interface. Use the /v1 OpenAI-compatible endpoint for MCP integration — it's the standard that all existing MCP adapter code targets, and it ensures your code works with both Ollama and cloud-hosted OpenAI-compatible APIs (Groq, Mistral, Together, etc.) without changes.

Tool-capable Ollama models

Not all Ollama models support function calling. The model's Modelfile and the underlying model weights determine whether tool calling works. Some models include tool-calling training in their weights but produce unreliable results with MCP's JSON schemas.

Always verify tool calling with a simple test before building an MCP integration on a new model. The verification test in the next section catches silent failures early.

Verifying tool calling before building

import asyncio
import json
import openai

ollama_client = openai.AsyncOpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",
)

# Minimal test tool to verify function calling works
TEST_TOOL = [{
    "type": "function",
    "function": {
        "name": "get_time",
        "description": "Returns the current time",
        "parameters": {"type": "object", "properties": {}, "required": []},
    }
}]

async def verify_tool_calling(model: str) -> bool:
    response = await ollama_client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": "What time is it right now?"}],
        tools=TEST_TOOL,
        tool_choice="required",  # Force tool use
    )
    choice = response.choices[0]
    has_tool_calls = bool(choice.message.tool_calls)
    print(f"{model}: {'PASS' if has_tool_calls else 'FAIL — returned plain text'}")
    return has_tool_calls

asyncio.run(verify_tool_calling("llama3.1:8b"))

Use tool_choice="required" in the verification test to force the model to call a tool. If the model still returns plain text with tool_choice="required", its tool calling implementation is broken or not trained into the weights — do not build an MCP integration on top of it. With tool_choice="auto" (the default), models that don't support tools will silently skip tool calls, making failures invisible until you're debugging production behavior.

MCP to OpenAI function format conversion

from mcp import ClientSession
from mcp.client.http import http_client

async def get_ollama_tools(mcp_url: str) -> tuple[list, dict]:
    """Fetch MCP tools and convert to OpenAI function format.

    Returns (tool_specs, tool_map) where tool_map maps name → MCP tool.
    """
    async with http_client(mcp_url) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()
            result = await session.list_tools()

    tool_specs = []
    tool_map = {}
    for tool in result.tools:
        spec = {
            "type": "function",
            "function": {
                "name": tool.name,
                "description": tool.description or f"Call {tool.name}",
                # MCP inputSchema is already JSON Schema — use directly
                "parameters": tool.inputSchema or {
                    "type": "object",
                    "properties": {},
                },
            }
        }
        tool_specs.append(spec)
        tool_map[tool.name] = tool
    return tool_specs, tool_map

Ollama models are more sensitive to tool schema quality than cloud models. Keep inputSchema flat — avoid nested objects more than one level deep. Write description fields that are clear and specific: "Search the web for recent information on a topic" outperforms "Search tool". Poor descriptions cause small Ollama models to choose wrong tools or hallucinate tool arguments more frequently than they would for simpler schemas.

The MCP tool-calling loop with Ollama

import asyncio
import json
from mcp import ClientSession
from mcp.client.http import http_client

MCP_URL = "https://search.internal/mcp"

async def run_ollama_agent(question: str, model: str = "llama3.1:8b") -> str:
    async with http_client(MCP_URL) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()

            tool_specs, _ = await get_ollama_tools_from_session(session)

            messages = [{"role": "user", "content": question}]
            max_iterations = 10

            for _ in range(max_iterations):
                response = await ollama_client.chat.completions.create(
                    model=model,
                    messages=messages,
                    tools=tool_specs if tool_specs else None,
                    tool_choice="auto",
                    temperature=0,
                )

                choice = response.choices[0]
                messages.append({"role": "assistant", "content": choice.message.content,
                                  "tool_calls": choice.message.tool_calls})

                if not choice.message.tool_calls:
                    # No tool calls — final answer
                    return choice.message.content or ""

                # Dispatch tool calls to MCP server (sequentially — Ollama models
                # rarely request multiple tools per turn)
                for tc in choice.message.tool_calls:
                    fn = tc.function
                    args = json.loads(fn.arguments) if fn.arguments else {}

                    mcp_result = await session.call_tool(fn.name, arguments=args)
                    result_text = "\n".join(
                        c.text for c in mcp_result.content if hasattr(c, "text")
                    )
                    if mcp_result.isError:
                        result_text = f"Error: {result_text}"

                    messages.append({
                        "role": "tool",
                        "tool_call_id": tc.id,
                        "content": result_text,
                    })

            return "Agent reached maximum iterations without a final answer."

asyncio.run(run_ollama_agent("Which MCP endpoints have been down this week?"))

Unlike Gemini, Ollama models typically generate one tool call per turn. The loop processes tool calls sequentially — this is simpler and is rarely a bottleneck because the local inference time (0.5–5 seconds depending on hardware and model size) dwarfs the MCP round-trip. The max_iterations guard is important: small local models occasionally loop on tool errors rather than admitting they can't answer.

Latency profile: local inference vs MCP round-trip

Ollama's inference time depends on hardware. The MCP server round-trip is typically 10–200 ms — the same as any network call. The ratio of inference time to tool call time determines where to optimize:

In local Ollama setups, inference is almost always the bottleneck — MCP tool calls are fast. The implication: tool call parallelism (like Gemini's multi-call architecture) provides minimal benefit for Ollama because you'd need to be running inference on multiple models simultaneously. Optimize inference first (quantization, GPU offloading, smaller model) before tuning MCP call patterns.

Monitoring MCP servers in Ollama deployments

Local Ollama deployments often run unattended — a developer sets up a local agent, and it runs periodically without anyone actively watching it. When an MCP server the agent depends on goes down, the failure is silent: the agent runs, finds no tools available or gets errors, and either returns a degraded response or stops working entirely. Without external monitoring, the failure isn't noticed until someone manually checks the agent's output.

This failure pattern is common even when the MCP server is remote and the Ollama instance is local. The MCP server might be a cloud-hosted SaaS tool (a search API, a database connector, a webhook service), and any of these can go down independently of the local Ollama instance. AliveMCP monitors the MCP server's initialize endpoint continuously and alerts on failure — so even in a local development setup or an unattended automated agent, you know when the MCP server your Ollama agent depends on becomes unavailable. Set alerts to a Slack channel or email so you're notified before investigating why the agent is producing degraded output.

Further reading

• MCP servers with Groq — OpenAI-compatible ultra-fast inference
• MCP servers with OpenAI Agents SDK — native MCPServerHTTP integration
• MCP server tool design — flat schemas for better LLM accuracy
• MCP server stdio transport — local process tool servers
• Open-source MCP monitoring — AliveMCP public dashboard and self-hosted options
• AliveMCP — continuous protocol monitoring for MCP servers