Abstract
A TypeScript extension for the Pi coding agent that dynamically discovers locally hosted LLM models across LM Studio, llama.cpp, and llama-swap backends, registers them with context-window metadata, and routes agent requests through a shared OpenAI-compatible API surface. The interesting problem it solves is making local model servers feel discoverable rather than requiring every model to be hard-coded into agent configuration.
1. What This Is
Pi Local LLM is a TypeScript plugin for the Pi coding-agent framework. Instead of listing every model manually in agent configuration, the extension queries a local inference server at startup, enumerates whatever models are currently loaded, reads their context-window metadata, and registers them as selectable providers.
It supports three backend families — LM Studio, llama.cpp, and llama-swap — behind a shared OpenAI-compatible API surface. Configuration covers local origin, port, and API-key behavior, so the extension can target another machine or a home-server endpoint as well as localhost. The package is distributed via npm for drop-in installation into an existing Pi environment.
2. How It Works
The extension follows a linear discovery-and-registration pipeline at agent startup, then hands off to the Pi framework for request routing.
| # | Stage | Input | Tool | Output |
|---|---|---|---|---|
| 01 | Load config | Local origin, port, API key | TypeScript config reader | Resolved endpoint |
| 02 | Discover models | Resolved endpoint | OpenAI-compatible /models query | Model list |
| 03 | Read metadata | Model list | Provider metadata fields | Context length, model info |
| 04 | Apply overrides | Metadata + user config | Override map | Corrected context values |
| 05 | Register with Pi | Enriched model list | Pi extension API | Selectable providers in agent |
| 06 | Route requests | Agent prompt + selected model | OpenAI-compatible chat API | Model response |
3. Implementation Notes
3.1 Dynamic discovery over static config
The core design choice is querying the server for its current model list rather than maintaining a static registry. Local inference servers frequently load and unload models during a session, so a hard-coded list goes stale quickly. The extension treats the server as the source of truth and re-enumerates on each agent session start.
3.2 Multi-backend abstraction
LM Studio, llama.cpp, and llama-swap all expose an OpenAI-compatible request/response surface, which lets the extension use a single query path for all three. The backend-specific differences are limited to how models are listed and what metadata fields are populated. This keeps the Pi integration decoupled from any single hosting application.
3.3 Context-window metadata and overrides
Not every local server advertises a useful context-length value. The extension reads the provider-supplied metadata when present and falls back to a user-supplied override map for cases where the value is missing, zero, or simply wrong. This matters because the Pi agent uses context length to decide how much of the conversation history to include in each request.
4. Constraints
-
Point-in-time discovery
Models are enumerated once at agent startup. A model loaded or unloaded on the server mid-session will not appear until the agent restarts or the extension re-queries.
-
OpenAI-compatible API assumption
All supported backends must conform to the OpenAI /models and /chat/completions surface. A local server with a non-standard API (e.g., raw GGUF loading without an HTTP wrapper) is not supported.
-
Single endpoint per configuration
The extension targets one origin/port pair. There is no built-in failover, load-balancing, or multi-server aggregation, so a downed server means no local models.
-
No quality or latency routing
The extension registers models but does not benchmark or rank them. Which model the agent uses is a user selection; there is no automatic preference for the fastest or highest-quality option.
5. Next
- a. Add periodic re-discovery or a hot-reload hook so models loaded after startup appear without restarting the Pi agent.
- b. Support multi-endpoint configuration with basic health-check probing, enabling failover between a home server and a local GPU box.
- c. Introduce a lightweight latency or token-throughput probe at registration time so the agent can surface a recommended default model rather than leaving selection entirely manual.
— end of report —