Chain AI Models Across Machines

The Multi-Model Problem

Modern AI applications chain specialized models together. A typical pipeline might look like this:

• Machine A (A100 80GB): Large language model for reasoning
• Machine B (T4 16GB): Whisper model for audio transcription
• Machine C (A10G 24GB): Stable Diffusion for image generation
• Machine D (CPU): Orchestrator, embedding model, and lightweight classifiers

The three problems with this architecture are OOM risk, latency compounding, and the KV cache incompatibility wall.

OOM risk. Loading multiple large models on the same GPU is a balancing act. VRAM is not fungible -- each model allocates its weights, activations, and KV cache independently. Two 7B models that theoretically fit in 28 GB of VRAM will in practice crash on a 32 GB GPU because of fragmentation, CUDA overhead, and activation memory spikes during inference. Spreading models across machines is the reliable solution, but it introduces a network dependency.

Latency compounding. If each model invocation takes 200ms of compute time, a 5-stage pipeline takes 1 second of pure compute. Add HTTP connection overhead -- DNS lookup, TCP handshake, TLS negotiation -- at each stage, and you add 50-150ms per hop. That is an extra 250-750ms on top of the compute time, a 25 to 75 percent overhead from networking alone. For real-time applications (voice assistants, interactive agents), this pushes response times past the threshold where users perceive delay.

KV cache incompatibility. Different model frameworks (vLLM, TGI, Ollama, TensorRT-LLM) use incompatible KV cache formats. You cannot pass the internal state from one model to the next. Every hop in the pipeline must serialize the output to text, send it over the wire, and have the next model re-tokenize and re-encode it. This is unavoidable, but it means the network transfer is the only thing you can optimize -- you are already paying the serialization tax.

Why HTTP Per Request Adds Unnecessary Overhead

The standard approach is to give each model an HTTP API (typically REST or gRPC) and have the orchestrator make HTTP calls to each one sequentially. This works, but every request pays the full connection lifecycle cost.

Consider a chain of three model calls over HTTPS:

# Each HTTP call includes:
# - DNS resolution:    ~5ms  (cached after first)
# - TCP handshake:     ~10ms (1 RTT)
# - TLS handshake:     ~30ms (2 RTTs)
# - Request/response:  ~200ms (model inference)
# - Connection close:  ~5ms
# Total per call:      ~250ms
# Total for 3 calls:   ~750ms
# Of which networking: ~150ms (20%)

# With HTTP keep-alive (connection reuse):
# First call:          ~250ms (full setup)
# Subsequent calls:    ~200ms (reuse connection)
# Total for 3 calls:   ~650ms
# But: keep-alive breaks across machines, through NAT, and after idle timeouts

HTTP keep-alive helps within the same client-server pair, but it has limitations. Keep-alive connections expire after idle timeouts (typically 60 seconds). Load balancers may reassign connections. And if the model services are behind NAT or on different networks, keep-alive requires maintaining public endpoints for each service.

For a pipeline that processes thousands of requests per day across the same set of models, the connection setup overhead is pure waste. What you want is a persistent tunnel: connect once, keep it open indefinitely, and stream data through it for every inference call.

Persistent Tunnels: Connect Once, Stream Continuously

A Pilot Protocol connection is persistent by design. When two agents connect, they establish an encrypted UDP tunnel, and the daemon sends a keepalive over it every 25 seconds. The tunnel survives NAT rebinding and transient packet loss. There is no per-request key exchange because the tunnel is already established.

The model for chaining is simple: each model service registers as a Pilot agent and listens on port 80 for HTTP requests over the overlay. The orchestrator connects to each model agent once, then reuses those persistent tunnels for every inference call. The connection setup cost (STUN discovery, key exchange, tunnel establishment) is paid once, not per request.

# Orchestrator connects to each model agent (one-time cost)
pilotctl ping llm-agent      # LLM agent
pilotctl ping whisper-agent  # Whisper agent
pilotctl ping image-agent    # Image gen agent

# After tunnels are established, every subsequent call
# goes directly through the existing encrypted tunnel.
# No public DNS, no TLS negotiation.

Architecture: Model Agents on Pilot

Each machine runs a Pilot daemon and a model server. The model server listens on port 80 over the Pilot overlay using the Go driver. The orchestrator calls each model sequentially, passing the output of one as the input to the next.

Here is the layout:

• Machine A -- LLM agent, hostname llm-agent, serves /v1/completions on port 80
• Machine B -- Whisper agent, hostname whisper-agent, serves /v1/transcribe on port 80
• Machine C -- Image agent, hostname image-agent, serves /v1/generate on port 80
• Machine D -- Orchestrator, chains calls across A, B, and C

Each model agent starts its daemon with a hostname that names its role. Tags are optional labels: a peer sees them when it looks the node up, but there is no search by tag.

# On Machine A
pilotctl daemon start --hostname llm-agent
pilotctl extras set-tags model-service llm reasoning

# On Machine B
pilotctl daemon start --hostname whisper-agent
pilotctl extras set-tags model-service whisper audio

# On Machine C
pilotctl daemon start --hostname image-agent
pilotctl extras set-tags model-service diffusion image

The orchestrator knows these hostnames and resolves each one through the registry. A private node resolves only for peers it trusts, so the orchestrator handshakes with each model agent first and each one approves:

# On Machine D
pilotctl handshake <llm-agent address> "inference pipeline"
# On Machine A
pilotctl approve <orchestrator node id>

Code Example: Go Service That Chains Model Calls

Here is a Go orchestrator that receives a user query, chains it through three model agents over persistent Pilot tunnels, and returns the final result.

package main

import (
    "bytes"
    "context"
    "encoding/json"
    "fmt"
    "io"
    "net"
    "net/http"
    "time"

    "github.com/pilot-protocol/common/driver"
)

// Model agent hostnames (resolved through the registry when dialing)
var (
    llmAddr     = "llm-agent"
    whisperAddr = "whisper-agent"
    imageAddr   = "image-agent"
)

type ChainRequest struct {
    AudioURL string `json:"audio_url"`
}

type ChainResponse struct {
    Transcript string `json:"transcript"`
    Analysis   string `json:"analysis"`
    ImageURL   string `json:"image_url"`
    TotalMs    int64  `json:"total_ms"`
}

func main() {
    d, err := driver.Connect("")
    if err != nil {
        panic(err)
    }

    // Listen on port 80 over the Pilot overlay
    ln, err := d.Listen(80)
    if err != nil {
        panic(err)
    }

    // Create an HTTP client that routes through Pilot tunnels
    transport := &http.Transport{
        DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) {
            // addr is "hostname:port"; resolve + dial over the Pilot overlay
            host, _, _ := net.SplitHostPort(addr)
            info, err := d.ResolveHostname(host)
            if err != nil {
                return nil, err
            }
            return d.Dial(fmt.Sprintf("%s:80", info["address"]))
        },
    }
    client := &http.Client{
        Transport: transport,
        Timeout:   60 * time.Second,
    }

    mux := http.NewServeMux()
    mux.HandleFunc("/chain", func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()

        var req ChainRequest
        json.NewDecoder(r.Body).Decode(&req)

        // Stage 1: Transcribe audio via Whisper agent
        transcript, err := callModel(client, whisperAddr,
            "/v1/transcribe", map[string]string{"audio_url": req.AudioURL})
        if err != nil {
            http.Error(w, err.Error(), 500)
            return
        }

        // Stage 2: Analyze transcript via LLM agent
        analysis, err := callModel(client, llmAddr,
            "/v1/completions", map[string]string{
                "prompt": "Analyze this transcript and summarize key points: " + transcript,
            })
        if err != nil {
            http.Error(w, err.Error(), 500)
            return
        }

        // Stage 3: Generate visualization via Image agent
        imageURL, err := callModel(client, imageAddr,
            "/v1/generate", map[string]string{
                "prompt": "Infographic summarizing: " + analysis,
            })
        if err != nil {
            http.Error(w, err.Error(), 500)
            return
        }

        resp := ChainResponse{
            Transcript: transcript,
            Analysis:   analysis,
            ImageURL:   imageURL,
            TotalMs:    time.Since(start).Milliseconds(),
        }
        w.Header().Set("Content-Type", "application/json")
        json.NewEncoder(w).Encode(resp)
    })

    fmt.Println("Orchestrator listening on port 80")
    http.Serve(ln, mux)
}

func callModel(client *http.Client, agentAddr, path string, payload any) (string, error) {
    body, _ := json.Marshal(payload)

    // Request goes through the persistent Pilot tunnel
    // No public DNS, no TLS negotiation
    url := fmt.Sprintf("http://%s%s", agentAddr, path)
    resp, err := client.Post(url, "application/json", bytes.NewReader(body))
    if err != nil {
        return "", fmt.Errorf("model call to %s failed: %w", agentAddr, err)
    }
    defer resp.Body.Close()

    result, _ := io.ReadAll(resp.Body)

    var parsed struct {
        Result string `json:"result"`
    }
    json.Unmarshal(result, &parsed)
    return parsed.Result, nil
}

The key detail is the custom http.Transport: its DialContext resolves the hostname through the Pilot registry and dials over the encrypted Pilot tunnel instead of the standard network stack. The http.Client uses it transparently -- the code looks like normal HTTP calls, but the transport is a persistent encrypted tunnel. There are no public DNS lookups and no TLS negotiations; the HTTP client reuses its connections, and a new one costs a single round trip over the existing tunnel.

Persistent Tunnel vs Per-Request HTTP

What changes is where the connection cost is paid. We haven't published measurements for this setup, so the comparison below is qualitative; time your own chain, and use pilotctl bench <peer> for raw tunnel throughput between two machines.

For latency-sensitive applications, the tunnel approach avoids sporadic TLS handshakes and DNS timeouts in the middle of a chain.

The more significant advantage is resilience. HTTP keep-alive connections break after idle timeouts or load balancer reshuffling. A Pilot tunnel stays up between calls because the daemon sends a keepalive every 25 seconds, which also keeps the NAT mapping open. For long-running inference pipelines that process requests over hours or days, this removes a class of transient connection failures.

When to Use This vs Single-Machine Serving

Distributed model chaining adds complexity. It is the right choice when:

• Models exceed single-machine VRAM. If your pipeline needs 3 models that each require 20+ GB of VRAM, you physically cannot fit them on one GPU. Distribute them.
• Models need different hardware. LLMs benefit from A100s. Whisper runs well on T4s. Image generation needs different batch sizes. Matching model to hardware saves cost.
• Models have different scaling profiles. Your LLM handles 10 requests/second. Your embedding model handles 1,000. Running them on the same machine wastes the embedding model's capacity or throttles the LLM.
• Teams own different models. The NLP team maintains the language model. The vision team maintains the image model. Letting each team deploy and scale independently reduces coordination overhead.

Single-machine serving is better when:

• Models fit in memory together. Two 3B models on an A100 80GB work fine. Avoid the distributed complexity.
• Latency is critical below 200ms. No network can match local function calls. If sub-200ms end-to-end latency is a hard requirement, keep everything local.
• Pipeline is simple. Two models in sequence do not justify the overhead of running multiple Pilot daemons. Use subprocess calls or in-process model loading.

The persistent tunnel architecture shines for medium-complexity pipelines (3-10 models) processing sustained traffic (hundreds to thousands of requests per hour) where models are too large to co-locate and need to run on heterogeneous hardware across multiple machines or regions.

Getting Started

Set up the pipeline in four steps:

# Step 1: Install Pilot on each machine
curl -fsSL https://pilotprotocol.network/install.sh | sh

# Step 2: Start each daemon with a hostname for its role
pilotctl daemon start --hostname llm-agent  # On Machine A

# Step 3: Establish trust between the orchestrator and each model agent
pilotctl handshake <llm-agent address> "inference pipeline"  # On Machine D
pilotctl approve <orchestrator node id>                    # On Machine A

# Step 4: Run your model server using driver.Listen(80)
# The orchestrator calls each model agent by hostname

Trust is set up pair by pair here. If you'd rather put the machines in a private network, networks are created by the operator of a registry, so run your own rendezvous and create one there.

Once the tunnels are established, model calls flow through persistent encrypted connections. The orchestrator code looks like standard HTTP, but the transport is Pilot. No certificate management, no firewall rules - just persistent tunnels that stay connected and keep your inference pipeline fast. The only infrastructure requirement is a registry and beacon for discovery and NAT traversal (the public ones run by Vulture Labs work out of the box, or you can run your own).