Contributing: A Codebase Tour

The Repository Map

The daemon and CLI live in pilotprotocol. Shared types live in common. The registry, the beacon and each daemon plugin are separate repositories that pilotprotocol imports as Go modules. Here is what each piece does and where the important files live.

common/protocol - Addresses, Headers, CRC

This is the foundation layer. It defines the 48-bit virtual address format ([16-bit network][32-bit node], text format N:NNNN.HHHH.LLLL), the 34-byte packet header structure, the well-known ports, and CRC32 checksum computation. Nearly every other package imports protocol. If you are fixing a serialization bug or adding a new message type, this is where you start. The address parsing is strict, and the text format must match exactly. The header includes fields for source/destination addresses, port numbers, sequence and acknowledgment numbers, flags, window size, and checksum. All multi-byte fields are big-endian.

pilotprotocol/pkg/daemon - The Core Engine

The daemon is the heart of Pilot Protocol. It manages the UDP tunnel, handles NAT traversal on the client side, runs the port manager, processes IPC commands from the CLI and SDKs, and coordinates encryption. This is the largest package and the one you will spend the most time reading. Key files handle tunnel read/write loops (tunnel.go), connection state machines (SYN/ACK/FIN lifecycle) and sliding window transport with congestion control, flow control with advertised receive windows, Nagle-style coalescing of small writes, segmentation (ports.go), keepalive probes, idle timeouts, and graceful shutdown. If you are working on transport reliability, congestion behavior, or connection lifecycle, this is the package.

rendezvous - The Registry

The registry is the rendezvous server's brain, in its own repository. It handles node registration, address assignment, hostname resolution, networks and their roles and policies, persistence (snapshots plus a write-ahead log), and hot-standby replication. Its cmd/rendezvous binary runs the registry and a beacon together; use -registry-addr and -beacon-addr to set their listen addresses. This binary is what you deploy if you run your own rendezvous. If you are working on discovery, node management, or multi-server coordination, start here.

beacon - STUN, Hole-Punch, Relay

The beacon handles the server side of NAT traversal. It answers STUN-style discovery requests so daemons learn their external address, coordinates UDP hole-punching (sending BeaconMsgPunchCommand to both peers), and relays traffic when no direct path works. Relayed packets travel in BeaconMsgRelay frames with the format [0x05][senderNodeID(4)][destNodeID(4)][payload...]; the payload stays encrypted end to end. The beacon can listen dual-stack (IPv4 and IPv6). The daemon's fallback chain (direct, hole-punch, relay) spans the daemon and the beacon, so changes here require careful testing across NAT setups.

common/driver - IPC Client and Go SDK

The driver is the Go SDK that applications use to talk to the daemon. It connects over a Unix domain socket (/tmp/pilot.sock by default) and provides an API for dialing and listening on ports, sending datagrams, resolving hostnames, and managing trust. It serializes IPC frame writes with a mutex so concurrent goroutines don't interleave frames. The Conn type implements net.Conn, with working read deadlines, and Listen returns a net.Listener, which is what lets http.Serve run over Pilot. If you are building an application on Pilot, the driver is your entry point.

pilotprotocol/cmd/pilotctl - The CLI

The command-line interface. It wraps the driver with commands like info, ping, connect, send-message, inbox, send-file, publish/subscribe, handshake/approve/trust, bench, lookup, and the appstore commands. Most commands are thin wrappers that parse flags, call the driver, and format output (text, or JSON with --json). Adding a new CLI command is one of the easiest contributions. Commands like set-hostname, set-public/set-private, and deregister go through the daemon IPC (not directly to the registry) because the daemon signs them with the node's key.

Plugins: dataexchange, eventstream, handshake, policy, webhook

The built-in services are daemon plugins, each in its own repository. A plugin implements coreapi.Service from common/coreapi (Name, Order, Start, Stop); Start receives the daemon's capabilities, including Streams for listening on and dialing ports. dataexchange (port 1001) carries messages and files and writes them to the inbox and ~/.pilot/received/. eventstream (port 1002) is the pub/sub broker. handshake runs trust handshakes, policy evaluates network policies, and webhook delivers event notifications. pilotprotocol/cmd/daemon/main.go registers them with the plugin runtime. The echo service on port 7 is built into the daemon itself.

tests/ - Integration Tests

The integration test suite in pilotprotocol/tests uses testenv.go to spin up a full in-process environment: beacon, registry, and multiple daemons with the standard plugins, all running in the same process with real UDP sockets. This gives you end-to-end testing without deploying anything. Run go test -parallel 4 -count=1 ./tests/ for the current tally. The test files are organized by feature with a zz_ prefix: zz_handshake_test.go, zz_privacy_test.go, zz_nat_traversal_test.go, zz_webhook_test.go, and so on. Every PR should include tests, and most features can be tested entirely through testenv without any external infrastructure.

The Test Environment

The test environment is the single most important file for contributors to understand. tests/testenv.go creates a miniature Pilot network inside a single Go test process. Here is how to use it:

# Run all integration tests (always use -parallel 4)
go test -parallel 4 -count=1 ./tests/

# Run a specific test
go test -parallel 4 -count=1 -run TestHandshake ./tests/

# Run with verbose output
go test -parallel 4 -count=1 -v -run TestHandshake ./tests/

NewTestEnv starts a beacon and a registry on random available ports, and AddDaemon starts as many daemons as the test needs. Each daemon gets its own virtual address, its own IPC socket, and its own tunnel. The daemons communicate over real UDP on localhost, so you are testing the actual network path, not mocks. Test daemons are public, so they can talk without a trust handshake unless the test is about trust.

A typical test follows this pattern:

func TestEchoRoundTrip(t *testing.T) {
    t.Parallel()

    // Beacon + registry; cleaned up automatically when the test ends
    env := NewTestEnv(t)
    a := env.AddDaemon()
    b := env.AddDaemon()

    // Open a connection from A to B's built-in echo service (port 7)
    conn, err := a.Driver.DialAddr(b.Daemon.Addr(), protocol.PortEcho)
    if err != nil {
        t.Fatalf("dial echo: %v", err)
    }
    defer conn.Close()

    // Send data and verify the echo
    conn.Write([]byte("hello"))
    buf := make([]byte, 5)
    if _, err := io.ReadFull(conn, buf); err != nil {
        t.Fatalf("read: %v", err)
    }
    if string(buf) != "hello" {
        t.Fatalf("echo = %q, want hello", buf)
    }
}

The environment handles all the setup: STUN discovery, registry registration, tunnel establishment, and cleanup. Your test just describes the scenario and checks the result.

Adding a New Service: A Walkthrough

Let us walk through adding a hypothetical new service on port 1004. Say you want a "latency" service that responds with one-way latency measurements. Here is the complete process.

Step 1: Write the Plugin

Every service is a plugin that implements coreapi.Service. The built-in plugins each live in their own Go module (look at eventstream/service.go for a complete one); a new plugin can start the same way. Start listens on the service's port and handles each incoming stream:

package latency

import (
    "context"
    "encoding/json"
    "fmt"
    "io"
    "time"

    "github.com/pilot-protocol/common/coreapi"
)

// PortLatency is the port the latency service listens on.
const PortLatency uint16 = 1004

// Response is the JSON payload returned to the caller.
type Response struct {
    ReceivedAt time.Time     `json:"received_at"`
    Latency    time.Duration `json:"latency_ns"`
}

// Service is the daemon plugin.
type Service struct{ ln coreapi.Listener }

func (s *Service) Name() string { return "latency" }
func (s *Service) Order() int   { return 200 }

// Start is called by the plugin runtime with the daemon's capabilities.
func (s *Service) Start(ctx context.Context, deps coreapi.Deps) error {
    ln, err := deps.Streams.Listen(PortLatency)
    if err != nil {
        return fmt.Errorf("latency: listen on port %d: %w", PortLatency, err)
    }
    s.ln = ln
    go func() {
        for {
            st, err := ln.Accept()
            if err != nil {
                return
            }
            go handle(st)
        }
    }()
    return nil
}

func (s *Service) Stop(ctx context.Context) error {
    if s.ln == nil {
        return nil
    }
    return s.ln.Close()
}

// handle answers each timestamp the peer sends with the measured latency.
func handle(conn io.ReadWriteCloser) {
    defer conn.Close()
    buf := make([]byte, 1024)
    for {
        n, err := conn.Read(buf)
        if err != nil {
            return
        }
        received := time.Now()

        // Parse the timestamp sent by the caller
        var sent time.Time
        if err := json.Unmarshal(buf[:n], &sent); err != nil {
            return
        }

        // Respond with latency measurement
        resp := Response{
            ReceivedAt: received,
            Latency:    received.Sub(sent),
        }
        data, _ := json.Marshal(resp)
        if _, err := conn.Write(data); err != nil {
            return
        }
    }
}

Step 2: Register with the Daemon

In cmd/daemon/main.go, register the plugin with the runtime alongside the others. The runtime starts plugins by Order and stops them in reverse registration order on shutdown.

// In cmd/daemon/main.go, next to the dataexchange and eventstream registrations:
if err := rt.Register(&latency.Service{}); err != nil {
    log.Fatalf("register latency: %v", err)
}

Step 3: Add the CLI Command

In cmd/pilotctl, add a latency subcommand that connects to port 1004 on a target, sends a timestamp, and reads back the response (connectDriver and parseAddrOrHostname are existing helpers there):

// pilotctl latency <address|hostname>
func cmdLatency(args []string) {
    d := connectDriver()
    defer d.Close()
    target, err := parseAddrOrHostname(d, args[0])
    if err != nil {
        fatalCode("not_found", "%v", err)
    }
    conn, err := d.DialAddr(target, latency.PortLatency)
    if err != nil {
        fatalCode("connection_failed", "dial: %v", err)
    }
    defer conn.Close()

    data, _ := json.Marshal(time.Now())
    conn.Write(data)

    buf := make([]byte, 1024)
    n, _ := conn.Read(buf)

    var resp latency.Response
    json.Unmarshal(buf[:n], &resp)
    fmt.Printf("Latency: %v\n", resp.Latency)
}

Step 4: Write the Test

The handler only needs a stream, so a unit test can drive it over an in-memory pipe:

func TestHandleReportsLatency(t *testing.T) {
    client, server := net.Pipe()
    defer client.Close()
    go handle(server)

    // Send timestamp
    data, _ := json.Marshal(time.Now())
    if _, err := client.Write(data); err != nil {
        t.Fatalf("write: %v", err)
    }

    // Read response
    buf := make([]byte, 1024)
    n, err := client.Read(buf)
    if err != nil {
        t.Fatalf("read: %v", err)
    }

    var resp Response
    if err := json.Unmarshal(buf[:n], &resp); err != nil {
        t.Fatalf("decode: %v", err)
    }
    if resp.ReceivedAt.IsZero() || resp.Latency < 0 {
        t.Fatalf("bad response: %+v", resp)
    }
}

For an end-to-end test, register the plugin in registerStandardPlugins in tests/testenv.go and dial port 1004 from a second daemon, the way the existing tests exercise the built-in services. That is it: plugin, registration, CLI, test. Look at dataexchange or eventstream for real-world examples of more complex service protocols.

Layering and Linting

The project enforces a layered architecture. layers.yaml in pilotprotocol assigns every package to a layer (datagram I/O, routing, transport, services, and so on), and imports must always point downward. The check-layers tool reads that file and fails CI on any violation, so if your change adds an import across layers, expect to restructure it rather than suppress the check.

Where to Start

Check the open issues in the GitHub repositories first. If nothing there fits, these kinds of change are self-contained and touch one area each:

• A new CLI command or flag. Commands in cmd/pilotctl are thin wrappers over the driver; add the usage text to the commandHelp map alongside the code.
• More test coverage. Pick a feature with thin tests in tests/ (broadcast in zz_broadcast_test.go, for example) and add cases through testenv.
• Better error messages. Walk through the daemon's connection establishment code and add context to errors: which address was being dialed, which port, whether the peer was resolved. Wrapping errors with fmt.Errorf("dial %s port %d: %w", addr, port, err) makes debugging much easier, and it is a good way to learn the codebase.

Development Workflow

Here is the workflow that regular contributors use:

# Clone and verify the build
git clone https://github.com/pilot-protocol/pilotprotocol.git
cd pilotprotocol
go build ./...

# Run the full test suite
go test -parallel 4 -count=1 ./tests/

# Make your changes, then verify
go vet ./...
go test -parallel 4 -count=1 ./tests/

# Run just the test for your feature
go test -parallel 4 -count=1 -v -run TestYourFeature ./tests/

The project has few direct dependencies and no code generation steps. The Makefile provides convenience targets (make build puts the daemon and pilotctl in bin/), but go build and go test are all you need.

For larger changes, read the documentation and technical specification linked from the research page, which describes the wire protocol and message types. If your change modifies protocol behavior, note it in your PR.

Every contribution goes through standard GitHub PR review. Keep PRs focused: one feature or fix per PR. Include tests. Update the spec if you change protocol behavior. And always run the full test suite with -parallel 4 before pushing.

Where to Go Next

If you want to understand the protocol itself before diving into code, read How Pilot Protocol Works for the architecture overview. For NAT traversal specifically, the NAT traversal deep dive explains how the beacon coordinates hole punching and relay. For the trust and encryption model, see Zero-Dependency Agent Encryption and The Trust Model.

The codebase is deliberately plain. No frameworks, no generated code, few direct dependencies. Just Go, the standard library, and the problem. Because each piece is its own module with strict layering, you can understand the part you are changing without reading everything else.