# Pando! 🌳
*Dynamic Schema Records API*

A lightweight, high-performance JSON records service built with **Rust**, **Axum**, and **SQLite**.

The service validates incoming JSON payloads against a dynamic schema specification defined in a JSON configuration file. Schemas support validation modes (`strict` vs. `loose`), data type constraints, and exact forced values (e.g., pinning API payload versions). Schemas can be reloaded at runtime without restarting the server.

Instances can run as a **primary** (accepts writes) or as a **read-only replica** that lazily pulls records from a primary. Replication is built for high-latency and unreliable links.

---

## Features

- **Dynamic Schema Validation**:
  - Enforce required types: `string`, `integer`, `number`, `boolean`, `object`, `array`, `null`.
  - Force exact values on any field (e.g., pin `"version": 2`).
  - Validation modes:
    - **`strict`**: All defined fields are required, and unknown fields are rejected.
    - **`loose`**: All defined fields are required, but additional unknown fields are accepted.
- **Hot-Reloadable Configuration**: Reload `config.json` via an HTTP admin endpoint with atomic swap and rollback on error.
- **Persistent Storage**: Validated records are stored with a generated UUID, an ISO-8601 UTC timestamp and a global sequence number (`seq`) in an SQLite database (`pando.db`).
- **Replication & Dynamic Role Switching**:
  - Pull-based and cursor-driven: the replica asks for everything after its last `seq`.
  - Append-only journal with a global tick (`seq`) that is preserved on replicas.
  - Seamless role switch / failover without data loss or split-brain.
  - Automatic CA credential sharing during dynamic join so promoted replicas can continue minting tokens and signing nodes.
  - Coordinated 1-step switchover API (`/api/admin/switchover`) and fine-grained controls (`/step-down`, `/promote`, `/repoint`).
  - Replicas can serve replication to other replicas (relay).
- **Structured Logging**: Built with `tracing` outputting to standard output.

---

## Prerequisites

- [Rust toolchain](https://rustup.rs/) (1.75+ recommended)
- SQLite development libraries (if linking dynamically)

### Cargo dependencies of note

```toml
reqwest    = { version = "0.12", default-features = false, features = ["json", "rustls-tls", "gzip"] }
tower-http = { version = "0.5", features = ["compression-gzip"] }
```

`reqwest` is used by the replica to pull changes. `tower-http` provides gzip response compression.

---

## Configuration

### Schema file

The service looks for a schema configuration file at the path specified by the `APP_CONFIG` environment variable, defaulting to `config.json` in the current working directory.

#### Example: `config.json`

```json
{
  "mode": "strict",
  "fields": {
    "version": {
      "type": "integer",
      "value": 2
    },
    "message": {
      "type": "string"
    },
    "enabled": {
      "type": "boolean"
    }
  }
}
```

#### Schema Specification

| Property | Type | Description |
| :--- | :--- | :--- |
| `mode` | `string` | `"strict"` or `"loose"`. |
| `fields` | `object` | Map of field names to validation specifications. Must contain at least one field. |
| `fields.<name>.type` | `string` | Expected JSON type: `string`, `integer`, `number`, `boolean`, `object`, `array`, `null`. |
| `fields.<name>.value` | *any* *(optional)* | Expected exact value. Rejects any request where the value does not strictly match. |

The schema is only enforced on writes to a primary. A replica still loads the file at startup, but it does not re-validate replicated records, since the primary already did.

### Environment variables

| Variable | Default | Description |
| :--- | :--- | :--- |
| `APP_CONFIG` | `pando.json` | Path to the schema file. |
| `APP_BIND` | `127.0.0.1:3000` | Listen address. Change this for a primary that replicas must reach over a network. |
| `APP_ROLE` | `primary` | `primary` or `replica`. |
| `APP_PRIMARY_URL` | *none* | Base URL of the replication source, e.g. `https://primary.example:3000`. **Required for a replica.** |
| `APP_JOIN_URL` | `APP_PRIMARY_URL` | CA enrollment URL used during first-boot mTLS join. Defaults to `APP_PRIMARY_URL`. Set this when the replication source differs from the Cluster CA holder (e.g., chain topologies where an intermediate replica serves data but cannot issue certificates). |
| `APP_REPL_TOKEN` | *none* | Shared bearer token for `/api/replication/changes`. **Required for a replica.** On a primary, replication stays disabled if it is unset. |
| `APP_SYNC_INTERVAL_SECS` | `5` | Replica idle poll interval (jittered) once caught up. |
| `APP_SYNC_MAX_BACKOFF_SECS` | `60` | Upper bound for retry backoff after failed pulls. |
| `APP_SYNC_BATCH` | `200` | Records per pull (clamped to 1–1000). Lower it for very poor links. |

---

## Getting Started

### 1. Build and Run

```bash
# Clone the repository and enter the directory
cd pando

# Create your config file
cat <<EOF > pando.json
{
  "mode": "strict",
  "fields": {
    "version": {
      "type": "integer",
      "value": 1
    },
    "message": {
      "type": "string"
    }
  }
}
EOF

# Run the server (default: 127.0.0.1:3000)
cargo run
```

To specify a custom configuration file path:

```bash
APP_CONFIG=custom_config.json cargo run
```

### 2. Run a primary and a replica

`pando.db` is opened relative to the working directory, so run each instance from its own directory.

```bash
# Primary
mkdir -p primary && cp pando.json primary/ && cd primary
APP_ROLE=primary \
APP_BIND=0.0.0.0:3000 \
APP_REPL_TOKEN=change-me \
  ../target/debug/pando

# Replica (in another terminal; start with an EMPTY database)
mkdir -p replica && cp pando.json replica/ && cd replica
APP_ROLE=replica \
APP_BIND=127.0.0.1:3001 \
APP_PRIMARY_URL=http://127.0.0.1:3000 \
APP_REPL_TOKEN=change-me \
  ../target/debug/pando
```

A ready-made script, `repl-test.sh`, starts both on localhost, writes a few records and checks that they arrive:

```bash
bash repl-test.sh
```

---

## API Reference

### Health / Service Info
- **Endpoint:** `GET /` or `GET /api/now`
- **Description:** Returns server timestamp and status payload.
- **Example Response:**
  ```json
  {
    "uuid": "4fc736f8-45e0-47da-8da9-6c30fef6a72e",
    "date": "2026-10-25T10:19:00Z",
    "data": {
      "version": 1,
      "message": "Hello from Rust!"
    }
  }
  ```

---

### Create Record
- **Endpoint:** `POST /api/records`
- **Headers:** `Content-Type: application/json`
- **Availability:** primary only. A replica answers `403 Forbidden` with the primary's URL in the message.
- **Request Body:**
  ```json
  {
    "data": {
      "version": 1,
      "message": "Hello, world!"
    }
  }
  ```
- **Response (`201 Created`):**
  ```json
  {
    "seq": 42,
    "uuid": "f3b392b9-dc81-42cb-b9ef-dc7e90aa7ce5",
    "date": "2026-10-25T10:20:15Z",
    "data": {
      "version": 1,
      "message": "Hello, world!"
    }
  }
  ```
- **Error Responses:**
  - `400 Bad Request`:
    - Missing field: `Missing required field: message`
    - Wrong type: `Field 'version' must be of type integer`
    - Value mismatch: `Field 'version' must equal 1`
    - Unknown field (in strict mode): `Unknown field: extra_field`
  - `403 Forbidden`: this instance is a read-only replica.

---

### List Records
- **Endpoint:** `GET /api/records`
- **Description:** Returns up to 100 most recent records ordered by `seq DESC`. Available on primaries and replicas.
- **Response (`200 OK`):**
  ```json
  [
    {
      "seq": 42,
      "uuid": "f3b392b9-dc81-42cb-b9ef-dc7e90aa7ce5",
      "date": "2026-10-25T10:20:15Z",
      "data": {
        "version": 1,
        "message": "Hello, world!"
      }
    }
  ]
  ```

---

### Get Active Configuration
- **Endpoint:** `GET /api/admin/config`
- **Description:** Inspect the currently running schema and mode.
- **Response (`200 OK`):**
  ```json
  {
    "mode": "strict",
    "fields": {
      "version": {
        "type": "integer",
        "value": 1
      },
      "message": {
        "type": "string"
      }
    }
  }
  ```

---

### Hot-Reload Configuration
- **Endpoint:** `POST /api/admin/reload-config`
- **Description:** Reads and validates the file at `APP_CONFIG`. If valid, atomically updates the active schema in memory.
- **Response (`200 OK`):**
  ```json
  {
    "status": "ok",
    "mode": "strict",
    "fields": 2
  }
  ```
- **Error Response (`422 Unprocessable Entity`):** Returned if the configuration file is unreadable, invalid JSON, or contains no fields. The active schema remains untouched.

---

### Coordinated Graceful Switchover
- **Endpoint:** `POST /api/admin/switchover`
- **Description:** Initiated on the **active primary** to execute a coordinated, zero-downtime role switch with a replica. The primary quiesces (rejects new writes), waits for the replica to catch up to its exact `head_seq`, issues a promotion call to the replica, and reconfigures itself as a replica following the new primary.
- **Request Body:**
  ```json
  {
    "replica_url": "https://127.0.0.1:3001",
    "timeout_secs": 15
  }
  ```
- **Response (`200 OK`):**
  ```json
  {
    "status": "ok",
    "message": "Role switchover completed successfully",
    "old_primary_role": "replica",
    "new_primary_url": "https://127.0.0.1:3001",
    "switched_at_seq": 42
  }
  ```

---

### Manual Role Management Endpoints

- **`POST /api/admin/step-down`**:
  Demotes a primary to a read-only replica. Optionally takes `{"primary_url": "https://..."}` to immediately begin pulling from a new primary.
- **`POST /api/admin/promote`**:
  Promotes a replica to a primary. Halts any active replication sync loop, increments cluster epoch, and begins accepting writes at `POST /api/records`.
- **`POST /api/admin/repoint`**:
  Updates a replica's source URL (`{"primary_url": "https://...", "reset_source": true}`). Resets the source instance guard and restarts the sync loop.

---

### Replication: Pull Changes
- **Endpoint:** `GET /api/replication/changes?after=<seq>&limit=<n>`
- **Headers:** `Authorization: Bearer <APP_REPL_TOKEN>`
- **Description:** Returns records with `seq > after` in ascending order. `after` defaults to `0`; `limit` defaults to `200` and is clamped to 1–1000. The server holds no per-replica state.
- **Response (`200 OK`):**
  ```json
  {
    "instance_id": "9d2c1f0e-3a41-4b8e-8f6e-5d1b7a0c2e11",
    "head_seq": 42,
    "has_more": false,
    "changes": [
      {
        "seq": 41,
        "uuid": "a1e5c0de-6c0b-4f43-9d5e-0f2f3c6b9a77",
        "date": "2026-10-25T10:19:58Z",
        "data": { "version": 1, "message": "earlier" }
      },
      {
        "seq": 42,
        "uuid": "f3b392b9-dc81-42cb-b9ef-dc7e90aa7ce5",
        "date": "2026-10-25T10:20:15Z",
        "data": { "version": 1, "message": "Hello, world!" }
      }
    ]
  }
  ```
- **Error Responses:** `401 Unauthorized` for a wrong or missing token; `403 Forbidden` if no `APP_REPL_TOKEN` is configured on this instance.

Note that `seq` values are increasing but not guaranteed to be contiguous, so consumers must use them as a cursor and must not count on gap-free numbering.

---

### Replication: Status
- **Endpoint:** `GET /api/replication/status`
- **Description:** Instance identity, role, cursor and lag. Unauthenticated; it contains no secrets.
- **Response (`200 OK`), on a replica:**
  ```json
  {
    "instance_id": "5b7f4a3c-0f1d-4c6e-9b1a-2e8d6a4f7c90",
    "role": "replica",
    "primary_url": "http://127.0.0.1:3000",
    "local_head_seq": 42,
    "lag_records": 0,
    "sync": {
      "state": "idle",
      "last_applied_seq": 42,
      "source_instance_id": "9d2c1f0e-3a41-4b8e-8f6e-5d1b7a0c2e11",
      "primary_head_seq": 42,
      "last_success": "2026-10-25T10:20:20Z",
      "last_error": null
    }
  }
  ```
- **`sync.state` values:**

  | State | Meaning |
  | :--- | :--- |
  | `primary` | This instance is a primary; no sync is running. |
  | `idle` | Replica is caught up and waiting for the next poll. |
  | `syncing` | Replica is catching up; more batches are pending. |
  | `retrying` | The last pull failed (network, 5xx, truncated body); retrying with backoff. |
  | `halted` | Fatal mismatch detected; the replica has stopped. See *Failure handling*. |

---

## Replication

### Model

```
        writes                    reads
   ┌───────────────┐         ┌───────────────┐
   │    primary    │ ◄─pull──│    replica    │
   │ seq 1..N      │ after=K │ seq 1..K..N   │
   └───────────────┘         └───────────────┘
```

- **Journal and global tick:** `records` is insert-only, so the table itself is the append-only journal. Each row has a `seq` (`INTEGER PRIMARY KEY AUTOINCREMENT`), which is never reused, even after a rollback. A replica stores the *primary's* `seq` values unchanged, so the tick is the same everywhere.
- **Instance ID:** each instance generates a permanent UUID on first start and keeps it in the `meta` table. Every `/changes` response carries the source's ID.
- **Pull, not push:** the replica opens all connections, so only the primary needs to be reachable, and it can sit behind NAT or a firewall on the replica side.
- **Lazy sync:** the replica polls on a timer and catches up whenever the link is available. Reads from a replica can be stale, never inconsistent: records appear in `seq` order with no reordering.

### Replica sync loop

1. Read the cursor (`last_applied_seq`) from memory (restored from the DB at startup).
2. `GET /changes?after=<cursor>&limit=<batch>`.
3. Check identity and history (see below).
4. In **one SQLite transaction**: `INSERT OR IGNORE` every row, then store the new cursor and the source's instance ID.
5. If `has_more`, pull again immediately; otherwise sleep for `APP_SYNC_INTERVAL_SECS` (jittered).

Step 4 is atomic, so a crash can never leave the cursor out of step with the data. `INSERT OR IGNORE` makes re-delivery harmless. A truncated or invalid response body fails to parse and is retried, so a partial batch is never applied.

### Failure handling

| Condition | Behaviour |
| :--- | :--- |
| Network error, timeout, non-2xx, bad body | Transient. Retry with exponential backoff (1s doubling up to `APP_SYNC_MAX_BACKOFF_SECS`) with jitter; state `retrying`. |
| Non-monotonic `seq` in a response | Treated as transient; batch discarded. |
| Source `instance_id` differs from the one stored | **Fatal.** Replica halts. |
| Source `instance_id` equals the replica's own | **Fatal.** Typically a cloned database file. |
| Source `head_seq` < replica cursor | **Fatal.** Source was probably restored from an older backup. |

On a fatal mismatch the replica halts, because retrying cannot fix it, and the reason is in `sync.last_error` and the log. To repoint a replica at a different source, stop it, delete its `pando.db` and start it again so it resyncs from `seq` 0.

### Operational notes

- **Bootstrapping:** start a replica with an empty database. Don't seed it by copying the primary's `pando.db`, because the copy shares the primary's instance ID and the replica will refuse to sync.
- **Transport security:** replication uses a bearer token over whatever transport you give it. Put TLS in front of the primary (reverse proxy) for anything beyond a trusted network.
- **Chaining:** a replica serves `/api/replication/changes` too (if it has a token), so replicas can follow other replicas. Downstream replicas see the upstream replica's instance ID as their source.
- **Promotion & Role Switching:** Replicas can be dynamically promoted at runtime without restart via `POST /api/admin/promote` or gracefully swapped via `POST /api/admin/switchover`. Nodes share Cluster CA credentials during initial join, allowing promoted primaries to issue join tokens and admit new nodes. Existing replicas can follow the new primary without resetting their databases via `POST /api/admin/repoint`.
- **Updates and deletes:** the design assumes records are only ever inserted. If updates or deletes are added later, the journal needs separate change entries (with tombstones), since replicas only observe inserts.
- **Fan-out:** polling is fine for a handful of replicas. For many, consider long-polling or SSE on `/changes`.

---

## Example Usage Workflow

```bash
# 1. Create a valid record
curl -i -X POST http://127.0.0.1:3000/api/records \
  -H "Content-Type: application/json" \
  -d '{"data": {"version": 1, "message": "First record"}}'

# 2. Modify config.json (e.g. bump required version to 2)
# Update "value": 2 in config.json...

# 3. Reload config without downtime
curl -i -X POST http://127.0.0.1:3000/api/admin/reload-config

# 4. Old version requests are now rejected
curl -i -X POST http://127.0.0.1:3000/api/records \
  -H "Content-Type: application/json" \
  -d '{"data": {"version": 1, "message": "Rejected"}}'
# -> HTTP 400: Field 'version' must equal 2

# 5. New version requests succeed
curl -i -X POST http://127.0.0.1:3000/api/records \
  -H "Content-Type: application/json" \
  -d '{"data": {"version": 2, "message": "Accepted"}}'

# 6. Read it back from a replica (once it has synced)
curl -s http://127.0.0.1:3001/api/records

# 7. Check replication health
curl -s http://127.0.0.1:3001/api/replication/status
```

---

## Database Architecture

On startup, the application creates (or opens) a local SQLite database named `pando.db`:

```sql
CREATE TABLE IF NOT EXISTS meta (
    key   TEXT PRIMARY KEY,
    value TEXT NOT NULL
);

CREATE TABLE IF NOT EXISTS records (
    seq  INTEGER PRIMARY KEY AUTOINCREMENT,
    uuid TEXT NOT NULL UNIQUE,
    date TEXT NOT NULL,
    data TEXT NOT NULL
);
```

`meta` holds the following keys:

| Key | Description |
| :--- | :--- |
| `instance_id` | Permanent UUID of this instance. |
| `last_applied_seq` | Replica cursor (replicas only). |
| `source_instance_id` | ID of the source this replica follows (replicas only). |

### Migration

Databases created by earlier versions (`uuid TEXT PRIMARY KEY`, no `seq`) are migrated automatically on startup, in a single transaction. Existing rows get `seq` values in their original insertion order.

---

## License

This project is licensed under the MIT License.