Replication!
This commit is contained in:
Kalzu Rekku committed
1 parent
427a824e45
commit
3d819c7b10
6 files changed
+1004
-71
No files matched your search
@@ -1,9 +1,11 @@
|
||||
# Dynamic Schema Records API
|
||||
|
||||
A lightweight, high-performance JSON records service built with **Rust**, **Axum**, and **SQLite**.
|
||||
A lightweight, high-performance JSON records service built with **Rust**, **Axum**, and **SQLite**.
|
||||
|
||||
The service validates incoming JSON payloads against a dynamic schema specification defined in a JSON configuration file. Schemas support validation modes (`strict` vs. `loose`), data type constraints, and exact forced values (e.g., pinning API payload versions). Schemas can be reloaded at runtime without restarting the server.
|
||||
|
||||
Instances can run as a **primary** (accepts writes) or as a **read-only replica** that lazily pulls records from a primary. Replication is built for high-latency and unreliable links.
|
||||
|
||||
---
|
||||
|
||||
## Features
|
||||
@@ -15,7 +17,14 @@ The service validates incoming JSON payloads against a dynamic schema specificat
|
||||
- **`strict`**: All defined fields are required, and unknown fields are rejected.
|
||||
- **`loose`**: All defined fields are required, but additional unknown fields are accepted.
|
||||
- **Hot-Reloadable Configuration**: Reload `config.json` via an HTTP admin endpoint with atomic swap and rollback on error.
|
||||
- **Persistent Storage**: Validated records are stored with a generated UUID and ISO-8601 UTC timestamp in an SQLite database (`messages.db`).
|
||||
- **Persistent Storage**: Validated records are stored with a generated UUID, an ISO-8601 UTC timestamp and a global sequence number (`seq`) in an SQLite database (`messages.db`).
|
||||
- **Read Replication**:
|
||||
- Pull-based and cursor-driven: the replica asks for everything after its last `seq`.
|
||||
- Append-only journal with a global tick (`seq`) that is preserved on replicas.
|
||||
- Permanent per-instance ID, used to detect wrong, cloned or restored sources.
|
||||
- Atomic, idempotent batch apply: safe across crashes, retries and duplicate deliveries.
|
||||
- Exponential backoff with jitter, small batches and gzip for flaky or slow links.
|
||||
- Replicas can serve replication to other replicas (relay).
|
||||
- **Structured Logging**: Built with `tracing` outputting to standard output.
|
||||
|
||||
---
|
||||
@@ -25,13 +34,24 @@ The service validates incoming JSON payloads against a dynamic schema specificat
|
||||
- [Rust toolchain](https://rustup.rs/) (1.75+ recommended)
|
||||
- SQLite development libraries (if linking dynamically)
|
||||
|
||||
### Cargo dependencies of note
|
||||
|
||||
```toml
|
||||
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls", "gzip"] }
|
||||
tower-http = { version = "0.5", features = ["compression-gzip"] }
|
||||
```
|
||||
|
||||
`reqwest` is used by the replica to pull changes. `tower-http` provides gzip response compression.
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
The service looks for a configuration file at the path specified by the `APP_CONFIG` environment variable, defaulting to `config.json` in the current working directory.
|
||||
### Schema file
|
||||
|
||||
### Example: `config.json`
|
||||
The service looks for a schema configuration file at the path specified by the `APP_CONFIG` environment variable, defaulting to `config.json` in the current working directory.
|
||||
|
||||
#### Example: `config.json`
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -51,7 +71,7 @@ The service looks for a configuration file at the path specified by the `APP_CON
|
||||
}
|
||||
```
|
||||
|
||||
### Schema Specification
|
||||
#### Schema Specification
|
||||
|
||||
| Property | Type | Description |
|
||||
| :--- | :--- | :--- |
|
||||
@@ -60,6 +80,21 @@ The service looks for a configuration file at the path specified by the `APP_CON
|
||||
| `fields.<name>.type` | `string` | Expected JSON type: `string`, `integer`, `number`, `boolean`, `object`, `array`, `null`. |
|
||||
| `fields.<name>.value` | *any* *(optional)* | Expected exact value. Rejects any request where the value does not strictly match. |
|
||||
|
||||
The schema is only enforced on writes to a primary. A replica still loads the file at startup, but it does not re-validate replicated records, since the primary already did.
|
||||
|
||||
### Environment variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
| :--- | :--- | :--- |
|
||||
| `APP_CONFIG` | `config.json` | Path to the schema file. |
|
||||
| `APP_BIND` | `127.0.0.1:3000` | Listen address. Change this for a primary that replicas must reach over a network. |
|
||||
| `APP_ROLE` | `primary` | `primary` or `replica`. |
|
||||
| `APP_PRIMARY_URL` | *none* | Base URL of the source, e.g. `https://primary.example:3000`. **Required for a replica.** |
|
||||
| `APP_REPL_TOKEN` | *none* | Shared bearer token for `/api/replication/changes`. **Required for a replica.** On a primary, replication stays disabled if it is unset. |
|
||||
| `APP_SYNC_INTERVAL_SECS` | `5` | Replica idle poll interval (jittered) once caught up. |
|
||||
| `APP_SYNC_MAX_BACKOFF_SECS` | `60` | Upper bound for retry backoff after failed pulls. |
|
||||
| `APP_SYNC_BATCH` | `200` | Records per pull (clamped to 1–1000). Lower it for very poor links. |
|
||||
|
||||
---
|
||||
|
||||
## Getting Started
|
||||
@@ -96,6 +131,33 @@ To specify a custom configuration file path:
|
||||
APP_CONFIG=custom_config.json cargo run
|
||||
```
|
||||
|
||||
### 2. Run a primary and a replica
|
||||
|
||||
`messages.db` is opened relative to the working directory, so run each instance from its own directory.
|
||||
|
||||
```bash
|
||||
# Primary
|
||||
mkdir -p primary && cp config.json primary/ && cd primary
|
||||
APP_ROLE=primary \
|
||||
APP_BIND=0.0.0.0:3000 \
|
||||
APP_REPL_TOKEN=change-me \
|
||||
../target/debug/rust_test
|
||||
|
||||
# Replica (in another terminal; start with an EMPTY database)
|
||||
mkdir -p replica && cp config.json replica/ && cd replica
|
||||
APP_ROLE=replica \
|
||||
APP_BIND=127.0.0.1:3001 \
|
||||
APP_PRIMARY_URL=http://127.0.0.1:3000 \
|
||||
APP_REPL_TOKEN=change-me \
|
||||
../target/debug/rust_test
|
||||
```
|
||||
|
||||
A ready-made script, `repl-test.sh`, starts both on localhost, writes a few records and checks that they arrive:
|
||||
|
||||
```bash
|
||||
bash repl-test.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Reference
|
||||
@@ -120,6 +182,7 @@ APP_CONFIG=custom_config.json cargo run
|
||||
### Create Record
|
||||
- **Endpoint:** `POST /api/records`
|
||||
- **Headers:** `Content-Type: application/json`
|
||||
- **Availability:** primary only. A replica answers `403 Forbidden` with the primary's URL in the message.
|
||||
- **Request Body:**
|
||||
```json
|
||||
{
|
||||
@@ -132,6 +195,7 @@ APP_CONFIG=custom_config.json cargo run
|
||||
- **Response (`201 Created`):**
|
||||
```json
|
||||
{
|
||||
"seq": 42,
|
||||
"uuid": "f3b392b9-dc81-42cb-b9ef-dc7e90aa7ce5",
|
||||
"date": "2026-10-25T10:20:15Z",
|
||||
"data": {
|
||||
@@ -140,21 +204,24 @@ APP_CONFIG=custom_config.json cargo run
|
||||
}
|
||||
}
|
||||
```
|
||||
- **Error Responses (`400 Bad Request`):**
|
||||
- Missing field: `Missing required field: message`
|
||||
- Wrong type: `Field 'version' must be of type integer`
|
||||
- Value mismatch: `Field 'version' must equal 1`
|
||||
- Unknown field (in strict mode): `Unknown field: extra_field`
|
||||
- **Error Responses:**
|
||||
- `400 Bad Request`:
|
||||
- Missing field: `Missing required field: message`
|
||||
- Wrong type: `Field 'version' must be of type integer`
|
||||
- Value mismatch: `Field 'version' must equal 1`
|
||||
- Unknown field (in strict mode): `Unknown field: extra_field`
|
||||
- `403 Forbidden`: this instance is a read-only replica.
|
||||
|
||||
---
|
||||
|
||||
### List Records
|
||||
- **Endpoint:** `GET /api/records`
|
||||
- **Description:** Returns up to 100 most recent records ordered by `date DESC`.
|
||||
- **Description:** Returns up to 100 most recent records ordered by `seq DESC`. Available on primaries and replicas.
|
||||
- **Response (`200 OK`):**
|
||||
```json
|
||||
[
|
||||
{
|
||||
"seq": 42,
|
||||
"uuid": "f3b392b9-dc81-42cb-b9ef-dc7e90aa7ce5",
|
||||
"date": "2026-10-25T10:20:15Z",
|
||||
"data": {
|
||||
@@ -203,6 +270,121 @@ APP_CONFIG=custom_config.json cargo run
|
||||
|
||||
---
|
||||
|
||||
### Replication: Pull Changes
|
||||
- **Endpoint:** `GET /api/replication/changes?after=<seq>&limit=<n>`
|
||||
- **Headers:** `Authorization: Bearer <APP_REPL_TOKEN>`
|
||||
- **Description:** Returns records with `seq > after` in ascending order. `after` defaults to `0`; `limit` defaults to `200` and is clamped to 1–1000. The server holds no per-replica state.
|
||||
- **Response (`200 OK`):**
|
||||
```json
|
||||
{
|
||||
"instance_id": "9d2c1f0e-3a41-4b8e-8f6e-5d1b7a0c2e11",
|
||||
"head_seq": 42,
|
||||
"has_more": false,
|
||||
"changes": [
|
||||
{
|
||||
"seq": 41,
|
||||
"uuid": "a1e5c0de-6c0b-4f43-9d5e-0f2f3c6b9a77",
|
||||
"date": "2026-10-25T10:19:58Z",
|
||||
"data": { "version": 1, "message": "earlier" }
|
||||
},
|
||||
{
|
||||
"seq": 42,
|
||||
"uuid": "f3b392b9-dc81-42cb-b9ef-dc7e90aa7ce5",
|
||||
"date": "2026-10-25T10:20:15Z",
|
||||
"data": { "version": 1, "message": "Hello, world!" }
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
- **Error Responses:** `401 Unauthorized` for a wrong or missing token; `403 Forbidden` if no `APP_REPL_TOKEN` is configured on this instance.
|
||||
|
||||
Note that `seq` values are increasing but not guaranteed to be contiguous, so consumers must use them as a cursor and must not count on gap-free numbering.
|
||||
|
||||
---
|
||||
|
||||
### Replication: Status
|
||||
- **Endpoint:** `GET /api/replication/status`
|
||||
- **Description:** Instance identity, role, cursor and lag. Unauthenticated; it contains no secrets.
|
||||
- **Response (`200 OK`), on a replica:**
|
||||
```json
|
||||
{
|
||||
"instance_id": "5b7f4a3c-0f1d-4c6e-9b1a-2e8d6a4f7c90",
|
||||
"role": "replica",
|
||||
"primary_url": "http://127.0.0.1:3000",
|
||||
"local_head_seq": 42,
|
||||
"lag_records": 0,
|
||||
"sync": {
|
||||
"state": "idle",
|
||||
"last_applied_seq": 42,
|
||||
"source_instance_id": "9d2c1f0e-3a41-4b8e-8f6e-5d1b7a0c2e11",
|
||||
"primary_head_seq": 42,
|
||||
"last_success": "2026-10-25T10:20:20Z",
|
||||
"last_error": null
|
||||
}
|
||||
}
|
||||
```
|
||||
- **`sync.state` values:**
|
||||
|
||||
| State | Meaning |
|
||||
| :--- | :--- |
|
||||
| `primary` | This instance is a primary; no sync is running. |
|
||||
| `idle` | Replica is caught up and waiting for the next poll. |
|
||||
| `syncing` | Replica is catching up; more batches are pending. |
|
||||
| `retrying` | The last pull failed (network, 5xx, truncated body); retrying with backoff. |
|
||||
| `halted` | Fatal mismatch detected; the replica has stopped. See *Failure handling*. |
|
||||
|
||||
---
|
||||
|
||||
## Replication
|
||||
|
||||
### Model
|
||||
|
||||
```
|
||||
writes reads
|
||||
┌───────────────┐ ┌───────────────┐
|
||||
│ primary │ ◄─pull──│ replica │
|
||||
│ seq 1..N │ after=K │ seq 1..K..N │
|
||||
└───────────────┘ └───────────────┘
|
||||
```
|
||||
|
||||
- **Journal and global tick:** `records` is insert-only, so the table itself is the append-only journal. Each row has a `seq` (`INTEGER PRIMARY KEY AUTOINCREMENT`), which is never reused, even after a rollback. A replica stores the *primary's* `seq` values unchanged, so the tick is the same everywhere.
|
||||
- **Instance ID:** each instance generates a permanent UUID on first start and keeps it in the `meta` table. Every `/changes` response carries the source's ID.
|
||||
- **Pull, not push:** the replica opens all connections, so only the primary needs to be reachable, and it can sit behind NAT or a firewall on the replica side.
|
||||
- **Lazy sync:** the replica polls on a timer and catches up whenever the link is available. Reads from a replica can be stale, never inconsistent: records appear in `seq` order with no reordering.
|
||||
|
||||
### Replica sync loop
|
||||
|
||||
1. Read the cursor (`last_applied_seq`) from memory (restored from the DB at startup).
|
||||
2. `GET /changes?after=<cursor>&limit=<batch>`.
|
||||
3. Check identity and history (see below).
|
||||
4. In **one SQLite transaction**: `INSERT OR IGNORE` every row, then store the new cursor and the source's instance ID.
|
||||
5. If `has_more`, pull again immediately; otherwise sleep for `APP_SYNC_INTERVAL_SECS` (jittered).
|
||||
|
||||
Step 4 is atomic, so a crash can never leave the cursor out of step with the data. `INSERT OR IGNORE` makes re-delivery harmless. A truncated or invalid response body fails to parse and is retried, so a partial batch is never applied.
|
||||
|
||||
### Failure handling
|
||||
|
||||
| Condition | Behaviour |
|
||||
| :--- | :--- |
|
||||
| Network error, timeout, non-2xx, bad body | Transient. Retry with exponential backoff (1s doubling up to `APP_SYNC_MAX_BACKOFF_SECS`) with jitter; state `retrying`. |
|
||||
| Non-monotonic `seq` in a response | Treated as transient; batch discarded. |
|
||||
| Source `instance_id` differs from the one stored | **Fatal.** Replica halts. |
|
||||
| Source `instance_id` equals the replica's own | **Fatal.** Typically a cloned database file. |
|
||||
| Source `head_seq` < replica cursor | **Fatal.** Source was probably restored from an older backup. |
|
||||
|
||||
On a fatal mismatch the replica halts, because retrying cannot fix it, and the reason is in `sync.last_error` and the log. To repoint a replica at a different source, stop it, delete its `messages.db` and start it again so it resyncs from `seq` 0.
|
||||
|
||||
### Operational notes
|
||||
|
||||
- **Bootstrapping:** start a replica with an empty database. Don't seed it by copying the primary's `messages.db`, because the copy shares the primary's instance ID and the replica will refuse to sync.
|
||||
- **Transport security:** replication uses a bearer token over whatever transport you give it. Put TLS in front of the primary (reverse proxy) for anything beyond a trusted network.
|
||||
- **Chaining:** a replica serves `/api/replication/changes` too (if it has a token), so replicas can follow other replicas. Downstream replicas see the upstream replica's instance ID as their source.
|
||||
- **Promotion:** a replica holds the same `seq` values as its source, so it can be restarted with `APP_ROLE=primary` to take over. Note that its instance ID differs from the old primary's, so other replicas will halt on an ID mismatch until they are reset against the new primary.
|
||||
- **Updates and deletes:** the design assumes records are only ever inserted. If updates or deletes are added later, the journal needs separate change entries (with tombstones), since replicas only observe inserts.
|
||||
- **Fan-out:** polling is fine for a handful of replicas. For many, consider long-polling or SSE on `/changes`.
|
||||
|
||||
---
|
||||
|
||||
## Example Usage Workflow
|
||||
|
||||
```bash
|
||||
@@ -227,23 +409,47 @@ curl -i -X POST http://127.0.0.1:3000/api/records \
|
||||
curl -i -X POST http://127.0.0.1:3000/api/records \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"data": {"version": 2, "message": "Accepted"}}'
|
||||
|
||||
# 6. Read it back from a replica (once it has synced)
|
||||
curl -s http://127.0.0.1:3001/api/records
|
||||
|
||||
# 7. Check replication health
|
||||
curl -s http://127.0.0.1:3001/api/replication/status
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Database Architecture
|
||||
|
||||
On startup, the application creates a local SQLite database named `messages.db` with the following schema:
|
||||
On startup, the application creates (or opens) a local SQLite database named `messages.db`:
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS meta (
|
||||
key TEXT PRIMARY KEY,
|
||||
value TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS records (
|
||||
uuid TEXT PRIMARY KEY,
|
||||
seq INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
uuid TEXT NOT NULL UNIQUE,
|
||||
date TEXT NOT NULL,
|
||||
data TEXT NOT NULL
|
||||
);
|
||||
```
|
||||
This project is licensed under the [MIT License](LICENSE) (or choose your preferred license).
|
||||
|
||||
`meta` holds the following keys:
|
||||
|
||||
| Key | Description |
|
||||
| :--- | :--- |
|
||||
| `instance_id` | Permanent UUID of this instance. |
|
||||
| `last_applied_seq` | Replica cursor (replicas only). |
|
||||
| `source_instance_id` | ID of the source this replica follows (replicas only). |
|
||||
|
||||
### Migration
|
||||
|
||||
Databases created by earlier versions (`uuid TEXT PRIMARY KEY`, no `seq`) are migrated automatically on startup, in a single transaction. Existing rows get `seq` values in their original insertion order.
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
|
||||
Reference in new issue
Block a user