Support for bigger clusters.

README.md

@@ -18,12 +18,12 @@ Instances can run as a **primary** (accepts writes) or as a **read-only replica*
     - **`loose`**: All defined fields are required, but additional unknown fields are accepted.
 - **Hot-Reloadable Configuration**: Reload `config.json` via an HTTP admin endpoint with atomic swap and rollback on error.
 - **Persistent Storage**: Validated records are stored with a generated UUID, an ISO-8601 UTC timestamp and a global sequence number (`seq`) in an SQLite database (`messages.db`).
-- **Read Replication**:
+- **Replication & Dynamic Role Switching**:
   - Pull-based and cursor-driven: the replica asks for everything after its last `seq`.
   - Append-only journal with a global tick (`seq`) that is preserved on replicas.
-  - Permanent per-instance ID, used to detect wrong, cloned or restored sources.
-  - Atomic, idempotent batch apply: safe across crashes, retries and duplicate deliveries.
-  - Exponential backoff with jitter, small batches and gzip for flaky or slow links.
+  - Seamless role switch / failover without data loss or split-brain.
+  - Automatic CA credential sharing during dynamic join so promoted replicas can continue minting tokens and signing nodes.
+  - Coordinated 1-step switchover API (`/api/admin/switchover`) and fine-grained controls (`/step-down`, `/promote`, `/repoint`).
   - Replicas can serve replication to other replicas (relay).
 - **Structured Logging**: Built with `tracing` outputting to standard output.
 
@@ -89,7 +89,8 @@ The schema is only enforced on writes to a primary. A replica still loads the fi
 | `APP_CONFIG` | `config.json` | Path to the schema file. |
 | `APP_BIND` | `127.0.0.1:3000` | Listen address. Change this for a primary that replicas must reach over a network. |
 | `APP_ROLE` | `primary` | `primary` or `replica`. |
-| `APP_PRIMARY_URL` | *none* | Base URL of the source, e.g. `https://primary.example:3000`. **Required for a replica.** |
+| `APP_PRIMARY_URL` | *none* | Base URL of the replication source, e.g. `https://primary.example:3000`. **Required for a replica.** |
+| `APP_JOIN_URL` | `APP_PRIMARY_URL` | CA enrollment URL used during first-boot mTLS join. Defaults to `APP_PRIMARY_URL`. Set this when the replication source differs from the Cluster CA holder (e.g., chain topologies where an intermediate replica serves data but cannot issue certificates). |
 | `APP_REPL_TOKEN` | *none* | Shared bearer token for `/api/replication/changes`. **Required for a replica.** On a primary, replication stays disabled if it is unset. |
 | `APP_SYNC_INTERVAL_SECS` | `5` | Replica idle poll interval (jittered) once caught up. |
 | `APP_SYNC_MAX_BACKOFF_SECS` | `60` | Upper bound for retry backoff after failed pulls. |
@@ -270,6 +271,40 @@ bash repl-test.sh
 
 ---
 
+### Coordinated Graceful Switchover
+- **Endpoint:** `POST /api/admin/switchover`
+- **Description:** Initiated on the **active primary** to execute a coordinated, zero-downtime role switch with a replica. The primary quiesces (rejects new writes), waits for the replica to catch up to its exact `head_seq`, issues a promotion call to the replica, and reconfigures itself as a replica following the new primary.
+- **Request Body:**
+  ```json
+  {
+    "replica_url": "https://127.0.0.1:3001",
+    "timeout_secs": 15
+  }
+  ```
+- **Response (`200 OK`):**
+  ```json
+  {
+    "status": "ok",
+    "message": "Role switchover completed successfully",
+    "old_primary_role": "replica",
+    "new_primary_url": "https://127.0.0.1:3001",
+    "switched_at_seq": 42
+  }
+  ```
+
+---
+
+### Manual Role Management Endpoints
+
+- **`POST /api/admin/step-down`**:
+  Demotes a primary to a read-only replica. Optionally takes `{"primary_url": "https://..."}` to immediately begin pulling from a new primary.
+- **`POST /api/admin/promote`**:
+  Promotes a replica to a primary. Halts any active replication sync loop, increments cluster epoch, and begins accepting writes at `POST /api/records`.
+- **`POST /api/admin/repoint`**:
+  Updates a replica's source URL (`{"primary_url": "https://...", "reset_source": true}`). Resets the source instance guard and restarts the sync loop.
+
+---
+
 ### Replication: Pull Changes
 - **Endpoint:** `GET /api/replication/changes?after=<seq>&limit=<n>`
 - **Headers:** `Authorization: Bearer <APP_REPL_TOKEN>`
@@ -379,7 +414,7 @@ On a fatal mismatch the replica halts, because retrying cannot fix it, and the r
 - **Bootstrapping:** start a replica with an empty database. Don't seed it by copying the primary's `messages.db`, because the copy shares the primary's instance ID and the replica will refuse to sync.
 - **Transport security:** replication uses a bearer token over whatever transport you give it. Put TLS in front of the primary (reverse proxy) for anything beyond a trusted network.
 - **Chaining:** a replica serves `/api/replication/changes` too (if it has a token), so replicas can follow other replicas. Downstream replicas see the upstream replica's instance ID as their source.
-- **Promotion:** a replica holds the same `seq` values as its source, so it can be restarted with `APP_ROLE=primary` to take over. Note that its instance ID differs from the old primary's, so other replicas will halt on an ID mismatch until they are reset against the new primary.
+- **Promotion & Role Switching:** Replicas can be dynamically promoted at runtime without restart via `POST /api/admin/promote` or gracefully swapped via `POST /api/admin/switchover`. Nodes share Cluster CA credentials during initial join, allowing promoted primaries to issue join tokens and admit new nodes. Existing replicas can follow the new primary without resetting their databases via `POST /api/admin/repoint`.
 - **Updates and deletes:** the design assumes records are only ever inserted. If updates or deletes are added later, the journal needs separate change entries (with tombstones), since replicas only observe inserts.
 - **Fan-out:** polling is fine for a handful of replicas. For many, consider long-polling or SSE on `/changes`.