From a9f50ff68ca3ec3dcd1f9ad350cdc48db7b71fbe Mon Sep 17 00:00:00 2001 From: Kalzu Rekku Date: Sat, 3 May 2025 18:26:12 +0300 Subject: [PATCH] Some documentation and stuff --- API.md | 359 +++++++++++++++++++++++++++++++++++++++++++++++ MiniDiscovery.py | 20 ++- README.md | 74 ++++++++++ key_rotation.md | 77 ++++++++++ requirements.txt | 35 +++++ 5 files changed, 564 insertions(+), 1 deletion(-) create mode 100644 API.md create mode 100644 README.md create mode 100644 key_rotation.md create mode 100644 requirements.txt diff --git a/API.md b/API.md new file mode 100644 index 0000000..6af8539 --- /dev/null +++ b/API.md @@ -0,0 +1,359 @@ +# MiniDiscovery API Documentation + +This document describes the HTTP API endpoints provided by MiniDiscovery. + +## Authentication + +Most API endpoints require authentication using an API token. Tokens must be passed in the `X-API-Token` HTTP header. + +```http +GET /v1/catalog/services HTTP/1.1 +Host: your-minidiscovery-host:8500 +X-API-Token: your_secret_api_token +``` + +Tokens have associated permissions: + +* **`read`**: Allows reading service information (catalog, health). +* **`write`**: Allows registering and deregistering services. +* **`admin`**: Allows all `read` and `write` operations, plus managing ACL tokens. + +The initial admin token is set using the `MINIDISCOVERY_ADMIN_TOKEN` environment variable on the *first* run when the database is empty. Subsequent tokens are managed via the API. + +## Data Models + +### `ServiceInstance` + +This model represents a single instance of a registered service. + +```json +{ + "id": "web-instance-1", + "name": "web", + "address": "192.168.1.100", + "port": 80, + "tags": ["frontend", "nginx"], + "metadata": { + "version": "1.2.3", + "region": "us-east-1" + }, + "health": "passing", + "last_updated": 1678886400.123456 +} +``` + +* `id` (string, required): Unique identifier for this specific instance. +* `name` (string, required): Logical name of the service (e.g., 'web', 'api', 'db'). +* `address` (string, required): IP address or resolvable hostname where the service listens. +* `port` (integer, required): Port number (1-65535). +* `tags` (array of strings, optional): List of tags for filtering/grouping. Defaults to `[]`. +* `metadata` (object, optional): Key-value string pairs for arbitrary metadata. Defaults to `{}`. +* `health` (string, optional): Current health status ('passing', 'failing', 'unknown'). Defaults to 'passing' on registration. Automatically updated by the health checker. +* `last_updated` (float, internal): Unix timestamp of the last update (set automatically). + +## Endpoints + +### Agent Endpoints + +These endpoints interact with the local agent state (registering/deregistering services). + +#### Register Service + +* **`POST /v1/agent/service/register`** +* **Description:** Registers a new service instance or updates an existing one with the same `id`. This operation is idempotent based on the service `id`. +* **Authentication:** Requires `write` permission. +* **Request Body:** `ServiceInstance` JSON object. The `health` and `last_updated` fields in the request body are ignored; health is managed internally, and `last_updated` is set automatically. +* **Success Response:** `200 OK` + ```json + { + "status": "registered", + "service_id": "web-instance-1" + } + ``` +* **Error Responses:** + * `400 Bad Request`: Invalid request body format. + * `401 Unauthorized`: Missing `X-API-Token`. + * `403 Forbidden`: Invalid token or insufficient permissions. + * `500 Internal Server Error`: Database error during registration. +* **`curl` Example:** + ```bash + curl -X POST http://localhost:8500/v1/agent/service/register \ + -H "X-API-Token: your_write_token" \ + -H "Content-Type: application/json" \ + -d '{ + "id": "api-instance-01", + "name": "api", + "address": "10.0.1.5", + "port": 8080, + "tags": ["backend", "v2"], + "metadata": {"git_sha": "a1b2c3d"} + }' + ``` + +#### Deregister Service + +* **`PUT /v1/agent/service/deregister/{service_id}`** + *(Note: Consul uses PUT here, although DELETE might feel more intuitive)* +* **Description:** Removes a service instance by its ID. +* **Authentication:** Requires `write` permission. +* **Path Parameters:** + * `service_id` (string): The unique ID of the service instance to deregister. +* **Success Response:** `200 OK` + ```json + { + "status": "deregistered", + "service_id": "api-instance-01" + } + ``` +* **Error Responses:** + * `401 Unauthorized`: Missing `X-API-Token`. + * `403 Forbidden`: Invalid token or insufficient permissions. + * `404 Not Found`: Service with the given `service_id` does not exist. +* **`curl` Example:** + ```bash + curl -X PUT http://localhost:8500/v1/agent/service/deregister/api-instance-01 \ + -H "X-API-Token: your_write_token" + ``` + +### Catalog Endpoints + +These endpoints provide information about registered services across the system. + +#### List Services + +* **`GET /v1/catalog/services`** +* **Description:** Returns a map of all registered service names to a list of unique tags associated with instances of that service. +* **Authentication:** Requires `read` permission. +* **Success Response:** `200 OK` + ```json + { + "web": ["frontend", "nginx"], + "api": ["backend", "v2"], + "db": [] + } + ``` +* **Error Responses:** + * `401 Unauthorized`: Missing `X-API-Token`. + * `403 Forbidden`: Invalid token or insufficient permissions. +* **`curl` Example:** + ```bash + curl http://localhost:8500/v1/catalog/services \ + -H "X-API-Token: your_read_token" + ``` + +#### List Service Instances + +* **`GET /v1/catalog/service/{service_name}`** +* **Description:** Returns a list of all registered instances for a given service name. +* **Authentication:** Requires `read` permission. +* **Path Parameters:** + * `service_name` (string): The name of the service (e.g., 'web', 'api'). +* **Query Parameters:** + * `tag` (string, optional): Filter instances by tag. Only instances with this tag will be returned. +* **Success Response:** `200 OK` (Returns an array of `ServiceInstance` objects) + ```json + [ + { + "id": "web-instance-1", + "name": "web", + "address": "192.168.1.100", + "port": 80, + "tags": ["frontend", "nginx"], + "metadata": {"version": "1.2.3"}, + "health": "passing", + "last_updated": 1678886400.123 + }, + { + "id": "web-instance-2", + "name": "web", + "address": "192.168.1.101", + "port": 80, + "tags": ["frontend"], + "metadata": {"version": "1.2.4"}, + "health": "failing", + "last_updated": 1678886405.456 + } + ] + ``` + * Returns `[]` if the service name is not found. +* **Error Responses:** + * `401 Unauthorized`: Missing `X-API-Token`. + * `403 Forbidden`: Invalid token or insufficient permissions. +* **`curl` Examples:** + ```bash + # Get all 'web' instances + curl http://localhost:8500/v1/catalog/service/web \ + -H "X-API-Token: your_read_token" + + # Get 'web' instances tagged with 'nginx' + curl http://localhost:8500/v1/catalog/service/web?tag=nginx \ + -H "X-API-Token: your_read_token" + ``` + +### Health Endpoints + +These endpoints provide service discovery filtered by health status. + +#### List Healthy Service Instances + +* **`GET /v1/health/service/{service_name}`** +* **Description:** Returns a list of service instances, similar to the catalog endpoint, but allows filtering by health status. +* **Authentication:** Requires `read` permission. +* **Path Parameters:** + * `service_name` (string): The name of the service. +* **Query Parameters:** + * `tag` (string, optional): Filter instances by tag. + * `passing` (boolean, optional): If `true`, only return instances with a `passing` health status. Defaults to `false` (returns all instances regardless of health). +* **Success Response:** `200 OK` (Returns an array of `ServiceInstance` objects) + * Response format is identical to `/v1/catalog/service/{service_name}` but filtered according to query parameters. +* **Error Responses:** + * `401 Unauthorized`: Missing `X-API-Token`. + * `403 Forbidden`: Invalid token or insufficient permissions. +* **`curl` Examples:** + ```bash + # Get all 'api' instances (healthy or not) + curl http://localhost:8500/v1/health/service/api \ + -H "X-API-Token: your_read_token" + + # Get only healthy ('passing') 'api' instances + curl http://localhost:8500/v1/health/service/api?passing=true \ + -H "X-API-Token: your_read_token" + + # Get only healthy 'api' instances tagged 'v2' + curl 'http://localhost:8500/v1/health/service/api?passing=true&tag=v2' \ + -H "X-API-Token: your_read_token" + ``` + +### ACL Token Endpoints + +These endpoints manage API access tokens. + +#### Create Token + +* **`POST /v1/acl/token`** +* **Description:** Creates a new API token with specified permissions. +* **Authentication:** Requires `admin` permission. +* **Request Body:** + ```json + { + "name": "my-app-token", + "permissions": ["read", "write"] + } + ``` + * `name` (string, required): A unique descriptive name for the token. + * `permissions` (array of strings, optional): List of permissions (`read`, `write`, `admin`). Defaults to `["read", "write"]`. +* **Success Response:** `201 Created` + ```json + { + "token": "YOUR_NEW_SECRET_TOKEN_VALUE", + "name": "my-app-token", + "permissions": ["read", "write"] + } + ``` + * **IMPORTANT:** The `token` value is the actual secret token. It is **only shown once** in this response. Store it securely immediately. +* **Error Responses:** + * `400 Bad Request`: Invalid request body or token name already exists. + * `401 Unauthorized`: Missing `X-API-Token`. + * `403 Forbidden`: Invalid token or insufficient permissions (not admin). + * `500 Internal Server Error`: Database error during token creation. +* **`curl` Example:** + ```bash + curl -X POST http://localhost:8500/v1/acl/token \ + -H "X-API-Token: your_admin_token" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "read-only-dashboard", + "permissions": ["read"] + }' + ``` + +#### Revoke Token + +* **`DELETE /v1/acl/token/{token_to_revoke}`** +* **Description:** Revokes (deletes) an existing API token. +* **Authentication:** Requires `admin` permission. +* **Path Parameters:** + * `token_to_revoke` (string): The **actual secret token value** of the token you want to revoke. +* **Success Response:** `200 OK` + ```json + { + "status": "revoked", + "token_info": "Token revoked successfully" + } + ``` +* **Error Responses:** + * `401 Unauthorized`: Missing `X-API-Token`. + * `403 Forbidden`: Invalid token or insufficient permissions (not admin). + * `404 Not Found`: The token specified in the path does not exist or was already revoked. +* **`curl` Example:** + ```bash + # Replace YOUR_OLD_SECRET_TOKEN_VALUE with the token to be deleted + curl -X DELETE http://localhost:8500/v1/acl/token/YOUR_OLD_SECRET_TOKEN_VALUE \ + -H "X-API-Token: your_admin_token" + ``` + +### DNS over HTTPS (DoH) + +* **`GET /dns-query`** +* **Description:** Provides a simplified DNS-over-HTTPS interface (RFC 8484 GET format) for querying service addresses and SRV records. Primarily intended for simple DNS clients or scripts. **Note:** Requires service names to end with the configured DNS suffix (default: `.laiska.local`). +* **Authentication:** None (typically). +* **Query Parameters:** + * `name` (string, required): The DNS query name (e.g., `web.laiska.local`, `_frontend._tcp.web.laiska.local`). + * `type` (string, optional): The DNS record type (e.g., `A`, `SRV`, `TXT`). Defaults to `A`. Case-insensitive. +* **Success Response:** `200 OK` (JSON object following RFC 8484 structure) + * **Example (A query for `web.laiska.local`):** + ```json + { + "Status": 0, "TC": false, "RD": true, "RA": false, "AD": false, "CD": false, + "Question": [{"name": "web.laiska.local.", "type": 1}], + "Answer": [ + {"name": "web.laiska.local.", "type": 1, "TTL": 60, "data": "192.168.1.100"}, + {"name": "web.laiska.local.", "type": 1, "TTL": 60, "data": "192.168.1.101"} + ] + } + ``` + * **Example (SRV query for `_frontend._tcp.web.laiska.local`):** + ```json + { + "Status": 0, "TC": false, "RD": true, "RA": false, "AD": false, "CD": false, + "Question": [{"name": "_frontend._tcp.web.laiska.local.", "type": 33}], + "Answer": [ + {"name": "_frontend._tcp.web.laiska.local.", "type": 33, "TTL": 60, "data": "0 10 80 web-instance-1.laiska.local."}, + {"name": "_frontend._tcp.web.laiska.local.", "type": 33, "TTL": 60, "data": "0 10 80 web-instance-2.laiska.local."} + ], + "Additional": [ // Note: MiniDiscovery doesn't currently populate Additional well for DoH + // Ideally A records for targets would be here + ] + } + ``` +* **Error/NXDOMAIN Response:** Returns JSON with `"Status"` other than `0` (e.g., `2` for NXDOMAIN, `4` for Not Implemented type). +* **`curl` Example:** + ```bash + # Query for A records + curl "http://localhost:8500/dns-query?name=web.laiska.local&type=A" + + # Query for SRV records + curl "http://localhost:8500/dns-query?name=_frontend._tcp.web.laiska.local&type=SRV" + ``` + +### Other Endpoints + +#### Root + +* **`GET /`** +* **Description:** Provides a basic entry point. Redirects browsers to `/status`. API clients receive a simple JSON message. +* **Authentication:** None. +* **Response (Browser):** HTTP 307 Redirect to `/status`. +* **Response (API Client):** `200 OK` + ```json + { + "message": "MiniDiscovery API Root. See /status for HTML view or /docs for API documentation." + } + ``` + +#### Status Page + +* **`GET /status`** +* **Description:** Serves a simple HTML status page showing node health based on registered services. Useful for a quick visual overview. Automatically refreshes. +* **Authentication:** None. +* **Response:** `200 OK` with `Content-Type: text/html`. diff --git a/MiniDiscovery.py b/MiniDiscovery.py index 273a8de..c8f05c6 100644 --- a/MiniDiscovery.py +++ b/MiniDiscovery.py @@ -18,7 +18,7 @@ from typing import Dict, List, Optional, Tuple, Generator, Any from fastapi import Depends, FastAPI, Header, HTTPException, Security, status from fastapi.security import APIKeyHeader -from fastapi.responses import HTMLResponse +from fastapi.responses import HTMLResponse, RedirectResponse from pydantic import BaseModel, Field from twisted.internet import reactor from twisted.names import dns, server, common @@ -624,6 +624,24 @@ def create_fastapi_app() -> FastAPI: version="0.2.0", ) + # --- Root Endpoint: Redirect Browsers to Status --- + @app.get("/", include_in_schema=False) # Exclude this from OpenAPI docs + async def read_root(accept: Optional[str] = Header(None)): + """ + Redirects browser users (based on Accept header) to /status. + API clients get a simple JSON response. + """ + # Check if the Accept header exists and contains 'text/html' + if accept and "text/html" in accept.lower(): + # Likely a browser, redirect to the status page + return RedirectResponse(url="/status") + else: + # Likely an API client or a request not explicitly accepting HTML + # Return a simple JSON message or potentially 404/403 if preferred + return { + "message": "MiniDiscovery API Root. See /status for HTML view or /docs for API documentation." + } + # --- Status Page Endpoint (Public) --- @app.get("/status", response_class=HTMLResponse) async def get_status_page( diff --git a/README.md b/README.md new file mode 100644 index 0000000..2802f73 --- /dev/null +++ b/README.md @@ -0,0 +1,74 @@ +# MiniDiscovery + +MiniDiscovery is a minimal, self-contained service discovery tool inspired by Consul, built with Python (FastAPI, Twisted, SQLite). + +It provides basic service registration, health checking, and discovery via both an HTTP API and a DNS interface. + +## Core Features + +* **Service Registration/Deregistration:** Register services with address, port, tags, and metadata via an HTTP API. +* **Health Checking:** Automatically performs simple TCP health checks on registered services. +* **HTTP API Discovery:** Query for services (including filtering by tags and health status). +* **DNS Discovery:** Query for healthy services using DNS A, SRV, and TXT records (e.g., `_web._tcp.laiska.local`, `api.laiska.local`). +* **Authentication:** API endpoints are protected using HMAC-signed API tokens. +* **Persistence:** Uses a simple SQLite database (`.db` file) for storing service and token information. + +## Components + +The script runs three main components as separate processes: + +1. **FastAPI Server:** Handles the HTTP API requests. +2. **Twisted DNS Server:** Responds to DNS queries for registered services. +3. **Health Checker:** Periodically checks the health of registered services. + +## Quick Start + +### 1. Prerequisites + +* Python 3.8+ +* Dependencies listed in `requirements.txt` (`pip install -r requirements.txt`) + +### 2. Running Directly + +```bash +# First run: Set the initial admin token environment variable +export MINIDISCOVERY_ADMIN_TOKEN="your_very_secret_admin_token_here" + +# Run the script (uses default ports and file paths) +python MiniDiscovery.py +``` + +* An HMAC key file (`minidiscovery.key`) will be generated if it doesn't exist. +* A database file (`minidiscovery_data.db`) will be created. + +### 3. Running with Docker (Recommended) + +(Requires the provided `Dockerfile` and `requirements.txt`) + +```bash +# Build the image +docker build -t minidiscovery:latest . + +# Create a directory on host for persistent data (key and db) +mkdir ./md_data +# !!! If you have an existing minidiscovery.key, place it in ./md_data !!! + +# Run the container +docker run -d --name minidiscovery \ + -p 8500:8500/tcp \ + -p 10053:10053/tcp \ + -p 10053:10053/udp \ + -v ./md_data:/data \ + -e MINIDISCOVERY_ADMIN_TOKEN="your_very_secret_admin_token_here" \ + minidiscovery:latest +``` + +* The container expects the database and HMAC key in the `/data` volume. +* The `MINIDISCOVERY_ADMIN_TOKEN` is required on the first run to create the initial admin token. + +## Configuration + +* **Database Path:** `--db-path` argument or `MINIDISCOVERY_DB_PATH` environment variable. (Defaults to `minidiscovery_data.db`, or `/data/minidiscovery_data.db` in Docker). +* **HMAC Key File:** `--hmac-key-file` argument. (Defaults to `minidiscovery.key`, or `/data/minidiscovery.key` in Docker). +* **Ports:** `--api-port`, `--dns-port`. +* **Initial Admin Token:** `MINIDISCOVERY_ADMIN_TOKEN` environment variable (needed for first run only). diff --git a/key_rotation.md b/key_rotation.md new file mode 100644 index 0000000..3dd534e --- /dev/null +++ b/key_rotation.md @@ -0,0 +1,77 @@ +# How to Rotate api tokens + +1. Create new token +2. Revoke old token + +**Assumptions:** + +* Your MiniDiscovery API is running at `http://localhost:8500`. +* Your *current* (initial) admin token is stored in the variable `OLD_ADMIN_TOKEN`. +* You want the *new* admin token to be named `admin-v2` (or similar). + +**Steps:** + +1. **Create a New Admin Token:** + * Use your *existing* admin token (`$OLD_ADMIN_TOKEN`) in the `X-API-Token` header. + * POST to the `/v1/acl/token` endpoint. + * Request the `admin` permission for the new token. + + ```bash + OLD_ADMIN_TOKEN="your_initial_secure_admin_token_here" + NEW_TOKEN_NAME="admin-v2" # Or any descriptive name + + # Make the API call + response=$(curl -s -X POST "http://localhost:8500/v1/acl/token" \ + -H "accept: application/json" \ + -H "X-API-Token: ${OLD_ADMIN_TOKEN}" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "'"${NEW_TOKEN_NAME}"'", + "permissions": ["read", "write", "admin"] + }') + + # Extract the new token (use jq if available for robustness, otherwise basic parsing) + # Using jq: + # NEW_ADMIN_TOKEN=$(echo $response | jq -r '.token') + # Using grep/sed (less robust): + NEW_ADMIN_TOKEN=$(echo $response | grep -o '"token": "[^"]*"' | sed 's/"token": "//;s/"$//') + + + if [ -z "$NEW_ADMIN_TOKEN" ] || [ "$NEW_ADMIN_TOKEN" = "null" ]; then + echo "Error creating new token. Response:" + echo "$response" + exit 1 + else + echo "Successfully created new admin token named '${NEW_TOKEN_NAME}'." + echo "NEW TOKEN (SAVE THIS SECURELY!): ${NEW_ADMIN_TOKEN}" + # !!! IMPORTANT: Securely store NEW_ADMIN_TOKEN now !!! + fi + ``` + +2. **Revoke the Old Admin Token:** + * You can use *either* the `$OLD_ADMIN_TOKEN` or the `$NEW_ADMIN_TOKEN` you just created for authentication in the `X-API-Token` header (since both have `admin` rights at this point). It's often good practice to use the new one to verify it works. + * Send a DELETE request to `/v1/acl/token/{token_to_revoke}`. + * The `{token_to_revoke}` path parameter MUST be the **plain text** value of the token you want to remove (i.e., the value of `$OLD_ADMIN_TOKEN`). + + ```bash + # Use the NEW token to authenticate the revocation request + curl -X DELETE "http://localhost:8500/v1/acl/token/${OLD_ADMIN_TOKEN}" \ + -H "accept: application/json" \ + -H "X-API-Token: ${NEW_ADMIN_TOKEN}" + + # Check the output, it should indicate success (e.g., {"status":"revoked", ...}) + # Or use the OLD token to authenticate: + # curl -X DELETE "http://localhost:8500/v1/acl/token/${OLD_ADMIN_TOKEN}" \ + # -H "accept: application/json" \ + # -H "X-API-Token: ${OLD_ADMIN_TOKEN}" + + echo "Attempted to revoke the old admin token. Verify the response." + ``` + +**After these steps:** + +* The initial admin token (`$OLD_ADMIN_TOKEN`) will no longer be valid. +* The new token (`$NEW_ADMIN_TOKEN`) will be the active token with admin privileges. +* You should update any scripts, configurations, or password managers that were using the old token to use the new one. + +This create-then-revoke process is the standard way to handle credential rotation in systems like this. diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..8912799 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,35 @@ +-i https://pypi.org/simple +annotated-types==0.7.0; python_version >= '3.8' +anyio==4.9.0; python_version >= '3.9' +attrs==25.3.0; python_version >= '3.8' +automat==25.4.16; python_version >= '3.9' +black==25.1.0; python_version >= '3.9' +certifi==2025.4.26; python_version >= '3.6' +charset-normalizer==3.4.2; python_version >= '3.7' +click==8.1.8; python_version >= '3.7' +constantly==23.10.4; python_version >= '3.8' +fastapi==0.115.12; python_version >= '3.8' +h11==0.16.0; python_version >= '3.8' +hyperlink==21.0.0 +idna==3.10; python_version >= '3.6' +incremental==24.7.2; python_version >= '3.8' +iniconfig==2.1.0; python_version >= '3.8' +mypy-extensions==1.1.0; python_version >= '3.8' +packaging==25.0; python_version >= '3.8' +pathspec==0.12.1; python_version >= '3.8' +platformdirs==4.3.7; python_version >= '3.9' +pluggy==1.5.0; python_version >= '3.8' +pydantic==2.11.4; python_version >= '3.9' +pydantic-core==2.33.2; python_version >= '3.9' +pytest==8.3.5; python_version >= '3.8' +pytest-mock==3.14.0; python_version >= '3.8' +requests==2.32.3; python_version >= '3.8' +setuptools==80.1.0; python_version >= '3.9' +sniffio==1.3.1; python_version >= '3.7' +starlette==0.46.2; python_version >= '3.9' +twisted==24.11.0; python_full_version >= '3.8.0' +typing-extensions==4.13.2; python_version >= '3.8' +typing-inspection==0.4.0; python_version >= '3.9' +urllib3==2.4.0; python_version >= '3.9' +uvicorn==0.34.2; python_version >= '3.9' +zope-interface==7.2; python_version >= '3.8'