MrKalzu/MiniDiscovery
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Some documentation and stuff

Browse Source
This commit is contained in:
Kalzu Rekku 2025-05-03 18:26:12 +03:00
parent 58bbef2837
commit a9f50ff68c
5 changed files with 564 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

359
API.md Normal file
View File

@ -0,0 +1,359 @@
# MiniDiscovery API Documentation
This document describes the HTTP API endpoints provided by MiniDiscovery.
## Authentication
Most API endpoints require authentication using an API token. Tokens must be passed in the `X-API-Token` HTTP header.
```http
GET /v1/catalog/services HTTP/1.1
Host: your-minidiscovery-host:8500
X-API-Token: your_secret_api_token
```
Tokens have associated permissions:
* **`read`**: Allows reading service information (catalog, health).
* **`write`**: Allows registering and deregistering services.
* **`admin`**: Allows all `read` and `write` operations, plus managing ACL tokens.
The initial admin token is set using the `MINIDISCOVERY_ADMIN_TOKEN` environment variable on the *first* run when the database is empty. Subsequent tokens are managed via the API.
## Data Models
### `ServiceInstance`
This model represents a single instance of a registered service.
```json
{
"id": "web-instance-1",
"name": "web",
"address": "192.168.1.100",
"port": 80,
"tags": ["frontend", "nginx"],
"metadata": {
"version": "1.2.3",
"region": "us-east-1"
},
"health": "passing",
"last_updated": 1678886400.123456
}
```
* `id` (string, required): Unique identifier for this specific instance.
* `name` (string, required): Logical name of the service (e.g., 'web', 'api', 'db').
* `address` (string, required): IP address or resolvable hostname where the service listens.
* `port` (integer, required): Port number (1-65535).
* `tags` (array of strings, optional): List of tags for filtering/grouping. Defaults to `[]`.
* `metadata` (object, optional): Key-value string pairs for arbitrary metadata. Defaults to `{}`.
* `health` (string, optional): Current health status ('passing', 'failing', 'unknown'). Defaults to 'passing' on registration. Automatically updated by the health checker.
* `last_updated` (float, internal): Unix timestamp of the last update (set automatically).
## Endpoints
### Agent Endpoints
These endpoints interact with the local agent state (registering/deregistering services).
#### Register Service
* **`POST /v1/agent/service/register`**
* **Description:** Registers a new service instance or updates an existing one with the same `id`. This operation is idempotent based on the service `id`.
* **Authentication:** Requires `write` permission.
* **Request Body:** `ServiceInstance` JSON object. The `health` and `last_updated` fields in the request body are ignored; health is managed internally, and `last_updated` is set automatically.
* **Success Response:** `200 OK`
```json
{
"status": "registered",
"service_id": "web-instance-1"
}
```
* **Error Responses:**
* `400 Bad Request`: Invalid request body format.
* `401 Unauthorized`: Missing `X-API-Token`.
* `403 Forbidden`: Invalid token or insufficient permissions.
* `500 Internal Server Error`: Database error during registration.
* **`curl` Example:**
```bash
curl -X POST http://localhost:8500/v1/agent/service/register \
-H "X-API-Token: your_write_token" \
-H "Content-Type: application/json" \
-d '{
"id": "api-instance-01",
"name": "api",
"address": "10.0.1.5",
"port": 8080,
"tags": ["backend", "v2"],
"metadata": {"git_sha": "a1b2c3d"}
}'
```
#### Deregister Service
* **`PUT /v1/agent/service/deregister/{service_id}`**
*(Note: Consul uses PUT here, although DELETE might feel more intuitive)*
* **Description:** Removes a service instance by its ID.
* **Authentication:** Requires `write` permission.
* **Path Parameters:**
* `service_id` (string): The unique ID of the service instance to deregister.
* **Success Response:** `200 OK`
```json
{
"status": "deregistered",
"service_id": "api-instance-01"
}
```
* **Error Responses:**
* `401 Unauthorized`: Missing `X-API-Token`.
* `403 Forbidden`: Invalid token or insufficient permissions.
* `404 Not Found`: Service with the given `service_id` does not exist.
* **`curl` Example:**
```bash
curl -X PUT http://localhost:8500/v1/agent/service/deregister/api-instance-01 \
-H "X-API-Token: your_write_token"
```
### Catalog Endpoints
These endpoints provide information about registered services across the system.
#### List Services
* **`GET /v1/catalog/services`**
* **Description:** Returns a map of all registered service names to a list of unique tags associated with instances of that service.
* **Authentication:** Requires `read` permission.
* **Success Response:** `200 OK`
```json
{
"web": ["frontend", "nginx"],
"api": ["backend", "v2"],
"db": []
}
```
* **Error Responses:**
* `401 Unauthorized`: Missing `X-API-Token`.
* `403 Forbidden`: Invalid token or insufficient permissions.
* **`curl` Example:**
```bash
curl http://localhost:8500/v1/catalog/services \
-H "X-API-Token: your_read_token"
```
#### List Service Instances
* **`GET /v1/catalog/service/{service_name}`**
* **Description:** Returns a list of all registered instances for a given service name.
* **Authentication:** Requires `read` permission.
* **Path Parameters:**
* `service_name` (string): The name of the service (e.g., 'web', 'api').
* **Query Parameters:**
* `tag` (string, optional): Filter instances by tag. Only instances with this tag will be returned.
* **Success Response:** `200 OK` (Returns an array of `ServiceInstance` objects)
```json
[
{
"id": "web-instance-1",
"name": "web",
"address": "192.168.1.100",
"port": 80,
"tags": ["frontend", "nginx"],
"metadata": {"version": "1.2.3"},
"health": "passing",
"last_updated": 1678886400.123
},
{
"id": "web-instance-2",
"name": "web",
"address": "192.168.1.101",
"port": 80,
"tags": ["frontend"],
"metadata": {"version": "1.2.4"},
"health": "failing",
"last_updated": 1678886405.456
}
]
```
* Returns `[]` if the service name is not found.
* **Error Responses:**
* `401 Unauthorized`: Missing `X-API-Token`.
* `403 Forbidden`: Invalid token or insufficient permissions.
* **`curl` Examples:**
```bash
# Get all 'web' instances
curl http://localhost:8500/v1/catalog/service/web \
-H "X-API-Token: your_read_token"
# Get 'web' instances tagged with 'nginx'
curl http://localhost:8500/v1/catalog/service/web?tag=nginx \
-H "X-API-Token: your_read_token"
```
### Health Endpoints
These endpoints provide service discovery filtered by health status.
#### List Healthy Service Instances
* **`GET /v1/health/service/{service_name}`**
* **Description:** Returns a list of service instances, similar to the catalog endpoint, but allows filtering by health status.
* **Authentication:** Requires `read` permission.
* **Path Parameters:**
* `service_name` (string): The name of the service.
* **Query Parameters:**
* `tag` (string, optional): Filter instances by tag.
* `passing` (boolean, optional): If `true`, only return instances with a `passing` health status. Defaults to `false` (returns all instances regardless of health).
* **Success Response:** `200 OK` (Returns an array of `ServiceInstance` objects)
* Response format is identical to `/v1/catalog/service/{service_name}` but filtered according to query parameters.
* **Error Responses:**
* `401 Unauthorized`: Missing `X-API-Token`.
* `403 Forbidden`: Invalid token or insufficient permissions.
* **`curl` Examples:**
```bash
# Get all 'api' instances (healthy or not)
curl http://localhost:8500/v1/health/service/api \
-H "X-API-Token: your_read_token"
# Get only healthy ('passing') 'api' instances
curl http://localhost:8500/v1/health/service/api?passing=true \
-H "X-API-Token: your_read_token"
# Get only healthy 'api' instances tagged 'v2'
curl 'http://localhost:8500/v1/health/service/api?passing=true&tag=v2' \
-H "X-API-Token: your_read_token"
```
### ACL Token Endpoints
These endpoints manage API access tokens.
#### Create Token
* **`POST /v1/acl/token`**
* **Description:** Creates a new API token with specified permissions.
* **Authentication:** Requires `admin` permission.
* **Request Body:**
```json
{
"name": "my-app-token",
"permissions": ["read", "write"]
}
```
* `name` (string, required): A unique descriptive name for the token.
* `permissions` (array of strings, optional): List of permissions (`read`, `write`, `admin`). Defaults to `["read", "write"]`.
* **Success Response:** `201 Created`
```json
{
"token": "YOUR_NEW_SECRET_TOKEN_VALUE",
"name": "my-app-token",
"permissions": ["read", "write"]
}
```
* **IMPORTANT:** The `token` value is the actual secret token. It is **only shown once** in this response. Store it securely immediately.
* **Error Responses:**
* `400 Bad Request`: Invalid request body or token name already exists.
* `401 Unauthorized`: Missing `X-API-Token`.
* `403 Forbidden`: Invalid token or insufficient permissions (not admin).
* `500 Internal Server Error`: Database error during token creation.
* **`curl` Example:**
```bash
curl -X POST http://localhost:8500/v1/acl/token \
-H "X-API-Token: your_admin_token" \
-H "Content-Type: application/json" \
-d '{
"name": "read-only-dashboard",
"permissions": ["read"]
}'
```
#### Revoke Token
* **`DELETE /v1/acl/token/{token_to_revoke}`**
* **Description:** Revokes (deletes) an existing API token.
* **Authentication:** Requires `admin` permission.
* **Path Parameters:**
* `token_to_revoke` (string): The **actual secret token value** of the token you want to revoke.
* **Success Response:** `200 OK`
```json
{
"status": "revoked",
"token_info": "Token revoked successfully"
}
```
* **Error Responses:**
* `401 Unauthorized`: Missing `X-API-Token`.
* `403 Forbidden`: Invalid token or insufficient permissions (not admin).
* `404 Not Found`: The token specified in the path does not exist or was already revoked.
* **`curl` Example:**
```bash
# Replace YOUR_OLD_SECRET_TOKEN_VALUE with the token to be deleted
curl -X DELETE http://localhost:8500/v1/acl/token/YOUR_OLD_SECRET_TOKEN_VALUE \
-H "X-API-Token: your_admin_token"
```
### DNS over HTTPS (DoH)
* **`GET /dns-query`**
* **Description:** Provides a simplified DNS-over-HTTPS interface (RFC 8484 GET format) for querying service addresses and SRV records. Primarily intended for simple DNS clients or scripts. **Note:** Requires service names to end with the configured DNS suffix (default: `.laiska.local`).
* **Authentication:** None (typically).
* **Query Parameters:**
* `name` (string, required): The DNS query name (e.g., `web.laiska.local`, `_frontend._tcp.web.laiska.local`).
* `type` (string, optional): The DNS record type (e.g., `A`, `SRV`, `TXT`). Defaults to `A`. Case-insensitive.
* **Success Response:** `200 OK` (JSON object following RFC 8484 structure)
* **Example (A query for `web.laiska.local`):**
```json
{
"Status": 0, "TC": false, "RD": true, "RA": false, "AD": false, "CD": false,
"Question": [{"name": "web.laiska.local.", "type": 1}],
"Answer": [
{"name": "web.laiska.local.", "type": 1, "TTL": 60, "data": "192.168.1.100"},
{"name": "web.laiska.local.", "type": 1, "TTL": 60, "data": "192.168.1.101"}
]
}
```
* **Example (SRV query for `_frontend._tcp.web.laiska.local`):**
```json
{
"Status": 0, "TC": false, "RD": true, "RA": false, "AD": false, "CD": false,
"Question": [{"name": "_frontend._tcp.web.laiska.local.", "type": 33}],
"Answer": [
{"name": "_frontend._tcp.web.laiska.local.", "type": 33, "TTL": 60, "data": "0 10 80 web-instance-1.laiska.local."},
{"name": "_frontend._tcp.web.laiska.local.", "type": 33, "TTL": 60, "data": "0 10 80 web-instance-2.laiska.local."}
],
"Additional": [ // Note: MiniDiscovery doesn't currently populate Additional well for DoH
// Ideally A records for targets would be here
]
}
```
* **Error/NXDOMAIN Response:** Returns JSON with `"Status"` other than `0` (e.g., `2` for NXDOMAIN, `4` for Not Implemented type).
* **`curl` Example:**
```bash
# Query for A records
curl "http://localhost:8500/dns-query?name=web.laiska.local&type=A"
# Query for SRV records
curl "http://localhost:8500/dns-query?name=_frontend._tcp.web.laiska.local&type=SRV"
```
### Other Endpoints
#### Root
* **`GET /`**
* **Description:** Provides a basic entry point. Redirects browsers to `/status`. API clients receive a simple JSON message.
* **Authentication:** None.
* **Response (Browser):** HTTP 307 Redirect to `/status`.
* **Response (API Client):** `200 OK`
```json
{
"message": "MiniDiscovery API Root. See /status for HTML view or /docs for API documentation."
}
```
#### Status Page
* **`GET /status`**
* **Description:** Serves a simple HTML status page showing node health based on registered services. Useful for a quick visual overview. Automatically refreshes.
* **Authentication:** None.
* **Response:** `200 OK` with `Content-Type: text/html`.

20
MiniDiscovery.py
View File

@ -18,7 +18,7 @@ from typing import Dict, List, Optional, Tuple, Generator, Any
from fastapi import Depends, FastAPI, Header, HTTPException, Security, status from fastapi import Depends, FastAPI, Header, HTTPException, Security, status
from fastapi.security import APIKeyHeader from fastapi.security import APIKeyHeader
from fastapi.responses import HTMLResponse from fastapi.responses import HTMLResponse, RedirectResponse
from pydantic import BaseModel, Field from pydantic import BaseModel, Field
from twisted.internet import reactor from twisted.internet import reactor
from twisted.names import dns, server, common from twisted.names import dns, server, common
@ -624,6 +624,24 @@ def create_fastapi_app() -> FastAPI:
version="0.2.0", version="0.2.0",
) )
# --- Root Endpoint: Redirect Browsers to Status ---
@app.get("/", include_in_schema=False) # Exclude this from OpenAPI docs
async def read_root(accept: Optional[str] = Header(None)):
"""
Redirects browser users (based on Accept header) to /status.
API clients get a simple JSON response.
"""
# Check if the Accept header exists and contains 'text/html'
if accept and "text/html" in accept.lower():
# Likely a browser, redirect to the status page
return RedirectResponse(url="/status")
else:
# Likely an API client or a request not explicitly accepting HTML
# Return a simple JSON message or potentially 404/403 if preferred
return {
"message": "MiniDiscovery API Root. See /status for HTML view or /docs for API documentation."
}
# --- Status Page Endpoint (Public) --- # --- Status Page Endpoint (Public) ---
@app.get("/status", response_class=HTMLResponse) @app.get("/status", response_class=HTMLResponse)
async def get_status_page( async def get_status_page(

74
README.md Normal file
View File

@ -0,0 +1,74 @@
# MiniDiscovery
MiniDiscovery is a minimal, self-contained service discovery tool inspired by Consul, built with Python (FastAPI, Twisted, SQLite).
It provides basic service registration, health checking, and discovery via both an HTTP API and a DNS interface.
## Core Features
* **Service Registration/Deregistration:** Register services with address, port, tags, and metadata via an HTTP API.
* **Health Checking:** Automatically performs simple TCP health checks on registered services.
* **HTTP API Discovery:** Query for services (including filtering by tags and health status).
* **DNS Discovery:** Query for healthy services using DNS A, SRV, and TXT records (e.g., `_web._tcp.laiska.local`, `api.laiska.local`).
* **Authentication:** API endpoints are protected using HMAC-signed API tokens.
* **Persistence:** Uses a simple SQLite database (`.db` file) for storing service and token information.
## Components
The script runs three main components as separate processes:
1. **FastAPI Server:** Handles the HTTP API requests.
2. **Twisted DNS Server:** Responds to DNS queries for registered services.
3. **Health Checker:** Periodically checks the health of registered services.
## Quick Start
### 1. Prerequisites
* Python 3.8+
* Dependencies listed in `requirements.txt` (`pip install -r requirements.txt`)
### 2. Running Directly
```bash
# First run: Set the initial admin token environment variable
export MINIDISCOVERY_ADMIN_TOKEN="your_very_secret_admin_token_here"
# Run the script (uses default ports and file paths)
python MiniDiscovery.py
```
* An HMAC key file (`minidiscovery.key`) will be generated if it doesn't exist.
* A database file (`minidiscovery_data.db`) will be created.
### 3. Running with Docker (Recommended)
(Requires the provided `Dockerfile` and `requirements.txt`)
```bash
# Build the image
docker build -t minidiscovery:latest .
# Create a directory on host for persistent data (key and db)
mkdir ./md_data
# !!! If you have an existing minidiscovery.key, place it in ./md_data !!!
# Run the container
docker run -d --name minidiscovery \
-p 8500:8500/tcp \
-p 10053:10053/tcp \
-p 10053:10053/udp \
-v ./md_data:/data \
-e MINIDISCOVERY_ADMIN_TOKEN="your_very_secret_admin_token_here" \
minidiscovery:latest
```
* The container expects the database and HMAC key in the `/data` volume.
* The `MINIDISCOVERY_ADMIN_TOKEN` is required on the first run to create the initial admin token.
## Configuration
* **Database Path:** `--db-path` argument or `MINIDISCOVERY_DB_PATH` environment variable. (Defaults to `minidiscovery_data.db`, or `/data/minidiscovery_data.db` in Docker).
* **HMAC Key File:** `--hmac-key-file` argument. (Defaults to `minidiscovery.key`, or `/data/minidiscovery.key` in Docker).
* **Ports:** `--api-port`, `--dns-port`.
* **Initial Admin Token:** `MINIDISCOVERY_ADMIN_TOKEN` environment variable (needed for first run only).

77
key_rotation.md Normal file
View File

@ -0,0 +1,77 @@
# How to Rotate api tokens
1. Create new token
2. Revoke old token
**Assumptions:**
* Your MiniDiscovery API is running at `http://localhost:8500`.
* Your *current* (initial) admin token is stored in the variable `OLD_ADMIN_TOKEN`.
* You want the *new* admin token to be named `admin-v2` (or similar).
**Steps:**
1. **Create a New Admin Token:**
* Use your *existing* admin token (`$OLD_ADMIN_TOKEN`) in the `X-API-Token` header.
* POST to the `/v1/acl/token` endpoint.
* Request the `admin` permission for the new token.
```bash
OLD_ADMIN_TOKEN="your_initial_secure_admin_token_here"
NEW_TOKEN_NAME="admin-v2" # Or any descriptive name
# Make the API call
response=$(curl -s -X POST "http://localhost:8500/v1/acl/token" \
-H "accept: application/json" \
-H "X-API-Token: ${OLD_ADMIN_TOKEN}" \
-H "Content-Type: application/json" \
-d '{
"name": "'"${NEW_TOKEN_NAME}"'",
"permissions": ["read", "write", "admin"]
}')
# Extract the new token (use jq if available for robustness, otherwise basic parsing)
# Using jq:
# NEW_ADMIN_TOKEN=$(echo $response | jq -r '.token')
# Using grep/sed (less robust):
NEW_ADMIN_TOKEN=$(echo $response | grep -o '"token": "[^"]*"' | sed 's/"token": "//;s/"$//')
if [ -z "$NEW_ADMIN_TOKEN" ] || [ "$NEW_ADMIN_TOKEN" = "null" ]; then
echo "Error creating new token. Response:"
echo "$response"
exit 1
else
echo "Successfully created new admin token named '${NEW_TOKEN_NAME}'."
echo "NEW TOKEN (SAVE THIS SECURELY!): ${NEW_ADMIN_TOKEN}"
# !!! IMPORTANT: Securely store NEW_ADMIN_TOKEN now !!!
fi
```
2. **Revoke the Old Admin Token:**
* You can use *either* the `$OLD_ADMIN_TOKEN` or the `$NEW_ADMIN_TOKEN` you just created for authentication in the `X-API-Token` header (since both have `admin` rights at this point). It's often good practice to use the new one to verify it works.
* Send a DELETE request to `/v1/acl/token/{token_to_revoke}`.
* The `{token_to_revoke}` path parameter MUST be the **plain text** value of the token you want to remove (i.e., the value of `$OLD_ADMIN_TOKEN`).
```bash
# Use the NEW token to authenticate the revocation request
curl -X DELETE "http://localhost:8500/v1/acl/token/${OLD_ADMIN_TOKEN}" \
-H "accept: application/json" \
-H "X-API-Token: ${NEW_ADMIN_TOKEN}"
# Check the output, it should indicate success (e.g., {"status":"revoked", ...})
# Or use the OLD token to authenticate:
# curl -X DELETE "http://localhost:8500/v1/acl/token/${OLD_ADMIN_TOKEN}" \
# -H "accept: application/json" \
# -H "X-API-Token: ${OLD_ADMIN_TOKEN}"
echo "Attempted to revoke the old admin token. Verify the response."
```
**After these steps:**
* The initial admin token (`$OLD_ADMIN_TOKEN`) will no longer be valid.
* The new token (`$NEW_ADMIN_TOKEN`) will be the active token with admin privileges.
* You should update any scripts, configurations, or password managers that were using the old token to use the new one.
This create-then-revoke process is the standard way to handle credential rotation in systems like this.

35
requirements.txt Normal file
View File

@ -0,0 +1,35 @@
-i https://pypi.org/simple
annotated-types==0.7.0; python_version >= '3.8'
anyio==4.9.0; python_version >= '3.9'
attrs==25.3.0; python_version >= '3.8'
automat==25.4.16; python_version >= '3.9'
black==25.1.0; python_version >= '3.9'
certifi==2025.4.26; python_version >= '3.6'
charset-normalizer==3.4.2; python_version >= '3.7'
click==8.1.8; python_version >= '3.7'
constantly==23.10.4; python_version >= '3.8'
fastapi==0.115.12; python_version >= '3.8'
h11==0.16.0; python_version >= '3.8'
hyperlink==21.0.0
idna==3.10; python_version >= '3.6'
incremental==24.7.2; python_version >= '3.8'
iniconfig==2.1.0; python_version >= '3.8'
mypy-extensions==1.1.0; python_version >= '3.8'
packaging==25.0; python_version >= '3.8'
pathspec==0.12.1; python_version >= '3.8'
platformdirs==4.3.7; python_version >= '3.9'
pluggy==1.5.0; python_version >= '3.8'
pydantic==2.11.4; python_version >= '3.9'
pydantic-core==2.33.2; python_version >= '3.9'
pytest==8.3.5; python_version >= '3.8'
pytest-mock==3.14.0; python_version >= '3.8'
requests==2.32.3; python_version >= '3.8'
setuptools==80.1.0; python_version >= '3.9'
sniffio==1.3.1; python_version >= '3.7'
starlette==0.46.2; python_version >= '3.9'
twisted==24.11.0; python_full_version >= '3.8.0'
typing-extensions==4.13.2; python_version >= '3.8'
typing-inspection==0.4.0; python_version >= '3.9'
urllib3==2.4.0; python_version >= '3.9'
uvicorn==0.34.2; python_version >= '3.9'
zope-interface==7.2; python_version >= '3.8'