From 829c6fae1f7933ca46b0a88e96978411ac64e2d3 Mon Sep 17 00:00:00 2001 From: Kalzu Rekku Date: Mon, 29 Sep 2025 22:06:18 +0300 Subject: [PATCH] Small write up about the two issues conserning missing api endpoints. --- issues/7and12.md | 120 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 120 insertions(+) create mode 100644 issues/7and12.md diff --git a/issues/7and12.md b/issues/7and12.md new file mode 100644 index 0000000..bef8bdc --- /dev/null +++ b/issues/7and12.md @@ -0,0 +1,120 @@ +#7 Add _ls and _tree Endpoints for Hierarchical Key Listing Using Merkle Tree +----------------------------------------- + +KVS supports hierarchical keys (e.g., /home/room/closet/socks), which is great for organizing data like a file system. However, there's currently no built-in way for clients to discover or list subkeys under a given prefix/path. This makes it hard to build intuitive tools or UIs that need to navigate the keyspace, such as a web-based explorer or CLI client. + +Add two new read-only endpoints that leverage the existing Merkle tree infrastructure for efficient prefix-based key listing. This aligns with KVS's modular design, eventual consistency model, and Merkle-based sync (no need for full DB scans—traverse the tree to identify relevant leaf nodes in O(log N) time). +Proposed Endpoints + + Direct Children Listing (_ls or _list): + Endpoint: GET /kv/{path}/_ls (or GET /kv/{path}/_list for clarity). + Purpose: Returns a sorted list of direct subkeys under the given path/prefix (non-recursive). + Query Params (optional): + limit: Max number of keys to return (default: 100, max: 1000). + include_metadata: If true, include basic metadata like timestamps (default: false). + Response (JSON): + + { + "path": "/home/room", + "children": [ + { "subkey": "closet", "timestamp": 1695280000000 }, + { "subkey": "bed", "timestamp": 1695279000000 } + ], + "total": 2, + "truncated": false + } + + Behavior: + Treat {path} as a prefix (e.g., /home/room/ → keys starting with /home/room/ but not /home/room/sub/). + Use the Merkle tree to find leaf nodes in the prefix range [prefix, prefix~] (where ~ is the next lexicographical prefix). + Skip index keys (e.g., _ts:*). + Respect auth: Use existing middleware (e.g., read scope if auth_enabled: true). + In read-only/syncing modes: Allow if not modifying data. + +Recursive Tree View (_tree): + + Endpoint: GET /kv/{path}/_tree. + Purpose: Returns a recursive tree structure of all subkeys under the given path (depth-first or breadth-first, configurable). + Query Params (optional): + depth: Max recursion depth (default: unlimited, but suggest 5 for safety). + limit: Max total keys (default: 500, max: 5000). + include_metadata: Include timestamps/UUIDs (default: false). + format: json (default) or nested (tree-like JSON). + Response (JSON, nested format): + + { + "path": "/home/room", + "children": [ + { + "subkey": "closet", + "children": [ + { "subkey": "socks", "timestamp": 1695281000000 } + ], + "timestamp": 1695280000000 + }, + { + "subkey": "bed", + "timestamp": 1695279000000 + } + ], + "total": 3, + "truncated": false + } + + Behavior: + Build on _ls logic: Recursively query sub-prefixes via Merkle tree traversal. + Prune at depth or limit to avoid overload. + Same auth and mode rules as _ls. + +Integration with Existing Systems + + Merkle Tree Usage: Extend cluster/merkle.go (e.g., add GetKeysInRange(startKey, endKey) []string method) to traverse nodes covering the prefix range without fetching full values. Reuse buildMerkleTreeFromPairs and filterPairsByRange from handlers.go. + Range Query Reuse: Build on existing KVRangeRequest/KVRangeResponse in types.go and getKVRangeHandler (strip values to return just keys for efficiency). + Auth & Permissions: Apply via authService.Middleware (e.g., read scope). Respect allow_anonymous_read. + Config Toggle: Add key_listing_enabled: true to types.Config for optional disable (e.g., for security in public clusters). + Distributed Consistency: Since Merkle trees are synced, listings will be eventually consistent across nodes. Add a consistent: true query param to force a quick Merkle refresh if needed. + + +#12 Missing API Endpoints for Resource Metadata Management (Ownership & Permissions) +----------------------------------------- + +The KVS system currently lacks API endpoints to manage ResourceMetadata for key-value paths (/kv/{path}). While the AuthService and permissions.go implement robust permission checking based on OwnerUUID, GroupUUID, and Permissions, there are no exposed routes to: + +Assign group-level permissions: Users cannot grant read/write access to specific groups for a given key-value path. + +Change resource ownership: Users cannot transfer ownership of a key-value entry to another user. + +This prevents administrators from fully leveraging the existing authentication and authorization framework for fine-grained access control over stored data. + +Impact: + +Limited administrative control over data access. + +Inability to implement granular, group-based access policies for KV data. + +Difficulty in reassigning data ownership when users or roles change. + +Proposed Solution: +Implement new API endpoints (e.g., /kv/{path}/metadata) to allow authenticated and authorized users to: + +Set/update the OwnerUUID for a given path. + +Set/update the GroupUUID for a given path. + +Set/update the Permissions bitmask for a given path. + +Relevant Files: + +server/routes.go (for new API routes) + +server/handlers.go (for implementing new handlers) + +auth/auth.go (for AuthService methods to interact with ResourceMetadata) + +auth/permissions.go (existing logic for permission checks) + +types/types.go (for ResourceMetadata structure) + + + +