Small write up about the two issues conserning missing api endpoints.

This commit is contained in:
Kalzu Rekku
2025-09-29 22:06:18 +03:00
parent d5a0eb7efe
commit 829c6fae1f

120
issues/7and12.md Normal file
View File

@@ -0,0 +1,120 @@
#7 Add _ls and _tree Endpoints for Hierarchical Key Listing Using Merkle Tree
-----------------------------------------
KVS supports hierarchical keys (e.g., /home/room/closet/socks), which is great for organizing data like a file system. However, there's currently no built-in way for clients to discover or list subkeys under a given prefix/path. This makes it hard to build intuitive tools or UIs that need to navigate the keyspace, such as a web-based explorer or CLI client.
Add two new read-only endpoints that leverage the existing Merkle tree infrastructure for efficient prefix-based key listing. This aligns with KVS's modular design, eventual consistency model, and Merkle-based sync (no need for full DB scans—traverse the tree to identify relevant leaf nodes in O(log N) time).
Proposed Endpoints
Direct Children Listing (_ls or _list):
Endpoint: GET /kv/{path}/_ls (or GET /kv/{path}/_list for clarity).
Purpose: Returns a sorted list of direct subkeys under the given path/prefix (non-recursive).
Query Params (optional):
limit: Max number of keys to return (default: 100, max: 1000).
include_metadata: If true, include basic metadata like timestamps (default: false).
Response (JSON):
{
"path": "/home/room",
"children": [
{ "subkey": "closet", "timestamp": 1695280000000 },
{ "subkey": "bed", "timestamp": 1695279000000 }
],
"total": 2,
"truncated": false
}
Behavior:
Treat {path} as a prefix (e.g., /home/room/ → keys starting with /home/room/ but not /home/room/sub/).
Use the Merkle tree to find leaf nodes in the prefix range [prefix, prefix~] (where ~ is the next lexicographical prefix).
Skip index keys (e.g., _ts:*).
Respect auth: Use existing middleware (e.g., read scope if auth_enabled: true).
In read-only/syncing modes: Allow if not modifying data.
Recursive Tree View (_tree):
Endpoint: GET /kv/{path}/_tree.
Purpose: Returns a recursive tree structure of all subkeys under the given path (depth-first or breadth-first, configurable).
Query Params (optional):
depth: Max recursion depth (default: unlimited, but suggest 5 for safety).
limit: Max total keys (default: 500, max: 5000).
include_metadata: Include timestamps/UUIDs (default: false).
format: json (default) or nested (tree-like JSON).
Response (JSON, nested format):
{
"path": "/home/room",
"children": [
{
"subkey": "closet",
"children": [
{ "subkey": "socks", "timestamp": 1695281000000 }
],
"timestamp": 1695280000000
},
{
"subkey": "bed",
"timestamp": 1695279000000
}
],
"total": 3,
"truncated": false
}
Behavior:
Build on _ls logic: Recursively query sub-prefixes via Merkle tree traversal.
Prune at depth or limit to avoid overload.
Same auth and mode rules as _ls.
Integration with Existing Systems
Merkle Tree Usage: Extend cluster/merkle.go (e.g., add GetKeysInRange(startKey, endKey) []string method) to traverse nodes covering the prefix range without fetching full values. Reuse buildMerkleTreeFromPairs and filterPairsByRange from handlers.go.
Range Query Reuse: Build on existing KVRangeRequest/KVRangeResponse in types.go and getKVRangeHandler (strip values to return just keys for efficiency).
Auth & Permissions: Apply via authService.Middleware (e.g., read scope). Respect allow_anonymous_read.
Config Toggle: Add key_listing_enabled: true to types.Config for optional disable (e.g., for security in public clusters).
Distributed Consistency: Since Merkle trees are synced, listings will be eventually consistent across nodes. Add a consistent: true query param to force a quick Merkle refresh if needed.
#12 Missing API Endpoints for Resource Metadata Management (Ownership & Permissions)
-----------------------------------------
The KVS system currently lacks API endpoints to manage ResourceMetadata for key-value paths (/kv/{path}). While the AuthService and permissions.go implement robust permission checking based on OwnerUUID, GroupUUID, and Permissions, there are no exposed routes to:
Assign group-level permissions: Users cannot grant read/write access to specific groups for a given key-value path.
Change resource ownership: Users cannot transfer ownership of a key-value entry to another user.
This prevents administrators from fully leveraging the existing authentication and authorization framework for fine-grained access control over stored data.
Impact:
Limited administrative control over data access.
Inability to implement granular, group-based access policies for KV data.
Difficulty in reassigning data ownership when users or roles change.
Proposed Solution:
Implement new API endpoints (e.g., /kv/{path}/metadata) to allow authenticated and authorized users to:
Set/update the OwnerUUID for a given path.
Set/update the GroupUUID for a given path.
Set/update the Permissions bitmask for a given path.
Relevant Files:
server/routes.go (for new API routes)
server/handlers.go (for implementing new handlers)
auth/auth.go (for AuthService methods to interact with ResourceMetadata)
auth/permissions.go (existing logic for permission checks)
types/types.go (for ResourceMetadata structure)