forked from ryyst/kalzu-value-store
Small write up about the two issues conserning missing api endpoints.
This commit is contained in:
120
issues/7and12.md
Normal file
120
issues/7and12.md
Normal file
@@ -0,0 +1,120 @@
|
|||||||
|
#7 Add _ls and _tree Endpoints for Hierarchical Key Listing Using Merkle Tree
|
||||||
|
-----------------------------------------
|
||||||
|
|
||||||
|
KVS supports hierarchical keys (e.g., /home/room/closet/socks), which is great for organizing data like a file system. However, there's currently no built-in way for clients to discover or list subkeys under a given prefix/path. This makes it hard to build intuitive tools or UIs that need to navigate the keyspace, such as a web-based explorer or CLI client.
|
||||||
|
|
||||||
|
Add two new read-only endpoints that leverage the existing Merkle tree infrastructure for efficient prefix-based key listing. This aligns with KVS's modular design, eventual consistency model, and Merkle-based sync (no need for full DB scans—traverse the tree to identify relevant leaf nodes in O(log N) time).
|
||||||
|
Proposed Endpoints
|
||||||
|
|
||||||
|
Direct Children Listing (_ls or _list):
|
||||||
|
Endpoint: GET /kv/{path}/_ls (or GET /kv/{path}/_list for clarity).
|
||||||
|
Purpose: Returns a sorted list of direct subkeys under the given path/prefix (non-recursive).
|
||||||
|
Query Params (optional):
|
||||||
|
limit: Max number of keys to return (default: 100, max: 1000).
|
||||||
|
include_metadata: If true, include basic metadata like timestamps (default: false).
|
||||||
|
Response (JSON):
|
||||||
|
|
||||||
|
{
|
||||||
|
"path": "/home/room",
|
||||||
|
"children": [
|
||||||
|
{ "subkey": "closet", "timestamp": 1695280000000 },
|
||||||
|
{ "subkey": "bed", "timestamp": 1695279000000 }
|
||||||
|
],
|
||||||
|
"total": 2,
|
||||||
|
"truncated": false
|
||||||
|
}
|
||||||
|
|
||||||
|
Behavior:
|
||||||
|
Treat {path} as a prefix (e.g., /home/room/ → keys starting with /home/room/ but not /home/room/sub/).
|
||||||
|
Use the Merkle tree to find leaf nodes in the prefix range [prefix, prefix~] (where ~ is the next lexicographical prefix).
|
||||||
|
Skip index keys (e.g., _ts:*).
|
||||||
|
Respect auth: Use existing middleware (e.g., read scope if auth_enabled: true).
|
||||||
|
In read-only/syncing modes: Allow if not modifying data.
|
||||||
|
|
||||||
|
Recursive Tree View (_tree):
|
||||||
|
|
||||||
|
Endpoint: GET /kv/{path}/_tree.
|
||||||
|
Purpose: Returns a recursive tree structure of all subkeys under the given path (depth-first or breadth-first, configurable).
|
||||||
|
Query Params (optional):
|
||||||
|
depth: Max recursion depth (default: unlimited, but suggest 5 for safety).
|
||||||
|
limit: Max total keys (default: 500, max: 5000).
|
||||||
|
include_metadata: Include timestamps/UUIDs (default: false).
|
||||||
|
format: json (default) or nested (tree-like JSON).
|
||||||
|
Response (JSON, nested format):
|
||||||
|
|
||||||
|
{
|
||||||
|
"path": "/home/room",
|
||||||
|
"children": [
|
||||||
|
{
|
||||||
|
"subkey": "closet",
|
||||||
|
"children": [
|
||||||
|
{ "subkey": "socks", "timestamp": 1695281000000 }
|
||||||
|
],
|
||||||
|
"timestamp": 1695280000000
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"subkey": "bed",
|
||||||
|
"timestamp": 1695279000000
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"total": 3,
|
||||||
|
"truncated": false
|
||||||
|
}
|
||||||
|
|
||||||
|
Behavior:
|
||||||
|
Build on _ls logic: Recursively query sub-prefixes via Merkle tree traversal.
|
||||||
|
Prune at depth or limit to avoid overload.
|
||||||
|
Same auth and mode rules as _ls.
|
||||||
|
|
||||||
|
Integration with Existing Systems
|
||||||
|
|
||||||
|
Merkle Tree Usage: Extend cluster/merkle.go (e.g., add GetKeysInRange(startKey, endKey) []string method) to traverse nodes covering the prefix range without fetching full values. Reuse buildMerkleTreeFromPairs and filterPairsByRange from handlers.go.
|
||||||
|
Range Query Reuse: Build on existing KVRangeRequest/KVRangeResponse in types.go and getKVRangeHandler (strip values to return just keys for efficiency).
|
||||||
|
Auth & Permissions: Apply via authService.Middleware (e.g., read scope). Respect allow_anonymous_read.
|
||||||
|
Config Toggle: Add key_listing_enabled: true to types.Config for optional disable (e.g., for security in public clusters).
|
||||||
|
Distributed Consistency: Since Merkle trees are synced, listings will be eventually consistent across nodes. Add a consistent: true query param to force a quick Merkle refresh if needed.
|
||||||
|
|
||||||
|
|
||||||
|
#12 Missing API Endpoints for Resource Metadata Management (Ownership & Permissions)
|
||||||
|
-----------------------------------------
|
||||||
|
|
||||||
|
The KVS system currently lacks API endpoints to manage ResourceMetadata for key-value paths (/kv/{path}). While the AuthService and permissions.go implement robust permission checking based on OwnerUUID, GroupUUID, and Permissions, there are no exposed routes to:
|
||||||
|
|
||||||
|
Assign group-level permissions: Users cannot grant read/write access to specific groups for a given key-value path.
|
||||||
|
|
||||||
|
Change resource ownership: Users cannot transfer ownership of a key-value entry to another user.
|
||||||
|
|
||||||
|
This prevents administrators from fully leveraging the existing authentication and authorization framework for fine-grained access control over stored data.
|
||||||
|
|
||||||
|
Impact:
|
||||||
|
|
||||||
|
Limited administrative control over data access.
|
||||||
|
|
||||||
|
Inability to implement granular, group-based access policies for KV data.
|
||||||
|
|
||||||
|
Difficulty in reassigning data ownership when users or roles change.
|
||||||
|
|
||||||
|
Proposed Solution:
|
||||||
|
Implement new API endpoints (e.g., /kv/{path}/metadata) to allow authenticated and authorized users to:
|
||||||
|
|
||||||
|
Set/update the OwnerUUID for a given path.
|
||||||
|
|
||||||
|
Set/update the GroupUUID for a given path.
|
||||||
|
|
||||||
|
Set/update the Permissions bitmask for a given path.
|
||||||
|
|
||||||
|
Relevant Files:
|
||||||
|
|
||||||
|
server/routes.go (for new API routes)
|
||||||
|
|
||||||
|
server/handlers.go (for implementing new handlers)
|
||||||
|
|
||||||
|
auth/auth.go (for AuthService methods to interact with ResourceMetadata)
|
||||||
|
|
||||||
|
auth/permissions.go (existing logic for permission checks)
|
||||||
|
|
||||||
|
types/types.go (for ResourceMetadata structure)
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
Reference in New Issue
Block a user