forked from ryyst/kalzu-value-store
		
	Small write up about the two issues conserning missing api endpoints.
This commit is contained in:
		
							
								
								
									
										120
									
								
								issues/7and12.md
									
									
									
									
									
										Normal file
									
								
							
							
						
						
									
										120
									
								
								issues/7and12.md
									
									
									
									
									
										Normal file
									
								
							@@ -0,0 +1,120 @@
 | 
			
		||||
#7 Add _ls and _tree Endpoints for Hierarchical Key Listing Using Merkle Tree
 | 
			
		||||
-----------------------------------------
 | 
			
		||||
 | 
			
		||||
KVS supports hierarchical keys (e.g., /home/room/closet/socks), which is great for organizing data like a file system. However, there's currently no built-in way for clients to discover or list subkeys under a given prefix/path. This makes it hard to build intuitive tools or UIs that need to navigate the keyspace, such as a web-based explorer or CLI client.
 | 
			
		||||
 | 
			
		||||
Add two new read-only endpoints that leverage the existing Merkle tree infrastructure for efficient prefix-based key listing. This aligns with KVS's modular design, eventual consistency model, and Merkle-based sync (no need for full DB scans—traverse the tree to identify relevant leaf nodes in O(log N) time).
 | 
			
		||||
Proposed Endpoints
 | 
			
		||||
 | 
			
		||||
    Direct Children Listing (_ls or _list):
 | 
			
		||||
        Endpoint: GET /kv/{path}/_ls (or GET /kv/{path}/_list for clarity).
 | 
			
		||||
        Purpose: Returns a sorted list of direct subkeys under the given path/prefix (non-recursive).
 | 
			
		||||
        Query Params (optional):
 | 
			
		||||
            limit: Max number of keys to return (default: 100, max: 1000).
 | 
			
		||||
            include_metadata: If true, include basic metadata like timestamps (default: false).
 | 
			
		||||
        Response (JSON):
 | 
			
		||||
 | 
			
		||||
        {
 | 
			
		||||
          "path": "/home/room",
 | 
			
		||||
          "children": [
 | 
			
		||||
            { "subkey": "closet", "timestamp": 1695280000000 },
 | 
			
		||||
            { "subkey": "bed", "timestamp": 1695279000000 }
 | 
			
		||||
          ],
 | 
			
		||||
          "total": 2,
 | 
			
		||||
          "truncated": false
 | 
			
		||||
        }
 | 
			
		||||
 | 
			
		||||
    Behavior:
 | 
			
		||||
        Treat {path} as a prefix (e.g., /home/room/ → keys starting with /home/room/ but not /home/room/sub/).
 | 
			
		||||
        Use the Merkle tree to find leaf nodes in the prefix range [prefix, prefix~] (where ~ is the next lexicographical prefix).
 | 
			
		||||
        Skip index keys (e.g., _ts:*).
 | 
			
		||||
        Respect auth: Use existing middleware (e.g., read scope if auth_enabled: true).
 | 
			
		||||
        In read-only/syncing modes: Allow if not modifying data.
 | 
			
		||||
 | 
			
		||||
Recursive Tree View (_tree):
 | 
			
		||||
 | 
			
		||||
    Endpoint: GET /kv/{path}/_tree.
 | 
			
		||||
    Purpose: Returns a recursive tree structure of all subkeys under the given path (depth-first or breadth-first, configurable).
 | 
			
		||||
    Query Params (optional):
 | 
			
		||||
        depth: Max recursion depth (default: unlimited, but suggest 5 for safety).
 | 
			
		||||
        limit: Max total keys (default: 500, max: 5000).
 | 
			
		||||
        include_metadata: Include timestamps/UUIDs (default: false).
 | 
			
		||||
        format: json (default) or nested (tree-like JSON).
 | 
			
		||||
    Response (JSON, nested format):
 | 
			
		||||
 | 
			
		||||
    {
 | 
			
		||||
      "path": "/home/room",
 | 
			
		||||
      "children": [
 | 
			
		||||
        {
 | 
			
		||||
          "subkey": "closet",
 | 
			
		||||
          "children": [
 | 
			
		||||
            { "subkey": "socks", "timestamp": 1695281000000 }
 | 
			
		||||
          ],
 | 
			
		||||
          "timestamp": 1695280000000
 | 
			
		||||
        },
 | 
			
		||||
        {
 | 
			
		||||
          "subkey": "bed",
 | 
			
		||||
          "timestamp": 1695279000000
 | 
			
		||||
        }
 | 
			
		||||
      ],
 | 
			
		||||
      "total": 3,
 | 
			
		||||
      "truncated": false
 | 
			
		||||
    }
 | 
			
		||||
 | 
			
		||||
        Behavior:
 | 
			
		||||
            Build on _ls logic: Recursively query sub-prefixes via Merkle tree traversal.
 | 
			
		||||
            Prune at depth or limit to avoid overload.
 | 
			
		||||
            Same auth and mode rules as _ls.
 | 
			
		||||
 | 
			
		||||
Integration with Existing Systems
 | 
			
		||||
 | 
			
		||||
    Merkle Tree Usage: Extend cluster/merkle.go (e.g., add GetKeysInRange(startKey, endKey) []string method) to traverse nodes covering the prefix range without fetching full values. Reuse buildMerkleTreeFromPairs and filterPairsByRange from handlers.go.
 | 
			
		||||
    Range Query Reuse: Build on existing KVRangeRequest/KVRangeResponse in types.go and getKVRangeHandler (strip values to return just keys for efficiency).
 | 
			
		||||
    Auth & Permissions: Apply via authService.Middleware (e.g., read scope). Respect allow_anonymous_read.
 | 
			
		||||
    Config Toggle: Add key_listing_enabled: true to types.Config for optional disable (e.g., for security in public clusters).
 | 
			
		||||
    Distributed Consistency: Since Merkle trees are synced, listings will be eventually consistent across nodes. Add a consistent: true query param to force a quick Merkle refresh if needed.
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
#12 Missing API Endpoints for Resource Metadata Management (Ownership & Permissions)
 | 
			
		||||
-----------------------------------------
 | 
			
		||||
 | 
			
		||||
The KVS system currently lacks API endpoints to manage ResourceMetadata for key-value paths (/kv/{path}). While the AuthService and permissions.go implement robust permission checking based on OwnerUUID, GroupUUID, and Permissions, there are no exposed routes to:
 | 
			
		||||
 | 
			
		||||
Assign group-level permissions: Users cannot grant read/write access to specific groups for a given key-value path.
 | 
			
		||||
 | 
			
		||||
Change resource ownership: Users cannot transfer ownership of a key-value entry to another user.
 | 
			
		||||
 | 
			
		||||
This prevents administrators from fully leveraging the existing authentication and authorization framework for fine-grained access control over stored data.
 | 
			
		||||
 | 
			
		||||
Impact:
 | 
			
		||||
 | 
			
		||||
Limited administrative control over data access.
 | 
			
		||||
 | 
			
		||||
Inability to implement granular, group-based access policies for KV data.
 | 
			
		||||
 | 
			
		||||
Difficulty in reassigning data ownership when users or roles change.
 | 
			
		||||
 | 
			
		||||
Proposed Solution:
 | 
			
		||||
Implement new API endpoints (e.g., /kv/{path}/metadata) to allow authenticated and authorized users to:
 | 
			
		||||
 | 
			
		||||
Set/update the OwnerUUID for a given path.
 | 
			
		||||
 | 
			
		||||
Set/update the GroupUUID for a given path.
 | 
			
		||||
 | 
			
		||||
Set/update the Permissions bitmask for a given path.
 | 
			
		||||
 | 
			
		||||
Relevant Files:
 | 
			
		||||
 | 
			
		||||
server/routes.go (for new API routes)
 | 
			
		||||
 | 
			
		||||
server/handlers.go (for implementing new handlers)
 | 
			
		||||
 | 
			
		||||
auth/auth.go (for AuthService methods to interact with ResourceMetadata)
 | 
			
		||||
 | 
			
		||||
auth/permissions.go (existing logic for permission checks)
 | 
			
		||||
 | 
			
		||||
types/types.go (for ResourceMetadata structure)
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
		Reference in New Issue
	
	Block a user