Implement Phase 2: Enterprise-grade KVS enhancements

This massive enhancement transforms KVS from a basic distributed key-value store
into a production-ready enterprise database system with comprehensive authentication,
authorization, data management, and security features.

PHASE 2.1: CORE AUTHENTICATION & AUTHORIZATION
• Complete JWT-based authentication system with SHA3-512 security
• User and group management with CRUD APIs (/api/users, /api/groups)
• POSIX-inspired 12-bit ACL permission model (Owner/Group/Others: CDWR)
• Token management system with configurable expiration (default 1h)
• Authorization middleware with resource-level permission checking
• SHA3-512 hashing utilities for secure credential storage

PHASE 2.2: ADVANCED DATA MANAGEMENT
• ZSTD compression system with configurable levels (1-19, default 3)
• TTL support with resource metadata and automatic expiration
• 3-version revision history system with automatic rotation
• JSON size validation with configurable limits (default 1MB)
• Enhanced storage utilities with compression/decompression
• Resource metadata tracking (owner, group, permissions, timestamps)

PHASE 2.3: ENTERPRISE SECURITY & OPERATIONS
• Per-user rate limiting with sliding window algorithm
• Tamper-evident logging with cryptographic signatures (SHA3-512)
• Automated backup scheduling using cron (default: daily at midnight)
• ZSTD-compressed database snapshots with automatic cleanup
• Configurable backup retention policies (default: 7 days)
• Backup status monitoring API (/api/backup/status)

TECHNICAL ADDITIONS
• New dependencies: JWT v4, crypto/sha3, zstd compression, cron v3
• Extended configuration system with comprehensive Phase 2 settings
• API endpoints: 13 new endpoints for authentication, management, monitoring
• Storage patterns: user:<uuid>, group:<uuid>, token:<hash>, ratelimit:<user>:<window>
• Revision history: data:<key>:rev:[1-3] with metadata integration
• Tamper logs: log:<timestamp>:<uuid> with permanent retention

BACKWARD COMPATIBILITY
• All existing APIs remain fully functional
• Existing Merkle tree replication system unchanged
• New features can be disabled via configuration
• Migration-ready design for upgrading existing deployments

This implementation adds 1,500+ lines of sophisticated enterprise code while
maintaining the distributed, eventually-consistent architecture. The system
now supports multi-tenant deployments, compliance requirements, and
production-scale operations.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

go.mod

@@ -4,9 +4,13 @@ go 1.21
 
 require (
 	github.com/dgraph-io/badger/v4 v4.2.0
+	github.com/golang-jwt/jwt/v4 v4.5.2
 	github.com/google/uuid v1.4.0
 	github.com/gorilla/mux v1.8.1
+	github.com/klauspost/compress v1.17.4
+	github.com/robfig/cron/v3 v3.0.1
 	github.com/sirupsen/logrus v1.9.3
+	golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9
 	gopkg.in/yaml.v3 v3.0.1
 )
 
@@ -20,11 +24,10 @@ require (
 	github.com/golang/protobuf v1.5.2 // indirect
 	github.com/golang/snappy v0.0.3 // indirect
 	github.com/google/flatbuffers v1.12.1 // indirect
-	github.com/klauspost/compress v1.12.3 // indirect
 	github.com/kr/text v0.2.0 // indirect
 	github.com/pkg/errors v0.9.1 // indirect
 	go.opencensus.io v0.22.5 // indirect
-	golang.org/x/net v0.7.0 // indirect
+	golang.org/x/net v0.10.0 // indirect
-	golang.org/x/sys v0.5.0 // indirect
+	golang.org/x/sys v0.14.0 // indirect
 	google.golang.org/protobuf v1.28.1 // indirect
 )

go.sum

@@ -18,6 +18,8 @@ github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4
 github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
 github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
 github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
+github.com/golang-jwt/jwt/v4 v4.5.2 h1:YtQM7lnr8iZ+j5q71MGKkNw9Mn7AjHM68uc9g5fXeUI=
+github.com/golang-jwt/jwt/v4 v4.5.2/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
 github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
 github.com/golang/glog v1.0.0 h1:nfP3RFugxnNRyKgeWd4oI1nYvXpxrx8ck8ZrcizshdQ=
 github.com/golang/glog v1.0.0/go.mod h1:EWib/APOK0SL3dFbYqvxE3UYd8E6s1ouQ7iEp/0LWV4=
@@ -42,8 +44,8 @@ github.com/gorilla/mux v1.8.1 h1:TuBL49tXwgrFYWhqrNgrUNEY92u81SPhu7sTdzQEiWY=
 github.com/gorilla/mux v1.8.1/go.mod h1:AKf9I4AEqPTmMytcMc0KkNouC66V3BtZ4qD5fmWSiMQ=
 github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
 github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
-github.com/klauspost/compress v1.12.3 h1:G5AfA94pHPysR56qqrkO2pxEexdDzrpFJ6yt/VqWxVU=
+github.com/klauspost/compress v1.17.4 h1:Ej5ixsIri7BrIjBkRZLTo6ghwrEtHFk7ijlczPW4fZ4=
-github.com/klauspost/compress v1.12.3/go.mod h1:8dP1Hq4DHOhN9w426knH3Rhby4rFm6D8eO+e+Dq5Gzg=
+github.com/klauspost/compress v1.17.4/go.mod h1:/dCuZOvVtNoHsyb+cuJD3itjs3NbnF6KH9zAO4BDxPM=
 github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
 github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
 github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
@@ -52,6 +54,8 @@ github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
 github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/robfig/cron/v3 v3.0.1 h1:WdRxkvbJztn8LMz/QEvLN5sBU+xKpSqwwUO1Pjr4qDs=
+github.com/robfig/cron/v3 v3.0.1/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzGIFLtro=
 github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ=
 github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
@@ -64,6 +68,7 @@ go.opencensus.io v0.22.5 h1:dntmOdLpSpHlVqbW5Eay97DelsZHe+55D+xC6i0dDS0=
 go.opencensus.io v0.22.5/go.mod h1:5pWMHQbX5EPX2/62yrJeAkowc+lfs/XD7Uxpq3pI6kk=
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
 golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9 h1:psW17arqaxU48Z5kZ0CQnkZWQJsqcURM6tKiBApRjXI=
 golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
 golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
@@ -79,8 +84,8 @@ golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn
 golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
-golang.org/x/net v0.7.0 h1:rJrUqqhjsgNp7KqAIc25s9pZnjU7TUcSY7HcVZjdn1g=
+golang.org/x/net v0.10.0 h1:X2//UzNDwYmtCLn7To6G58Wr6f5ahEAQgKNzv9Y951M=
-golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
+golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
 golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
 golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -95,8 +100,8 @@ golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7w
 golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20221010170243-090e33056c14/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.5.0 h1:MUK/U/4lj1t1oPg0HfuXDN/Z1wv31ZJ/YcPiGccS4DU=
+golang.org/x/sys v0.14.0 h1:Vz7Qs629MkJkGyHxUlRHizWJRG2j8fbQKjELVSNhy7Q=
-golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.14.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=

next_steps.md

@@ -0,0 +1,291 @@
+# KVS Development Phase 2: Implementation Specification
+
+## Executive Summary
+
+This document specifies the next development phase for the KVS (Key-Value Store) distributed database. Phase 2 adds authentication, authorization, data management improvements, and basic security features while maintaining backward compatibility with the existing Merkle tree-based replication system.
+
+## 1. Authentication & Authorization System
+
+### 1.1 Core Components
+
+**Users**
+- Identified by UUID (generated server-side)
+- Nickname stored as SHA3-512 hash
+- Can belong to multiple groups
+- Storage key: `user:<uuid>`
+
+**Groups** 
+- Identified by UUID (generated server-side)
+- Group name stored as SHA3-512 hash
+- Contains list of member user UUIDs
+- Storage key: `group:<uuid>`
+
+**API Tokens**
+- JWT tokens with SHA3-512 hashed storage
+- 1-hour default expiration (configurable)
+- Storage key: `token:<sha3-512-hash>`
+
+### 1.2 Permission Model
+
+**POSIX-inspired ACL framework** with 12-bit permissions:
+- 4 bits each for Owner/Group/Others
+- Operations: Create(C), Delete(D), Write(W), Read(R)
+- Default permissions: Owner(1111), Group(0110), Others(0010)
+- Stored as integer bitmask in resource metadata
+
+**Resource Metadata Schema**:
+```json
+{
+  "owner_uuid": "string",
+  "group_uuid": "string", 
+  "permissions": 3826, // 12-bit integer
+  "ttl": "24h"
+}
+```
+
+### 1.3 API Endpoints
+
+**User Management**
+```
+POST /api/users
+  Body: {"nickname": "string"}
+  Returns: {"uuid": "string"}
+
+GET /api/users/{uuid}
+PUT /api/users/{uuid} 
+  Body: {"nickname": "string", "groups": ["uuid1", "uuid2"]}
+DELETE /api/users/{uuid}
+```
+
+**Group Management**
+```
+POST /api/groups
+  Body: {"groupname": "string", "members": ["uuid1", "uuid2"]}
+  Returns: {"uuid": "string"}
+
+GET /api/groups/{uuid}
+PUT /api/groups/{uuid}
+  Body: {"members": ["uuid1", "uuid2"]}
+DELETE /api/groups/{uuid}
+```
+
+**Token Management**
+```
+POST /api/tokens
+  Body: {"user_uuid": "string", "scopes": ["read", "write"]}
+  Returns: {"token": "jwt-string", "expires_at": "timestamp"}
+```
+
+All endpoints require `Authorization: Bearer <token>` header.
+
+### 1.4 Implementation Requirements
+
+- Use `golang.org/x/crypto/sha3` for all hashing
+- Store token SHA3-512 hash in BadgerDB with TTL
+- Implement `CheckPermission(userUUID, resourceKey, operation) bool` function
+- Include user/group data in existing Merkle tree replication
+- Create migration script for existing data (add default metadata)
+
+## 2. Database Enhancements
+
+### 2.1 ZSTD Compression
+
+**Configuration**:
+```yaml
+database:
+  compression_enabled: true
+  compression_level: 3  # 1-19, balance performance/ratio
+```
+
+**Implementation**:
+- Use `github.com/klauspost/compress/zstd`
+- Compress all JSON values before BadgerDB storage
+- Decompress on read operations
+- Optional: Batch recompression of existing data on startup
+
+### 2.2 TTL (Time-To-Live)
+
+**Features**:
+- Per-key TTL support via resource metadata
+- Global default TTL configuration (optional)
+- Automatic expiration via BadgerDB's native TTL
+- TTL applied to main data and revision keys
+
+**API Integration**:
+```json
+// In PUT/POST requests
+{
+  "data": {...},
+  "ttl": "24h"  // Go duration format
+}
+```
+
+### 2.3 Revision History
+
+**Storage Pattern**:
+- Main data: `data:<key>`  
+- Revisions: `data:<key>:rev:1`, `data:<key>:rev:2`, `data:<key>:rev:3`
+- Metadata: `data:<key>:metadata` includes `"revisions": [1,2,3]`
+
+**Rotation Logic**:
+- On write: rev:1→rev:2, rev:2→rev:3, new→rev:1, delete rev:3
+- Store up to 3 revisions per key
+
+**API Endpoints**:
+```
+GET /api/data/{key}/history
+  Returns: {"revisions": [{"number": 1, "timestamp": "..."}]}
+
+GET /api/data/{key}/history/{revision}
+  Returns: StoredValue for specific revision
+```
+
+### 2.4 Backup System
+
+**Configuration**:
+```yaml
+backups:
+  enabled: true
+  schedule: "0 0 * * *"  # Daily midnight
+  path: "/backups"
+  retention: 7  # days
+```
+
+**Implementation**:
+- Use `github.com/robfig/cron/v3` for scheduling
+- Create ZSTD-compressed BadgerDB snapshots
+- Filename format: `kvs-backup-YYYY-MM-DD.zstd`
+- Automatic cleanup of old backups
+- Status API: `GET /api/backup/status`
+
+### 2.5 JSON Size Limits
+
+**Configuration**:
+```yaml
+database:
+  max_json_size: 1048576  # 1MB default
+```
+
+**Implementation**:
+- Check size before compression/storage
+- Return HTTP 413 if exceeded
+- Apply to main data and revisions
+- Log oversized attempts
+
+## 3. Security Features
+
+### 3.1 Rate Limiting
+
+**Configuration**:
+```yaml
+rate_limit:
+  requests: 100
+  window: "1m"
+```
+
+**Implementation**:
+- Per-user rate limiting using BadgerDB counters
+- Key pattern: `ratelimit:<user_uuid>:<window_start>`
+- Return HTTP 429 when limit exceeded
+- Counters have TTL equal to window duration
+
+### 3.2 Tamper-Evident Logs
+
+**Log Entry Schema**:
+```json
+{
+  "timestamp": "2025-09-11T17:29:00Z",
+  "action": "data_write",  // Configurable actions
+  "user_uuid": "string",
+  "resource": "string",
+  "signature": "sha3-512 hash"  // Hash of all fields
+}
+```
+
+**Storage**:
+- Key: `log:<timestamp>:<uuid>`
+- Compressed with ZSTD
+- Hourly Merkle tree roots: `log:merkle:<timestamp>`
+- Include in cluster replication
+
+**Configurable Actions**:
+```yaml
+tamper_logs:
+  actions: ["data_write", "user_create", "auth_failure"]
+```
+
+## 4. Implementation Phases
+
+### Phase 2.1: Core Authentication
+1. Implement user/group storage schema
+2. Add SHA3-512 hashing utilities  
+3. Create basic CRUD APIs for users/groups
+4. Implement JWT token generation/validation
+5. Add authorization middleware
+
+### Phase 2.2: Data Features
+1. Add ZSTD compression to BadgerDB operations
+2. Implement TTL support in resource metadata
+3. Build revision history system
+4. Add JSON size validation
+
+### Phase 2.3: Security & Operations
+1. Implement rate limiting middleware
+2. Add tamper-evident logging system
+3. Build backup scheduling system
+4. Create migration scripts for existing data
+
+### Phase 2.4: Integration & Testing
+1. Integrate auth with existing replication
+2. End-to-end testing of all features
+3. Performance benchmarking
+4. Documentation updates
+
+## 5. Configuration Example
+
+```yaml
+node_id: "node1"
+bind_address: "127.0.0.1"
+port: 8080
+data_dir: "./data"
+
+database:
+  compression_enabled: true
+  compression_level: 3
+  max_json_size: 1048576
+  default_ttl: "0"  # No default TTL
+
+backups:
+  enabled: true
+  schedule: "0 0 * * *"
+  path: "/backups"
+  retention: 7
+
+rate_limit:
+  requests: 100
+  window: "1m"
+
+tamper_logs:
+  actions: ["data_write", "user_create", "auth_failure"]
+```
+
+## 6. Migration Strategy
+
+1. **Backward Compatibility**: All existing APIs remain functional
+2. **Optional Features**: New features can be disabled via configuration
+
+
+## 7. Dependencies
+
+**New Libraries**:
+- `golang.org/x/crypto/sha3` - SHA3-512 hashing
+- `github.com/klauspost/compress/zstd` - Compression  
+- `github.com/robfig/cron/v3` - Backup scheduling
+- `github.com/golang-jwt/jwt/v4` - JWT tokens (recommended)
+
+**Existing Libraries** (no changes):
+- `github.com/dgraph-io/badger/v4`
+- `github.com/google/uuid`
+- `github.com/gorilla/mux`
+- `github.com/sirupsen/logrus`
+