MrKalzu/ping_service
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Claude Code session 1.

Browse Source
This commit is contained in:
Kalzu Rekku
2026-01-08 12:11:26 +02:00
parent c59523060d
commit 6db2e58dcd
20 changed files with 5497 additions and 83 deletions
Download Patch File Download Diff File Expand all files Collapse all files

146
input_service/README.md
View File

@@ -1,44 +1,112 @@
# HTTP Input Service
A lightweight HTTP server that serves individual IPv4 addresses from cloud provider CIDR ranges.
A lightweight HTTP server that serves individual IPv4 addresses from cloud provider CIDR ranges and accepts discovered hop IPs from traceroute results to organically grow the target pool.
## Purpose
Provides a continuous stream of IPv4 addresses to network scanning tools. Each consumer (identified by IP) receives addresses in randomized order from cloud provider IP ranges.
Provides a continuous stream of IPv4 addresses to network scanning tools. Each consumer (identified by IP) receives addresses in highly interleaved order from cloud provider IP ranges, avoiding consecutive IPs from the same subnet. Accepts discovered hop IPs from output_service to expand the target pool.
## Requirements
- Go 1.16+
- Go 1.25+
- Cloud provider IP repository cloned at `./cloud-provider-ip-addresses/`
## Usage
```bash
# Build
go build -ldflags="-s -w" -o ip-feeder main.go
## Building
# Run
./ip-feeder
```bash
go build -ldflags="-s -w" -o http_input_service http_input_service.go
```
## Usage
```bash
./http_input_service
```
Server starts on `http://localhost:8080`
## API
## API Endpoints
**GET /**
### `GET /`
Returns a single IPv4 address per request.
```bash
curl http://localhost:8080
# Output: 13.248.118.1
```
Each consumer (identified by source IP) gets their own independent sequence with interleaved IPs from different subnets.
### `POST /hops`
Accept discovered hop IPs from traceroute results.
**Request Body:**
```json
{
"hops": ["10.0.0.1", "172.16.5.3", "8.8.8.8"]
}
```
**Response:**
```json
{
"status": "ok",
"received": 3,
"added": 2,
"duplicates": 1
}
```
- Validates and filters out private, multicast, loopback IPs
- Global deduplication prevents re-adding seen IPs
- Automatically adds new hops to all consumer pools
### `GET /status`
View current service status and consumer information.
**Response:**
```json
{
"total_consumers": 2,
"consumers": [
{
"consumer": "192.168.1.100",
"remaining_cidrs": 1234,
"has_active_gen": true,
"total_cidrs": 5000
}
],
"state_directory": "progress_state",
"save_interval": "30s"
}
```
### `GET /export`
Export all consumer states for backup/migration.
Downloads a JSON file with all consumer progress states.
### `POST /import`
Import previously exported consumer states.
**Request:** Upload JSON from `/export` endpoint
## Features
- **Subnet Interleaving** - Maintains 10 active CIDR generators, rotating between them to avoid serving consecutive IPs from the same subnet
- **Per-consumer state** - Each client gets independent, deterministic sequence
- **Deduplication** - Both per-consumer and global deduplication to prevent serving duplicate IPs
- **Hop Discovery** - Accepts discovered traceroute hops via `/hops` endpoint to grow target pool organically
- **Memory efficient** - Loads CIDR files lazily (~5-15MB RAM usage)
- **Lazy expansion** - IPs generated on-demand from CIDR notation
- **Randomized order** - Interleaves IPs from multiple ranges randomly
- **IPv4 only** - Filters IPv6, multicast, network/broadcast addresses
- **Persistent state** - Progress saved every 30s, survives restarts
- **State export/import** - Backup and migrate consumer states between instances
- **IPv4 only** - Filters IPv6, multicast, network/broadcast, private addresses
- **Graceful shutdown** - Ctrl+C drains connections cleanly
## Expected Input Format
@@ -51,6 +119,54 @@ Scans `./cloud-provider-ip-addresses/` for `.txt` files containing IP ranges:
3.5.140.0/22
```
## Shutdown
## How Interleaving Works
Press `Ctrl+C` for graceful shutdown with 10s timeout.
To avoid consecutive IPs from the same subnet (e.g., `8.8.8.1`, `8.8.8.2`, `8.8.8.3`), the service:
1. Maintains **10 active CIDR generators** concurrently
2. **Rotates** between them in round-robin fashion
3. Each request pulls from the next generator in sequence
**Example output:**
```
9.9.9.1 # From CIDR 9.9.9.0/29
208.67.222.1 # From CIDR 208.67.222.0/29
1.1.1.1 # From CIDR 1.1.1.0/29
8.8.8.1 # From CIDR 8.8.8.0/29
8.8.4.1 # From CIDR 8.8.4.0/29
9.9.9.2 # Back to first CIDR
208.67.222.2 # Second CIDR
...
```
This ensures diverse network targeting and better coverage.
## Integration with Output Service
The `/hops` endpoint is designed to receive discovered hop IPs from `output_service`:
```bash
# Example from output_service
curl -X POST http://localhost:8080/hops \
-H "Content-Type: application/json" \
-d '{"hops": ["10.0.0.1", "172.16.5.3", "8.8.8.8"]}'
```
- Output service extracts intermediate hops from traceroute results
- POSTs them to input service `/hops` endpoint
- Input service validates, deduplicates, and adds to target pool
- Future consumers will receive these discovered IPs
This creates a feedback loop where the system organically discovers new targets through network exploration.
## Graceful Shutdown
Press `Ctrl+C` for graceful shutdown with 10s timeout. All consumer states are saved before exit.
## Multi-Instance Deployment
Each instance maintains its own consumer state files in `progress_state/` directory. For load-balanced deployments:
- Use **session affinity** (stick consumers to same instance) for optimal state consistency
- Or use **shared network storage** for `progress_state/` directory
- The `/hops` endpoint should be called on **all instances** to keep target pools synchronized

211
input_service/http_input_service.go
View File

@@ -31,6 +31,7 @@ const (
cleanupInterval = 5 * time.Minute
generatorTTL = 24 * time.Hour
maxImportSize = 10 * 1024 * 1024 // 10MB
interleavedGens = 10 // Number of concurrent CIDR generators to interleave
)
// GeneratorState represents the serializable state of a generator
@@ -53,8 +54,11 @@ type IPGenerator struct {
totalCIDRsCount int
remainingCIDRs []string
currentGen *hostGenerator
activeGens []*hostGenerator // Multiple active generators for interleaving
genRotationIdx int // Current rotation index
consumer string
dirty atomic.Bool
seenIPs map[string]bool // Deduplication map
}
type hostGenerator struct {
@@ -147,8 +151,10 @@ func (hg *hostGenerator) getState() HostGenState {
func newIPGenerator(s *Server, consumer string) (*IPGenerator, error) {
gen := &IPGenerator{
rng: rand.New(rand.NewSource(time.Now().UnixNano())),
consumer: consumer,
rng: rand.New(rand.NewSource(time.Now().UnixNano())),
consumer: consumer,
seenIPs: make(map[string]bool),
activeGens: make([]*hostGenerator, 0, interleavedGens),
}
// Try to load existing state
@@ -174,37 +180,90 @@ func (g *IPGenerator) Next() (string, error) {
g.mu.Lock()
defer g.mu.Unlock()
for {
if g.currentGen == nil || g.currentGen.done {
if len(g.remainingCIDRs) == 0 {
return "", fmt.Errorf("no more IPs available")
}
// Ensure we have enough active generators for interleaving
for len(g.activeGens) < interleavedGens && len(g.remainingCIDRs) > 0 {
cidr := g.remainingCIDRs[0]
g.remainingCIDRs = g.remainingCIDRs[1:]
cidr := g.remainingCIDRs[0]
g.remainingCIDRs = g.remainingCIDRs[1:]
if !strings.Contains(cidr, "/") {
cidr += "/32"
}
var err error
g.currentGen, err = newHostGenerator(cidr)
if err != nil {
g.dirty.Store(true)
continue
}
if !strings.Contains(cidr, "/") {
cidr += "/32"
}
ip, ok := g.currentGen.next()
if !ok {
g.currentGen = nil
newGen, err := newHostGenerator(cidr)
if err != nil {
g.dirty.Store(true)
continue
}
g.activeGens = append(g.activeGens, newGen)
g.dirty.Store(true)
}
// Try to get IP from rotating generators
maxAttempts := len(g.activeGens) * 100 // Avoid infinite loop
for attempt := 0; attempt < maxAttempts || len(g.activeGens) > 0; attempt++ {
if len(g.activeGens) == 0 {
if len(g.remainingCIDRs) == 0 {
return "", fmt.Errorf("no more IPs available")
}
// Refill active generators
for len(g.activeGens) < interleavedGens && len(g.remainingCIDRs) > 0 {
cidr := g.remainingCIDRs[0]
g.remainingCIDRs = g.remainingCIDRs[1:]
if !strings.Contains(cidr, "/") {
cidr += "/32"
}
newGen, err := newHostGenerator(cidr)
if err != nil {
g.dirty.Store(true)
continue
}
g.activeGens = append(g.activeGens, newGen)
g.dirty.Store(true)
}
if len(g.activeGens) == 0 {
return "", fmt.Errorf("no more IPs available")
}
}
// Round-robin through active generators
g.genRotationIdx = g.genRotationIdx % len(g.activeGens)
gen := g.activeGens[g.genRotationIdx]
ip, ok := gen.next()
if !ok {
// Remove exhausted generator
g.activeGens = append(g.activeGens[:g.genRotationIdx], g.activeGens[g.genRotationIdx+1:]...)
g.dirty.Store(true)
if g.genRotationIdx >= len(g.activeGens) && len(g.activeGens) > 0 {
g.genRotationIdx = 0
}
continue
}
// Check deduplication
if g.seenIPs[ip] {
g.genRotationIdx = (g.genRotationIdx + 1) % max(len(g.activeGens), 1)
continue
}
g.seenIPs[ip] = true
g.genRotationIdx = (g.genRotationIdx + 1) % max(len(g.activeGens), 1)
g.dirty.Store(true)
return ip, nil
}
return "", fmt.Errorf("no more unique IPs available")
}
func max(a, b int) int {
if a > b {
return a
}
return b
}
func (g *IPGenerator) buildState() GeneratorState {
@@ -314,6 +373,7 @@ type Server struct {
generators map[string]*IPGenerator
lastAccess map[string]time.Time
allCIDRs []string
globalSeen map[string]bool // Global deduplication across all sources
mu sync.RWMutex
stopSaver chan struct{}
stopCleanup chan struct{}
@@ -324,6 +384,7 @@ func newServer() *Server {
s := &Server{
generators: make(map[string]*IPGenerator),
lastAccess: make(map[string]time.Time),
globalSeen: make(map[string]bool),
stopSaver: make(chan struct{}),
stopCleanup: make(chan struct{}),
}
@@ -698,6 +759,106 @@ func (s *Server) handleImport(w http.ResponseWriter, r *http.Request) {
log.Printf("📥 Imported %d consumer states (%d failed)", imported, failed)
}
// HopsRequest is the payload from output_service
type HopsRequest struct {
Hops []string `json:"hops"`
}
func (s *Server) handleHops(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
return
}
defer r.Body.Close()
var req HopsRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
http.Error(w, "Invalid JSON", http.StatusBadRequest)
return
}
s.mu.Lock()
defer s.mu.Unlock()
added := 0
duplicates := 0
for _, hop := range req.Hops {
// Validate IP
addr, err := netip.ParseAddr(hop)
if err != nil {
log.Printf("⚠️ Invalid hop IP: %s", hop)
continue
}
// Skip if not IPv4
if !addr.Is4() {
continue
}
// Skip multicast, private, loopback
if addr.IsMulticast() || addr.IsLoopback() || addr.IsPrivate() {
continue
}
// Check global deduplication
if s.globalSeen[hop] {
duplicates++
continue
}
// Add to global pool
s.globalSeen[hop] = true
s.allCIDRs = append(s.allCIDRs, hop)
added++
}
log.Printf("🔍 Received %d hops: %d new, %d duplicates", len(req.Hops), added, duplicates)
response := map[string]interface{}{
"status": "ok",
"received": len(req.Hops),
"added": added,
"duplicates": duplicates,
}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(response)
}
// ServiceInfo represents service metadata for discovery
type ServiceInfo struct {
ServiceType string `json:"service_type"`
Version string `json:"version"`
Name string `json:"name"`
InstanceID string `json:"instance_id"`
Capabilities []string `json:"capabilities"`
}
func (s *Server) handleServiceInfo(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodGet {
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
return
}
hostname, _ := os.Hostname()
if hostname == "" {
hostname = "unknown"
}
info := ServiceInfo{
ServiceType: "input",
Version: "1.0.0",
Name: "http_input_service",
InstanceID: hostname,
Capabilities: []string{"target_generation", "cidr_import", "hop_discovery"},
}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(info)
}
func main() {
// Check if repo directory exists
if _, err := os.Stat(repoDir); os.IsNotExist(err) {
@@ -709,8 +870,10 @@ func main() {
mux := http.NewServeMux()
mux.HandleFunc("/", server.handleRequest)
mux.HandleFunc("/status", server.handleStatus)
mux.HandleFunc("/service-info", server.handleServiceInfo)
mux.HandleFunc("/export", server.handleExport)
mux.HandleFunc("/import", server.handleImport)
mux.HandleFunc("/hops", server.handleHops)
httpServer := &http.Server{
Addr: fmt.Sprintf(":%d", port),
@@ -742,10 +905,12 @@ func main() {
log.Printf("🌐 HTTP Input Server running on http://localhost:%d", port)
log.Printf(" Serving individual IPv4 host addresses lazily")
log.Printf(" In highly mixed random order per consumer")
log.Printf(" 🔄 Interleaving %d CIDRs to avoid same-subnet consecutive IPs", interleavedGens)
log.Printf(" 💾 Progress saved every %v to '%s' directory", saveInterval, stateDir)
log.Printf(" 📊 Status endpoint: http://localhost:%d/status", port)
log.Printf(" 📤 Export endpoint: http://localhost:%d/export", port)
log.Printf(" 📥 Import endpoint: http://localhost:%d/import (POST)", port)
log.Printf(" 🔍 Hops endpoint: http://localhost:%d/hops (POST)", port)
log.Printf(" Press Ctrl+C to stop")
if err := httpServer.ListenAndServe(); err != nil && err != http.ErrServerClosed {