BytbitBTC/input/README.md

WebSocket Streamer (JSONL Logger)

This Go program connects to a WebSocket endpoint, subscribes to a topic, and continuously writes incoming messages to hourly-rotated .jsonl files. It is designed for long-running, low-overhead data capture with basic observability and operational safety.

Typical use cases include:

• Market data capture (e.g. crypto trades)
• Event stream archiving
• Lightweight ingestion on small servers (VPS, Raspberry Pi, etc.)

---

Features

• WebSocket client with automatic reconnect
• Topic subscription (configurable)
• Hourly file rotation (.jsonl format)
• Buffered channel to decouple network I/O from disk writes
• Atomic message counters
• Periodic status logging
• Unix domain socket for live status queries
• Graceful shutdown on SIGINT / SIGTERM
• Configurable logging (file and/or stdout)

---

How It Works

1. Connects to a WebSocket endpoint

2. Sends a subscription message for the configured topic

3. Maintains connection with periodic ping

4. Reads messages and pushes them into a buffered channel

5. Writes messages line-by-line to hourly JSONL files

6. Exposes runtime status via:

   • Periodic log output
   • Unix socket query (--status mode)

---

Output Format

Messages are written verbatim as received, one JSON object per line:

output/
├── publicTrade.BTCUSDT_1700000000.jsonl
├── publicTrade.BTCUSDT_1700003600.jsonl
└── ...

Each file contains data for exactly one UTC hour.

---

Configuration

The application is configured via a JSON file.

Example config.json

{
  "output_dir": "./output",
  "topic": "publicTrade.BTCUSDT",
  "ws_url": "wss://stream.bybit.com/v5/public/linear",
  "buffer_size": 10000,
  "status_interval": 30,
  "log_file": "system.log",
  "log_to_stdout": false,
  "status_socket": "/tmp/streamer.sock"
}

Configuration Fields

Field	Description
output_dir	Directory for JSONL output files
topic	WebSocket subscription topic
ws_url	WebSocket endpoint URL
buffer_size	Size of internal message buffer
status_interval	Seconds between status log messages
log_file	Log file path
log_to_stdout	Also log to stdout
status_socket	Unix socket path for status queries

Defaults are applied automatically if fields are omitted.

---

Command Line Flags

Flag	Description
-config	Path to config file (default: config.json)
-debug	Force logs to stdout (overrides config)
-status	Query running instance status and exit

---

Running the Streamer

go run main.go -config config.json

Or build a binary:

go build -o streamer
./streamer -config config.json

---

Querying Runtime Status

While the streamer is running:

./streamer -status -config config.json

Example output:

Uptime: 12m34s | Total Msgs: 152340 | Rate: 7260.12 msg/min

This works via a Unix domain socket and does not interrupt the running process.

---

Logging

• Logs are written to log_file

• Optional stdout logging for debugging

• Includes:

  • Startup information
  • Connection errors and reconnects
  • Buffer overflow warnings
  • Periodic status summaries

---

Graceful Shutdown

On SIGINT or SIGTERM:

• WebSocket connection closes
• Status socket is removed
• Current output file is flushed and closed

Safe to run under systemd, Docker, or supervisord.

---

Dependencies

• Go 1.20+
• github.com/gorilla/websocket

---

Notes & Design Choices

• JSONL is used for easy streaming, compression, and downstream processing
• Hourly rotation avoids large files and simplifies retention policies
• Unix socket status avoids HTTP overhead and exposed ports
• Minimal memory footprint, suitable for low-end machines