MrKalzu/bybit_btcusdt_ingest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Added klines, liquidations, tickers and trades to be recorded. Bundled the stats as cli argument.

Browse Source
This commit is contained in:
Kalzu Rekku
2026-07-22 15:35:33 +03:00
parent 270922fb78
commit f6765fdf07
15 changed files with 2719 additions and 495 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+115 -155
View File
@@ -1,202 +1,162 @@
# Bybit BTC/USDT Tick Ingest Engine
# Bybit Multi-Stream Market Data Ingest Engine
A high-performance, self-cleaning market data ingestion service that captures real-time BTC/USDT trades from the Bybit V5 WebSocket API, stores raw ticks in a rolling SQLite hot database, aggregates 5-second feature bars for machine learning, and automatically archives historical data into weekly partitions.
A high-performance, self-cleaning market data ingestion service that captures real-time BTC/USDT streams from the Bybit V5 WebSocket API, stores hot raw data in per-stream SQLite databases, aggregates 5-second feature bars for machine learning, and automatically archives historical data into weekly partitions.
## Architecture
```
┌─────────────────────────┐
│ Bybit WebSocket │
│ publicTrade.BTCUSDT │
└────────────┬────────────┘
│ JSON Tick Stream
▼
┌─────────────────────────┐
│ Ingestor Goroutine │
│ (parse, fan-out) │
└──────┬────────────┬─────┘
│ │
Buffered Channel │ │ Inline 5s Aggregation
▼ ▼
┌────────────────────────────────────┐ ┌────────────────────────────────────┐
│ Writer Goroutine │ │ features.db │
│ (batch flush to hot DB) │ │ - Permanent 5s feature bars │
│ │ │ - 30-day rolling retention │
│ ▼ │ │ - ~300 MB/year │
│ hot_ticks.db │ └────────────────────────────────────┘
│ - Rolling 12h of raw ticks │
│ - WAL mode, high-throughput │
└──────────────────┬─────────────────┘
│
│ Hourly Maintenance Goroutine
▼
┌────────────────────────────────────┐
│ Weekly Archive DBs │
│ - data/archive/btc_ticks_YYYY_Www.db │
│ - Atomic ATTACH/DETACH migration │
│ - ISO week partitioning │
└────────────────────────────────────┘
┌─────────────────────────────────┐
│ Bybit WebSocket V5 │
│ (publicTrade, tickers, kline, │
│ orderbook.50, allLiquidation) │
└────────────────┬────────────────┘
│ Dynamic Multi-Stream JSON
▼
┌─────────────────────────────────┐
│ Ingestor Goroutine │
│ (topic-prefix JSON router) │
└────┬───┬────────┬────┬──────┬───┘
│ │ │ │ │
┌───────────────────────┘ │ │ │ └──────────────────────┐
▼ ▼ ▼ ▼ ▼
┌─────────────┐ ┌────────┐ ┌──────┐ ┌──────────┐ ┌──────────────┐
│ Trades │ │ Ticker │ │Klines│ │Orderbook │ │ Liquidations │
│ Handler │ │Handler │ │Handlr│ │ Handler │ │ Handler │
└──────┬──────┘ └───┬────┘ └──┬───┘ └────┬─────┘ └──────┬───────┘
│ │ │ │ │
data/trades/ data/ticker/ data/klines/ data/orderbook/ data/liquidations/
├── hot_ticks.db ├── hot_ticker.db ├── kline_5.db ├── hot_snapshots.db ├── hot_liquidations.db
├── features.db ├── features.db ├── kline_15.db └── features.db ├── features.db
└── archive/ └── archive/ ├── kline_60.db └── archive/
└── ticks_...db ├── features.db └── liquidations_...db
└── archive/
```
The system runs four concurrent goroutines:
## Features & Supported Streams
| Goroutine | Responsibility |
|-----------|---------------|
| **Ingestor** | Connects to Bybit WebSocket, parses trades, feeds aggregator and writer channel |
| **Writer** | Drains tick channel, batch-flushes to `hot_ticks.db` every 500ms or 100 ticks |
| **Aggregator** | Buckets ticks into 5-second windows, computes features, writes to `features.db` |
| **Maintenance** | Hourly migration of old ticks to weekly archives, feature pruning |
| Stream | Bybit Topic | Hot Retention | Features DB (5s resolution) | Archive Policy |
|--------|-------------|---------------|-----------------------------|----------------|
| **Trades** | `publicTrade.BTCUSDT` | 12h | Log return, realized vol, OFI, VWAP | Weekly rotated (`ticks_YYYY_Www.db`) |
| **Ticker** | `tickers.BTCUSDT` | 24h (snapshots @ 5s) | Spread, spread bps, mid-price, OI delta, funding rate, mark-index basis, bid/ask size imbalance | None |
| **Klines** | `kline.5/15/60.BTCUSDT` | 24h (5m, 15m), Weekly (60m) | Body ratio, upper/lower wicks, log return, volume/turnover | Weekly rotated (`kline_60_YYYY_Www.db`) |
| **Orderbook** | `orderbook.50.BTCUSDT` | 6h (snapshots @ 5s) | Top 5/20 depth imbalance, spread, weighted mid-price, top-10 VWAP | None |
| **Liquidations** | `allLiquidation.BTCUSDT` | 12h | Event counts, volume, USD value per side, net value, average price | Weekly rotated (`liquidations_YYYY_Www.db`) |
## Features
## Key Capabilities
- **Real-time ingestion** from Bybit V5 public linear WebSocket with automatic reconnection
- **Decoupled write pipeline** — buffered channel isolates network I/O from disk I/O
- **5-second feature bars** with log return, realized volatility, order flow imbalance (OFI), volume sum, and close price
- **Three-tier storage** — hot DB (12h), weekly archives, and permanent feature DB
- **Automatic maintenance** — hourly tick migration, feature pruning, incremental vacuum
- **Startup recovery** — detects and migrates stale ticks from previous runs
- **Graceful shutdown** — SIGINT/SIGTERM drains remaining data before exit
- **Real-time multi-stream ingestion** from Bybit V5 public linear WebSocket over a single connection with 20s ping heartbeats
- **Decoupled write pipeline** — buffered channels isolate network I/O from disk I/O
- **Per-stream directory isolation** (`data/trades/`, `data/ticker/`, `data/klines/`, `data/orderbook/`, `data/liquidations/`)
- **Automated backwards-compatible data migration** on first start
- **Automatic maintenance** — stream-specific retention policies, weekly partition migration, incremental vacuuming
- **Startup recovery** — migrates stale ticks left over from previous runs
- **Graceful shutdown** — SIGINT/SIGTERM cleanly flushes pending buckets and closes connections
- **Pure Go** — no CGO required (`modernc.org/sqlite`)
## Requirements
- Go 1.25.0+
- Linux amd64 (tested on Debian)
- Internet access to `stream.bybit.com`
## Quick Start
```bash
# Clone and build
git clone <repo-url>
cd bybit_btcusdt_ingest
# Build
make build
# Run (generates default config.json on first start)
# Run (generates multi-stream config.json on first start)
make run
# Run manual maintenance cycle
./bybit_btcusdt_ingest maintain
# Stop gracefully with Ctrl+C
```
## Configuration
A `config.json` file is auto-generated on first run with these defaults:
## Configuration (`config.json`)
```json
{
"websocket_url": "wss://stream.bybit.com/v5/public/linear",
"symbol": "BTCUSDT",
"data_dir": "data",
"log_file": "engine.log",
"hot_retention_hours": 12,
"feature_retention_days": 30,
"writer_flush_interval_ms": 500,
"writer_batch_size": 100,
"tick_channel_buffer": 10000,
"maintenance_interval_minutes": 60
"maintenance_interval_minutes": 60,
"streams": {
"trades": {
"enabled": true,
"hot_retention_hours": 12,
"feature_retention_days": 30
},
"ticker": {
"enabled": true,
"hot_retention_hours": 24,
"feature_retention_days": 30,
"snapshot_interval_ms": 5000
},
"klines": {
"enabled": true,
"intervals": ["5", "15", "60"],
"short_retention_hours": 24,
"long_retention_weeks": 4,
"feature_retention_days": 30
},
"orderbook": {
"enabled": true,
"depth": 50,
"snapshot_interval_ms": 5000,
"hot_retention_hours": 6,
"feature_retention_days": 30
},
"liquidations": {
"enabled": true,
"hot_retention_hours": 12,
"feature_retention_days": 30
}
}
}
```
| Parameter | Default | Description |
|-----------|---------|-------------|
| `websocket_url` | `wss://stream.bybit.com/v5/public/linear` | Bybit V5 public linear endpoint |
| `symbol` | `BTCUSDT` | Trading pair to subscribe to |
| `data_dir` | `data` | Base directory for all database files |
| `hot_retention_hours` | `12` | Hours of raw ticks to keep in hot DB before archiving |
| `feature_retention_days` | `30` | Days of 5s features to retain before pruning |
| `writer_flush_interval_ms` | `500` | Max time between batch flushes to disk |
| `writer_batch_size` | `100` | Max ticks per batch before forced flush |
| `tick_channel_buffer` | `10000` | Channel capacity to absorb traffic bursts |
| `maintenance_interval_minutes` | `60` | How often the maintenance cycle runs |
## Database Schemas
### Raw Ticks (`hot_ticks.db` and weekly archives)
```sql
CREATE TABLE btc_ticks (
timestamp INTEGER NOT NULL, -- Epoch milliseconds
price REAL NOT NULL, -- Trade price
volume REAL NOT NULL, -- Trade quantity
side TEXT NOT NULL -- "Buy" or "Sell"
);
CREATE INDEX idx_timestamp ON btc_ticks(timestamp);
```
### 5-Second Features (`features.db`)
```sql
CREATE TABLE five_second_features (
timestamp INTEGER PRIMARY KEY, -- Epoch ms (bucket start)
log_return REAL NOT NULL, -- ln(close / prev_close)
realized_vol REAL NOT NULL, -- Std dev of tick-to-tick log returns
ofi REAL NOT NULL, -- Buy volume − Sell volume
volume_sum REAL NOT NULL, -- Total volume in bucket
close_price REAL NOT NULL, -- Last trade price in bucket
vwap REAL NOT NULL -- Volume-Weighted Average Price in bucket
);
```
## Directory Layout
```
.
├── Makefile
├── README.md
├── config.json # Auto-generated on first run
├── go.mod
├── go.sum
├── main.go # Entry point, signal handling, goroutine orchestration
├── config.go # JSON configuration with defaults
├── types.go # Tick, FeatureBucket, Bybit message structs
├── websocket.go # Bybit V5 WebSocket client (auto-reconnect)
├── aggregator.go # 5-second feature bucketing and math
├── writer.go # Batch DB writer goroutine
├── storage.go # SQLite init, migration, pruning, recovery
└── data/ # Created at runtime
├── hot_ticks.db # Rolling 12h raw tick database
├── features.db # Permanent 5s feature database
└── archive/
├── btc_ticks_2026_W28.db # Weekly archive partitions
└── btc_ticks_2026_W29.db
├── config.json # Auto-generated on first run
├── main.go # Entry point & signal handling
├── config.go # Multi-stream configuration
├── types.go # Message structs and data types
├── storage.go # Per-stream storage manager & maintenance
├── websocket.go # Topic-routing WebSocket client
├── trade_handler.go # publicTrade stream handler
├── ticker_handler.go # tickers stream handler
├── kline_handler.go # kline stream handler
├── orderbook_handler.go # orderbook depth handler
├── liquidation_handler.go # allLiquidation stream handler
└── data/ # Data root directory
├── trades/
│ ├── hot_ticks.db
│ ├── features.db
│ └── archive/
├── ticker/
│ ├── hot_ticker.db
│ └── features.db
├── klines/
│ ├── kline_5.db
│ ├── kline_15.db
│ ├── kline_60.db
│ ├── features.db
│ └── archive/
├── orderbook/
│ ├── hot_snapshots.db
│ └── features.db
└── liquidations/
├── hot_liquidations.db
├── features.db
└── archive/
```
## Makefile Targets
```bash
make build # Fetch dependencies and compile
make run # Build and run
make test # Run tests with race detector
make vet # Static analysis
make fmt # Format source code
make clean # Remove binary and data directory
```
## Dependencies
| Module | Purpose |
|--------|---------|
| `modernc.org/sqlite` v1.53.0 | Pure Go SQLite driver (no CGO) |
| `nhooyr.io/websocket` v1.8.17 | Modern, context-aware WebSocket client |
## How It Works
1. **Startup** — Initializes databases with WAL mode and high-performance pragmas. Scans `hot_ticks.db` for any ticks older than the retention window (e.g., from a previous crash) and migrates them immediately.
2. **Ingestion** — Connects to the Bybit V5 WebSocket and subscribes to `publicTrade.BTCUSDT`. Each trade message (which may contain up to 1024 trades) is parsed and the ticks are:
- Passed to the **aggregator** inline for 5-second feature computation
- Sent to the **writer** via a buffered channel for raw storage
3. **Aggregation** — Ticks are bucketed by 5-second epoch boundaries. When a tick crosses into a new bucket, the previous bucket is finalized with computed features (log return, realized volatility, OFI, volume sum, close price) and written to `features.db`.
4. **Batch Writing** — The writer goroutine accumulates ticks and flushes them to `hot_ticks.db` in a single transaction every 500ms or 100 ticks, whichever comes first. This batching strategy minimizes SQLite write amplification.
5. **Maintenance** — Every hour, the maintenance goroutine:
- Migrates ticks older than 12 hours from `hot_ticks.db` to weekly archive databases using atomic `ATTACH`/`DETACH` transactions, handling cross-week boundaries correctly
- Prunes feature rows older than 30 days from `features.db`
- Runs `PRAGMA incremental_vacuum` to reclaim disk space
6. **Shutdown** — On SIGINT/SIGTERM, the engine cancels all goroutines, drains remaining ticks from the channel, flushes any partial aggregator bucket, and exits cleanly.
## License
MIT