Bybit BTC/USDT Tick Ingest Engine

A high-performance, self-cleaning market data ingestion service that captures real-time BTC/USDT trades from the Bybit V5 WebSocket API, stores raw ticks in a rolling SQLite hot database, aggregates 5-second feature bars for machine learning, and automatically archives historical data into weekly partitions.

Architecture

                            ┌─────────────────────────┐
                            │    Bybit WebSocket      │
                            │  publicTrade.BTCUSDT    │
                            └────────────┬────────────┘
                                         │ JSON Tick Stream
                                         ▼
                            ┌─────────────────────────┐
                            │    Ingestor Goroutine    │
                            │  (parse, fan-out)        │
                            └──────┬────────────┬─────┘
                                   │            │
             Buffered Channel      │            │  Inline 5s Aggregation
                                   ▼            ▼
┌────────────────────────────────────┐        ┌────────────────────────────────────┐
│     Writer Goroutine               │        │    features.db                     │
│  (batch flush to hot DB)           │        │  - Permanent 5s feature bars       │
│                                    │        │  - 30-day rolling retention        │
│          ▼                         │        │  - ~300 MB/year                    │
│   hot_ticks.db                     │        └────────────────────────────────────┘
│  - Rolling 12h of raw ticks       │
│  - WAL mode, high-throughput      │
└──────────────────┬─────────────────┘
                   │
                   │ Hourly Maintenance Goroutine
                   ▼
┌────────────────────────────────────┐
│     Weekly Archive DBs             │
│  - data/archive/btc_ticks_YYYY_Www.db │
│  - Atomic ATTACH/DETACH migration  │
│  - ISO week partitioning           │
└────────────────────────────────────┘

The system runs four concurrent goroutines:

Goroutine	Responsibility
Ingestor	Connects to Bybit WebSocket, parses trades, feeds aggregator and writer channel
Writer	Drains tick channel, batch-flushes to `hot_ticks.db` every 500ms or 100 ticks
Aggregator	Buckets ticks into 5-second windows, computes features, writes to `features.db`
Maintenance	Hourly migration of old ticks to weekly archives, feature pruning

Features

Real-time ingestion from Bybit V5 public linear WebSocket with automatic reconnection
Decoupled write pipeline — buffered channel isolates network I/O from disk I/O
5-second feature bars with log return, realized volatility, order flow imbalance (OFI), volume sum, and close price
Three-tier storage — hot DB (12h), weekly archives, and permanent feature DB
Automatic maintenance — hourly tick migration, feature pruning, incremental vacuum
Startup recovery — detects and migrates stale ticks from previous runs
Graceful shutdown — SIGINT/SIGTERM drains remaining data before exit
Pure Go — no CGO required (modernc.org/sqlite)

Requirements

Go 1.25.0+
Linux amd64 (tested on Debian)
Internet access to stream.bybit.com

Quick Start

# Clone and build
git clone <repo-url>
cd bybit_btcusdt_ingest
make build

# Run (generates default config.json on first start)
make run

# Stop gracefully with Ctrl+C

Configuration

A config.json file is auto-generated on first run with these defaults:

{
  "websocket_url": "wss://stream.bybit.com/v5/public/linear",
  "symbol": "BTCUSDT",
  "data_dir": "data",
  "hot_retention_hours": 12,
  "feature_retention_days": 30,
  "writer_flush_interval_ms": 500,
  "writer_batch_size": 100,
  "tick_channel_buffer": 10000,
  "maintenance_interval_minutes": 60
}

Parameter	Default	Description
`websocket_url`	`wss://stream.bybit.com/v5/public/linear`	Bybit V5 public linear endpoint
`symbol`	`BTCUSDT`	Trading pair to subscribe to
`data_dir`	`data`	Base directory for all database files
`hot_retention_hours`	`12`	Hours of raw ticks to keep in hot DB before archiving
`feature_retention_days`	`30`	Days of 5s features to retain before pruning
`writer_flush_interval_ms`	`500`	Max time between batch flushes to disk
`writer_batch_size`	`100`	Max ticks per batch before forced flush
`tick_channel_buffer`	`10000`	Channel capacity to absorb traffic bursts
`maintenance_interval_minutes`	`60`	How often the maintenance cycle runs

Database Schemas

Raw Ticks (`hot_ticks.db` and weekly archives)

CREATE TABLE btc_ticks (
    timestamp INTEGER NOT NULL,  -- Epoch milliseconds
    price     REAL    NOT NULL,  -- Trade price
    volume    REAL    NOT NULL,  -- Trade quantity
    side      TEXT    NOT NULL   -- "Buy" or "Sell"
);
CREATE INDEX idx_timestamp ON btc_ticks(timestamp);

5-Second Features (`features.db`)

CREATE TABLE five_second_features (
    timestamp    INTEGER PRIMARY KEY,  -- Epoch ms (bucket start)
    log_return   REAL    NOT NULL,     -- ln(close / prev_close)
    realized_vol REAL    NOT NULL,     -- Std dev of tick-to-tick log returns
    ofi          REAL    NOT NULL,     -- Buy volume − Sell volume
    volume_sum   REAL    NOT NULL,     -- Total volume in bucket
    close_price  REAL    NOT NULL      -- Last trade price in bucket
);

Directory Layout

.
├── Makefile
├── README.md
├── config.json                    # Auto-generated on first run
├── go.mod
├── go.sum
├── main.go                        # Entry point, signal handling, goroutine orchestration
├── config.go                      # JSON configuration with defaults
├── types.go                       # Tick, FeatureBucket, Bybit message structs
├── websocket.go                   # Bybit V5 WebSocket client (auto-reconnect)
├── aggregator.go                  # 5-second feature bucketing and math
├── writer.go                      # Batch DB writer goroutine
├── storage.go                     # SQLite init, migration, pruning, recovery
└── data/                          # Created at runtime
    ├── hot_ticks.db               # Rolling 12h raw tick database
    ├── features.db                # Permanent 5s feature database
    └── archive/
        ├── btc_ticks_2026_W28.db  # Weekly archive partitions
        └── btc_ticks_2026_W29.db

Makefile Targets

make build    # Fetch dependencies and compile
make run      # Build and run
make test     # Run tests with race detector
make vet      # Static analysis
make fmt      # Format source code
make clean    # Remove binary and data directory

Dependencies

Module	Purpose
`modernc.org/sqlite` v1.53.0	Pure Go SQLite driver (no CGO)
`nhooyr.io/websocket` v1.8.17	Modern, context-aware WebSocket client

How It Works

Startup — Initializes databases with WAL mode and high-performance pragmas. Scans hot_ticks.db for any ticks older than the retention window (e.g., from a previous crash) and migrates them immediately.
Ingestion — Connects to the Bybit V5 WebSocket and subscribes to publicTrade.BTCUSDT. Each trade message (which may contain up to 1024 trades) is parsed and the ticks are:
- Passed to the aggregator inline for 5-second feature computation
- Sent to the writer via a buffered channel for raw storage
Aggregation — Ticks are bucketed by 5-second epoch boundaries. When a tick crosses into a new bucket, the previous bucket is finalized with computed features (log return, realized volatility, OFI, volume sum, close price) and written to features.db.
Batch Writing — The writer goroutine accumulates ticks and flushes them to hot_ticks.db in a single transaction every 500ms or 100 ticks, whichever comes first. This batching strategy minimizes SQLite write amplification.
Maintenance — Every hour, the maintenance goroutine:
- Migrates ticks older than 12 hours from hot_ticks.db to weekly archive databases using atomic ATTACH/DETACH transactions, handling cross-week boundaries correctly
- Prunes feature rows older than 30 days from features.db
- Runs PRAGMA incremental_vacuum to reclaim disk space
Shutdown — On SIGINT/SIGTERM, the engine cancels all goroutines, drains remaining ticks from the channel, flushes any partial aggregator bucket, and exits cleanly.

License

MIT

9.3 KiB Raw Blame History Unescape Escape