129 lines
4.9 KiB
Markdown
129 lines
4.9 KiB
Markdown
This is a comprehensive `README.md` designed for the root of your repository. It ties together the Go and Python services, explains the data pipeline, and provides the first documentation for your `mother.sh` orchestration script.
|
|
|
|
***
|
|
|
|
# Bybit BTCUSDT Signal Engine
|
|
|
|
A high-performance, multi-stage pipeline for streaming, processing, and generating trading signals from Bybit's BTCUSDT public trade WebSocket.
|
|
|
|
This project is architected as a series of decoupled microservices that move data from raw WebSocket packets to actionable trading signals via SQLite persistence and Unix Domain Sockets.
|
|
|
|
## 🏗 Architecture & Data Flow
|
|
|
|
The engine operates as a linear pipeline, ensuring data integrity and separation of concerns:
|
|
|
|
1. **`input` (Go)**: Connects to Bybit V5 WS. Streams raw JSON payloads into hourly-rotated `.jsonl` files.
|
|
2. **`onramp` (Go)**: Tails the JSONL files. Aggregates individual trades into OHLCV candles across multiple timeframes (1m, 5m, 15m, 1h) and saves them to `candles.db`.
|
|
3. **`analysis` (Python)**: Reads `candles.db`. Computes technical indicators (EMA, RSI, MACD, Bollinger Bands) and saves them to `analysis.db`.
|
|
4. **`signals` (Python)**: Monitors both databases. Applies "Personalities" (Scalping or Swing logic) to generate signals, broadcasted via Unix Domain Sockets.
|
|
5. **`monitor` (Python)**: Provides a real-time dashboard of the system health and signal flow.
|
|
|
|
```mermaid
|
|
graph LR
|
|
Bybit((Bybit WS)) --> input[input service]
|
|
input --> JSONL[(JSONL Files)]
|
|
JSONL --> onramp[onramp service]
|
|
onramp --> CDB[(candles.db)]
|
|
CDB --> analysis[analysis service]
|
|
analysis --> ADB[(analysis.db)]
|
|
ADB --> signals[signals service]
|
|
CDB --> signals
|
|
signals --> Socket((/tmp/signals.sock))
|
|
Socket --> Bot[Trading Bot / Monitor]
|
|
```
|
|
|
|
---
|
|
|
|
## 🛠 The "Mother" Script (`mother.sh`)
|
|
|
|
The entire stack is managed by `mother.sh`, a centralized orchestration script located in the root. It handles dependency management, system health checks, and process multiplexing using **tmux**.
|
|
|
|
### Key Features
|
|
* **Dependency Management**: One-command setup for both Go modules and Python `pipenv` environments.
|
|
* **Tick Verification**: Before starting, the script polls `pool.ntp.org` to ensure system clock drift is `< 0.5s`, preventing timestamp rejection from Bybit.
|
|
* **Sequential Bootstrapping**: Starts services in the correct order (`input` -> `onramp` -> `analysis` -> `signals`) with specific delays to allow database initialization.
|
|
* **Process Isolation**: Each service runs in its own named window within a `tmux` session.
|
|
|
|
### Usage
|
|
| Command | Description |
|
|
| :--- | :--- |
|
|
| `./mother.sh setup` | Installs all Go and Python dependencies in their respective subdirectories. |
|
|
| `./mother.sh start` | Verifies NTP time and launches the full pipeline in a tmux session. |
|
|
| `./mother.sh status` | Checks if the services are currently running. |
|
|
| `./mother.sh stop` | Gracefully kills the tmux session and all underlying processes. |
|
|
| `./mother.sh restart` | Full stop and sequential restart. |
|
|
|
|
---
|
|
|
|
## 🚀 Getting Started
|
|
|
|
### Prerequisites
|
|
* **Go** 1.20+
|
|
* **Python** 3.10+ & **pipenv**
|
|
* **tmux** (for process management)
|
|
* **TA-Lib** (System library required for Python signal analysis)
|
|
```bash
|
|
# Ubuntu/Debian
|
|
sudo apt-get install ntpdate bc
|
|
# See signals/README.md for TA-Lib compilation instructions
|
|
```
|
|
|
|
### Installation & Launch
|
|
1. **Clone the repo**:
|
|
```bash
|
|
git clone <repo-url>
|
|
cd bybitbtc
|
|
```
|
|
2. **Run Setup**:
|
|
```bash
|
|
./mother.sh setup
|
|
```
|
|
3. **Start the Engine**:
|
|
```bash
|
|
./mother.sh start
|
|
```
|
|
4. **Attach to Logs**:
|
|
```bash
|
|
tmux attach -t bybitbtc
|
|
```
|
|
|
|
---
|
|
|
|
## 📊 Service Details
|
|
|
|
### 1. Ingestion (`input/`)
|
|
* **Language**: Go
|
|
* **Output**: `./input/output/publicTrade.BTCUSDT_TIMESTAMP.jsonl`
|
|
* **Rotation**: Hourly.
|
|
* **Compression**: Automatically gzips files older than 2 hours to save disk space.
|
|
|
|
### 2. Aggregation (`onramp/`)
|
|
* **Language**: Go
|
|
* **Database**: `candles.db` (SQLite with WAL mode).
|
|
* **Features**: Real-time OHLCV calculation, Buy/Sell volume tracking, and a Janitor loop that prunes data older than 30 days.
|
|
|
|
### 3. Analysis (`analysis/`)
|
|
* **Language**: Python (pandas, talib)
|
|
* **Database**: `analysis.db`.
|
|
* **Indicators**: EMA (9, 21), SMA (50, 200), RSI (14), MACD (12, 26, 9), Bollinger Bands with Squeeze detection.
|
|
|
|
### 4. Signals (`signals/`)
|
|
* **Language**: Python
|
|
* **Personalities**:
|
|
* **Scalping**: High-frequency (1m/5m), EMA-cross and Stochastic focused.
|
|
* **Swing**: Medium-frequency (15m/1h), Trend regime and BB-Squeeze focused.
|
|
* **Delivery**: Streams JSON signals over `/tmp/signals.sock`.
|
|
|
|
---
|
|
|
|
## 📡 Monitoring & Health
|
|
|
|
Each service exposes a status interface:
|
|
* **Input**: `go run input.go -status`
|
|
* **Onramp**: `echo "status" | nc 127.0.0.1 9999`
|
|
* **Signals**: `python signals/signal_health_client.py` (via `/tmp/signals_health.sock`)
|
|
|
|
---
|
|
|
|
## ⚖️ License
|
|
MIT |