BytbitBTC/README.md

This is a comprehensive README.md designed for the root of your repository. It ties together the Go and Python services, explains the data pipeline, and provides the first documentation for your mother.sh orchestration script.

---

Bybit BTCUSDT Signal Engine

A high-performance, multi-stage pipeline for streaming, processing, and generating trading signals from Bybit's BTCUSDT public trade WebSocket.

This project is architected as a series of decoupled microservices that move data from raw WebSocket packets to actionable trading signals via SQLite persistence and Unix Domain Sockets.

🏗 Architecture & Data Flow

The engine operates as a linear pipeline, ensuring data integrity and separation of concerns:

1. input (Go): Connects to Bybit V5 WS. Streams raw JSON payloads into hourly-rotated .jsonl files.
2. onramp (Go): Tails the JSONL files. Aggregates individual trades into OHLCV candles across multiple timeframes (1m, 5m, 15m, 1h) and saves them to candles.db.
3. analysis (Python): Reads candles.db. Computes technical indicators (EMA, RSI, MACD, Bollinger Bands) and saves them to analysis.db.
4. signals (Python): Monitors both databases. Applies "Personalities" (Scalping or Swing logic) to generate signals, broadcasted via Unix Domain Sockets.
5. monitor (Python): Provides a real-time dashboard of the system health and signal flow.

graph LR
    Bybit((Bybit WS)) --> input[input service]
    input --> JSONL[(JSONL Files)]
    JSONL --> onramp[onramp service]
    onramp --> CDB[(candles.db)]
    CDB --> analysis[analysis service]
    analysis --> ADB[(analysis.db)]
    ADB --> signals[signals service]
    CDB --> signals
    signals --> Socket((/tmp/signals.sock))
    Socket --> Bot[Trading Bot / Monitor]

---

🛠 The "Mother" Script (mother.sh)

The entire stack is managed by mother.sh, a centralized orchestration script located in the root. It handles dependency management, system health checks, and process multiplexing using tmux.

Key Features

• Dependency Management: One-command setup for both Go modules and Python pipenv environments.
• Tick Verification: Before starting, the script polls pool.ntp.org to ensure system clock drift is < 0.5s, preventing timestamp rejection from Bybit.
• Sequential Bootstrapping: Starts services in the correct order (input -> onramp -> analysis -> signals) with specific delays to allow database initialization.
• Process Isolation: Each service runs in its own named window within a tmux session.

Usage

Command	Description
./mother.sh setup	Installs all Go and Python dependencies in their respective subdirectories.
./mother.sh start	Verifies NTP time and launches the full pipeline in a tmux session.
./mother.sh status	Checks if the services are currently running.
./mother.sh stop	Gracefully kills the tmux session and all underlying processes.
./mother.sh restart	Full stop and sequential restart.

---

🚀 Getting Started

Prerequisites

• Go 1.20+
• Python 3.10+ & pipenv
• tmux (for process management)
• TA-Lib (System library required for Python signal analysis)

  # Ubuntu/Debian
  sudo apt-get install ntpdate bc
  # See signals/README.md for TA-Lib compilation instructions
  

Installation & Launch

1. Clone the repo:

   git clone <repo-url>
   cd bybitbtc
   

2. Run Setup:

   ./mother.sh setup
   

3. Start the Engine:

   ./mother.sh start
   

4. Attach to Logs:

   tmux attach -t bybitbtc
   

---

📊 Service Details

1. Ingestion (input/)

• Language: Go
• Output: ./input/output/publicTrade.BTCUSDT_TIMESTAMP.jsonl
• Rotation: Hourly.
• Compression: Automatically gzips files older than 2 hours to save disk space.

2. Aggregation (onramp/)

• Language: Go
• Database: candles.db (SQLite with WAL mode).
• Features: Real-time OHLCV calculation, Buy/Sell volume tracking, and a Janitor loop that prunes data older than 30 days.

3. Analysis (analysis/)

• Language: Python (pandas, talib)
• Database: analysis.db.
• Indicators: EMA (9, 21), SMA (50, 200), RSI (14), MACD (12, 26, 9), Bollinger Bands with Squeeze detection.

4. Signals (signals/)

• Language: Python
• Personalities:
  • Scalping: High-frequency (1m/5m), EMA-cross and Stochastic focused.
  • Swing: Medium-frequency (15m/1h), Trend regime and BB-Squeeze focused.
• Delivery: Streams JSON signals over /tmp/signals.sock.

---

📡 Monitoring & Health

Each service exposes a status interface:

• Input: go run input.go -status
• Onramp: echo "status" | nc 127.0.0.1 9999
• Signals: python signals/signal_health_client.py (via /tmp/signals_health.sock)

---

⚖️ License

MIT