BytbitBTC/DESIGN.md at 9834272df3118b091ee286a74406c79e43112ec5

Files

Kalzu Rekku aa216981d2 Bug fixes in input and onramp. Hot config reload on signals. Added example utility scripts for signals.

2026-01-17 14:47:13 +02:00

12 KiB

Raw Blame History

Key Points on Signal Generator Design

Core Functionality: The system processes real-time BTCUSDT data from SQLite databases to generate buy/sell signals based on technical indicators like EMAs, RSI, MACD, Bollinger Bands, and Stochastic Oscillator, supporting scalping (short-term, high-frequency) and swing (medium-term, trend-focused) personalities.
Low-Latency Focus: Uses Unix Domain Sockets (UDS) for local streaming to minimize delays (typically 130-200μs RTT), ideal for your 0.5-1.7s pipeline lag and single-trade limit.
Implementation Choice: Python 3 with pandas, TA-Lib, and sqlite3 for rapid development; incorporate backtesting via pandas or Backtesting.py to validate strategies before live use.
Risk Considerations: Strategies like scalping may yield quick wins but face higher fees and noise; swing trading offers better risk-reward in trends but requires patience—backtest extensively to assess win rates (often 50-60%).

System Overview

This design outlines a modular signal generator that ingests data from candles.db and analysis.db, computes signals using rule-based logic, and streams them to a paper trading bot. It emphasizes determinism, low overhead, and extensibility for your local setup.

Architecture Components

Input: Polls databases every 0.5-1s for closed candles.
Processing: Applies personality-specific rules with confluence scoring.
Output: JSON signals via UDS (fallback TCP for dual machines).

Recommended Strategies

For scalping: Focus on 1m/5m timeframes with EMA crossovers and Stochastic. For swing: Use 15m/1h with regime filters and squeezes. Validate via backtesting to ensure profitability.

Comprehensive Design and Plan for BTCUSDT Signal Generator

1. Introduction and Objectives

The signal generator is a critical component in your Bybit BTCUSDT websocket data pipeline, transforming aggregated OHLCV and precomputed indicators into actionable buy/sell signals for a paper trading bot. This document provides a ready-to-use blueprint, incorporating best practices from Python-based trading systems, such as modular code for indicators, efficient data handling with pandas, and low-latency communication. Objectives include:

Supporting "scalping" (high-frequency, short-term trades) and "swing" (medium-term, trend-capturing) personalities.
Ensuring sub-second latency for signal delivery, aligning with your 0.5-1.7s pipeline.
Enabling backtesting to evaluate strategy performance, targeting win rates of 50-60% in crypto markets.
Facilitating easy extension for new indicators or personalities.

The system runs as a single-threaded daemon on one or two local machines, handling ~600 messages/minute without high concurrency needs.

2. System Architecture

The architecture follows a layered, event-driven design inspired by high-performance trading systems: input, processing, and output. It uses Python 3 for prototyping, with pandas for data manipulation and TA-Lib for indicators.

Input Layer: Connects to candles.db and analysis.db using WAL mode for concurrent reads. Polls every 0.5-1s, fetching the last 200-500 rows per timeframe to compute trends.
Processing Layer: Loads data into pandas DataFrames, computes additional indicators (e.g., Stochastic), applies rules, and scores signals (0-1 confidence based on confluence).
Output Layer: Streams JSON signals (e.g., {"signal": "BUY", "timeframe": "1m", "confidence": 0.75}) via UDS for lowest latency (~130μs RTT); fallback to TCP with NODELAY for dual machines.
Configuration: YAML file for personalities, thresholds (e.g., RSI<30), and polling intervals.
Monitoring: Logs to file; optional metrics for signal frequency.

This setup ensures determinism—no repainting—and handles your throughput reduction goal by filtering to 1-10 signals/hour.

3. Data Integration and Handling

Leverage your existing schemas: Join candles and analysis tables on timeframe/timestamp for unified DataFrames. Use TA-Lib for Stochastic (%K=14, %D=3) and extend precomputed indicators (EMA9/21, RSI14, MACD, Bollinger, volume MA20).

Example Fetch Function:

import sqlite3
import pandas as pd
import talib

def fetch_and_enrich(timeframe, limit=200, candles_db='candles.db', analysis_db='analysis.db'):
    conn_c = sqlite3.connect(candles_db, timeout=10)
    conn_a = sqlite3.connect(analysis_db, timeout=10)
    query = """
        SELECT c.*, a.*
        FROM candles c
        JOIN analysis a ON c.timeframe = a.timeframe AND c.timestamp = a.timestamp
        WHERE c.timeframe = ?
        ORDER BY c.timestamp DESC
        LIMIT ?
    """
    df = pd.read_sql_query(query, conn_c, params=(timeframe, limit))
    conn_c.close(); conn_a.close()
    df['datetime'] = pd.to_datetime(df['timestamp'], unit='s')
    df = df.sort_values('timestamp').reset_index(drop=True)
    # Add Stochastic
    df['stoch_k'], df['stoch_d'] = talib.STOCH(df['high'], df['low'], df['close'], fastk_period=14, slowk_period=3, slowd_period=3)
    return df

Filter for closed candles (timestamp < current time) to avoid instability.

4. Indicators and Signal Logic

Build on your analysis.db with rule-based logic for confluence. Key indicators:

MA Crossings: EMA9 > EMA21 (buy); reverse for sell.
RSI: <30 (oversold, buy); >70 (overbought, sell).
MACD: Line > signal (buy).
Bollinger Squeeze: bb_squeeze=1 + breakout.
Stochastic: %K > %D and <20 (buy).
Volume: >1.5 * MA20; buy_ratio >0.6.

Confidence: Weighted sum (e.g., 0.3 for MA, 0.2 for volume) >0.6 triggers signal.

Example Logic:

def generate_signal(df, personality='scalping'):
    latest = df.iloc[-1]
    score = 0
    reasons = []
    if personality == 'scalping':
        if latest['ema_9'] > latest['ema_21'] and latest['stoch_k'] > latest['stoch_d'] and latest['stoch_k'] < 30:
            score += 0.4; reasons.append('EMA crossover + Stochastic buy')
        if latest['rsi_14'] < 50 and latest['volume'] > 1.5 * latest['volume_ma_20']:
            score += 0.3; reasons.append('RSI undersold + volume surge')
    elif personality == 'swing':
        if latest['close'] > latest['sma_50'] and latest['bb_squeeze'] == 1 and latest['close'] > latest['bb_upper']:
            score += 0.4; reasons.append('Regime bull + BB breakout')
        if latest['macd'] > latest['macd_signal'] and latest['buy_volume'] / latest['volume'] > 0.55:
            score += 0.3; reasons.append('MACD crossover + buy bias')
    if score > 0.6:
        return {'signal': 'BUY' if score > 0 else 'SELL', 'confidence': score, 'reasons': reasons}
    return None

5. Trading Personalities

Tailor strategies to crypto volatility.

Personality	Timeframes	Focus Indicators	Trade Frequency	Risk Notes
Scalping	1m, 5m	EMA9/21, Stochastic, MACD hist, volume	10-50/day	High fees/slippage; aim 0.25-1% per trade, 65% win rate possible with BB/RSI.
Swing	15m, 1h	SMA50/200, BB squeeze, RSI, net flow	2-10/week	Lower stress; target 5-10% moves, suits trends but lags in ranges.

Switch via config; add filters like regime bias (close > SMA200 for longs).

6. Streaming and Bot Integration

Use UDS for polling-based feed: Generator writes signals; bot polls every 50-100ms. Defer stops/positioning to bot.

Server Code:

import socket
import json
import time

SOCKET_PATH = '/tmp/signal.sock'
server = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
try: os.remove(SOCKET_PATH)
except: pass
server.bind(SOCKET_PATH)
server.listen(1)
conn, _ = server.accept()
while True:
    signal = generate_signal(fetch_and_enrich('1m'))  # Example
    if signal:
        conn.sendall(json.dumps(signal).encode() + b'\n')
    time.sleep(0.5)  # Poll interval

Bot polls non-blockingly.

7. Implementation Plan

Dependencies: pandas, sqlite3, talib, backtesting.py.
Phases: 1) Setup DB connections (1 day); 2) Indicator/signal logic (2-3 days); 3) Streaming (1 day); 4) Backtesting (2 days); 5) Testing/deployment (1-2 days).
Code Structure: Main loop in signal_generator.py; separate modules for data, logic, config.

8. Backtesting and Validation

Use pandas for simple backtests or Backtesting.py for advanced. Simulate slippage (0.1%) and fees; track Sharpe ratio, drawdown.

Example with Backtesting.py:

from backtesting import Backtest, Strategy

class ScalpStrategy(Strategy):
    def init(self):
        self.ema9 = self.I(talib.EMA, self.data.Close, 9)
        self.stoch_k, self.stoch_d = self.I(talib.STOCH, self.data.High, self.data.Low, self.data.Close)
    def next(self):
        if crossover(self.ema9, self.ema21) and self.stoch_k > self.stoch_d and self.stoch_k < 30:
            self.buy()

bt = Backtest(df, ScalpStrategy, cash=10000, commission=.001)
stats = bt.run()
print(stats)

Aim for positive expectancy; re-optimize thresholds quarterly.

9. Deployment and Maintenance

Run as systemd service; monitor with logs. Vacuum DBs monthly. Scale by adding personalities or ML later.

10. Risks and Mitigations

Market Risks: Crypto volatility; mitigate with volume filters and backtesting.
Technical Risks: DB locks; use timeouts. Overfitting; use out-of-sample data.
Performance: CPU spikes from polling; adaptive intervals.
Disclaimer: Signals for paper trading only; not financial advice.

This plan equips you to build a robust system, validated by industry practices.

12 KiB Raw Blame History Unescape Escape