Bug fixes in input and onramp. Hot config reload on signals. Added example utility scripts for signals.

2026-01-17 14:47:13 +02:00
parent 7d7038d6bd
commit aa216981d2
16 changed files with 2339 additions and 306 deletions
--- a/signals/DESIGN.md
+++ b/signals/DESIGN.md
@@ -0,0 +1,196 @@
+### Key Points on Signal Generator Design
+- **Core Functionality**: The system processes real-time BTCUSDT data from SQLite databases to generate buy/sell signals based on technical indicators like EMAs, RSI, MACD, Bollinger Bands, and Stochastic Oscillator, supporting scalping (short-term, high-frequency) and swing (medium-term, trend-focused) personalities.
+- **Low-Latency Focus**: Uses Unix Domain Sockets (UDS) for local streaming to minimize delays (typically 130-200μs RTT), ideal for your 0.5-1.7s pipeline lag and single-trade limit.
+- **Implementation Choice**: Python 3 with pandas, TA-Lib, and sqlite3 for rapid development; incorporate backtesting via pandas or Backtesting.py to validate strategies before live use.
+- **Risk Considerations**: Strategies like scalping may yield quick wins but face higher fees and noise; swing trading offers better risk-reward in trends but requires patience—backtest extensively to assess win rates (often 50-60%).
+
+### System Overview
+This design outlines a modular signal generator that ingests data from `candles.db` and `analysis.db`, computes signals using rule-based logic, and streams them to a paper trading bot. It emphasizes determinism, low overhead, and extensibility for your local setup.
+
+### Architecture Components
+- **Input**: Polls databases every 0.5-1s for closed candles.
+- **Processing**: Applies personality-specific rules with confluence scoring.
+- **Output**: JSON signals via UDS (fallback TCP for dual machines).
+
+### Recommended Strategies
+For scalping: Focus on 1m/5m timeframes with EMA crossovers and Stochastic. For swing: Use 15m/1h with regime filters and squeezes. Validate via backtesting to ensure profitability.
+
+---
+### Comprehensive Design and Plan for BTCUSDT Signal Generator
+
+#### 1. Introduction and Objectives
+The signal generator is a critical component in your Bybit BTCUSDT websocket data pipeline, transforming aggregated OHLCV and precomputed indicators into actionable buy/sell signals for a paper trading bot. This document provides a ready-to-use blueprint, incorporating best practices from Python-based trading systems, such as modular code for indicators, efficient data handling with pandas, and low-latency communication. Objectives include:
+- Supporting "scalping" (high-frequency, short-term trades) and "swing" (medium-term, trend-capturing) personalities.
+- Ensuring sub-second latency for signal delivery, aligning with your 0.5-1.7s pipeline.
+- Enabling backtesting to evaluate strategy performance, targeting win rates of 50-60% in crypto markets.
+- Facilitating easy extension for new indicators or personalities.
+
+The system runs as a single-threaded daemon on one or two local machines, handling ~600 messages/minute without high concurrency needs.
+
+#### 2. System Architecture
+The architecture follows a layered, event-driven design inspired by high-performance trading systems: input, processing, and output. It uses Python 3 for prototyping, with pandas for data manipulation and TA-Lib for indicators.
+
+- **Input Layer**: Connects to `candles.db` and `analysis.db` using WAL mode for concurrent reads. Polls every 0.5-1s, fetching the last 200-500 rows per timeframe to compute trends.
+- **Processing Layer**: Loads data into pandas DataFrames, computes additional indicators (e.g., Stochastic), applies rules, and scores signals (0-1 confidence based on confluence).
+- **Output Layer**: Streams JSON signals (e.g., {"signal": "BUY", "timeframe": "1m", "confidence": 0.75}) via UDS for lowest latency (~130μs RTT); fallback to TCP with NODELAY for dual machines.
+- **Configuration**: YAML file for personalities, thresholds (e.g., RSI<30), and polling intervals.
+- **Monitoring**: Logs to file; optional metrics for signal frequency.
+
+This setup ensures determinism—no repainting—and handles your throughput reduction goal by filtering to 1-10 signals/hour.
+
+#### 3. Data Integration and Handling
+Leverage your existing schemas: Join `candles` and `analysis` tables on timeframe/timestamp for unified DataFrames. Use TA-Lib for Stochastic (%K=14, %D=3) and extend precomputed indicators (EMA9/21, RSI14, MACD, Bollinger, volume MA20).
+
+Example Fetch Function:
+```python
+import sqlite3
+import pandas as pd
+import talib
+
+def fetch_and_enrich(timeframe, limit=200, candles_db='candles.db', analysis_db='analysis.db'):
+    conn_c = sqlite3.connect(candles_db, timeout=10)
+    conn_a = sqlite3.connect(analysis_db, timeout=10)
+    query = """
+        SELECT c.*, a.*
+        FROM candles c
+        JOIN analysis a ON c.timeframe = a.timeframe AND c.timestamp = a.timestamp
+        WHERE c.timeframe = ?
+        ORDER BY c.timestamp DESC
+        LIMIT ?
+    """
+    df = pd.read_sql_query(query, conn_c, params=(timeframe, limit))
+    conn_c.close(); conn_a.close()
+    df['datetime'] = pd.to_datetime(df['timestamp'], unit='s')
+    df = df.sort_values('timestamp').reset_index(drop=True)
+    # Add Stochastic
+    df['stoch_k'], df['stoch_d'] = talib.STOCH(df['high'], df['low'], df['close'], fastk_period=14, slowk_period=3, slowd_period=3)
+    return df
+```
+
+Filter for closed candles (timestamp < current time) to avoid instability.
+
+#### 4. Indicators and Signal Logic
+Build on your `analysis.db` with rule-based logic for confluence. Key indicators:
+- MA Crossings: EMA9 > EMA21 (buy); reverse for sell.
+- RSI: <30 (oversold, buy); >70 (overbought, sell).
+- MACD: Line > signal (buy).
+- Bollinger Squeeze: bb_squeeze=1 + breakout.
+- Stochastic: %K > %D and <20 (buy).
+- Volume: >1.5 * MA20; buy_ratio >0.6.
+
+Confidence: Weighted sum (e.g., 0.3 for MA, 0.2 for volume) >0.6 triggers signal.
+
+Example Logic:
+```python
+def generate_signal(df, personality='scalping'):
+    latest = df.iloc[-1]
+    score = 0
+    reasons = []
+    if personality == 'scalping':
+        if latest['ema_9'] > latest['ema_21'] and latest['stoch_k'] > latest['stoch_d'] and latest['stoch_k'] < 30:
+            score += 0.4; reasons.append('EMA crossover + Stochastic buy')
+        if latest['rsi_14'] < 50 and latest['volume'] > 1.5 * latest['volume_ma_20']:
+            score += 0.3; reasons.append('RSI undersold + volume surge')
+    elif personality == 'swing':
+        if latest['close'] > latest['sma_50'] and latest['bb_squeeze'] == 1 and latest['close'] > latest['bb_upper']:
+            score += 0.4; reasons.append('Regime bull + BB breakout')
+        if latest['macd'] > latest['macd_signal'] and latest['buy_volume'] / latest['volume'] > 0.55:
+            score += 0.3; reasons.append('MACD crossover + buy bias')
+    if score > 0.6:
+        return {'signal': 'BUY' if score > 0 else 'SELL', 'confidence': score, 'reasons': reasons}
+    return None
+```
+
+#### 5. Trading Personalities
+Tailor strategies to crypto volatility.
+
+| Personality | Timeframes | Focus Indicators | Trade Frequency | Risk Notes |
+|-------------|------------|------------------|-----------------|------------|
+| Scalping   | 1m, 5m    | EMA9/21, Stochastic, MACD hist, volume | 10-50/day | High fees/slippage; aim 0.25-1% per trade, 65% win rate possible with BB/RSI. |
+| Swing      | 15m, 1h   | SMA50/200, BB squeeze, RSI, net flow | 2-10/week | Lower stress; target 5-10% moves, suits trends but lags in ranges. |
+
+Switch via config; add filters like regime bias (close > SMA200 for longs).
+
+#### 6. Streaming and Bot Integration
+Use UDS for polling-based feed: Generator writes signals; bot polls every 50-100ms. Defer stops/positioning to bot.
+
+Server Code:
+```python
+import socket
+import json
+import time
+
+SOCKET_PATH = '/tmp/signal.sock'
+server = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+try: os.remove(SOCKET_PATH)
+except: pass
+server.bind(SOCKET_PATH)
+server.listen(1)
+conn, _ = server.accept()
+while True:
+    signal = generate_signal(fetch_and_enrich('1m'))  # Example
+    if signal:
+        conn.sendall(json.dumps(signal).encode() + b'\n')
+    time.sleep(0.5)  # Poll interval
+```
+
+Bot polls non-blockingly.
+
+#### 7. Implementation Plan
+- **Dependencies**: pandas, sqlite3, talib, backtesting.py.
+- **Phases**: 1) Setup DB connections (1 day); 2) Indicator/signal logic (2-3 days); 3) Streaming (1 day); 4) Backtesting (2 days); 5) Testing/deployment (1-2 days).
+- **Code Structure**: Main loop in `signal_generator.py`; separate modules for data, logic, config.
+
+#### 8. Backtesting and Validation
+Use pandas for simple backtests or Backtesting.py for advanced. Simulate slippage (0.1%) and fees; track Sharpe ratio, drawdown.
+
+Example with Backtesting.py:
+```python
+from backtesting import Backtest, Strategy
+
+class ScalpStrategy(Strategy):
+    def init(self):
+        self.ema9 = self.I(talib.EMA, self.data.Close, 9)
+        self.stoch_k, self.stoch_d = self.I(talib.STOCH, self.data.High, self.data.Low, self.data.Close)
+    def next(self):
+        if crossover(self.ema9, self.ema21) and self.stoch_k > self.stoch_d and self.stoch_k < 30:
+            self.buy()
+
+bt = Backtest(df, ScalpStrategy, cash=10000, commission=.001)
+stats = bt.run()
+print(stats)
+```
+
+Aim for positive expectancy; re-optimize thresholds quarterly.
+
+#### 9. Deployment and Maintenance
+Run as systemd service; monitor with logs. Vacuum DBs monthly. Scale by adding personalities or ML later.
+
+#### 10. Risks and Mitigations
+- **Market Risks**: Crypto volatility; mitigate with volume filters and backtesting.
+- **Technical Risks**: DB locks; use timeouts. Overfitting; use out-of-sample data.
+- **Performance**: CPU spikes from polling; adaptive intervals.
+- **Disclaimer**: Signals for paper trading only; not financial advice.
+
+This plan equips you to build a robust system, validated by industry practices.
+
+### Key Citations
+- [Creating a Custom Python Library for Trading Indicators and Signal Generation - Medium](https://medium.com/@deepml1818/creating-a-custom-python-library-for-trading-indicators-and-signal-generation-2097d0d526f7)
+- [How to build a macro trading strategy (with open-source Python) - Macrosynergy](https://macrosynergy.com/research/how-to-build-a-macro-trading-strategy-with-open-source-python)
+- [Python in FinTech: Building High-Performance Trading Systems - CMARIX](https://www.cmarix.com/blog/build-trading-systems-with-python-in-fintech)
+- [Python in High-Frequency Trading: Low-Latency Techniques - PyQuant News](https://www.pyquantnews.com/free-python-resources/python-in-high-frequency-trading-low-latency-techniques)
+- [A Look at the Differences Between Scalping and Swing Trading in Crypto | by JeNovation](https://medium.com/@ByTrade.io/a-look-at-the-differences-between-scalping-and-swing-trading-in-crypto-d7a83623c277)
+- [Day trade - swing or scalp? : r/BitcoinMarkets - Reddit](https://www.reddit.com/r/BitcoinMarkets/comments/1jklil5/day_trade_swing_or_scalp)
+- [Best Crypto Trading Strategies (Beginner to Intermediate Guide) - Changelly](https://changelly.com/blog/cryptocurrency-trading-strategies)
+- [Crypto Scalping vs Swing Trading: Which is More Profitable for Retail Investors?](https://www.blockchain-council.org/cryptocurrency/crypto-scalping-vs-swing-trading)
+- [Crypto Scalping: What It Is, How It Works, Strategies & Tools (2025) - CryptoNinjas](https://www.cryptoninjas.net/crypto/cryptocurrency-scalping)
+- [7 Best Crypto Trading Strategies for 2025 - CMC Markets](https://www.cmcmarkets.com/en/cryptocurrencies/7-crypto-trading-strategies)
+- [I Tried 1-Minute Scalping on BTC — My Honest Results - InsiderFinance Wire](https://wire.insiderfinance.io/i-tried-1-minute-scalping-on-btc-my-honest-results-c80989f7335b)
+- [Algorithmic Trading with Stochastic Oscillator in Python | by Nikhil Adithyan - Medium](https://medium.com/codex/algorithmic-trading-with-stochastic-oscillator-in-python-7e2bec49b60d)
+- [Python Trading Strategy: Synergizing Stochastic Oscillator and MACD Indicator - EODHD](https://eodhd.com/financial-academy/backtesting-strategies-examples/using-python-to-create-an-innovative-trading-strategy-and-achieve-better-results)
+- [Using the Stochastic Oscillator in Python for Algorithmic Trading - αlphαrithms](https://www.alpharithms.com/stochastic-oscillator-in-python-483214)
+- [Momentum Indicators - Technical Analysis Library in Python's documentation!](https://technical-analysis-library-in-python.readthedocs.io/en/latest/ta.html)
+- [Backtesting with Pandas and TA-lib - YouTube](https://www.youtube.com/watch?v=WBPnN8DIMYI)
+- [Backtesting.py - Backtest trading strategies in Python](https://kernc.github.io/backtesting.py)
+- [Mastering Python Backtesting for Trading Strategies | by Time Money Code | Medium](https://medium.com/@timemoneycode/mastering-python-backtesting-for-trading-strategies-1f7df773fdf5)
+- [Python Libraries for Quantitative Trading | QuantStart](https://www.quantstart.com/articles/python-libraries-for-quantitative-trading)