196 lines
12 KiB
Markdown
196 lines
12 KiB
Markdown
### Key Points on Signal Generator Design
|
||
- **Core Functionality**: The system processes real-time BTCUSDT data from SQLite databases to generate buy/sell signals based on technical indicators like EMAs, RSI, MACD, Bollinger Bands, and Stochastic Oscillator, supporting scalping (short-term, high-frequency) and swing (medium-term, trend-focused) personalities.
|
||
- **Low-Latency Focus**: Uses Unix Domain Sockets (UDS) for local streaming to minimize delays (typically 130-200μs RTT), ideal for your 0.5-1.7s pipeline lag and single-trade limit.
|
||
- **Implementation Choice**: Python 3 with pandas, TA-Lib, and sqlite3 for rapid development; incorporate backtesting via pandas or Backtesting.py to validate strategies before live use.
|
||
- **Risk Considerations**: Strategies like scalping may yield quick wins but face higher fees and noise; swing trading offers better risk-reward in trends but requires patience—backtest extensively to assess win rates (often 50-60%).
|
||
|
||
### System Overview
|
||
This design outlines a modular signal generator that ingests data from `candles.db` and `analysis.db`, computes signals using rule-based logic, and streams them to a paper trading bot. It emphasizes determinism, low overhead, and extensibility for your local setup.
|
||
|
||
### Architecture Components
|
||
- **Input**: Polls databases every 0.5-1s for closed candles.
|
||
- **Processing**: Applies personality-specific rules with confluence scoring.
|
||
- **Output**: JSON signals via UDS (fallback TCP for dual machines).
|
||
|
||
### Recommended Strategies
|
||
For scalping: Focus on 1m/5m timeframes with EMA crossovers and Stochastic. For swing: Use 15m/1h with regime filters and squeezes. Validate via backtesting to ensure profitability.
|
||
|
||
---
|
||
### Comprehensive Design and Plan for BTCUSDT Signal Generator
|
||
|
||
#### 1. Introduction and Objectives
|
||
The signal generator is a critical component in your Bybit BTCUSDT websocket data pipeline, transforming aggregated OHLCV and precomputed indicators into actionable buy/sell signals for a paper trading bot. This document provides a ready-to-use blueprint, incorporating best practices from Python-based trading systems, such as modular code for indicators, efficient data handling with pandas, and low-latency communication. Objectives include:
|
||
- Supporting "scalping" (high-frequency, short-term trades) and "swing" (medium-term, trend-capturing) personalities.
|
||
- Ensuring sub-second latency for signal delivery, aligning with your 0.5-1.7s pipeline.
|
||
- Enabling backtesting to evaluate strategy performance, targeting win rates of 50-60% in crypto markets.
|
||
- Facilitating easy extension for new indicators or personalities.
|
||
|
||
The system runs as a single-threaded daemon on one or two local machines, handling ~600 messages/minute without high concurrency needs.
|
||
|
||
#### 2. System Architecture
|
||
The architecture follows a layered, event-driven design inspired by high-performance trading systems: input, processing, and output. It uses Python 3 for prototyping, with pandas for data manipulation and TA-Lib for indicators.
|
||
|
||
- **Input Layer**: Connects to `candles.db` and `analysis.db` using WAL mode for concurrent reads. Polls every 0.5-1s, fetching the last 200-500 rows per timeframe to compute trends.
|
||
- **Processing Layer**: Loads data into pandas DataFrames, computes additional indicators (e.g., Stochastic), applies rules, and scores signals (0-1 confidence based on confluence).
|
||
- **Output Layer**: Streams JSON signals (e.g., {"signal": "BUY", "timeframe": "1m", "confidence": 0.75}) via UDS for lowest latency (~130μs RTT); fallback to TCP with NODELAY for dual machines.
|
||
- **Configuration**: YAML file for personalities, thresholds (e.g., RSI<30), and polling intervals.
|
||
- **Monitoring**: Logs to file; optional metrics for signal frequency.
|
||
|
||
This setup ensures determinism—no repainting—and handles your throughput reduction goal by filtering to 1-10 signals/hour.
|
||
|
||
#### 3. Data Integration and Handling
|
||
Leverage your existing schemas: Join `candles` and `analysis` tables on timeframe/timestamp for unified DataFrames. Use TA-Lib for Stochastic (%K=14, %D=3) and extend precomputed indicators (EMA9/21, RSI14, MACD, Bollinger, volume MA20).
|
||
|
||
Example Fetch Function:
|
||
```python
|
||
import sqlite3
|
||
import pandas as pd
|
||
import talib
|
||
|
||
def fetch_and_enrich(timeframe, limit=200, candles_db='candles.db', analysis_db='analysis.db'):
|
||
conn_c = sqlite3.connect(candles_db, timeout=10)
|
||
conn_a = sqlite3.connect(analysis_db, timeout=10)
|
||
query = """
|
||
SELECT c.*, a.*
|
||
FROM candles c
|
||
JOIN analysis a ON c.timeframe = a.timeframe AND c.timestamp = a.timestamp
|
||
WHERE c.timeframe = ?
|
||
ORDER BY c.timestamp DESC
|
||
LIMIT ?
|
||
"""
|
||
df = pd.read_sql_query(query, conn_c, params=(timeframe, limit))
|
||
conn_c.close(); conn_a.close()
|
||
df['datetime'] = pd.to_datetime(df['timestamp'], unit='s')
|
||
df = df.sort_values('timestamp').reset_index(drop=True)
|
||
# Add Stochastic
|
||
df['stoch_k'], df['stoch_d'] = talib.STOCH(df['high'], df['low'], df['close'], fastk_period=14, slowk_period=3, slowd_period=3)
|
||
return df
|
||
```
|
||
|
||
Filter for closed candles (timestamp < current time) to avoid instability.
|
||
|
||
#### 4. Indicators and Signal Logic
|
||
Build on your `analysis.db` with rule-based logic for confluence. Key indicators:
|
||
- MA Crossings: EMA9 > EMA21 (buy); reverse for sell.
|
||
- RSI: <30 (oversold, buy); >70 (overbought, sell).
|
||
- MACD: Line > signal (buy).
|
||
- Bollinger Squeeze: bb_squeeze=1 + breakout.
|
||
- Stochastic: %K > %D and <20 (buy).
|
||
- Volume: >1.5 * MA20; buy_ratio >0.6.
|
||
|
||
Confidence: Weighted sum (e.g., 0.3 for MA, 0.2 for volume) >0.6 triggers signal.
|
||
|
||
Example Logic:
|
||
```python
|
||
def generate_signal(df, personality='scalping'):
|
||
latest = df.iloc[-1]
|
||
score = 0
|
||
reasons = []
|
||
if personality == 'scalping':
|
||
if latest['ema_9'] > latest['ema_21'] and latest['stoch_k'] > latest['stoch_d'] and latest['stoch_k'] < 30:
|
||
score += 0.4; reasons.append('EMA crossover + Stochastic buy')
|
||
if latest['rsi_14'] < 50 and latest['volume'] > 1.5 * latest['volume_ma_20']:
|
||
score += 0.3; reasons.append('RSI undersold + volume surge')
|
||
elif personality == 'swing':
|
||
if latest['close'] > latest['sma_50'] and latest['bb_squeeze'] == 1 and latest['close'] > latest['bb_upper']:
|
||
score += 0.4; reasons.append('Regime bull + BB breakout')
|
||
if latest['macd'] > latest['macd_signal'] and latest['buy_volume'] / latest['volume'] > 0.55:
|
||
score += 0.3; reasons.append('MACD crossover + buy bias')
|
||
if score > 0.6:
|
||
return {'signal': 'BUY' if score > 0 else 'SELL', 'confidence': score, 'reasons': reasons}
|
||
return None
|
||
```
|
||
|
||
#### 5. Trading Personalities
|
||
Tailor strategies to crypto volatility.
|
||
|
||
| Personality | Timeframes | Focus Indicators | Trade Frequency | Risk Notes |
|
||
|-------------|------------|------------------|-----------------|------------|
|
||
| Scalping | 1m, 5m | EMA9/21, Stochastic, MACD hist, volume | 10-50/day | High fees/slippage; aim 0.25-1% per trade, 65% win rate possible with BB/RSI. |
|
||
| Swing | 15m, 1h | SMA50/200, BB squeeze, RSI, net flow | 2-10/week | Lower stress; target 5-10% moves, suits trends but lags in ranges. |
|
||
|
||
Switch via config; add filters like regime bias (close > SMA200 for longs).
|
||
|
||
#### 6. Streaming and Bot Integration
|
||
Use UDS for polling-based feed: Generator writes signals; bot polls every 50-100ms. Defer stops/positioning to bot.
|
||
|
||
Server Code:
|
||
```python
|
||
import socket
|
||
import json
|
||
import time
|
||
|
||
SOCKET_PATH = '/tmp/signal.sock'
|
||
server = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||
try: os.remove(SOCKET_PATH)
|
||
except: pass
|
||
server.bind(SOCKET_PATH)
|
||
server.listen(1)
|
||
conn, _ = server.accept()
|
||
while True:
|
||
signal = generate_signal(fetch_and_enrich('1m')) # Example
|
||
if signal:
|
||
conn.sendall(json.dumps(signal).encode() + b'\n')
|
||
time.sleep(0.5) # Poll interval
|
||
```
|
||
|
||
Bot polls non-blockingly.
|
||
|
||
#### 7. Implementation Plan
|
||
- **Dependencies**: pandas, sqlite3, talib, backtesting.py.
|
||
- **Phases**: 1) Setup DB connections (1 day); 2) Indicator/signal logic (2-3 days); 3) Streaming (1 day); 4) Backtesting (2 days); 5) Testing/deployment (1-2 days).
|
||
- **Code Structure**: Main loop in `signal_generator.py`; separate modules for data, logic, config.
|
||
|
||
#### 8. Backtesting and Validation
|
||
Use pandas for simple backtests or Backtesting.py for advanced. Simulate slippage (0.1%) and fees; track Sharpe ratio, drawdown.
|
||
|
||
Example with Backtesting.py:
|
||
```python
|
||
from backtesting import Backtest, Strategy
|
||
|
||
class ScalpStrategy(Strategy):
|
||
def init(self):
|
||
self.ema9 = self.I(talib.EMA, self.data.Close, 9)
|
||
self.stoch_k, self.stoch_d = self.I(talib.STOCH, self.data.High, self.data.Low, self.data.Close)
|
||
def next(self):
|
||
if crossover(self.ema9, self.ema21) and self.stoch_k > self.stoch_d and self.stoch_k < 30:
|
||
self.buy()
|
||
|
||
bt = Backtest(df, ScalpStrategy, cash=10000, commission=.001)
|
||
stats = bt.run()
|
||
print(stats)
|
||
```
|
||
|
||
Aim for positive expectancy; re-optimize thresholds quarterly.
|
||
|
||
#### 9. Deployment and Maintenance
|
||
Run as systemd service; monitor with logs. Vacuum DBs monthly. Scale by adding personalities or ML later.
|
||
|
||
#### 10. Risks and Mitigations
|
||
- **Market Risks**: Crypto volatility; mitigate with volume filters and backtesting.
|
||
- **Technical Risks**: DB locks; use timeouts. Overfitting; use out-of-sample data.
|
||
- **Performance**: CPU spikes from polling; adaptive intervals.
|
||
- **Disclaimer**: Signals for paper trading only; not financial advice.
|
||
|
||
This plan equips you to build a robust system, validated by industry practices.
|
||
|
||
### Key Citations
|
||
- [Creating a Custom Python Library for Trading Indicators and Signal Generation - Medium](https://medium.com/@deepml1818/creating-a-custom-python-library-for-trading-indicators-and-signal-generation-2097d0d526f7)
|
||
- [How to build a macro trading strategy (with open-source Python) - Macrosynergy](https://macrosynergy.com/research/how-to-build-a-macro-trading-strategy-with-open-source-python)
|
||
- [Python in FinTech: Building High-Performance Trading Systems - CMARIX](https://www.cmarix.com/blog/build-trading-systems-with-python-in-fintech)
|
||
- [Python in High-Frequency Trading: Low-Latency Techniques - PyQuant News](https://www.pyquantnews.com/free-python-resources/python-in-high-frequency-trading-low-latency-techniques)
|
||
- [A Look at the Differences Between Scalping and Swing Trading in Crypto | by JeNovation](https://medium.com/@ByTrade.io/a-look-at-the-differences-between-scalping-and-swing-trading-in-crypto-d7a83623c277)
|
||
- [Day trade - swing or scalp? : r/BitcoinMarkets - Reddit](https://www.reddit.com/r/BitcoinMarkets/comments/1jklil5/day_trade_swing_or_scalp)
|
||
- [Best Crypto Trading Strategies (Beginner to Intermediate Guide) - Changelly](https://changelly.com/blog/cryptocurrency-trading-strategies)
|
||
- [Crypto Scalping vs Swing Trading: Which is More Profitable for Retail Investors?](https://www.blockchain-council.org/cryptocurrency/crypto-scalping-vs-swing-trading)
|
||
- [Crypto Scalping: What It Is, How It Works, Strategies & Tools (2025) - CryptoNinjas](https://www.cryptoninjas.net/crypto/cryptocurrency-scalping)
|
||
- [7 Best Crypto Trading Strategies for 2025 - CMC Markets](https://www.cmcmarkets.com/en/cryptocurrencies/7-crypto-trading-strategies)
|
||
- [I Tried 1-Minute Scalping on BTC — My Honest Results - InsiderFinance Wire](https://wire.insiderfinance.io/i-tried-1-minute-scalping-on-btc-my-honest-results-c80989f7335b)
|
||
- [Algorithmic Trading with Stochastic Oscillator in Python | by Nikhil Adithyan - Medium](https://medium.com/codex/algorithmic-trading-with-stochastic-oscillator-in-python-7e2bec49b60d)
|
||
- [Python Trading Strategy: Synergizing Stochastic Oscillator and MACD Indicator - EODHD](https://eodhd.com/financial-academy/backtesting-strategies-examples/using-python-to-create-an-innovative-trading-strategy-and-achieve-better-results)
|
||
- [Using the Stochastic Oscillator in Python for Algorithmic Trading - αlphαrithms](https://www.alpharithms.com/stochastic-oscillator-in-python-483214)
|
||
- [Momentum Indicators - Technical Analysis Library in Python's documentation!](https://technical-analysis-library-in-python.readthedocs.io/en/latest/ta.html)
|
||
- [Backtesting with Pandas and TA-lib - YouTube](https://www.youtube.com/watch?v=WBPnN8DIMYI)
|
||
- [Backtesting.py - Backtest trading strategies in Python](https://kernc.github.io/backtesting.py)
|
||
- [Mastering Python Backtesting for Trading Strategies | by Time Money Code | Medium](https://medium.com/@timemoneycode/mastering-python-backtesting-for-trading-strategies-1f7df773fdf5)
|
||
- [Python Libraries for Quantitative Trading | QuantStart](https://www.quantstart.com/articles/python-libraries-for-quantitative-trading) |