Bug fixes in input and onramp. Hot config reload on signals. Added example utility scripts for signals.
This commit is contained in:
Kalzu Rekku
173
input/README.md
Normal file
173
input/README.md
Normal file
@@ -0,0 +1,173 @@
|
||||
# WebSocket Streamer (JSONL Logger)
|
||||
|
||||
This Go program connects to a WebSocket endpoint, subscribes to a topic, and continuously writes incoming messages to hourly-rotated `.jsonl` files.
|
||||
It is designed for **long-running, low-overhead data capture** with basic observability and operational safety.
|
||||
|
||||
Typical use cases include:
|
||||
|
||||
* Market data capture (e.g. crypto trades)
|
||||
* Event stream archiving
|
||||
* Lightweight ingestion on small servers (VPS, Raspberry Pi, etc.)
|
||||
|
||||
---
|
||||
|
||||
## Features
|
||||
|
||||
* WebSocket client with automatic reconnect
|
||||
* Topic subscription (configurable)
|
||||
* Hourly file rotation (`.jsonl` format)
|
||||
* Buffered channel to decouple network I/O from disk writes
|
||||
* Atomic message counters
|
||||
* Periodic status logging
|
||||
* Unix domain socket for live status queries
|
||||
* Graceful shutdown on SIGINT / SIGTERM
|
||||
* Configurable logging (file and/or stdout)
|
||||
|
||||
---
|
||||
|
||||
## How It Works
|
||||
|
||||
1. Connects to a WebSocket endpoint
|
||||
2. Sends a subscription message for the configured topic
|
||||
3. Maintains connection with periodic `ping`
|
||||
4. Reads messages and pushes them into a buffered channel
|
||||
5. Writes messages line-by-line to hourly JSONL files
|
||||
6. Exposes runtime status via:
|
||||
|
||||
* Periodic log output
|
||||
* Unix socket query (`--status` mode)
|
||||
|
||||
---
|
||||
|
||||
## Output Format
|
||||
|
||||
Messages are written **verbatim** as received, one JSON object per line:
|
||||
|
||||
```
|
||||
output/
|
||||
├── publicTrade.BTCUSDT_1700000000.jsonl
|
||||
├── publicTrade.BTCUSDT_1700003600.jsonl
|
||||
└── ...
|
||||
```
|
||||
|
||||
Each file contains data for exactly one UTC hour.
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
The application is configured via a JSON file.
|
||||
|
||||
### Example `config.json`
|
||||
|
||||
```json
|
||||
{
|
||||
"output_dir": "./output",
|
||||
"topic": "publicTrade.BTCUSDT",
|
||||
"ws_url": "wss://stream.bybit.com/v5/public/linear",
|
||||
"buffer_size": 10000,
|
||||
"status_interval": 30,
|
||||
"log_file": "system.log",
|
||||
"log_to_stdout": false,
|
||||
"status_socket": "/tmp/streamer.sock"
|
||||
}
|
||||
```
|
||||
|
||||
### Configuration Fields
|
||||
|
||||
| Field | Description |
|
||||
| ----------------- | ----------------------------------- |
|
||||
| `output_dir` | Directory for JSONL output files |
|
||||
| `topic` | WebSocket subscription topic |
|
||||
| `ws_url` | WebSocket endpoint URL |
|
||||
| `buffer_size` | Size of internal message buffer |
|
||||
| `status_interval` | Seconds between status log messages |
|
||||
| `log_file` | Log file path |
|
||||
| `log_to_stdout` | Also log to stdout |
|
||||
| `status_socket` | Unix socket path for status queries |
|
||||
|
||||
Defaults are applied automatically if fields are omitted.
|
||||
|
||||
---
|
||||
|
||||
## Command Line Flags
|
||||
|
||||
| Flag | Description |
|
||||
| --------- | -------------------------------------------- |
|
||||
| `-config` | Path to config file (default: `config.json`) |
|
||||
| `-debug` | Force logs to stdout (overrides config) |
|
||||
| `-status` | Query running instance status and exit |
|
||||
|
||||
---
|
||||
|
||||
## Running the Streamer
|
||||
|
||||
```bash
|
||||
go run main.go -config config.json
|
||||
```
|
||||
|
||||
Or build a binary:
|
||||
|
||||
```bash
|
||||
go build -o streamer
|
||||
./streamer -config config.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Querying Runtime Status
|
||||
|
||||
While the streamer is running:
|
||||
|
||||
```bash
|
||||
./streamer -status -config config.json
|
||||
```
|
||||
|
||||
Example output:
|
||||
|
||||
```
|
||||
Uptime: 12m34s | Total Msgs: 152340 | Rate: 7260.12 msg/min
|
||||
```
|
||||
|
||||
This works via a Unix domain socket and does **not** interrupt the running process.
|
||||
|
||||
---
|
||||
|
||||
## Logging
|
||||
|
||||
* Logs are written to `log_file`
|
||||
* Optional stdout logging for debugging
|
||||
* Includes:
|
||||
|
||||
* Startup information
|
||||
* Connection errors and reconnects
|
||||
* Buffer overflow warnings
|
||||
* Periodic status summaries
|
||||
|
||||
---
|
||||
|
||||
## Graceful Shutdown
|
||||
|
||||
On `SIGINT` or `SIGTERM`:
|
||||
|
||||
* WebSocket connection closes
|
||||
* Status socket is removed
|
||||
* Current output file is flushed and closed
|
||||
|
||||
Safe to run under systemd, Docker, or supervisord.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
* Go 1.20+
|
||||
* [`github.com/gorilla/websocket`](https://github.com/gorilla/websocket)
|
||||
|
||||
---
|
||||
|
||||
## Notes & Design Choices
|
||||
|
||||
* **JSONL** is used for easy streaming, compression, and downstream processing
|
||||
* Hourly rotation avoids large files and simplifies retention policies
|
||||
* Unix socket status avoids HTTP overhead and exposed ports
|
||||
* Minimal memory footprint, suitable for low-end machines
|
||||
@@ -5,5 +5,7 @@
|
||||
"buffer_size": 10000,
|
||||
"log_file": "system.log",
|
||||
"log_to_stdout": false,
|
||||
"status_interval": 30
|
||||
"status_interval": 30,
|
||||
"gzip_after_hours": 6,
|
||||
"gzip_check_interval": 3000
|
||||
}
|
||||
|
||||
@@ -2,4 +2,4 @@ module input
|
||||
|
||||
go 1.25.0
|
||||
|
||||
require github.com/gorilla/websocket v1.5.3 // indirect
|
||||
require github.com/gorilla/websocket v1.5.3
|
||||
|
||||
@@ -2,19 +2,22 @@ package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"compress/gzip"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net"
|
||||
"os"
|
||||
"sort"
|
||||
"strconv"
|
||||
"os/signal"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/gorilla/websocket"
|
||||
)
|
||||
|
||||
@@ -233,6 +236,100 @@ func (s *Streamer) rotate(t time.Time) {
|
||||
return
|
||||
}
|
||||
s.currentFile = f
|
||||
|
||||
// After rotation, compress old files (keeping current and N-1 as plaintext)
|
||||
go s.compressOldFiles()
|
||||
}
|
||||
|
||||
func (s *Streamer) compressOldFiles() {
|
||||
entries, err := os.ReadDir(s.config.OutputDir)
|
||||
if err != nil {
|
||||
log.Printf("Gzip scan error: %v", err)
|
||||
return
|
||||
}
|
||||
|
||||
// Collect all .jsonl files with their timestamps
|
||||
type fileInfo struct {
|
||||
path string
|
||||
timestamp int64
|
||||
}
|
||||
var jsonlFiles []fileInfo
|
||||
|
||||
for _, e := range entries {
|
||||
if e.IsDir() {
|
||||
continue
|
||||
}
|
||||
|
||||
name := e.Name()
|
||||
if !strings.HasSuffix(name, ".jsonl") {
|
||||
continue
|
||||
}
|
||||
|
||||
// Extract timestamp from filename: topic_TIMESTAMP.jsonl
|
||||
parts := strings.Split(name, "_")
|
||||
if len(parts) < 2 {
|
||||
continue
|
||||
}
|
||||
|
||||
tsStr := strings.TrimSuffix(parts[len(parts)-1], ".jsonl")
|
||||
ts, err := strconv.ParseInt(tsStr, 10, 64)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
fullPath := filepath.Join(s.config.OutputDir, name)
|
||||
jsonlFiles = append(jsonlFiles, fileInfo{path: fullPath, timestamp: ts})
|
||||
}
|
||||
|
||||
// Sort by timestamp (newest first)
|
||||
sort.Slice(jsonlFiles, func(i, j int) bool {
|
||||
return jsonlFiles[i].timestamp > jsonlFiles[j].timestamp
|
||||
})
|
||||
|
||||
// Keep the 2 newest files (current + N-1) as plaintext, gzip the rest
|
||||
for i, fi := range jsonlFiles {
|
||||
if i < 2 {
|
||||
// Skip the 2 newest files
|
||||
continue
|
||||
}
|
||||
|
||||
if err := gzipFile(fi.path); err != nil {
|
||||
log.Printf("Gzip failed for %s: %v", filepath.Base(fi.path), err)
|
||||
} else {
|
||||
log.Printf("Compressed %s", filepath.Base(fi.path))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func gzipFile(path string) error {
|
||||
in, err := os.Open(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer in.Close()
|
||||
|
||||
outPath := path + ".gz"
|
||||
out, err := os.Create(outPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
gw := gzip.NewWriter(out)
|
||||
if _, err := io.Copy(gw, in); err != nil {
|
||||
gw.Close()
|
||||
out.Close()
|
||||
return err
|
||||
}
|
||||
|
||||
if err := gw.Close(); err != nil {
|
||||
out.Close()
|
||||
return err
|
||||
}
|
||||
if err := out.Close(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return os.Remove(path)
|
||||
}
|
||||
|
||||
func setupLogger(c *Config, debugFlag bool) {
|
||||
|
||||
@@ -1,75 +0,0 @@
|
||||
# Technical Design: Bybit WebSocket Streamer
|
||||
|
||||
1. System Overview
|
||||
|
||||
The software acts as a dedicated bridge between the Bybit V5 WebSocket API and a local filesystem. Its primary goal is to provide a "hot" data stream for downstream consumers who read from the disk every ~80ms.
|
||||
2. Core Architecture
|
||||
|
||||
The system follows a Producer-Consumer pattern to decouple network ingestion from disk I/O. This prevents disk latency spikes from causing packet loss on the WebSocket buffer.
|
||||
|
||||
Producer (WS Client): Manages the connection, sends heartbeats, and pushes raw messages into a high-speed queue.
|
||||
|
||||
Consumer (File Writer): Pulls messages from the queue, determines the target file, and writes data to disk.
|
||||
|
||||
3. Functional Components
|
||||
A. Connection Manager (Self-Healing)
|
||||
|
||||
To ensure "solid" performance, the manager must implement:
|
||||
|
||||
Heartbeat Mechanism: Send periodic ping messages to Bybit (usually every 20-30 seconds) to prevent idle timeouts.
|
||||
|
||||
Auto-Reconnect: On network loss or socket error, the client must automatically attempt to reconnect.
|
||||
|
||||
Exponential Backoff: To avoid spamming the API during an outage, use a delay formula:
|
||||
Delay=min(2attempts,MaxDelay)
|
||||
|
||||
State Tracking: Maintain the subscription state so it automatically re-subscribes to the topic upon reconnection.
|
||||
|
||||
B. Stream Processor & Memory Management
|
||||
|
||||
Streaming I/O: Messages must be handled as raw byte streams. Do not parse the JSON into deep objects unless necessary for validation, as this creates garbage collection overhead.
|
||||
|
||||
Bounded Buffer: The queue between the network and disk should have a fixed capacity. If the disk fails, the queue should drop old data rather than growing infinitely and crashing the system (RAM hoarding).
|
||||
|
||||
C. Atomic File Rotator
|
||||
|
||||
The rotator manages the lifecycle of the .jsonl files.
|
||||
|
||||
Naming Convention: {topic}_{unix_timestamp}.jsonl
|
||||
|
||||
Rotation Logic: On every message receive, compare the current system time against the active file's creation hour. If a new hour has begun:
|
||||
|
||||
Flush and Close the current file handle.
|
||||
|
||||
Open/Create the new file for the current hour.
|
||||
|
||||
Immediate Visibility: Because downstream programs read every 80ms, the writer must Flush the stream buffer immediately after writing each JSON line to ensure the data is visible to other processes without waiting for the OS buffer to fill.
|
||||
|
||||
4. Configuration Requirements
|
||||
|
||||
The software should read from a configuration file (YAML/JSON) or environment variables:
|
||||
Option Description Example
|
||||
OUTPUT_DIR Absolute path for data storage /data/bybit/
|
||||
WS_URL Bybit WebSocket endpoint wss://stream.bybit.com/v5/public/linear
|
||||
TOPIC Topic to subscribe to publicTrade.BTCUSDT
|
||||
ROTATION_SECONDS Interval for file creation 3600
|
||||
LOG_LEVEL Verbosity of internal logs INFO, DEBUG, ERROR
|
||||
5. Logging & Diagnostics
|
||||
|
||||
Operational logs must be written to a separate rolling log file (e.g., system.log) to track:
|
||||
|
||||
Connection Events: Timestamps of successful handshakes and disconnections.
|
||||
|
||||
Subscription Status: Confirmation of topic subscription.
|
||||
|
||||
Rotation Events: Filenames of newly created files.
|
||||
|
||||
Errors: Socket timeouts, disk full errors, or malformed JSON received from the exchange.
|
||||
|
||||
6. Implementation Considerations for the Programmer
|
||||
|
||||
Concurrency: Use non-blocking I/O or green threads (Goroutines, Asyncio, etc.) to ensure the heartbeat doesn't get stuck behind a slow disk write.
|
||||
|
||||
File Permissions: Ensure created files have read permissions for the downstream programs.
|
||||
|
||||
Graceful Shutdown: On SIGTERM, the software must flush all buffers and close the current file properly to avoid data corruption.
|
||||
@@ -16,6 +16,7 @@ from rich.text import Text
|
||||
|
||||
# --- CONFIGURATION ---
|
||||
INPUT_SOCKET = "/tmp/streamer.sock"
|
||||
SIGNALS_HEALTH_SOCKET = "/tmp/signals_health.sock"
|
||||
|
||||
ONRAMP_HOST = "127.0.0.1"
|
||||
ONRAMP_PORT = 9999
|
||||
@@ -44,6 +45,24 @@ def query_input_go():
|
||||
return None
|
||||
|
||||
|
||||
def query_signals_health():
|
||||
if not os.path.exists(SIGNALS_HEALTH_SOCKET):
|
||||
return None
|
||||
try:
|
||||
with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as s:
|
||||
s.settimeout(0.5)
|
||||
s.connect(SIGNALS_HEALTH_SOCKET)
|
||||
chunks = []
|
||||
while True:
|
||||
chunk = s.recv(4096)
|
||||
if not chunk:
|
||||
break
|
||||
chunks.append(chunk)
|
||||
return json.loads(b"".join(chunks).decode())
|
||||
except:
|
||||
return None
|
||||
|
||||
|
||||
def query_tcp_json(host, port, payload=b"status"):
|
||||
try:
|
||||
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
|
||||
@@ -69,14 +88,23 @@ def make_layout():
|
||||
layout = Layout()
|
||||
layout.split(
|
||||
Layout(name="header", size=3),
|
||||
Layout(name="main", size=10),
|
||||
Layout(name="services", size=18),
|
||||
Layout(name="signals_table", size=8),
|
||||
Layout(name="market", size=12),
|
||||
Layout(name="footer", size=3),
|
||||
)
|
||||
layout["main"].split_row(
|
||||
# Split services into 2x2 grid
|
||||
layout["services"].split_column(
|
||||
Layout(name="services_row1"),
|
||||
Layout(name="services_row2"),
|
||||
)
|
||||
layout["services_row1"].split_row(
|
||||
Layout(name="input_svc"),
|
||||
Layout(name="onramp_svc"),
|
||||
)
|
||||
layout["services_row2"].split_row(
|
||||
Layout(name="analyst_svc"),
|
||||
Layout(name="signals_svc"),
|
||||
)
|
||||
return layout
|
||||
|
||||
@@ -156,6 +184,93 @@ def get_analyst_panel():
|
||||
return Panel(content, title="[3] Analyst Service (Python)", border_style="magenta")
|
||||
|
||||
|
||||
def get_signals_panel():
|
||||
data = query_signals_health()
|
||||
if not data:
|
||||
return Panel(Text("OFFLINE", style="bold red"),
|
||||
title="[4] Signal Generator (Python)", border_style="red")
|
||||
|
||||
status_color = "green" if data.get("status") == "running" else "red"
|
||||
|
||||
content = Text()
|
||||
content.append(f"Status : ")
|
||||
content.append(f"{data.get('status', 'unknown').upper()}", style=f"bold {status_color}")
|
||||
content.append(f"\nPersonality : {data.get('personality', 'n/a')}\n")
|
||||
content.append(f"Timeframes : {', '.join(data.get('timeframes', []))}\n")
|
||||
content.append(f"Uptime : {data.get('uptime_seconds', 0)}s ({data.get('uptime_seconds', 0)//60}m)\n")
|
||||
|
||||
total = data.get('total_signals', 0)
|
||||
buy = data.get('buy_signals', 0)
|
||||
sell = data.get('sell_signals', 0)
|
||||
|
||||
content.append(f"Total Sigs : {total} (")
|
||||
content.append(f"↑{buy}", style="green")
|
||||
content.append(" / ")
|
||||
content.append(f"↓{sell}", style="red")
|
||||
content.append(")\n")
|
||||
|
||||
content.append(f"Errors : {data.get('errors', 0)}\n")
|
||||
content.append(f"Clients : {data.get('connected_clients', 0)}\n")
|
||||
|
||||
last_sig = data.get('last_signal')
|
||||
if last_sig:
|
||||
try:
|
||||
last_time = datetime.fromisoformat(last_sig.replace('Z', '+00:00'))
|
||||
time_ago = (datetime.now(last_time.tzinfo) - last_time).total_seconds()
|
||||
content.append(f"Last Signal : {int(time_ago)}s ago")
|
||||
except:
|
||||
content.append(f"Last Signal : {last_sig}")
|
||||
else:
|
||||
content.append(f"Last Signal : None")
|
||||
|
||||
border_color = "green" if data.get("status") == "running" else "red"
|
||||
return Panel(content, title="[4] Signal Generator (Python)", border_style=border_color)
|
||||
|
||||
|
||||
# ------------------- SIGNALS TABLE -------------------
|
||||
|
||||
def get_signals_table():
|
||||
data = query_signals_health()
|
||||
table = Table(expand=True, border_style="yellow", header_style="bold yellow")
|
||||
|
||||
table.add_column("Time", justify="center", style="dim")
|
||||
table.add_column("TF", justify="center")
|
||||
table.add_column("Signal", justify="center", style="bold")
|
||||
table.add_column("Price", justify="right")
|
||||
table.add_column("Conf", justify="right")
|
||||
table.add_column("Reason", justify="left", no_wrap=False)
|
||||
|
||||
if data and "recent_signals" in data and data["recent_signals"]:
|
||||
for sig in data["recent_signals"][-5:]: # Last 5 signals
|
||||
try:
|
||||
sig_time = datetime.fromisoformat(sig['generated_at'].replace('Z', '+00:00'))
|
||||
time_str = sig_time.strftime('%H:%M:%S')
|
||||
except:
|
||||
time_str = "n/a"
|
||||
|
||||
signal_type = sig.get('signal', '?')
|
||||
signal_color = "green" if signal_type == "BUY" else "red"
|
||||
|
||||
# Take first 2 reasons
|
||||
reasons = sig.get('reasons', [])
|
||||
reason_str = ", ".join(reasons[:2])
|
||||
if len(reasons) > 2:
|
||||
reason_str += "..."
|
||||
|
||||
table.add_row(
|
||||
time_str,
|
||||
sig.get('timeframe', '?'),
|
||||
Text(signal_type, style=f"bold {signal_color}"),
|
||||
f"${sig.get('price', 0):.2f}",
|
||||
f"{sig.get('confidence', 0)*100:.0f}%",
|
||||
reason_str
|
||||
)
|
||||
else:
|
||||
table.add_row("—", "—", "—", "—", "—", "No signals yet")
|
||||
|
||||
return Panel(table, title="Recent Signals", border_style="yellow")
|
||||
|
||||
|
||||
# ------------------- MARKET TABLE -------------------
|
||||
|
||||
def get_market_table():
|
||||
@@ -194,39 +309,47 @@ def get_market_table():
|
||||
else:
|
||||
table.add_row("waiting...", "-", "-", "-", "-", "-", "-", "-")
|
||||
|
||||
return table
|
||||
return Panel(table, title="Live Market Data", border_style="cyan")
|
||||
|
||||
|
||||
# ------------------- MAIN -------------------
|
||||
|
||||
def main():
|
||||
layout = make_layout()
|
||||
with Live(layout, refresh_per_second=2, screen=True):
|
||||
while True:
|
||||
layout["header"].update(
|
||||
Panel(
|
||||
Text(
|
||||
f"BYBIT BTCUSDT UNIFIED MONITOR | {datetime.now().strftime('%H:%M:%S')}",
|
||||
justify="center",
|
||||
style="bold white on blue"
|
||||
try:
|
||||
with Live(layout, refresh_per_second=2, screen=True):
|
||||
while True:
|
||||
layout["header"].update(
|
||||
Panel(
|
||||
Text(
|
||||
f"BYBIT BTCUSDT UNIFIED MONITOR | {datetime.now().strftime('%H:%M:%S')}",
|
||||
justify="center",
|
||||
style="bold white on blue"
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
layout["input_svc"].update(get_input_panel())
|
||||
layout["onramp_svc"].update(get_onramp_panel())
|
||||
layout["analyst_svc"].update(get_analyst_panel())
|
||||
layout["market"].update(get_market_table())
|
||||
layout["input_svc"].update(get_input_panel())
|
||||
layout["onramp_svc"].update(get_onramp_panel())
|
||||
layout["analyst_svc"].update(get_analyst_panel())
|
||||
layout["signals_svc"].update(get_signals_panel())
|
||||
|
||||
layout["footer"].update(
|
||||
Text(
|
||||
"Ctrl+C to exit | Pipeline: Input → Onramp → Analyst",
|
||||
justify="center",
|
||||
style="dim"
|
||||
layout["signals_table"].update(get_signals_table())
|
||||
layout["market"].update(get_market_table())
|
||||
|
||||
layout["footer"].update(
|
||||
Text(
|
||||
"Ctrl+C to exit | Pipeline: Input → Onramp → Analyst → Signals",
|
||||
justify="center",
|
||||
style="dim"
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
time.sleep(REFRESH_RATE)
|
||||
time.sleep(REFRESH_RATE)
|
||||
except KeyboardInterrupt:
|
||||
console.print("\n[yellow]Shutting down monitor...[/yellow]")
|
||||
finally:
|
||||
console.print("[green]Monitor stopped.[/green]")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -211,9 +211,19 @@ func (a *Aggregator) serve(host string, port int) {
|
||||
var response interface{}
|
||||
a.mu.RLock()
|
||||
if cmd == "live" {
|
||||
// Deep copy the cache
|
||||
cacheCopy := make(map[string]map[int64]*Candle)
|
||||
for tf, candles := range a.cache {
|
||||
cacheCopy[tf] = make(map[int64]*Candle)
|
||||
for ts, candle := range candles {
|
||||
// Copy the candle struct
|
||||
candleCopy := *candle
|
||||
cacheCopy[tf][ts] = &candleCopy
|
||||
}
|
||||
}
|
||||
response = map[string]interface{}{
|
||||
"type": "live_candles",
|
||||
"data": a.cache,
|
||||
"data": cacheCopy,
|
||||
}
|
||||
} else {
|
||||
response = map[string]interface{}{
|
||||
|
||||
199
onramp/onramp.py
199
onramp/onramp.py
@@ -1,199 +0,0 @@
|
||||
import os
|
||||
import json
|
||||
import time
|
||||
import glob
|
||||
import sqlite3
|
||||
import logging
|
||||
import socket
|
||||
import threading
|
||||
from datetime import datetime
|
||||
from collections import defaultdict
|
||||
|
||||
# Setup Logging
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format='%(asctime)s [%(levelname)s] %(message)s',
|
||||
handlers=[logging.FileHandler("processor.log"), logging.StreamHandler()]
|
||||
)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
class CandleAggregator:
|
||||
def __init__(self, db_path):
|
||||
self.db_path = db_path
|
||||
self.lock = threading.Lock() # Ensure thread safety
|
||||
self.timeframes = {
|
||||
"1m": 60,
|
||||
"5m": 300,
|
||||
"15m": 900,
|
||||
"1h": 3600
|
||||
}
|
||||
self.init_db()
|
||||
# Cache structure: {timeframe: {timestamp: {data}}}
|
||||
self.cache = defaultdict(dict)
|
||||
|
||||
def init_db(self):
|
||||
with sqlite3.connect(self.db_path) as conn:
|
||||
conn.execute("""
|
||||
CREATE TABLE IF NOT EXISTS candles (
|
||||
timeframe TEXT,
|
||||
timestamp INTEGER,
|
||||
open REAL, high REAL, low REAL, close REAL,
|
||||
volume REAL, buy_volume REAL,
|
||||
PRIMARY KEY (timeframe, timestamp)
|
||||
)
|
||||
""")
|
||||
conn.commit()
|
||||
|
||||
def process_trade(self, ts_ms, price, volume, side):
|
||||
ts_s = ts_ms // 1000
|
||||
is_buy = 1 if side.lower() == "buy" else 0
|
||||
|
||||
with self.lock:
|
||||
for tf_name, seconds in self.timeframes.items():
|
||||
candle_ts = (ts_s // seconds) * seconds
|
||||
current = self.cache[tf_name].get(candle_ts)
|
||||
|
||||
if not current:
|
||||
current = {
|
||||
"timestamp": candle_ts,
|
||||
"open": price, "high": price, "low": price, "close": price,
|
||||
"volume": volume, "buy_volume": volume if is_buy else 0.0
|
||||
}
|
||||
else:
|
||||
current["high"] = max(current["high"], price)
|
||||
current["low"] = min(current["low"], price)
|
||||
current["close"] = price
|
||||
current["volume"] += volume
|
||||
if is_buy:
|
||||
current["buy_volume"] += volume
|
||||
|
||||
self.cache[tf_name][candle_ts] = current
|
||||
self.save_to_db(tf_name, candle_ts, current)
|
||||
|
||||
def save_to_db(self, timeframe, ts, data):
|
||||
try:
|
||||
with sqlite3.connect(self.db_path) as conn:
|
||||
conn.execute("""
|
||||
INSERT INTO candles (timeframe, timestamp, open, high, low, close, volume, buy_volume)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(timeframe, timestamp) DO UPDATE SET
|
||||
high = excluded.high, low = excluded.low, close = excluded.close,
|
||||
volume = excluded.volume, buy_volume = excluded.buy_volume
|
||||
""", (timeframe, ts, data['open'], data['high'], data['low'],
|
||||
data['close'], data['volume'], data['buy_volume']))
|
||||
except Exception as e:
|
||||
logger.error(f"Database error: {e}")
|
||||
|
||||
def get_live_snapshot(self):
|
||||
"""Returns the current state of all active candles in the cache."""
|
||||
with self.lock:
|
||||
# We return a copy to avoid dictionary size mutation errors during JSON serialization
|
||||
return json.loads(json.dumps(self.cache))
|
||||
|
||||
class StatusServer:
|
||||
def __init__(self, host, port, stats_ref, aggregator):
|
||||
self.host = host
|
||||
self.port = port
|
||||
self.stats = stats_ref
|
||||
self.aggregator = aggregator
|
||||
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
||||
self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
|
||||
|
||||
def start(self):
|
||||
self.sock.bind((self.host, self.port))
|
||||
self.sock.listen(5)
|
||||
threading.Thread(target=self._serve, daemon=True).start()
|
||||
logger.info(f"Network server started on {self.host}:{self.port}")
|
||||
|
||||
def _serve(self):
|
||||
while True:
|
||||
try:
|
||||
client, addr = self.sock.accept()
|
||||
# Set a timeout so a slow client doesn't hang the server
|
||||
client.settimeout(2.0)
|
||||
|
||||
# Receive command (e.g., "status" or "live")
|
||||
data = client.recv(1024).decode('utf-8').strip()
|
||||
|
||||
response = {}
|
||||
if data == "live":
|
||||
response = {
|
||||
"type": "live_candles",
|
||||
"data": self.aggregator.get_live_snapshot()
|
||||
}
|
||||
else:
|
||||
# Default to status info
|
||||
response = {
|
||||
"type": "status",
|
||||
"uptime_start": self.stats['start_time'],
|
||||
"last_file": self.stats['last_file'],
|
||||
"total_trades": self.stats['lines_count'],
|
||||
"last_ts": self.stats['last_ts']
|
||||
}
|
||||
|
||||
client.send(json.dumps(response).encode('utf-8'))
|
||||
client.close()
|
||||
except Exception as e:
|
||||
logger.error(f"Server error: {e}")
|
||||
|
||||
class FileTailer:
|
||||
def __init__(self, config):
|
||||
self.config = config
|
||||
self.aggregator = CandleAggregator(config['database_path'])
|
||||
self.stats = {
|
||||
"start_time": datetime.now().isoformat(),
|
||||
"last_file": None, "lines_count": 0, "last_ts": None
|
||||
}
|
||||
self.status_server = StatusServer(
|
||||
config['status_host'],
|
||||
config['status_port'],
|
||||
self.stats,
|
||||
self.aggregator
|
||||
)
|
||||
|
||||
def get_latest_file(self):
|
||||
path_pattern = os.path.join(self.config['input_directory'], self.config['file_pattern'])
|
||||
files = sorted(glob.glob(path_pattern))
|
||||
return files[-1] if files else None
|
||||
|
||||
def run(self):
|
||||
self.status_server.start()
|
||||
current_file_path = self.get_latest_file()
|
||||
last_position = 0
|
||||
|
||||
while True:
|
||||
newest_file = self.get_latest_file()
|
||||
if newest_file and newest_file != current_file_path:
|
||||
logger.info(f"Rotating to: {newest_file}")
|
||||
current_file_path = newest_file
|
||||
last_position = 0
|
||||
|
||||
if current_file_path and os.path.exists(current_file_path):
|
||||
self.stats['last_file'] = current_file_path
|
||||
with open(current_file_path, 'r') as f:
|
||||
f.seek(last_position)
|
||||
while True:
|
||||
line = f.readline()
|
||||
if not line: break
|
||||
self.process_line(line)
|
||||
last_position = f.tell()
|
||||
|
||||
time.sleep(self.config['poll_interval_ms'] / 1000.0)
|
||||
|
||||
def process_line(self, line):
|
||||
try:
|
||||
payload = json.loads(line)
|
||||
if "data" not in payload: return
|
||||
for trade in payload["data"]:
|
||||
self.aggregator.process_trade(
|
||||
trade["T"], float(trade["p"]), float(trade["v"]), trade["S"]
|
||||
)
|
||||
self.stats['last_ts'] = trade["T"]
|
||||
self.stats['lines_count'] += 1
|
||||
except Exception as e:
|
||||
logger.error(f"Line processing error: {e}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
with open("config.json", "r") as cf:
|
||||
conf = json.load(cf)
|
||||
FileTailer(conf).run()
|
||||
196
signals/DESIGN.md
Normal file
196
signals/DESIGN.md
Normal file
@@ -0,0 +1,196 @@
|
||||
### Key Points on Signal Generator Design
|
||||
- **Core Functionality**: The system processes real-time BTCUSDT data from SQLite databases to generate buy/sell signals based on technical indicators like EMAs, RSI, MACD, Bollinger Bands, and Stochastic Oscillator, supporting scalping (short-term, high-frequency) and swing (medium-term, trend-focused) personalities.
|
||||
- **Low-Latency Focus**: Uses Unix Domain Sockets (UDS) for local streaming to minimize delays (typically 130-200μs RTT), ideal for your 0.5-1.7s pipeline lag and single-trade limit.
|
||||
- **Implementation Choice**: Python 3 with pandas, TA-Lib, and sqlite3 for rapid development; incorporate backtesting via pandas or Backtesting.py to validate strategies before live use.
|
||||
- **Risk Considerations**: Strategies like scalping may yield quick wins but face higher fees and noise; swing trading offers better risk-reward in trends but requires patience—backtest extensively to assess win rates (often 50-60%).
|
||||
|
||||
### System Overview
|
||||
This design outlines a modular signal generator that ingests data from `candles.db` and `analysis.db`, computes signals using rule-based logic, and streams them to a paper trading bot. It emphasizes determinism, low overhead, and extensibility for your local setup.
|
||||
|
||||
### Architecture Components
|
||||
- **Input**: Polls databases every 0.5-1s for closed candles.
|
||||
- **Processing**: Applies personality-specific rules with confluence scoring.
|
||||
- **Output**: JSON signals via UDS (fallback TCP for dual machines).
|
||||
|
||||
### Recommended Strategies
|
||||
For scalping: Focus on 1m/5m timeframes with EMA crossovers and Stochastic. For swing: Use 15m/1h with regime filters and squeezes. Validate via backtesting to ensure profitability.
|
||||
|
||||
---
|
||||
### Comprehensive Design and Plan for BTCUSDT Signal Generator
|
||||
|
||||
#### 1. Introduction and Objectives
|
||||
The signal generator is a critical component in your Bybit BTCUSDT websocket data pipeline, transforming aggregated OHLCV and precomputed indicators into actionable buy/sell signals for a paper trading bot. This document provides a ready-to-use blueprint, incorporating best practices from Python-based trading systems, such as modular code for indicators, efficient data handling with pandas, and low-latency communication. Objectives include:
|
||||
- Supporting "scalping" (high-frequency, short-term trades) and "swing" (medium-term, trend-capturing) personalities.
|
||||
- Ensuring sub-second latency for signal delivery, aligning with your 0.5-1.7s pipeline.
|
||||
- Enabling backtesting to evaluate strategy performance, targeting win rates of 50-60% in crypto markets.
|
||||
- Facilitating easy extension for new indicators or personalities.
|
||||
|
||||
The system runs as a single-threaded daemon on one or two local machines, handling ~600 messages/minute without high concurrency needs.
|
||||
|
||||
#### 2. System Architecture
|
||||
The architecture follows a layered, event-driven design inspired by high-performance trading systems: input, processing, and output. It uses Python 3 for prototyping, with pandas for data manipulation and TA-Lib for indicators.
|
||||
|
||||
- **Input Layer**: Connects to `candles.db` and `analysis.db` using WAL mode for concurrent reads. Polls every 0.5-1s, fetching the last 200-500 rows per timeframe to compute trends.
|
||||
- **Processing Layer**: Loads data into pandas DataFrames, computes additional indicators (e.g., Stochastic), applies rules, and scores signals (0-1 confidence based on confluence).
|
||||
- **Output Layer**: Streams JSON signals (e.g., {"signal": "BUY", "timeframe": "1m", "confidence": 0.75}) via UDS for lowest latency (~130μs RTT); fallback to TCP with NODELAY for dual machines.
|
||||
- **Configuration**: YAML file for personalities, thresholds (e.g., RSI<30), and polling intervals.
|
||||
- **Monitoring**: Logs to file; optional metrics for signal frequency.
|
||||
|
||||
This setup ensures determinism—no repainting—and handles your throughput reduction goal by filtering to 1-10 signals/hour.
|
||||
|
||||
#### 3. Data Integration and Handling
|
||||
Leverage your existing schemas: Join `candles` and `analysis` tables on timeframe/timestamp for unified DataFrames. Use TA-Lib for Stochastic (%K=14, %D=3) and extend precomputed indicators (EMA9/21, RSI14, MACD, Bollinger, volume MA20).
|
||||
|
||||
Example Fetch Function:
|
||||
```python
|
||||
import sqlite3
|
||||
import pandas as pd
|
||||
import talib
|
||||
|
||||
def fetch_and_enrich(timeframe, limit=200, candles_db='candles.db', analysis_db='analysis.db'):
|
||||
conn_c = sqlite3.connect(candles_db, timeout=10)
|
||||
conn_a = sqlite3.connect(analysis_db, timeout=10)
|
||||
query = """
|
||||
SELECT c.*, a.*
|
||||
FROM candles c
|
||||
JOIN analysis a ON c.timeframe = a.timeframe AND c.timestamp = a.timestamp
|
||||
WHERE c.timeframe = ?
|
||||
ORDER BY c.timestamp DESC
|
||||
LIMIT ?
|
||||
"""
|
||||
df = pd.read_sql_query(query, conn_c, params=(timeframe, limit))
|
||||
conn_c.close(); conn_a.close()
|
||||
df['datetime'] = pd.to_datetime(df['timestamp'], unit='s')
|
||||
df = df.sort_values('timestamp').reset_index(drop=True)
|
||||
# Add Stochastic
|
||||
df['stoch_k'], df['stoch_d'] = talib.STOCH(df['high'], df['low'], df['close'], fastk_period=14, slowk_period=3, slowd_period=3)
|
||||
return df
|
||||
```
|
||||
|
||||
Filter for closed candles (timestamp < current time) to avoid instability.
|
||||
|
||||
#### 4. Indicators and Signal Logic
|
||||
Build on your `analysis.db` with rule-based logic for confluence. Key indicators:
|
||||
- MA Crossings: EMA9 > EMA21 (buy); reverse for sell.
|
||||
- RSI: <30 (oversold, buy); >70 (overbought, sell).
|
||||
- MACD: Line > signal (buy).
|
||||
- Bollinger Squeeze: bb_squeeze=1 + breakout.
|
||||
- Stochastic: %K > %D and <20 (buy).
|
||||
- Volume: >1.5 * MA20; buy_ratio >0.6.
|
||||
|
||||
Confidence: Weighted sum (e.g., 0.3 for MA, 0.2 for volume) >0.6 triggers signal.
|
||||
|
||||
Example Logic:
|
||||
```python
|
||||
def generate_signal(df, personality='scalping'):
|
||||
latest = df.iloc[-1]
|
||||
score = 0
|
||||
reasons = []
|
||||
if personality == 'scalping':
|
||||
if latest['ema_9'] > latest['ema_21'] and latest['stoch_k'] > latest['stoch_d'] and latest['stoch_k'] < 30:
|
||||
score += 0.4; reasons.append('EMA crossover + Stochastic buy')
|
||||
if latest['rsi_14'] < 50 and latest['volume'] > 1.5 * latest['volume_ma_20']:
|
||||
score += 0.3; reasons.append('RSI undersold + volume surge')
|
||||
elif personality == 'swing':
|
||||
if latest['close'] > latest['sma_50'] and latest['bb_squeeze'] == 1 and latest['close'] > latest['bb_upper']:
|
||||
score += 0.4; reasons.append('Regime bull + BB breakout')
|
||||
if latest['macd'] > latest['macd_signal'] and latest['buy_volume'] / latest['volume'] > 0.55:
|
||||
score += 0.3; reasons.append('MACD crossover + buy bias')
|
||||
if score > 0.6:
|
||||
return {'signal': 'BUY' if score > 0 else 'SELL', 'confidence': score, 'reasons': reasons}
|
||||
return None
|
||||
```
|
||||
|
||||
#### 5. Trading Personalities
|
||||
Tailor strategies to crypto volatility.
|
||||
|
||||
| Personality | Timeframes | Focus Indicators | Trade Frequency | Risk Notes |
|
||||
|-------------|------------|------------------|-----------------|------------|
|
||||
| Scalping | 1m, 5m | EMA9/21, Stochastic, MACD hist, volume | 10-50/day | High fees/slippage; aim 0.25-1% per trade, 65% win rate possible with BB/RSI. |
|
||||
| Swing | 15m, 1h | SMA50/200, BB squeeze, RSI, net flow | 2-10/week | Lower stress; target 5-10% moves, suits trends but lags in ranges. |
|
||||
|
||||
Switch via config; add filters like regime bias (close > SMA200 for longs).
|
||||
|
||||
#### 6. Streaming and Bot Integration
|
||||
Use UDS for polling-based feed: Generator writes signals; bot polls every 50-100ms. Defer stops/positioning to bot.
|
||||
|
||||
Server Code:
|
||||
```python
|
||||
import socket
|
||||
import json
|
||||
import time
|
||||
|
||||
SOCKET_PATH = '/tmp/signal.sock'
|
||||
server = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||||
try: os.remove(SOCKET_PATH)
|
||||
except: pass
|
||||
server.bind(SOCKET_PATH)
|
||||
server.listen(1)
|
||||
conn, _ = server.accept()
|
||||
while True:
|
||||
signal = generate_signal(fetch_and_enrich('1m')) # Example
|
||||
if signal:
|
||||
conn.sendall(json.dumps(signal).encode() + b'\n')
|
||||
time.sleep(0.5) # Poll interval
|
||||
```
|
||||
|
||||
Bot polls non-blockingly.
|
||||
|
||||
#### 7. Implementation Plan
|
||||
- **Dependencies**: pandas, sqlite3, talib, backtesting.py.
|
||||
- **Phases**: 1) Setup DB connections (1 day); 2) Indicator/signal logic (2-3 days); 3) Streaming (1 day); 4) Backtesting (2 days); 5) Testing/deployment (1-2 days).
|
||||
- **Code Structure**: Main loop in `signal_generator.py`; separate modules for data, logic, config.
|
||||
|
||||
#### 8. Backtesting and Validation
|
||||
Use pandas for simple backtests or Backtesting.py for advanced. Simulate slippage (0.1%) and fees; track Sharpe ratio, drawdown.
|
||||
|
||||
Example with Backtesting.py:
|
||||
```python
|
||||
from backtesting import Backtest, Strategy
|
||||
|
||||
class ScalpStrategy(Strategy):
|
||||
def init(self):
|
||||
self.ema9 = self.I(talib.EMA, self.data.Close, 9)
|
||||
self.stoch_k, self.stoch_d = self.I(talib.STOCH, self.data.High, self.data.Low, self.data.Close)
|
||||
def next(self):
|
||||
if crossover(self.ema9, self.ema21) and self.stoch_k > self.stoch_d and self.stoch_k < 30:
|
||||
self.buy()
|
||||
|
||||
bt = Backtest(df, ScalpStrategy, cash=10000, commission=.001)
|
||||
stats = bt.run()
|
||||
print(stats)
|
||||
```
|
||||
|
||||
Aim for positive expectancy; re-optimize thresholds quarterly.
|
||||
|
||||
#### 9. Deployment and Maintenance
|
||||
Run as systemd service; monitor with logs. Vacuum DBs monthly. Scale by adding personalities or ML later.
|
||||
|
||||
#### 10. Risks and Mitigations
|
||||
- **Market Risks**: Crypto volatility; mitigate with volume filters and backtesting.
|
||||
- **Technical Risks**: DB locks; use timeouts. Overfitting; use out-of-sample data.
|
||||
- **Performance**: CPU spikes from polling; adaptive intervals.
|
||||
- **Disclaimer**: Signals for paper trading only; not financial advice.
|
||||
|
||||
This plan equips you to build a robust system, validated by industry practices.
|
||||
|
||||
### Key Citations
|
||||
- [Creating a Custom Python Library for Trading Indicators and Signal Generation - Medium](https://medium.com/@deepml1818/creating-a-custom-python-library-for-trading-indicators-and-signal-generation-2097d0d526f7)
|
||||
- [How to build a macro trading strategy (with open-source Python) - Macrosynergy](https://macrosynergy.com/research/how-to-build-a-macro-trading-strategy-with-open-source-python)
|
||||
- [Python in FinTech: Building High-Performance Trading Systems - CMARIX](https://www.cmarix.com/blog/build-trading-systems-with-python-in-fintech)
|
||||
- [Python in High-Frequency Trading: Low-Latency Techniques - PyQuant News](https://www.pyquantnews.com/free-python-resources/python-in-high-frequency-trading-low-latency-techniques)
|
||||
- [A Look at the Differences Between Scalping and Swing Trading in Crypto | by JeNovation](https://medium.com/@ByTrade.io/a-look-at-the-differences-between-scalping-and-swing-trading-in-crypto-d7a83623c277)
|
||||
- [Day trade - swing or scalp? : r/BitcoinMarkets - Reddit](https://www.reddit.com/r/BitcoinMarkets/comments/1jklil5/day_trade_swing_or_scalp)
|
||||
- [Best Crypto Trading Strategies (Beginner to Intermediate Guide) - Changelly](https://changelly.com/blog/cryptocurrency-trading-strategies)
|
||||
- [Crypto Scalping vs Swing Trading: Which is More Profitable for Retail Investors?](https://www.blockchain-council.org/cryptocurrency/crypto-scalping-vs-swing-trading)
|
||||
- [Crypto Scalping: What It Is, How It Works, Strategies & Tools (2025) - CryptoNinjas](https://www.cryptoninjas.net/crypto/cryptocurrency-scalping)
|
||||
- [7 Best Crypto Trading Strategies for 2025 - CMC Markets](https://www.cmcmarkets.com/en/cryptocurrencies/7-crypto-trading-strategies)
|
||||
- [I Tried 1-Minute Scalping on BTC — My Honest Results - InsiderFinance Wire](https://wire.insiderfinance.io/i-tried-1-minute-scalping-on-btc-my-honest-results-c80989f7335b)
|
||||
- [Algorithmic Trading with Stochastic Oscillator in Python | by Nikhil Adithyan - Medium](https://medium.com/codex/algorithmic-trading-with-stochastic-oscillator-in-python-7e2bec49b60d)
|
||||
- [Python Trading Strategy: Synergizing Stochastic Oscillator and MACD Indicator - EODHD](https://eodhd.com/financial-academy/backtesting-strategies-examples/using-python-to-create-an-innovative-trading-strategy-and-achieve-better-results)
|
||||
- [Using the Stochastic Oscillator in Python for Algorithmic Trading - αlphαrithms](https://www.alpharithms.com/stochastic-oscillator-in-python-483214)
|
||||
- [Momentum Indicators - Technical Analysis Library in Python's documentation!](https://technical-analysis-library-in-python.readthedocs.io/en/latest/ta.html)
|
||||
- [Backtesting with Pandas and TA-lib - YouTube](https://www.youtube.com/watch?v=WBPnN8DIMYI)
|
||||
- [Backtesting.py - Backtest trading strategies in Python](https://kernc.github.io/backtesting.py)
|
||||
- [Mastering Python Backtesting for Trading Strategies | by Time Money Code | Medium](https://medium.com/@timemoneycode/mastering-python-backtesting-for-trading-strategies-1f7df773fdf5)
|
||||
- [Python Libraries for Quantitative Trading | QuantStart](https://www.quantstart.com/articles/python-libraries-for-quantitative-trading)
|
||||
15
signals/Pipfile
Normal file
15
signals/Pipfile
Normal file
@@ -0,0 +1,15 @@
|
||||
[[source]]
|
||||
url = "https://pypi.org/simple"
|
||||
verify_ssl = true
|
||||
name = "pypi"
|
||||
|
||||
[packages]
|
||||
pandas = "*"
|
||||
numpy = "*"
|
||||
ta-lib = "*"
|
||||
black = "*"
|
||||
|
||||
[dev-packages]
|
||||
|
||||
[requires]
|
||||
python_version = "3.13"
|
||||
405
signals/README.md
Normal file
405
signals/README.md
Normal file
@@ -0,0 +1,405 @@
|
||||
# BTCUSDT Signal Generator
|
||||
|
||||
Production-ready signal generation service for the Bybit BTCUSDT paper trading pipeline.
|
||||
|
||||
## Features
|
||||
|
||||
- **Dual Personality Support**: Scalping (1m/5m) and Swing (15m/1h) strategies
|
||||
- **Real-time Processing**: Sub-second polling of candles.db and analysis.db
|
||||
- **Unix Socket Streaming**: Low-latency signal delivery (~130μs RTT)
|
||||
- **Health Check Interface**: Separate socket for status monitoring
|
||||
- **Comprehensive Logging**: File and stdout logging with detailed diagnostics
|
||||
- **Signal Cooldown**: Prevents signal spam with configurable cooldown periods
|
||||
- **Graceful Shutdown**: SIGINT/SIGTERM handling with proper cleanup
|
||||
- **Error Recovery**: Robust error handling with automatic recovery
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌──────────────┐ ┌──────────────┐
|
||||
│ candles.db │────▶│ │
|
||||
└──────────────┘ │ Signal │ ┌─────────────────┐
|
||||
│ Generator │────▶│ Unix Socket │
|
||||
┌──────────────┐ │ │ │ /tmp/signals.sock│
|
||||
│ analysis.db │────▶│ │ └─────────────────┘
|
||||
└──────────────┘ └──────────────┘ │
|
||||
│ ▼
|
||||
│ ┌─────────────────┐
|
||||
│ │ Trading Bot │
|
||||
▼ └─────────────────┘
|
||||
┌─────────────────┐
|
||||
│ Health Socket │
|
||||
│ /tmp/signals_ │
|
||||
│ health.sock │
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
### 1. Install TA-Lib System Library
|
||||
|
||||
**Ubuntu/Debian:**
|
||||
```bash
|
||||
wget http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz
|
||||
tar -xzf ta-lib-0.4.0-src.tar.gz
|
||||
cd ta-lib/
|
||||
./configure --prefix=/usr
|
||||
make
|
||||
sudo make install
|
||||
```
|
||||
|
||||
**macOS:**
|
||||
```bash
|
||||
brew install ta-lib
|
||||
```
|
||||
|
||||
### 2. Install Python Dependencies
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Edit `config.json` to customize behavior:
|
||||
|
||||
### Key Parameters
|
||||
|
||||
| Parameter | Description | Default |
|
||||
|-----------|-------------|---------|
|
||||
| `personality` | Trading style: "scalping" or "swing" | "scalping" |
|
||||
| `timeframes` | List of timeframes to monitor | ["1m", "5m"] |
|
||||
| `poll_interval` | Seconds between checks | 0.5 |
|
||||
| `min_confidence` | Minimum confidence threshold (0-1) | 0.6 |
|
||||
| `cooldown_seconds` | Cooldown between signals | 60 |
|
||||
| `lookback` | Candles to fetch for analysis | 200 |
|
||||
|
||||
### Personality Configurations
|
||||
|
||||
**Scalping** (Short-term, high-frequency):
|
||||
- Timeframes: 1m, 5m
|
||||
- Focus: EMA crossovers, Stochastic, volume surges
|
||||
- Best for: Quick moves, scalp trades
|
||||
- Signal frequency: Higher (10-50/day)
|
||||
|
||||
**Swing** (Medium-term, trend-following):
|
||||
- Timeframes: 15m, 1h
|
||||
- Focus: Regime filters, Bollinger squeezes, MACD
|
||||
- Best for: Trend captures, larger moves
|
||||
- Signal frequency: Lower (2-10/week)
|
||||
|
||||
### Signal Weights
|
||||
|
||||
Adjust weights in `config.json` to tune signal generation:
|
||||
|
||||
```json
|
||||
"weights": {
|
||||
"scalping": {
|
||||
"ema_cross": 0.3, // EMA 9/21 crossover
|
||||
"stoch": 0.25, // Stochastic oscillator
|
||||
"rsi": 0.2, // RSI oversold/overbought
|
||||
"volume": 0.15, // Volume surge
|
||||
"macd": 0.1 // MACD confirmation
|
||||
},
|
||||
"swing": {
|
||||
"regime": 0.35, // SMA 50/200 filter
|
||||
"bb_squeeze": 0.25, // Bollinger squeeze
|
||||
"macd": 0.2, // MACD crossover
|
||||
"flow": 0.15, // Buy/sell pressure
|
||||
"rsi": 0.05 // RSI filter
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Running
|
||||
|
||||
### Start the Generator
|
||||
|
||||
```bash
|
||||
python signal_generator.py
|
||||
```
|
||||
|
||||
Or with custom config:
|
||||
```bash
|
||||
python signal_generator.py -c custom_config.json
|
||||
```
|
||||
|
||||
### Run as Background Service
|
||||
|
||||
```bash
|
||||
nohup python signal_generator.py > /dev/null 2>&1 &
|
||||
```
|
||||
|
||||
### Systemd Service (Recommended)
|
||||
|
||||
Create `/etc/systemd/system/signal-generator.service`:
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=BTCUSDT Signal Generator
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=youruser
|
||||
WorkingDirectory=/path/to/bybitbtc/signals
|
||||
ExecStart=/usr/bin/python3 signal_generator.py
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
Then:
|
||||
```bash
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable signal-generator
|
||||
sudo systemctl start signal-generator
|
||||
sudo systemctl status signal-generator
|
||||
```
|
||||
|
||||
## Health Checks
|
||||
|
||||
### Query Current Status
|
||||
|
||||
```bash
|
||||
python health_check.py
|
||||
```
|
||||
|
||||
Output example:
|
||||
```
|
||||
============================================================
|
||||
SIGNAL GENERATOR HEALTH STATUS
|
||||
============================================================
|
||||
Status: running
|
||||
Personality: scalping
|
||||
Timeframes: 1m, 5m
|
||||
Uptime: 3456s (57m)
|
||||
Total Signals: 23
|
||||
- Buy: 14
|
||||
- Sell: 9
|
||||
Last Signal: 2026-01-15T14:32:11.234Z
|
||||
Errors: 0
|
||||
Connected Clients: 1
|
||||
|
||||
Recent Signals:
|
||||
[1m] BUY @ $43250.50 (conf: 0.75)
|
||||
Reasons: EMA9 crossed above EMA21, Stochastic oversold crossover
|
||||
[5m] SELL @ $43180.25 (conf: 0.68)
|
||||
Reasons: EMA9 crossed below EMA21, RSI oversold (62.3)
|
||||
============================================================
|
||||
```
|
||||
|
||||
### Monitor Logs
|
||||
|
||||
```bash
|
||||
tail -f logs/signal_generator.log
|
||||
```
|
||||
|
||||
## Testing the Signal Stream
|
||||
|
||||
### Connect Test Client
|
||||
|
||||
```bash
|
||||
python test_client.py
|
||||
```
|
||||
|
||||
This will connect and display all incoming signals in real-time:
|
||||
|
||||
```
|
||||
======================================================================
|
||||
[2026-01-15 14:32:11] BUY SIGNAL
|
||||
======================================================================
|
||||
Timeframe: 1m
|
||||
Price: $43250.50
|
||||
Confidence: 75.0%
|
||||
Personality: scalping
|
||||
Reasons:
|
||||
• EMA9 crossed above EMA21
|
||||
• Stochastic oversold crossover
|
||||
• Volume surge detected
|
||||
======================================================================
|
||||
```
|
||||
|
||||
### Integration with Trading Bot
|
||||
|
||||
Your trading bot should connect to `/tmp/signals.sock`:
|
||||
|
||||
```python
|
||||
import socket
|
||||
import json
|
||||
|
||||
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||||
sock.connect("/tmp/signals.sock")
|
||||
|
||||
buffer = ""
|
||||
while True:
|
||||
chunk = sock.recv(4096).decode('utf-8')
|
||||
buffer += chunk
|
||||
|
||||
while '\n' in buffer:
|
||||
line, buffer = buffer.split('\n', 1)
|
||||
if line.strip():
|
||||
signal = json.loads(line)
|
||||
# Process signal
|
||||
print(f"Received: {signal['signal']} @ ${signal['price']}")
|
||||
```
|
||||
|
||||
## Signal Format
|
||||
|
||||
Each signal is a JSON object with the following structure:
|
||||
|
||||
```json
|
||||
{
|
||||
"signal": "BUY",
|
||||
"timeframe": "1m",
|
||||
"confidence": 0.75,
|
||||
"price": 43250.50,
|
||||
"timestamp": 1705329131,
|
||||
"reasons": [
|
||||
"EMA9 crossed above EMA21",
|
||||
"Stochastic oversold crossover",
|
||||
"Volume surge detected"
|
||||
],
|
||||
"personality": "scalping",
|
||||
"generated_at": "2026-01-15T14:32:11.234567Z"
|
||||
}
|
||||
```
|
||||
|
||||
## Signal Logic
|
||||
|
||||
### Scalping Strategy
|
||||
|
||||
Generates signals on:
|
||||
- **EMA Crossover**: 9/21 cross with direction confirmation
|
||||
- **Stochastic**: Oversold (<30) or overbought (>70) with crossover
|
||||
- **RSI**: Confirmation filter (<40 buy, >60 sell)
|
||||
- **Volume**: 1.5x average indicates momentum
|
||||
- **MACD**: Directional confirmation
|
||||
|
||||
### Swing Strategy
|
||||
|
||||
Generates signals on:
|
||||
- **Regime Filter**: Price vs SMA 50/200 trend structure
|
||||
- **BB Squeeze**: Volatility compression + breakout
|
||||
- **MACD**: Crossover for momentum shift
|
||||
- **Flow**: Buy/sell pressure ratio (55% threshold)
|
||||
- **RSI**: Light filter to avoid extremes
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
### Reduce Signal Frequency
|
||||
|
||||
Increase `min_confidence`:
|
||||
```json
|
||||
"min_confidence": 0.75
|
||||
```
|
||||
|
||||
Increase `cooldown_seconds`:
|
||||
```json
|
||||
"cooldown_seconds": 300
|
||||
```
|
||||
|
||||
### Increase Sensitivity
|
||||
|
||||
Lower confidence threshold:
|
||||
```json
|
||||
"min_confidence": 0.5
|
||||
```
|
||||
|
||||
Reduce cooldown:
|
||||
```json
|
||||
"cooldown_seconds": 30
|
||||
```
|
||||
|
||||
### Change Polling Rate
|
||||
|
||||
Faster updates (higher CPU):
|
||||
```json
|
||||
"poll_interval": 0.2
|
||||
```
|
||||
|
||||
Slower updates (lower CPU):
|
||||
```json
|
||||
"poll_interval": 1.0
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Socket not found" Error
|
||||
|
||||
Check if generator is running:
|
||||
```bash
|
||||
python health_check.py
|
||||
```
|
||||
|
||||
Check for socket file:
|
||||
```bash
|
||||
ls -la /tmp/signals*.sock
|
||||
```
|
||||
|
||||
### No Signals Generated
|
||||
|
||||
1. Check database connections:
|
||||
```bash
|
||||
sqlite3 ../onramp/candles.db "SELECT COUNT(*) FROM candles WHERE timeframe='1m';"
|
||||
```
|
||||
|
||||
2. Lower confidence threshold temporarily:
|
||||
```json
|
||||
"min_confidence": 0.4
|
||||
```
|
||||
|
||||
3. Check logs for errors:
|
||||
```bash
|
||||
grep ERROR logs/signal_generator.log
|
||||
```
|
||||
|
||||
### Database Locked Errors
|
||||
|
||||
Ensure WAL mode is enabled (automatic in code):
|
||||
```bash
|
||||
sqlite3 ../onramp/candles.db "PRAGMA journal_mode=WAL;"
|
||||
```
|
||||
|
||||
### High CPU Usage
|
||||
|
||||
Increase poll interval:
|
||||
```json
|
||||
"poll_interval": 1.0
|
||||
```
|
||||
|
||||
Reduce lookback:
|
||||
```json
|
||||
"lookback": 100
|
||||
```
|
||||
|
||||
## Maintenance
|
||||
|
||||
### Log Rotation
|
||||
|
||||
Use logrotate or manually clear:
|
||||
```bash
|
||||
echo "" > logs/signal_generator.log
|
||||
```
|
||||
|
||||
### Database Optimization
|
||||
|
||||
The generator reads in WAL mode and doesn't modify data, so no maintenance needed on its side.
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Backtest**: Use historical data to validate signal quality
|
||||
2. **Tune Weights**: Adjust based on paper trading results
|
||||
3. **Add Personalities**: Create custom strategies (e.g., "momentum", "mean-reversion")
|
||||
4. **ML Integration**: Replace rule-based logic with trained models
|
||||
5. **Multi-Asset**: Extend to other trading pairs
|
||||
|
||||
## Notes
|
||||
|
||||
- Signals are based on **closed candles only** - no repainting risk
|
||||
- The latest forming candle is automatically excluded
|
||||
- All timestamps are UTC
|
||||
- Confidence scores are additive from multiple indicators
|
||||
- Cooldown prevents signal spam per timeframe/personality combination
|
||||
31
signals/config.json
Normal file
31
signals/config.json
Normal file
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"candles_db": "../onramp/market_data.db",
|
||||
"analysis_db": "../analysis/analysis.db",
|
||||
"socket_path": "/tmp/signals.sock",
|
||||
"health_socket_path": "/tmp/signals_health.sock",
|
||||
"control_socket_path": "/tmp/signals_control.sock",
|
||||
"log_file": "logs/signal_generator.log",
|
||||
"log_to_stdout": true,
|
||||
"poll_interval": 0.5,
|
||||
"personality": "scalping",
|
||||
"timeframes": ["1m", "5m"],
|
||||
"lookback": 200,
|
||||
"min_confidence": 0.45,
|
||||
"cooldown_seconds": 30,
|
||||
"weights": {
|
||||
"scalping": {
|
||||
"ema_cross": 0.25,
|
||||
"stoch": 0.2,
|
||||
"rsi": 0.2,
|
||||
"volume": 0.2,
|
||||
"macd": 0.15
|
||||
},
|
||||
"swing": {
|
||||
"regime": 0.35,
|
||||
"bb_squeeze": 0.25,
|
||||
"macd": 0.2,
|
||||
"flow": 0.15,
|
||||
"rsi": 0.05
|
||||
}
|
||||
}
|
||||
}
|
||||
83
signals/example_client.py
Executable file
83
signals/example_client.py
Executable file
@@ -0,0 +1,83 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test client for signal generator
|
||||
Connects to Unix socket and prints incoming signals
|
||||
"""
|
||||
|
||||
import socket
|
||||
import sys
|
||||
import json
|
||||
from datetime import datetime
|
||||
|
||||
|
||||
def listen_signals(socket_path="/tmp/signals.sock"):
|
||||
"""Connect and listen for signals"""
|
||||
print(f"Connecting to {socket_path}...")
|
||||
|
||||
try:
|
||||
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||||
sock.connect(socket_path)
|
||||
print("Connected! Listening for signals...\n")
|
||||
|
||||
buffer = ""
|
||||
while True:
|
||||
chunk = sock.recv(4096).decode("utf-8")
|
||||
if not chunk:
|
||||
print("Connection closed by server")
|
||||
break
|
||||
|
||||
buffer += chunk
|
||||
|
||||
# Process complete messages (newline-delimited)
|
||||
while "\n" in buffer:
|
||||
line, buffer = buffer.split("\n", 1)
|
||||
if line.strip():
|
||||
try:
|
||||
signal = json.loads(line)
|
||||
print_signal(signal)
|
||||
except json.JSONDecodeError as e:
|
||||
print(f"Invalid JSON: {e}")
|
||||
|
||||
sock.close()
|
||||
|
||||
except FileNotFoundError:
|
||||
print(f"Error: Socket not found at {socket_path}")
|
||||
print("Is the signal generator running?")
|
||||
return 1
|
||||
except ConnectionRefusedError:
|
||||
print(f"Error: Connection refused at {socket_path}")
|
||||
return 1
|
||||
except KeyboardInterrupt:
|
||||
print("\nDisconnecting...")
|
||||
return 0
|
||||
except Exception as e:
|
||||
print(f"Error: {e}")
|
||||
return 1
|
||||
|
||||
|
||||
def print_signal(signal):
|
||||
"""Pretty print a signal"""
|
||||
timestamp = datetime.fromisoformat(signal["generated_at"])
|
||||
|
||||
# Color coding
|
||||
color = "\033[92m" if signal["signal"] == "BUY" else "\033[91m"
|
||||
reset = "\033[0m"
|
||||
|
||||
print(f"{color}{'=' * 70}{reset}")
|
||||
print(
|
||||
f"{color}[{timestamp.strftime('%Y-%m-%d %H:%M:%S')}] "
|
||||
f"{signal['signal']} SIGNAL{reset}"
|
||||
)
|
||||
print(f"Timeframe: {signal['timeframe']}")
|
||||
print(f"Price: ${signal['price']:.2f}")
|
||||
print(f"Confidence: {signal['confidence']:.1%}")
|
||||
print(f"Personality: {signal['personality']}")
|
||||
print(f"Reasons:")
|
||||
for reason in signal["reasons"]:
|
||||
print(f" • {reason}")
|
||||
print(f"{color}{'=' * 70}{reset}\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
socket_path = sys.argv[1] if len(sys.argv) > 1 else "/tmp/signals.sock"
|
||||
sys.exit(listen_signals(socket_path))
|
||||
115
signals/signal_health_client.py
Executable file
115
signals/signal_health_client.py
Executable file
@@ -0,0 +1,115 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Health Check Client for Signal Generator
|
||||
Query the running signal generator status
|
||||
"""
|
||||
|
||||
import socket
|
||||
import sys
|
||||
import json
|
||||
|
||||
def check_health(socket_path="/tmp/signals_health.sock"):
|
||||
"""Query signal generator health status"""
|
||||
try:
|
||||
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||||
sock.settimeout(5)
|
||||
sock.connect(socket_path)
|
||||
|
||||
# Receive response
|
||||
response = b""
|
||||
while True:
|
||||
chunk = sock.recv(4096)
|
||||
if not chunk:
|
||||
break
|
||||
response += chunk
|
||||
|
||||
sock.close()
|
||||
|
||||
# Parse and display
|
||||
health = json.loads(response.decode('utf-8'))
|
||||
|
||||
print("=" * 60)
|
||||
print("SIGNAL GENERATOR HEALTH STATUS")
|
||||
print("=" * 60)
|
||||
print(f"Status: {health['status']}")
|
||||
print(f"Personality: {health['personality']}")
|
||||
print(f"Timeframes: {', '.join(health['timeframes'])}")
|
||||
print(f"Uptime: {health['uptime_seconds']}s ({health['uptime_seconds']//60}m)")
|
||||
print(f"Total Signals: {health['total_signals']}")
|
||||
print(f" - Buy: {health['buy_signals']}")
|
||||
print(f" - Sell: {health['sell_signals']}")
|
||||
print(f"Last Signal: {health['last_signal'] or 'None'}")
|
||||
print(f"Errors: {health['errors']}")
|
||||
print(f"Connected Clients: {health['connected_clients']}")
|
||||
|
||||
# Database Status
|
||||
print("\n" + "=" * 70)
|
||||
print("DATABASE STATUS")
|
||||
print("=" * 70)
|
||||
|
||||
db = health.get('databases', {})
|
||||
|
||||
# Candles DB
|
||||
candles = db.get('candles_db', {})
|
||||
print(f"Candles DB: {'✓ OK' if candles.get('accessible') else '✗ FAILED'}")
|
||||
if candles.get('error'):
|
||||
print(f" Error: {candles['error']}")
|
||||
else:
|
||||
print(f" Total Rows: {candles.get('row_count', 0)}")
|
||||
for tf, info in candles.get('timeframes', {}).items():
|
||||
age = info.get('age_seconds')
|
||||
age_str = f"{age}s ago" if age else "N/A"
|
||||
print(f" [{tf}]: {info['count']} rows, latest: {age_str}")
|
||||
|
||||
# Analysis DB
|
||||
analysis = db.get('analysis_db', {})
|
||||
print(f"Analysis DB: {'✓ OK' if analysis.get('accessible') else '✗ FAILED'}")
|
||||
if analysis.get('error'):
|
||||
print(f" Error: {analysis['error']}")
|
||||
else:
|
||||
print(f" Total Rows: {analysis.get('row_count', 0)}")
|
||||
for tf, info in analysis.get('timeframes', {}).items():
|
||||
age = info.get('age_seconds')
|
||||
age_str = f"{age}s ago" if age else "N/A"
|
||||
print(f" [{tf}]: {info['count']} rows, latest: {age_str}")
|
||||
|
||||
# Configuration
|
||||
print("\n" + "=" * 70)
|
||||
print("CONFIGURATION")
|
||||
print("=" * 70)
|
||||
cfg = health.get('config', {})
|
||||
print(f"Min Confidence: {cfg.get('min_confidence', 'N/A')}")
|
||||
print(f"Cooldown: {cfg.get('cooldown_seconds', 'N/A')}s")
|
||||
print(f"Lookback: {cfg.get('lookback', 'N/A')} candles")
|
||||
if cfg.get('weights'):
|
||||
print(f"Weights:")
|
||||
for indicator, weight in cfg['weights'].items():
|
||||
print(f" {indicator:12s} {weight}")
|
||||
|
||||
if health['recent_signals']:
|
||||
print("\n" + "=" * 60)
|
||||
print("RECENT SIGNALS")
|
||||
print("=" * 60)
|
||||
for sig in health['recent_signals']:
|
||||
print(f" [{sig['timeframe']}] {sig['signal']} @ ${sig['price']:.2f} "
|
||||
f"(conf: {sig['confidence']:.2f})")
|
||||
print(f" Reasons: {', '.join(sig['reasons'])}")
|
||||
|
||||
print("=" * 60)
|
||||
|
||||
return 0
|
||||
|
||||
except FileNotFoundError:
|
||||
print(f"Error: Socket not found at {socket_path}")
|
||||
print("Is the signal generator running?")
|
||||
return 1
|
||||
except ConnectionRefusedError:
|
||||
print(f"Error: Connection refused at {socket_path}")
|
||||
return 1
|
||||
except Exception as e:
|
||||
print(f"Error: {e}")
|
||||
return 1
|
||||
|
||||
if __name__ == "__main__":
|
||||
socket_path = sys.argv[1] if len(sys.argv) > 1 else "/tmp/signals_health.sock"
|
||||
sys.exit(check_health(socket_path))
|
||||
155
signals/signal_live_config.py
Executable file
155
signals/signal_live_config.py
Executable file
@@ -0,0 +1,155 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Control Client for Signal Generator
|
||||
Send runtime commands to the signal generator
|
||||
"""
|
||||
|
||||
import socket
|
||||
import sys
|
||||
import json
|
||||
|
||||
|
||||
def send_command(command: dict, socket_path="/tmp/signals_control.sock"):
|
||||
"""Send a command to the signal generator"""
|
||||
try:
|
||||
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||||
sock.settimeout(5)
|
||||
sock.connect(socket_path)
|
||||
|
||||
# Send command
|
||||
sock.sendall(json.dumps(command).encode('utf-8'))
|
||||
|
||||
# Receive response
|
||||
response = b""
|
||||
while True:
|
||||
chunk = sock.recv(4096)
|
||||
if not chunk:
|
||||
break
|
||||
response += chunk
|
||||
|
||||
sock.close()
|
||||
|
||||
# Parse and display
|
||||
result = json.loads(response.decode('utf-8'))
|
||||
|
||||
if result.get("status") == "success":
|
||||
print(f"✓ {result['message']}")
|
||||
if "changes" in result:
|
||||
print("\nChanges:")
|
||||
for key, value in result["changes"].items():
|
||||
print(f" {key}: {value['old']} -> {value['new']}")
|
||||
else:
|
||||
print(f"✗ Error: {result.get('message', 'Unknown error')}")
|
||||
|
||||
return 0
|
||||
|
||||
except FileNotFoundError:
|
||||
print(f"Error: Socket not found at {socket_path}")
|
||||
print("Is the signal generator running?")
|
||||
return 1
|
||||
except ConnectionRefusedError:
|
||||
print(f"Error: Connection refused at {socket_path}")
|
||||
return 1
|
||||
except Exception as e:
|
||||
print(f"Error: {e}")
|
||||
return 1
|
||||
|
||||
|
||||
def print_usage():
|
||||
print("""
|
||||
Signal Generator Control Client
|
||||
|
||||
Usage:
|
||||
./signal_control_client.py <command> [arguments]
|
||||
|
||||
Commands:
|
||||
reload - Reload config.json
|
||||
personality <type> - Set personality (scalping/swing)
|
||||
confidence <value> - Set min confidence (0.0-1.0)
|
||||
cooldown <seconds> - Set cooldown period
|
||||
debug - Toggle debug logging
|
||||
clear-cooldowns - Clear all signal cooldowns
|
||||
reset-stats - Reset statistics
|
||||
|
||||
Examples:
|
||||
./signal_control_client.py reload
|
||||
./signal_control_client.py personality swing
|
||||
./signal_control_client.py confidence 0.45
|
||||
./signal_control_client.py cooldown 30
|
||||
./signal_control_client.py debug
|
||||
|
||||
You can also use SIGUSR1 and SIGUSR2:
|
||||
kill -USR1 <pid> # Reload config
|
||||
kill -USR2 <pid> # Toggle debug
|
||||
""")
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) < 2:
|
||||
print_usage()
|
||||
return 1
|
||||
|
||||
command = sys.argv[1].lower()
|
||||
|
||||
if command in ["help", "-h", "--help"]:
|
||||
print_usage()
|
||||
return 0
|
||||
|
||||
# Build command dictionary
|
||||
cmd = {}
|
||||
|
||||
if command == "reload":
|
||||
cmd = {"action": "reload"}
|
||||
|
||||
elif command == "personality":
|
||||
if len(sys.argv) < 3:
|
||||
print("Error: personality requires a value (scalping/swing)")
|
||||
return 1
|
||||
cmd = {"action": "set_personality", "value": sys.argv[2]}
|
||||
|
||||
elif command == "confidence":
|
||||
if len(sys.argv) < 3:
|
||||
print("Error: confidence requires a value (0.0-1.0)")
|
||||
return 1
|
||||
try:
|
||||
value = float(sys.argv[2])
|
||||
cmd = {"action": "set_confidence", "value": value}
|
||||
except ValueError:
|
||||
print("Error: confidence must be a number")
|
||||
return 1
|
||||
|
||||
elif command == "cooldown":
|
||||
if len(sys.argv) < 3:
|
||||
print("Error: cooldown requires a value in seconds")
|
||||
return 1
|
||||
try:
|
||||
value = int(sys.argv[2])
|
||||
cmd = {"action": "set_cooldown", "value": value}
|
||||
except ValueError:
|
||||
print("Error: cooldown must be an integer")
|
||||
return 1
|
||||
|
||||
elif command == "debug":
|
||||
cmd = {"action": "toggle_debug"}
|
||||
|
||||
elif command == "clear-cooldowns":
|
||||
cmd = {"action": "clear_cooldowns"}
|
||||
|
||||
elif command == "reset-stats":
|
||||
cmd = {"action": "reset_stats"}
|
||||
|
||||
else:
|
||||
print(f"Unknown command: {command}")
|
||||
print_usage()
|
||||
return 1
|
||||
|
||||
# Send command
|
||||
socket_path = "/tmp/signals_control.sock"
|
||||
if len(sys.argv) > 2 and sys.argv[-1].startswith("/"):
|
||||
socket_path = sys.argv[-1]
|
||||
|
||||
return send_command(cmd, socket_path)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
902
signals/signals.py
Executable file
902
signals/signals.py
Executable file
@@ -0,0 +1,902 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
BTCUSDT Signal Generator
|
||||
Generates trading signals from candles.db and analysis.db
|
||||
Streams signals via Unix Domain Socket
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
import talib
|
||||
import json
|
||||
import time
|
||||
import logging
|
||||
import signal
|
||||
import sys
|
||||
import os
|
||||
import socket
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Optional, Dict, List
|
||||
import threading
|
||||
from collections import deque
|
||||
|
||||
|
||||
# Configuration
|
||||
class Config:
|
||||
def __init__(self, config_path: str = "config.json"):
|
||||
self.config_path = config_path
|
||||
self.reload()
|
||||
|
||||
def reload(self):
|
||||
"""Reload configuration from file"""
|
||||
with open(self.config_path, "r") as f:
|
||||
data = json.load(f)
|
||||
|
||||
self.candles_db = data.get("candles_db", "../onramp/candles.db")
|
||||
self.analysis_db = data.get("analysis_db", "../analysis/analysis.db")
|
||||
self.socket_path = data.get("socket_path", "/tmp/signals.sock")
|
||||
self.health_socket_path = data.get(
|
||||
"health_socket_path", "/tmp/signals_health.sock"
|
||||
)
|
||||
self.control_socket_path = data.get(
|
||||
"control_socket_path", "/tmp/signals_control.sock"
|
||||
)
|
||||
self.log_file = data.get("log_file", "logs/signal_generator.log")
|
||||
self.log_to_stdout = data.get("log_to_stdout", True)
|
||||
self.poll_interval = data.get("poll_interval", 0.5)
|
||||
self.personality = data.get("personality", "scalping")
|
||||
self.timeframes = data.get("timeframes", ["1m", "5m"])
|
||||
self.lookback = data.get("lookback", 200)
|
||||
|
||||
# Signal thresholds
|
||||
self.min_confidence = data.get("min_confidence", 0.6)
|
||||
self.cooldown_seconds = data.get("cooldown_seconds", 60)
|
||||
|
||||
# Personality-specific weights
|
||||
self.weights = data.get(
|
||||
"weights",
|
||||
{
|
||||
"scalping": {
|
||||
"ema_cross": 0.3,
|
||||
"stoch": 0.25,
|
||||
"rsi": 0.2,
|
||||
"volume": 0.15,
|
||||
"macd": 0.1,
|
||||
},
|
||||
"swing": {
|
||||
"regime": 0.35,
|
||||
"bb_squeeze": 0.25,
|
||||
"macd": 0.2,
|
||||
"flow": 0.15,
|
||||
"rsi": 0.05,
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
# Logging setup
|
||||
def setup_logging(config: Config):
|
||||
Path(config.log_file).parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
handlers = [logging.FileHandler(config.log_file)]
|
||||
if config.log_to_stdout:
|
||||
handlers.append(logging.StreamHandler(sys.stdout))
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s [%(levelname)s] %(message)s",
|
||||
handlers=handlers,
|
||||
)
|
||||
return logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Signal Generator Class
|
||||
class SignalGenerator:
|
||||
def __init__(self, config: Config, logger: logging.Logger):
|
||||
self.config = config
|
||||
self.logger = logger
|
||||
self.running = False
|
||||
self.debug_mode = False
|
||||
self.last_signal_time = {}
|
||||
self.signal_history = deque(maxlen=100)
|
||||
self.stats = {
|
||||
"total_signals": 0,
|
||||
"buy_signals": 0,
|
||||
"sell_signals": 0,
|
||||
"last_signal_time": None,
|
||||
"uptime_start": datetime.now(timezone.utc),
|
||||
"errors": 0,
|
||||
"config_reloads": 0,
|
||||
}
|
||||
|
||||
# Unix socket
|
||||
self.socket = None
|
||||
self.connections = []
|
||||
|
||||
# Health check socket
|
||||
self.health_socket = None
|
||||
|
||||
# Control socket
|
||||
self.control_socket = None
|
||||
|
||||
def fetch_and_enrich(self, timeframe: str) -> Optional[pd.DataFrame]:
|
||||
"""Fetch data from databases and enrich with additional indicators"""
|
||||
try:
|
||||
conn_c = sqlite3.connect(
|
||||
f"file:{self.config.candles_db}?mode=ro",
|
||||
uri=True,
|
||||
timeout=10,
|
||||
)
|
||||
|
||||
# ATTACH the analysis database so we can JOIN across them
|
||||
conn_c.execute(
|
||||
f"ATTACH DATABASE 'file:{self.config.analysis_db}?mode=ro' AS analysis_db"
|
||||
)
|
||||
|
||||
query = """
|
||||
SELECT
|
||||
c.timeframe, c.timestamp, c.open, c.high, c.low, c.close,
|
||||
c.volume, c.buy_volume,
|
||||
a.ema_9, a.ema_21, a.sma_50, a.sma_200,
|
||||
a.rsi_14, a.macd, a.macd_signal, a.macd_hist,
|
||||
a.bb_upper, a.bb_middle, a.bb_lower, a.bb_squeeze,
|
||||
a.volume_ma_20
|
||||
FROM candles c
|
||||
JOIN analysis_db.analysis a
|
||||
ON c.timeframe = a.timeframe
|
||||
AND c.timestamp = a.timestamp
|
||||
WHERE c.timeframe = ?
|
||||
ORDER BY c.timestamp DESC
|
||||
LIMIT ?
|
||||
"""
|
||||
|
||||
df = pd.read_sql_query(
|
||||
query, conn_c, params=(timeframe, self.config.lookback)
|
||||
)
|
||||
|
||||
conn_c.close()
|
||||
|
||||
if df.empty:
|
||||
return None
|
||||
|
||||
# Sort chronologically
|
||||
df = df.sort_values("timestamp").reset_index(drop=True)
|
||||
df["datetime"] = pd.to_datetime(df["timestamp"], unit="s")
|
||||
|
||||
# Filter only closed candles (exclude current forming candle)
|
||||
current_time = int(time.time())
|
||||
if timeframe == "1m":
|
||||
window = 60
|
||||
elif timeframe == "5m":
|
||||
window = 300
|
||||
elif timeframe == "15m":
|
||||
window = 900
|
||||
elif timeframe == "1h":
|
||||
window = 3600
|
||||
else:
|
||||
window = 60
|
||||
|
||||
df = df[df["timestamp"] < (current_time - window)]
|
||||
|
||||
if len(df) < 50:
|
||||
self.logger.debug(f"Not enough data for {timeframe}: {len(df)} rows")
|
||||
return None
|
||||
|
||||
# Drop rows with NULL in critical columns
|
||||
df = df.dropna(subset=["open", "high", "low", "close", "volume"])
|
||||
|
||||
if len(df) < 50:
|
||||
self.logger.debug(
|
||||
f"Not enough valid data after NULL filtering for {timeframe}"
|
||||
)
|
||||
return None
|
||||
|
||||
# Add Stochastic Oscillator
|
||||
df["stoch_k"], df["stoch_d"] = talib.STOCH(
|
||||
df["high"].values,
|
||||
df["low"].values,
|
||||
df["close"].values,
|
||||
fastk_period=14,
|
||||
slowk_period=3,
|
||||
slowd_period=3,
|
||||
)
|
||||
|
||||
# Calculate buy/sell ratio
|
||||
df["buy_ratio"] = df["buy_volume"] / df["volume"].replace(0, np.nan)
|
||||
df["net_flow"] = df["buy_volume"] - (df["volume"] - df["buy_volume"])
|
||||
|
||||
return df
|
||||
|
||||
except Exception as e:
|
||||
self.logger.error(f"Error fetching data for {timeframe}: {e}")
|
||||
self.stats["errors"] += 1
|
||||
return None
|
||||
|
||||
def generate_signal_scalping(
|
||||
self, df: pd.DataFrame, timeframe: str
|
||||
) -> Optional[Dict]:
|
||||
"""Generate signal using scalping personality"""
|
||||
if len(df) < 21:
|
||||
self.logger.debug(f"[{timeframe}] Insufficient data: {len(df)} rows")
|
||||
return None
|
||||
|
||||
latest = df.iloc[-1]
|
||||
prev = df.iloc[-2]
|
||||
|
||||
# Check for NULL indicators - skip if essential indicators are missing
|
||||
required_cols = [
|
||||
"ema_9",
|
||||
"ema_21",
|
||||
"rsi_14",
|
||||
"stoch_k",
|
||||
"stoch_d",
|
||||
"macd",
|
||||
"macd_signal",
|
||||
]
|
||||
if any(pd.isna(latest[col]) for col in required_cols):
|
||||
self.logger.debug(f"[{timeframe}] Skipping: missing required indicators")
|
||||
return None
|
||||
|
||||
score = 0
|
||||
reasons = []
|
||||
signal_type = None
|
||||
|
||||
weights = self.config.weights["scalping"]
|
||||
|
||||
# EMA Crossover (9/21)
|
||||
ema_cross_up = (
|
||||
latest["ema_9"] > latest["ema_21"] and prev["ema_9"] <= prev["ema_21"]
|
||||
)
|
||||
ema_cross_down = (
|
||||
latest["ema_9"] < latest["ema_21"] and prev["ema_9"] >= prev["ema_21"]
|
||||
)
|
||||
|
||||
if ema_cross_up:
|
||||
score += weights["ema_cross"]
|
||||
reasons.append("EMA9 crossed above EMA21")
|
||||
signal_type = "BUY"
|
||||
elif ema_cross_down:
|
||||
score += weights["ema_cross"]
|
||||
reasons.append("EMA9 crossed below EMA21")
|
||||
signal_type = "SELL"
|
||||
|
||||
# Log EMA status for debugging
|
||||
self.logger.debug(
|
||||
f"[{timeframe}] EMA9={latest['ema_9']:.2f} vs EMA21={latest['ema_21']:.2f}, "
|
||||
f"Prev: EMA9={prev['ema_9']:.2f} vs EMA21={prev['ema_21']:.2f}"
|
||||
)
|
||||
|
||||
# Stochastic
|
||||
if signal_type == "BUY":
|
||||
if latest["stoch_k"] > latest["stoch_d"] and latest["stoch_k"] < 30:
|
||||
score += weights["stoch"]
|
||||
reasons.append("Stochastic oversold crossover")
|
||||
elif signal_type == "SELL":
|
||||
if latest["stoch_k"] < latest["stoch_d"] and latest["stoch_k"] > 70:
|
||||
score += weights["stoch"]
|
||||
reasons.append("Stochastic overbought crossover")
|
||||
|
||||
# RSI
|
||||
if signal_type == "BUY" and latest["rsi_14"] < 40:
|
||||
score += weights["rsi"]
|
||||
reasons.append(f"RSI undersold ({latest['rsi_14']:.1f})")
|
||||
elif signal_type == "SELL" and latest["rsi_14"] > 60:
|
||||
score += weights["rsi"]
|
||||
reasons.append(f"RSI oversold ({latest['rsi_14']:.1f})")
|
||||
|
||||
# Volume surge
|
||||
if latest["volume"] > 1.5 * latest["volume_ma_20"]:
|
||||
score += weights["volume"]
|
||||
reasons.append("Volume surge detected")
|
||||
|
||||
# MACD confirmation
|
||||
if signal_type == "BUY" and latest["macd"] > latest["macd_signal"]:
|
||||
score += weights["macd"]
|
||||
reasons.append("MACD bullish")
|
||||
elif signal_type == "SELL" and latest["macd"] < latest["macd_signal"]:
|
||||
score += weights["macd"]
|
||||
reasons.append("MACD bearish")
|
||||
|
||||
# Debug output
|
||||
if signal_type:
|
||||
self.logger.debug(
|
||||
f"[{timeframe}] Potential {signal_type} signal - Score: {score:.3f} "
|
||||
f"(threshold: {self.config.min_confidence}), Reasons: {len(reasons)}"
|
||||
)
|
||||
|
||||
if signal_type and score >= self.config.min_confidence:
|
||||
return {
|
||||
"signal": signal_type,
|
||||
"timeframe": timeframe,
|
||||
"confidence": round(score, 3),
|
||||
"price": float(latest["close"]),
|
||||
"timestamp": int(latest["timestamp"]),
|
||||
"reasons": reasons,
|
||||
"personality": "scalping",
|
||||
}
|
||||
|
||||
return None
|
||||
|
||||
def generate_signal_swing(self, df: pd.DataFrame, timeframe: str) -> Optional[Dict]:
|
||||
"""Generate signal using swing personality"""
|
||||
if len(df) < 200:
|
||||
return None
|
||||
|
||||
latest = df.iloc[-1]
|
||||
prev = df.iloc[-2]
|
||||
|
||||
# Check for NULL indicators - skip if essential indicators are missing
|
||||
required_cols = [
|
||||
"sma_50",
|
||||
"sma_200",
|
||||
"bb_upper",
|
||||
"bb_lower",
|
||||
"bb_squeeze",
|
||||
"macd",
|
||||
"macd_signal",
|
||||
"buy_ratio",
|
||||
]
|
||||
if any(pd.isna(latest[col]) for col in required_cols):
|
||||
self.logger.debug(f"Skipping {timeframe}: missing required indicators")
|
||||
return None
|
||||
|
||||
score = 0
|
||||
reasons = []
|
||||
signal_type = None
|
||||
|
||||
weights = self.config.weights["swing"]
|
||||
|
||||
# Regime filter (SMA50/200)
|
||||
bull_regime = latest["close"] > latest["sma_50"] > latest["sma_200"]
|
||||
bear_regime = latest["close"] < latest["sma_50"] < latest["sma_200"]
|
||||
|
||||
if bull_regime:
|
||||
signal_type = "BUY"
|
||||
score += weights["regime"]
|
||||
reasons.append("Bull regime (price > SMA50 > SMA200)")
|
||||
elif bear_regime:
|
||||
signal_type = "SELL"
|
||||
score += weights["regime"]
|
||||
reasons.append("Bear regime (price < SMA50 < SMA200)")
|
||||
|
||||
# Bollinger Squeeze breakout
|
||||
if latest["bb_squeeze"] == 1 or prev["bb_squeeze"] == 1:
|
||||
if signal_type == "BUY" and latest["close"] > latest["bb_upper"]:
|
||||
score += weights["bb_squeeze"]
|
||||
reasons.append("BB squeeze breakout to upside")
|
||||
elif signal_type == "SELL" and latest["close"] < latest["bb_lower"]:
|
||||
score += weights["bb_squeeze"]
|
||||
reasons.append("BB squeeze breakout to downside")
|
||||
|
||||
# MACD crossover
|
||||
if (
|
||||
signal_type == "BUY"
|
||||
and latest["macd"] > latest["macd_signal"]
|
||||
and prev["macd"] <= prev["macd_signal"]
|
||||
):
|
||||
score += weights["macd"]
|
||||
reasons.append("MACD bullish crossover")
|
||||
elif (
|
||||
signal_type == "SELL"
|
||||
and latest["macd"] < latest["macd_signal"]
|
||||
and prev["macd"] >= prev["macd_signal"]
|
||||
):
|
||||
score += weights["macd"]
|
||||
reasons.append("MACD bearish crossover")
|
||||
|
||||
# Net flow
|
||||
if signal_type == "BUY" and latest["buy_ratio"] > 0.55:
|
||||
score += weights["flow"]
|
||||
reasons.append(f"Buy pressure ({latest['buy_ratio']:.2%})")
|
||||
elif signal_type == "SELL" and latest["buy_ratio"] < 0.45:
|
||||
score += weights["flow"]
|
||||
reasons.append(f"Sell pressure ({latest['buy_ratio']:.2%})")
|
||||
|
||||
# RSI (light filter for swing)
|
||||
if signal_type == "BUY" and latest["rsi_14"] < 50:
|
||||
score += weights["rsi"]
|
||||
reasons.append("RSI not overbought")
|
||||
elif signal_type == "SELL" and latest["rsi_14"] > 50:
|
||||
score += weights["rsi"]
|
||||
reasons.append("RSI not oversold")
|
||||
|
||||
if signal_type and score >= self.config.min_confidence:
|
||||
return {
|
||||
"signal": signal_type,
|
||||
"timeframe": timeframe,
|
||||
"confidence": round(score, 3),
|
||||
"price": float(latest["close"]),
|
||||
"timestamp": int(latest["timestamp"]),
|
||||
"reasons": reasons,
|
||||
"personality": "swing",
|
||||
}
|
||||
|
||||
return None
|
||||
|
||||
def generate_signal(self, timeframe: str) -> Optional[Dict]:
|
||||
"""Main signal generation dispatcher"""
|
||||
# Check cooldown
|
||||
cooldown_key = f"{self.config.personality}_{timeframe}"
|
||||
if cooldown_key in self.last_signal_time:
|
||||
elapsed = time.time() - self.last_signal_time[cooldown_key]
|
||||
if elapsed < self.config.cooldown_seconds:
|
||||
return None
|
||||
|
||||
df = self.fetch_and_enrich(timeframe)
|
||||
if df is None:
|
||||
return None
|
||||
|
||||
if self.config.personality == "scalping":
|
||||
signal = self.generate_signal_scalping(df, timeframe)
|
||||
elif self.config.personality == "swing":
|
||||
signal = self.generate_signal_swing(df, timeframe)
|
||||
else:
|
||||
self.logger.error(f"Unknown personality: {self.config.personality}")
|
||||
return None
|
||||
|
||||
if signal:
|
||||
self.last_signal_time[cooldown_key] = time.time()
|
||||
signal["generated_at"] = datetime.now(timezone.utc).isoformat()
|
||||
|
||||
# Update stats
|
||||
self.stats["total_signals"] += 1
|
||||
if signal["signal"] == "BUY":
|
||||
self.stats["buy_signals"] += 1
|
||||
else:
|
||||
self.stats["sell_signals"] += 1
|
||||
self.stats["last_signal_time"] = signal["generated_at"]
|
||||
|
||||
self.signal_history.append(signal)
|
||||
|
||||
return signal
|
||||
|
||||
def broadcast_signal(self, signal: Dict):
|
||||
"""Broadcast signal to all connected clients"""
|
||||
message = json.dumps(signal) + "\n"
|
||||
message_bytes = message.encode("utf-8")
|
||||
|
||||
disconnected = []
|
||||
for conn in self.connections:
|
||||
try:
|
||||
conn.sendall(message_bytes)
|
||||
self.logger.info(
|
||||
f"Sent {signal['signal']} signal: {signal['timeframe']} @ {signal['price']} (conf: {signal['confidence']})"
|
||||
)
|
||||
except Exception as e:
|
||||
self.logger.warning(f"Failed to send to client: {e}")
|
||||
disconnected.append(conn)
|
||||
|
||||
# Remove disconnected clients
|
||||
for conn in disconnected:
|
||||
try:
|
||||
conn.close()
|
||||
except:
|
||||
pass
|
||||
self.connections.remove(conn)
|
||||
|
||||
def setup_signal_socket(self):
|
||||
"""Setup Unix domain socket for signal streaming"""
|
||||
try:
|
||||
if os.path.exists(self.config.socket_path):
|
||||
os.remove(self.config.socket_path)
|
||||
|
||||
self.socket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||||
self.socket.bind(self.config.socket_path)
|
||||
self.socket.listen(5)
|
||||
self.socket.settimeout(0.1) # Non-blocking accept
|
||||
|
||||
self.logger.info(f"Signal socket listening on {self.config.socket_path}")
|
||||
|
||||
except Exception as e:
|
||||
self.logger.error(f"Failed to setup signal socket: {e}")
|
||||
raise
|
||||
|
||||
def setup_health_socket(self):
|
||||
"""Setup Unix domain socket for health checks"""
|
||||
try:
|
||||
if os.path.exists(self.config.health_socket_path):
|
||||
os.remove(self.config.health_socket_path)
|
||||
|
||||
self.health_socket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||||
self.health_socket.bind(self.config.health_socket_path)
|
||||
self.health_socket.listen(5)
|
||||
self.health_socket.settimeout(0.1)
|
||||
|
||||
self.logger.info(
|
||||
f"Health socket listening on {self.config.health_socket_path}"
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
self.logger.error(f"Failed to setup health socket: {e}")
|
||||
raise
|
||||
|
||||
def setup_control_socket(self):
|
||||
"""Setup Unix domain socket for control commands"""
|
||||
try:
|
||||
if os.path.exists(self.config.control_socket_path):
|
||||
os.remove(self.config.control_socket_path)
|
||||
|
||||
self.control_socket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||||
self.control_socket.bind(self.config.control_socket_path)
|
||||
self.control_socket.listen(5)
|
||||
self.control_socket.settimeout(0.1)
|
||||
|
||||
self.logger.info(
|
||||
f"Control socket listening on {self.config.control_socket_path}"
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
self.logger.error(f"Failed to setup control socket: {e}")
|
||||
raise
|
||||
|
||||
def check_database_status(self) -> Dict:
|
||||
"""Check database connectivity and data availability"""
|
||||
status = {
|
||||
"candles_db": {"accessible": False, "row_count": 0, "timeframes": {}},
|
||||
"analysis_db": {"accessible": False, "row_count": 0, "timeframes": {}},
|
||||
}
|
||||
|
||||
# Check candles DB
|
||||
try:
|
||||
conn_c = sqlite3.connect(
|
||||
f"file:{self.config.candles_db}?mode=ro", uri=True, timeout=5
|
||||
)
|
||||
cursor = conn_c.cursor()
|
||||
|
||||
# Check total rows
|
||||
cursor.execute("SELECT COUNT(*) FROM candles")
|
||||
status["candles_db"]["row_count"] = cursor.fetchone()[0]
|
||||
status["candles_db"]["accessible"] = True
|
||||
|
||||
# Check per-timeframe data
|
||||
for tf in self.config.timeframes:
|
||||
cursor.execute(
|
||||
"SELECT COUNT(*), MAX(timestamp) FROM candles WHERE timeframe = ?",
|
||||
(tf,),
|
||||
)
|
||||
count, max_ts = cursor.fetchone()
|
||||
status["candles_db"]["timeframes"][tf] = {
|
||||
"count": count,
|
||||
"latest_timestamp": max_ts,
|
||||
"age_seconds": int(time.time() - max_ts) if max_ts else None,
|
||||
}
|
||||
|
||||
conn_c.close()
|
||||
except Exception as e:
|
||||
status["candles_db"]["error"] = str(e)
|
||||
|
||||
# Check analysis DB
|
||||
try:
|
||||
conn_a = sqlite3.connect(
|
||||
f"file:{self.config.analysis_db}?mode=ro", uri=True, timeout=5
|
||||
)
|
||||
cursor = conn_a.cursor()
|
||||
|
||||
cursor.execute("SELECT COUNT(*) FROM analysis")
|
||||
status["analysis_db"]["row_count"] = cursor.fetchone()[0]
|
||||
status["analysis_db"]["accessible"] = True
|
||||
|
||||
# Check per-timeframe data
|
||||
for tf in self.config.timeframes:
|
||||
cursor.execute(
|
||||
"SELECT COUNT(*), MAX(timestamp) FROM analysis WHERE timeframe = ?",
|
||||
(tf,),
|
||||
)
|
||||
count, max_ts = cursor.fetchone()
|
||||
status["analysis_db"]["timeframes"][tf] = {
|
||||
"count": count,
|
||||
"latest_timestamp": max_ts,
|
||||
"age_seconds": int(time.time() - max_ts) if max_ts else None,
|
||||
}
|
||||
|
||||
conn_a.close()
|
||||
except Exception as e:
|
||||
status["analysis_db"]["error"] = str(e)
|
||||
|
||||
return status
|
||||
|
||||
def handle_health_checks(self):
|
||||
"""Handle incoming health check requests"""
|
||||
try:
|
||||
conn, _ = self.health_socket.accept()
|
||||
|
||||
uptime = datetime.now(timezone.utc) - self.stats["uptime_start"]
|
||||
db_status = self.check_database_status()
|
||||
|
||||
health = {
|
||||
"status": "running",
|
||||
"personality": self.config.personality,
|
||||
"timeframes": self.config.timeframes,
|
||||
"uptime_seconds": int(uptime.total_seconds()),
|
||||
"total_signals": self.stats["total_signals"],
|
||||
"buy_signals": self.stats["buy_signals"],
|
||||
"sell_signals": self.stats["sell_signals"],
|
||||
"last_signal": self.stats["last_signal_time"],
|
||||
"errors": self.stats["errors"],
|
||||
"connected_clients": len(self.connections),
|
||||
"recent_signals": list(self.signal_history)[-5:],
|
||||
"databases": db_status,
|
||||
"config": {
|
||||
"min_confidence": self.config.min_confidence,
|
||||
"cooldown_seconds": self.config.cooldown_seconds,
|
||||
"lookback": self.config.lookback,
|
||||
"weights": self.config.weights[self.config.personality],
|
||||
"reloads": self.stats["config_reloads"],
|
||||
},
|
||||
"debug_mode": self.debug_mode,
|
||||
}
|
||||
|
||||
conn.sendall(json.dumps(health, indent=2).encode("utf-8") + b"\n")
|
||||
conn.close()
|
||||
|
||||
except socket.timeout:
|
||||
pass
|
||||
except Exception as e:
|
||||
self.logger.debug(f"Health check error: {e}")
|
||||
|
||||
def handle_control_commands(self):
|
||||
"""Handle incoming control commands"""
|
||||
try:
|
||||
conn, _ = self.control_socket.accept()
|
||||
|
||||
# Receive command
|
||||
data = conn.recv(4096).decode("utf-8").strip()
|
||||
|
||||
if not data:
|
||||
conn.close()
|
||||
return
|
||||
|
||||
try:
|
||||
cmd = json.loads(data)
|
||||
response = self.process_command(cmd)
|
||||
except json.JSONDecodeError:
|
||||
response = {"status": "error", "message": "Invalid JSON"}
|
||||
|
||||
conn.sendall(json.dumps(response, indent=2).encode("utf-8") + b"\n")
|
||||
conn.close()
|
||||
|
||||
except socket.timeout:
|
||||
pass
|
||||
except Exception as e:
|
||||
self.logger.debug(f"Control command error: {e}")
|
||||
|
||||
def process_command(self, cmd: Dict) -> Dict:
|
||||
"""Process control commands"""
|
||||
action = cmd.get("action")
|
||||
|
||||
if action == "reload":
|
||||
try:
|
||||
old_personality = self.config.personality
|
||||
old_confidence = self.config.min_confidence
|
||||
|
||||
self.config.reload()
|
||||
self.stats["config_reloads"] += 1
|
||||
|
||||
self.logger.info(
|
||||
f"Config reloaded: personality={self.config.personality}, "
|
||||
f"min_confidence={self.config.min_confidence}"
|
||||
)
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"message": "Configuration reloaded",
|
||||
"changes": {
|
||||
"personality": {
|
||||
"old": old_personality,
|
||||
"new": self.config.personality,
|
||||
},
|
||||
"min_confidence": {
|
||||
"old": old_confidence,
|
||||
"new": self.config.min_confidence,
|
||||
},
|
||||
},
|
||||
}
|
||||
except Exception as e:
|
||||
return {"status": "error", "message": str(e)}
|
||||
|
||||
elif action == "set_personality":
|
||||
personality = cmd.get("value")
|
||||
if personality in ["scalping", "swing"]:
|
||||
self.config.personality = personality
|
||||
self.logger.info(f"Personality changed to: {personality}")
|
||||
return {
|
||||
"status": "success",
|
||||
"message": f"Personality set to {personality}",
|
||||
}
|
||||
else:
|
||||
return {
|
||||
"status": "error",
|
||||
"message": "Invalid personality (use 'scalping' or 'swing')",
|
||||
}
|
||||
|
||||
elif action == "set_confidence":
|
||||
try:
|
||||
confidence = float(cmd.get("value"))
|
||||
if 0 <= confidence <= 1:
|
||||
self.config.min_confidence = confidence
|
||||
self.logger.info(f"Min confidence changed to: {confidence}")
|
||||
return {
|
||||
"status": "success",
|
||||
"message": f"Min confidence set to {confidence}",
|
||||
}
|
||||
else:
|
||||
return {
|
||||
"status": "error",
|
||||
"message": "Confidence must be between 0 and 1",
|
||||
}
|
||||
except (TypeError, ValueError):
|
||||
return {"status": "error", "message": "Invalid confidence value"}
|
||||
|
||||
elif action == "set_cooldown":
|
||||
try:
|
||||
cooldown = int(cmd.get("value"))
|
||||
if cooldown >= 0:
|
||||
self.config.cooldown_seconds = cooldown
|
||||
self.logger.info(f"Cooldown changed to: {cooldown}s")
|
||||
return {
|
||||
"status": "success",
|
||||
"message": f"Cooldown set to {cooldown}s",
|
||||
}
|
||||
else:
|
||||
return {"status": "error", "message": "Cooldown must be >= 0"}
|
||||
except (TypeError, ValueError):
|
||||
return {"status": "error", "message": "Invalid cooldown value"}
|
||||
|
||||
elif action == "toggle_debug":
|
||||
self.debug_mode = not self.debug_mode
|
||||
level = logging.DEBUG if self.debug_mode else logging.INFO
|
||||
self.logger.setLevel(level)
|
||||
self.logger.info(f"Debug mode: {'ON' if self.debug_mode else 'OFF'}")
|
||||
return {
|
||||
"status": "success",
|
||||
"message": f"Debug mode: {'ON' if self.debug_mode else 'OFF'}",
|
||||
}
|
||||
|
||||
elif action == "clear_cooldowns":
|
||||
self.last_signal_time.clear()
|
||||
self.logger.info("Signal cooldowns cleared")
|
||||
return {"status": "success", "message": "All cooldowns cleared"}
|
||||
|
||||
elif action == "reset_stats":
|
||||
self.stats = {
|
||||
"total_signals": 0,
|
||||
"buy_signals": 0,
|
||||
"sell_signals": 0,
|
||||
"last_signal_time": None,
|
||||
"uptime_start": datetime.now(timezone.utc),
|
||||
"errors": 0,
|
||||
"config_reloads": self.stats["config_reloads"],
|
||||
}
|
||||
self.signal_history.clear()
|
||||
self.logger.info("Statistics reset")
|
||||
return {"status": "success", "message": "Statistics reset"}
|
||||
|
||||
else:
|
||||
return {"status": "error", "message": f"Unknown action: {action}"}
|
||||
|
||||
def accept_connections(self):
|
||||
"""Accept new client connections"""
|
||||
try:
|
||||
conn, _ = self.socket.accept()
|
||||
self.connections.append(conn)
|
||||
self.logger.info(
|
||||
f"New client connected. Total clients: {len(self.connections)}"
|
||||
)
|
||||
except socket.timeout:
|
||||
pass
|
||||
except Exception as e:
|
||||
self.logger.debug(f"Accept error: {e}")
|
||||
|
||||
def run(self):
|
||||
"""Main processing loop"""
|
||||
self.running = True
|
||||
self.setup_signal_socket()
|
||||
self.setup_health_socket()
|
||||
self.setup_control_socket()
|
||||
|
||||
self.logger.info(
|
||||
f"Signal generator started - Personality: {self.config.personality}"
|
||||
)
|
||||
self.logger.info(f"Monitoring timeframes: {', '.join(self.config.timeframes)}")
|
||||
self.logger.info(f"Poll interval: {self.config.poll_interval}s")
|
||||
|
||||
try:
|
||||
while self.running:
|
||||
# Accept new connections
|
||||
self.accept_connections()
|
||||
|
||||
# Handle health checks
|
||||
self.handle_health_checks()
|
||||
|
||||
# Handle control commands
|
||||
self.handle_control_commands()
|
||||
|
||||
# Generate signals for each timeframe
|
||||
for timeframe in self.config.timeframes:
|
||||
try:
|
||||
signal = self.generate_signal(timeframe)
|
||||
if signal:
|
||||
self.broadcast_signal(signal)
|
||||
except Exception as e:
|
||||
self.logger.error(f"Error processing {timeframe}: {e}")
|
||||
self.stats["errors"] += 1
|
||||
|
||||
time.sleep(self.config.poll_interval)
|
||||
|
||||
except KeyboardInterrupt:
|
||||
self.logger.info("Received interrupt signal")
|
||||
finally:
|
||||
self.cleanup()
|
||||
|
||||
def cleanup(self):
|
||||
"""Cleanup resources"""
|
||||
self.logger.info("Shutting down...")
|
||||
|
||||
for conn in self.connections:
|
||||
try:
|
||||
conn.close()
|
||||
except:
|
||||
pass
|
||||
|
||||
if self.socket:
|
||||
self.socket.close()
|
||||
if os.path.exists(self.config.socket_path):
|
||||
os.remove(self.config.socket_path)
|
||||
|
||||
if self.health_socket:
|
||||
self.health_socket.close()
|
||||
if os.path.exists(self.config.health_socket_path):
|
||||
os.remove(self.config.health_socket_path)
|
||||
|
||||
if self.control_socket:
|
||||
self.control_socket.close()
|
||||
if os.path.exists(self.config.control_socket_path):
|
||||
os.remove(self.config.control_socket_path)
|
||||
|
||||
self.logger.info("Shutdown complete")
|
||||
|
||||
|
||||
def main():
|
||||
config = Config()
|
||||
logger = setup_logging(config)
|
||||
|
||||
generator = SignalGenerator(config, logger)
|
||||
|
||||
# Signal handlers for hot-reload and control
|
||||
def reload_config(sig, frame):
|
||||
"""SIGUSR1: Reload configuration"""
|
||||
logger.info("Received SIGUSR1 - Reloading configuration...")
|
||||
try:
|
||||
old_personality = config.personality
|
||||
config.reload()
|
||||
generator.stats["config_reloads"] += 1
|
||||
logger.info(
|
||||
f"Configuration reloaded successfully "
|
||||
f"(personality: {old_personality} -> {config.personality})"
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to reload configuration: {e}")
|
||||
|
||||
def toggle_debug(sig, frame):
|
||||
"""SIGUSR2: Toggle debug logging"""
|
||||
generator.debug_mode = not generator.debug_mode
|
||||
level = logging.DEBUG if generator.debug_mode else logging.INFO
|
||||
logger.setLevel(level)
|
||||
logger.info(f"Debug mode {'enabled' if generator.debug_mode else 'disabled'}")
|
||||
|
||||
def shutdown(sig, frame):
|
||||
"""SIGINT/SIGTERM: Graceful shutdown"""
|
||||
generator.running = False
|
||||
|
||||
signal.signal(signal.SIGUSR1, reload_config)
|
||||
signal.signal(signal.SIGUSR2, toggle_debug)
|
||||
signal.signal(signal.SIGINT, shutdown)
|
||||
signal.signal(signal.SIGTERM, shutdown)
|
||||
|
||||
generator.run()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user