home-assistant-controller/docs/superpowers/specs/2026-08-20-concurrent-runtime-design.md

Concurrent runtime: multiple controllers, channels, supervision, reconnect

Date: 2026-08-20 Status: Approved (pending spec review)

Goal

Make the HASS runtime production-quality:

• Support multiple controllers.
• A single websocket read loop; no per-controller connections.
• The read loop broadcasts messages to controllers over channels (TChan).
• Each controller runs on its own thread (async).
• Controllers know nothing of the websocket connection: service calls go through an outbound channel; a writer thread does the actual sends.
• Crashed worker threads are restarted with backoff (annotated-exception for context).
• The websocket reconnects on network failure.

Implemented as two slices, each independently buildable and committed:

1. Concurrency architecture: bus, reader/writer threads, controller threads, channel interpreter, wiring. Any worker crash exits the process (today's behavior).
2. Robustness: supervisor with backoff, annotated-exception, fatal vs retryable failures, reconnect-as-restart.

Current state

HomeAssistant.Runtime.app runs everything on one thread inside WS.runClient: auth handshake, a get_states debug dump, one subscribe_events, then a loop that steps a single lightController via dryRunHassEval. The get_states dump is dead debug code and is removed (state pre-seeding is a separate future concern).

Architecture

                HA websocket
                 ▲          │
        sends    │          ▼ receives
   ┌─────────────┴──┐  ┌──────────────┐
   │ writer thread  │  │ reader thread │
   └───────▲────────┘  └───────┬──────┘
           │                   │ decode → Value
           │ TChan Service     │ broadcast (single write)
           │                   ▼
   ┌───────┴───────────────────────────┐
   │ busInbound (broadcast TChan)      │
   └──┬──────────────┬──────────────┬──┘
      ▼ dupTChan     ▼              ▼
  controller 1   controller 2   controller N   (1 thread each)

Bus

data Bus = Bus
  { busInbound  :: TChan Value          -- newBroadcastTChanIO; reader writes only
  , busOutbound :: TChan Service        -- controllers → writer; fire-and-forget
  , busConn     :: TVar (Maybe WS.Connection)
  , busGen      :: CallIdGen            -- existing IORef-based, thread-safe
  }

• busInbound comes from newBroadcastTChanIO: a write-only broadcast channel. A plain never-read TChan would pin its entire history; the broadcast variant does not. Controllers receive via dupTChanIO busInbound; the reader writes each message once.

• busConn is a plain current-value cell (no MVar blocking semantics). Maybe + STM retry lets the writer block until a connection exists and lets defaultMain spawn all workers up front: the initial connect is just the reader's first attempt, so first-connect failures and reconnect failures take the same backoff path.

• The controller list is static. Each entry is an existential:

  data Controller = forall b. Controller T.Text (HASS (Event Value) b)
  

  No Show constraint: machine outputs are discarded; observability will come from a logging effect later. The name tags restart logs.

Send-safety without locks

No two threads ever send on the same live connection:

• The reader performs all setup sends (auth, subscribe) before swapping the connection into busConn; afterwards it only receives.
• The writer only sends on connections read from busConn.

On reconnect the reader builds a new connection, handshakes, then atomically swaps busConn. A writer mid-send on the dead connection throws, its supervisor restarts it, and it picks up the new connection.

Outbound backpressure

Fire-and-forget: services queue in busOutbound during outages and are sent after reconnect. The queue is naturally bounded in practice: no inbound events means controllers produce no calls.

Module layout

Module	Concern
HomeAssistant.Runtime.Supervisor	Generic restart-with-backoff combinator, Backoff, Fatal; no HA knowledge
HomeAssistant.Runtime.Bus	Bus, newBus, channel interpreter for HASSEff
HomeAssistant.Runtime.Connection	readerAction, writerAction; connect/auth/subscribe, receive-decode-broadcast loop, send loop
HomeAssistant.Runtime	Glue: defaultMain, controllers, Controller, runController, step, dryRunHassEval

Exports removed from HomeAssistant.Runtime: app, hassEval, wsCallService, receiveJSON (internal or superseded). Kept: defaultMain, step, dryRunHassEval, CallIdGen, mkCallIdGen. app/Main.hs unchanged.

Connection lifecycle

Reader action (restartable unit; restart = reconnect):

connect → expect auth_required → send token → expect auth_ok
        → send subscribe_events state_changed (id from busGen)
        → swap busConn
        → forever: receive → decode → broadcast to busInbound

• auth_invalid (and undecodable handshake messages) is fatal: a bad token cannot be fixed by retrying. The reader throws Fatal; the supervisor rethrows it (with annotations) and the process exits.
• Undecodable messages in the receive loop are not fatal: log a warning and skip. Reconnecting cannot fix a decode problem, so crash-restarting would just be a hot loop.
• Controller Mealy state survives reconnects (explicit decision). State may be stale until each watched entity's next state_changed event. Re-seeding via get_states is a separate future concern.

Writer action:

forever: readTChan busOutbound
       → readTVar busConn (retry until Just)
       → encode with fresh id from busGen → send

Encoding fixes a latent bug: today's wsCallService drops serviceData; the writer encodes it as "service_data" when present.

Supervision

supervised :: Text -> Backoff -> IO a -> IO Void   -- never returns normally

• Catches synchronous exceptions; rethrows SomeAsyncException (no restarting on cancellation).

• On Fatal: rethrow with annotations; process exits.

• Otherwise: log (component name, attempt, annotated exception), sleep per backoff, restart the action.

• Backoff: exponential from base, capped; resets after a quiet period.

  data Backoff = Backoff
    { backoffBase :: NominalDiffTime   -- first restart delay, e.g. 100ms
    , backoffCap  :: NominalDiffTime   -- max delay, e.g. 30s
    , backoffQuiet :: NominalDiffTime  -- uptime that resets delay, e.g. 30s
    }
  

  Delay after the n-th consecutive crash: min cap (base * 2^(n-1)).

• annotated-exception adds context at catch sites (component, phase such as "authenticating" / "receiving") so logs read like [reader] attempt 3: ConnectionClosed while receiving.

Wiring

• Slice 1: defaultMain = newBus → mapConcurrently_ over reader, writer, and controller actions. First worker crash cancels the rest and exits the process.
• Slice 2: each action wrapped in supervised; main waits forever. The worker IO actions are identical in both slices; only the spawning changes.

Controller runner

runController :: Bus -> Controller -> IO Void
-- dupTChanIO busInbound
-- loop: readTChan → step (channelHassEval bus) → discard output

On a supervised restart the action re-dups: a fresh port sees only messages written after the dup, consistent with the machine also restarting from its initial state. Messages broadcast during a restart window are lost (accepted, documented).

Interpreter

channelHassEval :: Bus -> HASSEff a -> IO a
channelHassEval bus (CallService svc) = atomically (writeTChan (busOutbound bus) svc)
channelHassEval _   (Pure a)          = pure a

dryRunHassEval stays exported for experiments/tests.

Logging

Tagged plain lines ([reader] …) via putStrLn. No framework.

Dependencies

Added: stm, async (slice 1); annotated-exception (slice 2). After editing the cabal file, regenerate default.nix via nix run nixpkgs#cabal2nix -- ./. > default.nix (AGENTS.md).

Testing

hspec for unit tests, hedgehog for property tests (AGENTS.md):

• hspec:
  • Broadcast semantics: two dupTChan ports see all writes, in order.
  • channelHassEval writes Service to the outbound chan.
  • runController end-to-end on a real Bus with a test controller: feed values into busInbound, assert services land in busOutbound.
  • Writer encoding: golden JSON for call_service incl. service_data.
  • Supervisor (tiny backoff values): crash-twice-then-succeed restarts; Fatal rethrown, not restarted; async exceptions not restarted.
• hedgehog:
  • Backoff property: delays double from base, clamp at cap, reset after quiet period.

Slices

1. Concurrency architecture: Bus (already reconnect-ready: TVar (Maybe Connection)), channelHassEval, readerAction, writerAction, runController, module split, cabal deps, wiring via mapConcurrently_, tests for bus/runner/encoding. Decode log-and-skip included.
2. Robustness: Supervisor module, annotated-exception dep, backoff, Fatal auth failures, rewire defaultMain to supervised workers, supervisor tests and backoff property.

In slice 1 an auth_invalid response exits the process like any other crash; the fatal/retryable distinction only exists once the supervisor does (slice 2).

Each slice: builds clean, tests green, committed separately.

Non-goals

• Request/response correlation for service calls (approach B, deferred; outbound message type can grow later).
• Dynamic controller registration (static list).
• Structured logging / logging effect (future).
• Host/port/token configuration via env (beyond current HA_TOKEN).
• State pre-seeding via get_states (user's future concern).
• Multiple websocket connections.