home-assistant-controller/docs/superpowers/specs/2026-08-20-concurrent-runtime-design.md

# Concurrent runtime: multiple controllers, channels, supervision, reconnect

Date: 2026-08-20
Status: Approved (pending spec review)

## Goal

Make the HASS runtime production-quality:

- Support multiple controllers.
- A single websocket read loop; no per-controller connections.
- The read loop broadcasts messages to controllers over channels (`TChan`).
- Each controller runs on its own thread (`async`).
- Controllers know nothing of the websocket connection: service calls go
  through an outbound channel; a writer thread does the actual sends.
- Crashed worker threads are restarted with backoff (`annotated-exception`
  for context).
- The websocket reconnects on network failure.

Implemented as two slices, each independently buildable and committed:

1. **Concurrency architecture**: bus, reader/writer threads, controller
   threads, channel interpreter, wiring. Any worker crash exits the
   process (today's behavior).
2. **Robustness**: supervisor with backoff, `annotated-exception`, fatal
   vs retryable failures, reconnect-as-restart.

## Current state

`HomeAssistant.Runtime.app` runs everything on one thread inside
`WS.runClient`: auth handshake, a `get_states` debug dump, one
`subscribe_events`, then a loop that steps a single `lightController`
via `dryRunHassEval`. The `get_states` dump is dead debug code and is
removed (state pre-seeding is a separate future concern).

## Architecture

```
                HA websocket
                 ▲          │
        sends    │          ▼ receives
   ┌─────────────┴──┐  ┌──────────────┐
   │ writer thread  │  │ reader thread │
   └───────▲────────┘  └───────┬──────┘
           │                   │ decode → Value
           │ TChan Service     │ broadcast (single write)
           │                   ▼
   ┌───────┴───────────────────────────┐
   │ busInbound (broadcast TChan)      │
   └──┬──────────────┬──────────────┬──┘
      ▼ dupTChan     ▼              ▼
  controller 1   controller 2   controller N   (1 thread each)
```

### Bus

```haskell
data Bus = Bus
  { busInbound  :: TChan Value          -- newBroadcastTChanIO; reader writes only
  , busOutbound :: TChan Service        -- controllers → writer; fire-and-forget
  , busConn     :: TVar (Maybe WS.Connection)
  , busGen      :: CallIdGen            -- existing IORef-based, thread-safe
  }
```

- `busInbound` comes from `newBroadcastTChanIO`: a write-only broadcast
  channel. A plain never-read `TChan` would pin its entire history; the
  broadcast variant does not. Controllers receive via
  `dupTChanIO busInbound`; the reader writes each message once.
- `busConn` is a plain current-value cell (no `MVar` blocking semantics).
  `Maybe` + STM `retry` lets the writer block until a connection exists
  and lets `defaultMain` spawn all workers up front: the initial connect
  is just the reader's first attempt, so first-connect failures and
  reconnect failures take the same backoff path.
- The controller list is static. Each entry is an existential:

  ```haskell
  data Controller = forall b. Controller T.Text (HASS (Event Value) b)
  ```

  No `Show` constraint: machine outputs are discarded; observability will
  come from a logging effect later. The name tags restart logs.

### Send-safety without locks

No two threads ever send on the same live connection:

- The reader performs all setup sends (auth, subscribe) **before**
  swapping the connection into `busConn`; afterwards it only receives.
- The writer only sends on connections read from `busConn`.

On reconnect the reader builds a new connection, handshakes, then
atomically swaps `busConn`. A writer mid-send on the dead connection
throws, its supervisor restarts it, and it picks up the new connection.

### Outbound backpressure

Fire-and-forget: services queue in `busOutbound` during outages and are
sent after reconnect. The queue is naturally bounded in practice: no
inbound events means controllers produce no calls.

## Module layout

| Module | Concern |
|---|---|
| `HomeAssistant.Runtime.Supervisor` | Generic restart-with-backoff combinator, `Backoff`, `Fatal`; no HA knowledge |
| `HomeAssistant.Runtime.Bus` | `Bus`, `newBus`, channel interpreter for `HASSEff` |
| `HomeAssistant.Runtime.Connection` | `readerAction`, `writerAction`; connect/auth/subscribe, receive-decode-broadcast loop, send loop |
| `HomeAssistant.Runtime` | Glue: `defaultMain`, `controllers`, `Controller`, `runController`, `step`, `dryRunHassEval` |

Exports removed from `HomeAssistant.Runtime`: `app`, `hassEval`,
`wsCallService`, `receiveJSON` (internal or superseded). Kept:
`defaultMain`, `step`, `dryRunHassEval`, `CallIdGen`, `mkCallIdGen`.
`app/Main.hs` unchanged.

## Connection lifecycle

**Reader action** (restartable unit; restart = reconnect):

```
connect → expect auth_required → send token → expect auth_ok
        → send subscribe_events state_changed (id from busGen)
        → swap busConn
        → forever: receive → decode → broadcast to busInbound
```

- `auth_invalid` (and undecodable handshake messages) is **fatal**: a bad
  token cannot be fixed by retrying. The reader throws `Fatal`; the
  supervisor rethrows it (with annotations) and the process exits.
- Undecodable messages in the receive loop are **not** fatal: log a
  warning and skip. Reconnecting cannot fix a decode problem, so
  crash-restarting would just be a hot loop.
- Controller Mealy state survives reconnects (explicit decision). State
  may be stale until each watched entity's next `state_changed` event.
  Re-seeding via `get_states` is a separate future concern.

**Writer action**:

```
forever: readTChan busOutbound
       → readTVar busConn (retry until Just)
       → encode with fresh id from busGen → send
```

Encoding fixes a latent bug: today's `wsCallService` drops
`serviceData`; the writer encodes it as `"service_data"` when present.

## Supervision

```haskell
supervised :: Text -> Backoff -> IO a -> IO Void   -- never returns normally
```

- Catches synchronous exceptions; rethrows `SomeAsyncException`
  (no restarting on cancellation).
- On `Fatal`: rethrow with annotations; process exits.
- Otherwise: log (component name, attempt, annotated exception), sleep
  per backoff, restart the action.
- Backoff: exponential from base, capped; resets after a quiet period.

  ```haskell
  data Backoff = Backoff
    { backoffBase :: NominalDiffTime   -- first restart delay, e.g. 100ms
    , backoffCap  :: NominalDiffTime   -- max delay, e.g. 30s
    , backoffQuiet :: NominalDiffTime  -- uptime that resets delay, e.g. 30s
    }
  ```

  Delay after the n-th consecutive crash: `min cap (base * 2^(n-1))`.

- `annotated-exception` adds context at catch sites (component, phase
  such as "authenticating" / "receiving") so logs read like
  `[reader] attempt 3: ConnectionClosed while receiving`.

### Wiring

- Slice 1: `defaultMain` = `newBus` → `mapConcurrently_` over reader,
  writer, and controller actions. First worker crash cancels the rest
  and exits the process.
- Slice 2: each action wrapped in `supervised`; main waits forever.
  The worker `IO` actions are identical in both slices; only the
  spawning changes.

## Controller runner

```haskell
runController :: Bus -> Controller -> IO Void
-- dupTChanIO busInbound
-- loop: readTChan → step (channelHassEval bus) → discard output
```

On a supervised restart the action re-dups: a fresh port sees only
messages written after the dup, consistent with the machine also
restarting from its initial state. Messages broadcast during a restart
window are lost (accepted, documented).

## Interpreter

```haskell
channelHassEval :: Bus -> HASSEff a -> IO a
channelHassEval bus (CallService svc) = atomically (writeTChan (busOutbound bus) svc)
channelHassEval _   (Pure a)          = pure a
```

`dryRunHassEval` stays exported for experiments/tests.

## Logging

Tagged plain lines (`[reader] …`) via `putStrLn`. No framework.

## Dependencies

Added: `stm`, `async` (slice 1); `annotated-exception` (slice 2).
After editing the cabal file, regenerate `default.nix` via
`nix run nixpkgs#cabal2nix -- ./. > default.nix` (AGENTS.md).

## Testing

hspec for unit tests, hedgehog for property tests (AGENTS.md):

- hspec:
  - Broadcast semantics: two `dupTChan` ports see all writes, in order.
  - `channelHassEval` writes `Service` to the outbound chan.
  - `runController` end-to-end on a real `Bus` with a test controller:
    feed values into `busInbound`, assert services land in `busOutbound`.
  - Writer encoding: golden JSON for `call_service` incl. `service_data`.
  - Supervisor (tiny backoff values): crash-twice-then-succeed restarts;
    `Fatal` rethrown, not restarted; async exceptions not restarted.
- hedgehog:
  - Backoff property: delays double from base, clamp at cap, reset after
    quiet period.

## Slices

1. **Concurrency architecture**: `Bus` (already reconnect-ready:
   `TVar (Maybe Connection)`), `channelHassEval`, `readerAction`,
   `writerAction`, `runController`, module split, cabal deps, wiring via
   `mapConcurrently_`, tests for bus/runner/encoding. Decode
   log-and-skip included.
2. **Robustness**: `Supervisor` module, `annotated-exception` dep,
   backoff, `Fatal` auth failures, rewire `defaultMain` to supervised
   workers, supervisor tests and backoff property.

In slice 1 an `auth_invalid` response exits the process like any other
crash; the fatal/retryable distinction only exists once the supervisor
does (slice 2).

Each slice: builds clean, tests green, committed separately.

## Non-goals

- Request/response correlation for service calls (approach B, deferred;
  outbound message type can grow later).
- Dynamic controller registration (static list).
- Structured logging / logging effect (future).
- Host/port/token configuration via env (beyond current `HA_TOKEN`).
- State pre-seeding via `get_states` (user's future concern).
- Multiple websocket connections.