espresso_frame/docs/architecture.md

# Architecture

Two independent pieces talk over HTTP or HTTPS (the server itself always
speaks plain HTTP; HTTPS means a reverse proxy in front of it, see
[`firmware/README.md`](../firmware/README.md#http-vs-https)) on the local
network: the ESP32-C6 firmware, and a small FastAPI server that sits
between it and Immich.

```mermaid
sequenceDiagram
    participant Immich
    participant Server as ESPresso Frame Server
    participant Frame as ESP32-C6 Frame

    Note over Frame: First boot / never provisioned
    Frame->>Frame: Generate AP SSID/password, draw QR + config QR on panel
    Frame->>Frame: Bring up ESPRESSO_XXXXXX softAP + captive portal
    Note over Frame: User scans WiFi QR, then config QR -> fills in<br/>home WiFi + "Tools Server" host:port
    Frame->>Frame: Save config to NVS, reboot

    Note over Frame: Every wake (deep sleep timer, next/back-photo button,<br/>or any other reboot)
    Frame->>Frame: Connect to home WiFi
    alt next-photo button pressed
        Frame->>Server: POST /frame/advance
        Server->>Server: Force-advance to next queued photo, reset interval clock
    else back-photo button pressed
        Frame->>Server: POST /frame/back
        Server->>Server: Return to previously-current photo (bounded history),<br/>reset interval clock
    else normal wake
        Frame->>Server: GET /frame/image
        Server->>Server: Advance only if refresh_interval_s has elapsed<br/>since the current photo was set -- otherwise a no-op
    end
    Server->>Immich: List album assets / download preview / faces
    Immich-->>Server: JPEG + face bounding boxes
    Server->>Server: Crop (face-aware) + quantize (dither) + pack 4bpp
    Server-->>Frame: 192,000 raw bytes, streamed
    Frame->>Frame: Write to panel SPI buffer, compute CRC32
    alt CRC unchanged since last physical refresh
        Frame->>Frame: Skip refresh (nothing visually changed)
    else CRC changed
        Frame->>Frame: Trigger physical refresh, store new CRC
    end
    Frame->>Server: GET /frame/config
    Server-->>Frame: {"refresh_interval_s": ...}
    Frame->>Frame: Deep sleep (server-configured interval, or a short<br/>retry interval on any failure)
```

## Firmware boot flow

1. **No stored config** (first boot, or NVS erased): bring up the display,
   render a WiFi-join QR code + plaintext password (left) and a
   captive-portal config QR code (right), *then* start the `ESPRESSO_XXXXXX`
   softAP + DNS redirect + HTTP server. The display goes up before the AP
   so the join instructions are visible before the network is joinable.
   The captive portal form saves SSID/password/toolsserver to NVS and
   reboots.
2. **Stored config exists**: connect to the saved WiFi network (a few
   retries before falling back to provisioning if it fails), then run the
   fetch cycle in `frame_client.c`:
   - Check the next-photo and back-photo buttons (`next_button_check()`,
     `back_button_check()`) -- if either was what woke the device
     (checked via the latched `esp_sleep_get_gpio_wakeup_status()`, not
     a live pin read, since a quick tap can release before boot gets
     around to polling it) or is currently held, the fetch below hits
     `POST /frame/advance` or `POST /frame/back` instead of
     `GET /frame/image`, forcing the server to move in that direction
     immediately (next takes priority if somehow both read pressed at
     once).
   - Fetch the frame and write it into the panel's SPI buffer
     (`epd_write_frame()`), computing a CRC32 as it streams -- never
     buffering the full ~192KB frame in RAM. The panel driver refuses to
     write a short/wrong-size response into the buffer at all, so a
     truncated fetch can't corrupt what's already there.
   - Compare the new CRC32 against the last one that was actually
     refreshed onto the panel (persisted in NVS). If it matches -- the
     same photo is already visibly on screen, e.g. the device rebooted
     before the server's refresh interval elapsed -- skip the physical
     refresh entirely (`epd_turn_on_display()`), avoiding its visible
     flash and 15-30s duration for no visual change. Otherwise trigger
     the refresh and store the new CRC.
   - `GET /frame/config` for the refresh interval, used to set the deep
     sleep duration -- deliberately fetched *after* the image, not
     before: its timeout is much tighter (3s vs. the image fetch's 15s),
     and fetching second lets it ride the connection the image fetch just
     warmed up rather than eating the latency spike common on the first
     request after waking from a long sleep.
   - Deep sleep for the server-configured interval on success, or a
     shorter retry interval on any failure.

   The menu/reset button's soft-reset and factory-reset tiers (held ~3s
   or ~15s) are handled earlier, before any of this, and never return --
   see [`firmware/README.md`](../firmware/README.md#managing-the-queue-soft-resetting-and-factory-resetting).

See [`docs/hardware.md`](hardware.md) for wiring and
[`server/README.md`](../server/README.md) for the server side.

## Why image processing happens server-side

The ESP32-C6 has no PSRAM and a tight SRAM budget (already tight enough
that a single 4KB stack buffer caused a crash during development -- see
git history). Decoding a JPEG, then resizing/dithering/quantizing it to
the panel's 6-color palette, would be expensive on-device in both memory
and battery. Instead, the server does all of that with Pillow and hands
the frame a pre-packed, ready-to-stream buffer -- the device never
decodes an image at all.

## Why face detection isn't run on-device (or even on the server)

Immich already runs face detection for its own "People" feature. The
server just asks Immich for the bounding boxes it already computed
(`GET /api/faces?id=...`) and biases the crop to keep them on screen,
rather than bundling a detector (OpenCV/dlib) anywhere in this project.