# Architecture

Two independent pieces talk over HTTP or HTTPS (the server itself always
speaks plain HTTP; HTTPS means a reverse proxy in front of it, see
[`firmware/README.md`](../firmware/README.md#http-vs-https)) on the local
network: the ESP32 firmware (ESP32-C6 for the devkit/xiao boards,
ESP32-S3 for ee02 -- see [`docs/hardware.md`](hardware.md)), and a small
FastAPI server that sits between it and Immich.

```mermaid
sequenceDiagram
    participant Immich
    participant Server as ESPresso Frame Server
    participant Frame as ESP32 Frame

    Note over Frame: First boot / never provisioned
    Frame->>Frame: Generate AP SSID/password, draw QR + config QR on panel
    Frame->>Frame: Bring up ESPRESSO_XXXXXX softAP + captive portal
    Note over Frame: User scans WiFi QR, then config QR -> fills in<br/>home WiFi + "Tools Server" host:port
    Frame->>Frame: Save config to NVS, reboot

    Note over Frame: Every wake (deep sleep timer, next/back-photo button,<br/>or any other reboot)
    Frame->>Frame: Connect to home WiFi
    alt next-photo button pressed
        Frame->>Server: POST /frame/advance
        Server->>Server: Run every action assigned to NEXT, in order<br/>(may span several widgets -- see docs/widgets.md)
    else back-photo button pressed
        Frame->>Server: POST /frame/back
        Server->>Server: Run every action assigned to BACK, in order
    else normal wake
        Frame->>Server: GET /frame/image
        Server->>Server: Render every widget on the panel into its own region<br/>(each independently idempotent -- a photo widget only<br/>actually advances once its own refresh_interval_s has elapsed)
    end
    Server->>Immich: List album assets / download preview / faces<br/>(once per photo widget on the panel)
    Immich-->>Server: JPEG + face bounding boxes
    Server->>Server: Composite every widget's region onto one canvas,<br/>then enhance/overlay/quantize (dither)/pack 4bpp once
    Server-->>Frame: packed 4bpp bytes, streamed<br/>(192,000 for the 7.3" panel; sized to whichever<br/>panel this frame's device reports, see Frame.panel_type)
    Frame->>Frame: Write to panel SPI buffer, compute CRC32
    alt CRC unchanged since last physical refresh
        Frame->>Frame: Skip refresh (nothing visually changed)
    else CRC changed
        Frame->>Frame: Trigger physical refresh, store new CRC
    end
    Frame->>Server: GET /frame/config
    Server-->>Frame: {"refresh_interval_s": ...}
    Frame->>Frame: Deep sleep (server-configured interval, or a short<br/>retry interval on any failure)
```

The device-facing endpoints above (`/frame/image`, `/frame/advance`,
`/frame/back`, `/frame/config`) are frozen -- baked into deployed firmware
-- and unchanged by any of this. What *does* change server-side: a frame's
panel isn't a single fixed "mode" anymore, it holds an arbitrary
arrangement of independently placed/sized widgets (photos/calendar/
whiteboard, including several of the same type), each rendered into its
own region and composited together, with NEXT/BACK each mapped to their
own ordered list of per-widget actions rather than one fixed meaning. See
[`docs/widgets.md`](widgets.md) for the widget system's data model,
placement grid, and button-action dispatch.

## Firmware boot flow

1. **No stored config** (first boot, or NVS erased): bring up the display,
   render a WiFi-join QR code + plaintext password (left) and a
   captive-portal config QR code (right), *then* start the `ESPRESSO_XXXXXX`
   softAP + DNS redirect + HTTP server. The display goes up before the AP
   so the join instructions are visible before the network is joinable.
   The captive portal form saves SSID/password/toolsserver to NVS and
   reboots.
2. **Stored config exists**: connect to the saved WiFi network (a few
   retries before falling back to provisioning if it fails). The first
   attempt tries a cached BSSID/channel + static IP from the last
   successful connection, skipping the scan and DHCP; a bad cache falls
   back to a normal attempt and gets cleared (see `firmware/README.md`'s
   "WiFi fast-connect" section). Then run the fetch cycle in
   `frame_client.c`:
   - Check the next-photo and back-photo buttons (`next_button_check()`,
     `back_button_check()`) -- if either was what woke the device
     (checked via the latched `esp_sleep_get_gpio_wakeup_status()`, not
     a live pin read, since a quick tap can release before boot gets
     around to polling it) or is currently held, the fetch below hits
     `POST /frame/advance` or `POST /frame/back` instead of
     `GET /frame/image`, forcing the server to move in that direction
     immediately (next takes priority if somehow both read pressed at
     once).
   - Fetch the frame and write it into the panel's SPI buffer
     (`epd_write_frame()`), computing a CRC32 as it streams -- never
     buffering the full packed frame in RAM (~192KB for the 7.3" panel;
     proportionally more for the 13.3" panel). The panel driver refuses
     to write a short/wrong-size response into the buffer at all, so a
     truncated fetch can't corrupt what's already there.
   - Compare the new CRC32 against the last one that was actually
     refreshed onto the panel (persisted in NVS). If it matches -- the
     same photo is already visibly on screen, e.g. the device rebooted
     before the server's refresh interval elapsed -- skip the physical
     refresh entirely (`epd_turn_on_display()`), avoiding its visible
     flash and 15-30s duration for no visual change. Otherwise trigger
     the refresh and store the new CRC.
   - `GET /frame/config` for the refresh interval, used to set the deep
     sleep duration -- deliberately fetched *after* the image, not
     before: its timeout is much tighter (3s vs. the image fetch's 15s),
     and fetching second lets it ride the connection the image fetch just
     warmed up rather than eating the latency spike common on the first
     request after waking from a long sleep.
   - Deep sleep for the server-configured interval on success, or a
     shorter retry interval on any failure.

   The menu/reset button's soft-reset (quick press) and factory-reset
   (held ~15s) tiers are handled earlier, before any of this, and never
   return -- see [`firmware/README.md`](../firmware/README.md#managing-the-queue-soft-resetting-and-factory-resetting).

See [`docs/hardware.md`](hardware.md) for wiring and
[`server/README.md`](../server/README.md) for the server side.

## Why image processing happens server-side

The ESP32-C6 has no PSRAM and a tight SRAM budget (already tight enough
that a single 4KB stack buffer caused a crash during development -- see
git history). Decoding a JPEG, then resizing/dithering/quantizing it to
the panel's 6-color palette, would be expensive on-device in both memory
and battery. Instead, the server does all of that with Pillow and hands
the frame a pre-packed, ready-to-stream buffer -- the device never
decodes an image at all. The ee02 board's ESP32-S3 does have PSRAM, but
the same server-side design applies there too, for consistency and
battery reasons rather than because the C6's memory limit forces it.

## Why face detection isn't run on-device (or even on the server)

Immich already runs face detection for its own "People" feature. The
server just asks Immich for the bounding boxes it already computed
(`GET /api/faces?id=...`) and biases the crop to keep them on screen,
rather than bundling a detector (OpenCV/dlib) anywhere in this project.