espresso_frame/docs/architecture.md

Architecture

Two independent pieces talk over HTTP or HTTPS (the server itself always speaks plain HTTP; HTTPS means a reverse proxy in front of it, see firmware/README.md) on the local network: the ESP32 firmware (ESP32-C6 for the devkit/xiao boards, ESP32-S3 for ee02 -- see docs/hardware.md), and a small FastAPI server that sits between it and Immich.

sequenceDiagram
    participant Immich
    participant Server as ESPresso Frame Server
    participant Frame as ESP32 Frame

    Note over Frame: First boot / never provisioned
    Frame->>Frame: Generate AP SSID/password, draw QR + config QR on panel
    Frame->>Frame: Bring up ESPRESSO_XXXXXX softAP + captive portal
    Note over Frame: User scans WiFi QR, then config QR -> fills in<br/>home WiFi + "Tools Server" host:port
    Frame->>Frame: Save config to NVS, reboot

    Note over Frame: Every wake (deep sleep timer, next/back-photo button,<br/>or any other reboot)
    Frame->>Frame: Connect to home WiFi
    alt next-photo button pressed
        Frame->>Server: POST /frame/advance
        Server->>Server: Run every action assigned to NEXT, in order<br/>(may span several widgets -- see docs/widgets.md)
    else back-photo button pressed
        Frame->>Server: POST /frame/back
        Server->>Server: Run every action assigned to BACK, in order
    else normal wake
        Frame->>Server: GET /frame/image
        Server->>Server: Render every widget on the panel into its own region<br/>(each independently idempotent -- a photo widget only<br/>actually advances once its own refresh_interval_s has elapsed)
    end
    Server->>Immich: List album assets / download preview / faces<br/>(once per photo widget on the panel)
    Immich-->>Server: JPEG + face bounding boxes
    Server->>Server: Composite every widget's region onto one canvas,<br/>then enhance/overlay/quantize (dither)/pack 4bpp once
    Server-->>Frame: packed 4bpp bytes, streamed<br/>(192,000 for the 7.3" panel; sized to whichever<br/>panel this frame's device reports, see Frame.panel_type)
    Frame->>Frame: Write to panel SPI buffer, compute CRC32
    alt CRC unchanged since last physical refresh
        Frame->>Frame: Skip refresh (nothing visually changed)
    else CRC changed
        Frame->>Frame: Trigger physical refresh, store new CRC
    end
    Frame->>Server: GET /frame/config
    Server-->>Frame: {"refresh_interval_s": ...}
    Frame->>Frame: Deep sleep (server-configured interval, or a short<br/>retry interval on any failure)

The device-facing endpoints above (/frame/image, /frame/advance, /frame/back, /frame/config) are frozen -- baked into deployed firmware -- and unchanged by any of this. What does change server-side: a frame's panel isn't a single fixed "mode" anymore, it holds an arbitrary arrangement of independently placed/sized widgets (photos/calendar/ whiteboard, including several of the same type), each rendered into its own region and composited together, with NEXT/BACK each mapped to their own ordered list of per-widget actions rather than one fixed meaning. See docs/widgets.md for the widget system's data model, placement grid, and button-action dispatch.

Firmware boot flow

1. No stored config (first boot, or NVS erased): bring up the display, render a WiFi-join QR code + plaintext password (left) and a captive-portal config QR code (right), then start the ESPRESSO_XXXXXX softAP + DNS redirect + HTTP server. The display goes up before the AP so the join instructions are visible before the network is joinable. The captive portal form saves SSID/password/toolsserver to NVS and reboots.

2. Stored config exists: connect to the saved WiFi network (a few retries before falling back to provisioning if it fails). The first attempt tries a cached BSSID/channel + static IP from the last successful connection, skipping the scan and DHCP; a bad cache falls back to a normal attempt and gets cleared (see firmware/README.md's "WiFi fast-connect" section). Then run the fetch cycle in frame_client.c:

   • Check the next-photo and back-photo buttons (next_button_check(), back_button_check()) -- if either was what woke the device (checked via the latched esp_sleep_get_gpio_wakeup_status(), not a live pin read, since a quick tap can release before boot gets around to polling it) or is currently held, the fetch below hits POST /frame/advance or POST /frame/back instead of GET /frame/image, forcing the server to move in that direction immediately (next takes priority if somehow both read pressed at once).
   • Fetch the frame and write it into the panel's SPI buffer (epd_write_frame()), computing a CRC32 as it streams -- never buffering the full packed frame in RAM (~192KB for the 7.3" panel; proportionally more for the 13.3" panel). The panel driver refuses to write a short/wrong-size response into the buffer at all, so a truncated fetch can't corrupt what's already there.
   • Compare the new CRC32 against the last one that was actually refreshed onto the panel (persisted in NVS). If it matches -- the same photo is already visibly on screen, e.g. the device rebooted before the server's refresh interval elapsed -- skip the physical refresh entirely (epd_turn_on_display()), avoiding its visible flash and 15-30s duration for no visual change. Otherwise trigger the refresh and store the new CRC.
   • GET /frame/config for the refresh interval, used to set the deep sleep duration -- deliberately fetched after the image, not before: its timeout is much tighter (3s vs. the image fetch's 15s), and fetching second lets it ride the connection the image fetch just warmed up rather than eating the latency spike common on the first request after waking from a long sleep.
   • Deep sleep for the server-configured interval on success, or a shorter retry interval on any failure.

   The menu/reset button's soft-reset (quick press) and factory-reset (held ~15s) tiers are handled earlier, before any of this, and never return -- see firmware/README.md.

See docs/hardware.md for wiring and server/README.md for the server side.

Why image processing happens server-side

The ESP32-C6 has no PSRAM and a tight SRAM budget (already tight enough that a single 4KB stack buffer caused a crash during development -- see git history). Decoding a JPEG, then resizing/dithering/quantizing it to the panel's 6-color palette, would be expensive on-device in both memory and battery. Instead, the server does all of that with Pillow and hands the frame a pre-packed, ready-to-stream buffer -- the device never decodes an image at all. The ee02 board's ESP32-S3 does have PSRAM, but the same server-side design applies there too, for consistency and battery reasons rather than because the C6's memory limit forces it.

Why face detection isn't run on-device (or even on the server)

Immich already runs face detection for its own "People" feature. The server just asks Immich for the bounding boxes it already computed (GET /api/faces?id=...) and biases the crop to keep them on screen, rather than bundling a detector (OpenCV/dlib) anywhere in this project.