espresso_frame/docs/architecture.md

Architecture

Two independent pieces talk over plain HTTP on the local network: the ESP32-C6 firmware, and a small FastAPI server that sits between it and Immich.

sequenceDiagram
    participant Immich
    participant Server as ESPresso Frame Server
    participant Frame as ESP32-C6 Frame

    Note over Frame: First boot / never provisioned
    Frame->>Frame: Generate AP SSID/password, draw QR + config QR on panel
    Frame->>Frame: Bring up ESPRESSO_XXXXXX softAP + captive portal
    Note over Frame: User scans WiFi QR, then config QR -> fills in<br/>home WiFi + "Tools Server" host:port
    Frame->>Frame: Save config to NVS, reboot

    Note over Frame: Every wake (deep sleep timer, next-photo button,<br/>or any other reboot)
    Frame->>Frame: Connect to home WiFi
    alt next-photo button pressed
        Frame->>Server: POST /frame/advance
        Server->>Server: Force-advance to next queued photo, reset interval clock
    else normal wake
        Frame->>Server: GET /frame/image
        Server->>Server: Advance only if refresh_interval_s has elapsed<br/>since the current photo was set -- otherwise a no-op
    end
    Server->>Immich: List album assets / download preview / faces
    Immich-->>Server: JPEG + face bounding boxes
    Server->>Server: Crop (face-aware) + quantize (dither) + pack 4bpp
    Server-->>Frame: 192,000 raw bytes, streamed
    Frame->>Frame: Write to panel SPI buffer, compute CRC32
    alt CRC unchanged since last physical refresh
        Frame->>Frame: Skip refresh (nothing visually changed)
    else CRC changed
        Frame->>Frame: Trigger physical refresh, store new CRC
    end
    Frame->>Server: GET /frame/config
    Server-->>Frame: {"refresh_interval_s": ...}
    Frame->>Frame: Deep sleep (server-configured interval, or a short<br/>retry interval on any failure)

Firmware boot flow

1. No stored config (first boot, or NVS erased): bring up the display, render a WiFi-join QR code + plaintext password (left) and a captive-portal config QR code (right), then start the ESPRESSO_XXXXXX softAP + DNS redirect + HTTP server. The display goes up before the AP so the join instructions are visible before the network is joinable. The captive portal form saves SSID/password/toolsserver to NVS and reboots.

2. Stored config exists: connect to the saved WiFi network (a few retries before falling back to provisioning if it fails), then run the fetch cycle in frame_client.c:

   • Check the next-photo button (next_button_check()) -- if it was what woke the device (checked via the latched esp_sleep_get_gpio_wakeup_status(), not a live pin read, since a quick tap can release before boot gets around to polling it) or is currently held, the fetch below hits POST /frame/advance instead of GET /frame/image, forcing the server to skip ahead immediately.
   • Fetch the frame and write it into the panel's SPI buffer (epd_write_frame()), computing a CRC32 as it streams -- never buffering the full ~192KB frame in RAM. The panel driver refuses to write a short/wrong-size response into the buffer at all, so a truncated fetch can't corrupt what's already there.
   • Compare the new CRC32 against the last one that was actually refreshed onto the panel (persisted in NVS). If it matches -- the same photo is already visibly on screen, e.g. the device rebooted before the server's refresh interval elapsed -- skip the physical refresh entirely (epd_turn_on_display()), avoiding its visible flash and 15-30s duration for no visual change. Otherwise trigger the refresh and store the new CRC.
   • GET /frame/config for the refresh interval, used to set the deep sleep duration -- deliberately fetched after the image, not before: its timeout is much tighter (3s vs. the image fetch's 15s), and fetching second lets it ride the connection the image fetch just warmed up rather than eating the latency spike common on the first request after waking from a long sleep.
   • Deep sleep for the server-configured interval on success, or a shorter retry interval on any failure.

   The factory-reset button (hold 10s) is checked earlier, before any of this -- see firmware/README.md.

See docs/hardware.md for wiring and server/README.md for the server side.

Why image processing happens server-side

The ESP32-C6 has no PSRAM and a tight SRAM budget (already tight enough that a single 4KB stack buffer caused a crash during development -- see git history). Decoding a JPEG, then resizing/dithering/quantizing it to the panel's 6-color palette, would be expensive on-device in both memory and battery. Instead, the server does all of that with Pillow and hands the frame a pre-packed, ready-to-stream buffer -- the device never decodes an image at all.

Why face detection isn't run on-device (or even on the server)

Immich already runs face detection for its own "People" feature. The server just asks Immich for the bounding boxes it already computed (GET /api/faces?id=...) and biases the crop to keep them on screen, rather than bundling a detector (OpenCV/dlib) anywhere in this project.