espresso_frame/docs/architecture.md

Architecture

Two independent pieces talk over plain HTTP on the local network: the ESP32-C6 firmware, and a small FastAPI server that sits between it and Immich.

sequenceDiagram
    participant Immich
    participant Server as ESPresso Frame Server
    participant Frame as ESP32-C6 Frame

    Note over Frame: First boot / never provisioned
    Frame->>Frame: Generate AP SSID/password, draw QR + config QR on panel
    Frame->>Frame: Bring up ESPRESSO_XXXXXX softAP + captive portal
    Note over Frame: User scans WiFi QR, then config QR -> fills in<br/>home WiFi + "Tools Server" host:port
    Frame->>Frame: Save config to NVS, reboot

    Note over Frame: Every wake (deep sleep timer)
    Frame->>Frame: Connect to home WiFi
    Frame->>Server: GET /frame/config
    Server-->>Frame: {"refresh_interval_s": ...}
    alt server unreachable
        Frame->>Frame: Show "SERVER: FAILED" status screen
        Frame->>Frame: Deep sleep (short retry interval)
    else server reachable
        Frame->>Server: GET /frame/image
        Server->>Immich: List album assets / download preview / faces
        Immich-->>Server: JPEG + face bounding boxes
        Server->>Server: Crop (face-aware) + quantize (dither) + pack 4bpp
        Server-->>Frame: 192,000 raw bytes, streamed
        Frame->>Frame: Stream straight to panel SPI, refresh
        Frame->>Frame: Deep sleep (server-configured interval)
    end

Firmware boot flow

1. No stored config (first boot, or NVS erased): bring up the display, render a WiFi-join QR code + plaintext password (left) and a captive-portal config QR code (right), then start the ESPRESSO_XXXXXX softAP + DNS redirect + HTTP server. The display goes up before the AP so the join instructions are visible before the network is joinable. The captive portal form saves SSID/password/toolsserver to NVS and reboots.
2. Stored config exists: connect to the saved WiFi network (a few retries before falling back to provisioning if it fails), then run the fetch cycle in frame_client.c:
   • GET /frame/config on the configured tools server -- doubles as a reachability check and the source of the refresh interval (a Kconfig value is only used as a fallback).
   • If reachable, GET /frame/image and stream the response directly into the panel over SPI (epd_display_stream()), never buffering the full ~192KB frame in RAM.
   • The panel driver refuses to physically refresh unless the stream supplied exactly the expected byte count -- a truncated or wrong-size response leaves the previous image on screen instead of painting garbage.
   • Deep sleep for the server-configured interval on success, or a shorter retry interval on any failure.

See docs/hardware.md for wiring and server/README.md for the server side.

Why image processing happens server-side

The ESP32-C6 has no PSRAM and a tight SRAM budget (already tight enough that a single 4KB stack buffer caused a crash during development -- see git history). Decoding a JPEG, then resizing/dithering/quantizing it to the panel's 6-color palette, would be expensive on-device in both memory and battery. Instead, the server does all of that with Pillow and hands the frame a pre-packed, ready-to-stream buffer -- the device never decodes an image at all.

Why face detection isn't run on-device (or even on the server)

Immich already runs face detection for its own "People" feature. The server just asks Immich for the bounding boxes it already computed (GET /api/faces?id=...) and biases the crop to keep them on screen, rather than bundling a detector (OpenCV/dlib) anywhere in this project.