tfaour
fe5a2df074
docs/hardware.md and firmware/README.md were updated already; this catches the root README, docs/architecture.md, docs/widgets.md, and server/README.md -- all still described the project as single-panel/ single-chip (800x480, ESP32-C6 only) even after image_pipeline.py stopped hardcoding that.
132 lines
7.3 KiB
Markdown
132 lines
7.3 KiB
Markdown
# Architecture
|
|
|
|
Two independent pieces talk over HTTP or HTTPS (the server itself always
|
|
speaks plain HTTP; HTTPS means a reverse proxy in front of it, see
|
|
[`firmware/README.md`](../firmware/README.md#http-vs-https)) on the local
|
|
network: the ESP32 firmware (ESP32-C6 for the devkit/xiao boards,
|
|
ESP32-S3 for ee02 -- see [`docs/hardware.md`](hardware.md)), and a small
|
|
FastAPI server that sits between it and Immich.
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant Immich
|
|
participant Server as ESPresso Frame Server
|
|
participant Frame as ESP32 Frame
|
|
|
|
Note over Frame: First boot / never provisioned
|
|
Frame->>Frame: Generate AP SSID/password, draw QR + config QR on panel
|
|
Frame->>Frame: Bring up ESPRESSO_XXXXXX softAP + captive portal
|
|
Note over Frame: User scans WiFi QR, then config QR -> fills in<br/>home WiFi + "Tools Server" host:port
|
|
Frame->>Frame: Save config to NVS, reboot
|
|
|
|
Note over Frame: Every wake (deep sleep timer, next/back-photo button,<br/>or any other reboot)
|
|
Frame->>Frame: Connect to home WiFi
|
|
alt next-photo button pressed
|
|
Frame->>Server: POST /frame/advance
|
|
Server->>Server: Run every action assigned to NEXT, in order<br/>(may span several widgets -- see docs/widgets.md)
|
|
else back-photo button pressed
|
|
Frame->>Server: POST /frame/back
|
|
Server->>Server: Run every action assigned to BACK, in order
|
|
else normal wake
|
|
Frame->>Server: GET /frame/image
|
|
Server->>Server: Render every widget on the panel into its own region<br/>(each independently idempotent -- a photo widget only<br/>actually advances once its own refresh_interval_s has elapsed)
|
|
end
|
|
Server->>Immich: List album assets / download preview / faces<br/>(once per photo widget on the panel)
|
|
Immich-->>Server: JPEG + face bounding boxes
|
|
Server->>Server: Composite every widget's region onto one canvas,<br/>then enhance/overlay/quantize (dither)/pack 4bpp once
|
|
Server-->>Frame: packed 4bpp bytes, streamed<br/>(192,000 for the 7.3" panel; sized to whichever<br/>panel this frame's device reports, see Frame.panel_type)
|
|
Frame->>Frame: Write to panel SPI buffer, compute CRC32
|
|
alt CRC unchanged since last physical refresh
|
|
Frame->>Frame: Skip refresh (nothing visually changed)
|
|
else CRC changed
|
|
Frame->>Frame: Trigger physical refresh, store new CRC
|
|
end
|
|
Frame->>Server: GET /frame/config
|
|
Server-->>Frame: {"refresh_interval_s": ...}
|
|
Frame->>Frame: Deep sleep (server-configured interval, or a short<br/>retry interval on any failure)
|
|
```
|
|
|
|
The device-facing endpoints above (`/frame/image`, `/frame/advance`,
|
|
`/frame/back`, `/frame/config`) are frozen -- baked into deployed firmware
|
|
-- and unchanged by any of this. What *does* change server-side: a frame's
|
|
panel isn't a single fixed "mode" anymore, it holds an arbitrary
|
|
arrangement of independently placed/sized widgets (photos/calendar/
|
|
whiteboard, including several of the same type), each rendered into its
|
|
own region and composited together, with NEXT/BACK each mapped to their
|
|
own ordered list of per-widget actions rather than one fixed meaning. See
|
|
[`docs/widgets.md`](widgets.md) for the widget system's data model,
|
|
placement grid, and button-action dispatch.
|
|
|
|
## Firmware boot flow
|
|
|
|
1. **No stored config** (first boot, or NVS erased): bring up the display,
|
|
render a WiFi-join QR code + plaintext password (left) and a
|
|
captive-portal config QR code (right), *then* start the `ESPRESSO_XXXXXX`
|
|
softAP + DNS redirect + HTTP server. The display goes up before the AP
|
|
so the join instructions are visible before the network is joinable.
|
|
The captive portal form saves SSID/password/toolsserver to NVS and
|
|
reboots.
|
|
2. **Stored config exists**: connect to the saved WiFi network (a few
|
|
retries before falling back to provisioning if it fails). The first
|
|
attempt tries a cached BSSID/channel + static IP from the last
|
|
successful connection, skipping the scan and DHCP; a bad cache falls
|
|
back to a normal attempt and gets cleared (see `firmware/README.md`'s
|
|
"WiFi fast-connect" section). Then run the fetch cycle in
|
|
`frame_client.c`:
|
|
- Check the next-photo and back-photo buttons (`next_button_check()`,
|
|
`back_button_check()`) -- if either was what woke the device
|
|
(checked via the latched `esp_sleep_get_gpio_wakeup_status()`, not
|
|
a live pin read, since a quick tap can release before boot gets
|
|
around to polling it) or is currently held, the fetch below hits
|
|
`POST /frame/advance` or `POST /frame/back` instead of
|
|
`GET /frame/image`, forcing the server to move in that direction
|
|
immediately (next takes priority if somehow both read pressed at
|
|
once).
|
|
- Fetch the frame and write it into the panel's SPI buffer
|
|
(`epd_write_frame()`), computing a CRC32 as it streams -- never
|
|
buffering the full packed frame in RAM (~192KB for the 7.3" panel;
|
|
proportionally more for the 13.3" panel). The panel driver refuses
|
|
to write a short/wrong-size response into the buffer at all, so a
|
|
truncated fetch can't corrupt what's already there.
|
|
- Compare the new CRC32 against the last one that was actually
|
|
refreshed onto the panel (persisted in NVS). If it matches -- the
|
|
same photo is already visibly on screen, e.g. the device rebooted
|
|
before the server's refresh interval elapsed -- skip the physical
|
|
refresh entirely (`epd_turn_on_display()`), avoiding its visible
|
|
flash and 15-30s duration for no visual change. Otherwise trigger
|
|
the refresh and store the new CRC.
|
|
- `GET /frame/config` for the refresh interval, used to set the deep
|
|
sleep duration -- deliberately fetched *after* the image, not
|
|
before: its timeout is much tighter (3s vs. the image fetch's 15s),
|
|
and fetching second lets it ride the connection the image fetch just
|
|
warmed up rather than eating the latency spike common on the first
|
|
request after waking from a long sleep.
|
|
- Deep sleep for the server-configured interval on success, or a
|
|
shorter retry interval on any failure.
|
|
|
|
The menu/reset button's soft-reset (quick press) and factory-reset
|
|
(held ~15s) tiers are handled earlier, before any of this, and never
|
|
return -- see [`firmware/README.md`](../firmware/README.md#managing-the-queue-soft-resetting-and-factory-resetting).
|
|
|
|
See [`docs/hardware.md`](hardware.md) for wiring and
|
|
[`server/README.md`](../server/README.md) for the server side.
|
|
|
|
## Why image processing happens server-side
|
|
|
|
The ESP32-C6 has no PSRAM and a tight SRAM budget (already tight enough
|
|
that a single 4KB stack buffer caused a crash during development -- see
|
|
git history). Decoding a JPEG, then resizing/dithering/quantizing it to
|
|
the panel's 6-color palette, would be expensive on-device in both memory
|
|
and battery. Instead, the server does all of that with Pillow and hands
|
|
the frame a pre-packed, ready-to-stream buffer -- the device never
|
|
decodes an image at all. The ee02 board's ESP32-S3 does have PSRAM, but
|
|
the same server-side design applies there too, for consistency and
|
|
battery reasons rather than because the C6's memory limit forces it.
|
|
|
|
## Why face detection isn't run on-device (or even on the server)
|
|
|
|
Immich already runs face detection for its own "People" feature. The
|
|
server just asks Immich for the bounding boxes it already computed
|
|
(`GET /api/faces?id=...`) and biases the crop to keep them on screen,
|
|
rather than bundling a detector (OpenCV/dlib) anywhere in this project.
|