Add CLAUDE.md and docs/widgets.md; fix stale single-mode architecture docs

Repo-tracked context so a fresh Claude Code session (this machine or a
remote/cloud one) gets accurate project context without relying on this
session's local, machine-specific memory: repo conventions (no
co-author trailers, flag copyleft deps explicitly, scope security
gates broadly -- each backed by a real past incident), testing/deploy
workflow, and pointers into the existing docs.

docs/architecture.md's sequence diagram and boot-flow text still
described the pre-widget-system single-photo-queue model (e.g. "force-
advance to next queued photo") even though that was fully replaced by
the widget system across this branch's recent history -- fixed, and
added docs/widgets.md distilling the widget system's actual design
(data model, grid placement, compositor, button-action dispatch) as
current-state documentation, including the still-open Phase 6 cleanup
(legacy Frame columns not yet dropped) as a known gap.
This commit is contained in:
2026-07-24 20:04:08 -04:00
parent 289d308b57
commit 914eaed71c
4 changed files with 239 additions and 7 deletions
+71
View File
@@ -0,0 +1,71 @@
# espresso_frame
A DIY e-ink photo frame: an ESP32-C6 (`firmware/`, ESP-IDF) driving a
Waveshare 7.3" E Ink Spectra 6 panel (800x480, 6-color, SPI), paired with a
self-hosted FastAPI server (`server/`) that pulls from Immich, does all
image processing (crop/dither/quantize/pack), and serves a placeable
photos/calendar/whiteboard widget system to the device.
Start here, don't re-derive from scratch:
- [`docs/architecture.md`](docs/architecture.md) -- how firmware and
server talk (sequence diagram, boot flow).
- [`docs/widgets.md`](docs/widgets.md) -- the server-side widget system
(data model, grid placement, compositor, button-action dispatch). Notes
a known gap at the bottom (legacy `Frame` columns not yet dropped).
- [`docs/hardware.md`](docs/hardware.md) -- wiring.
- [`server/README.md`](server/README.md), [`firmware/README.md`](firmware/README.md)
-- per-component setup, config, and a lot of accumulated gotchas
(Immich API shape, TLS trust-anchor details, button GPIO wakeup
quirks, etc.) -- check these before assuming something is a new bug.
## Conventions specific to this repo
- **No `Co-Authored-By: Claude` trailers in commits.** Attribution lives
in the root [`README.md`](README.md) instead (see its last line) --
the maintainer's explicit preference, not the default.
- **Copyleft dependencies need an explicit flag, not a silent decision.**
Before adding anything LGPL/GPL/AGPL (or unclear), verify the actual
license via `pip show`/package metadata -- including transitive deps,
not just the top-level package -- and present the finding and tradeoff
in plain text rather than picking an approach unilaterally (hand-rolling
an alternative, swapping packages, silently accepting it). This project
has knowingly accepted AGPL-3.0-or-later exposure once already
(`icalendar-searcher`, a transitive dep of `caldav`) as a deliberate,
explicit call -- not a precedent for skipping the check next time.
- **Scope new auth/access-control broadly, not just to the literal
endpoint named.** When a request changes the trust model (e.g. adding
public-internet exposure), apply the new gate to every endpoint serving
real data or performing a real action, and call out anything you're
tempted to exclude and why. This repo shipped a token gate once that
covered `/api/*` but left `/frame/image` -- the actual photo bytes --
open; caught immediately in production.
## Working in this repo
- **Server tests**: `cd server && pytest` (SQLite, fixtures wipe/reseed
between tests -- see `tests/conftest.py`). Migration changes need a
matching test in `tests/test_migrations.py`; anything touching
`require_frame_view`/`require_frame_control` boundaries needs a
same-shape permission test (see `tests/test_permission_boundaries.py`
and `tests/test_button_actions.py` for the pattern: owner, linked user,
unrelated user, logged out).
- **UI changes**: verify in a real browser (Playwright), not just by
reading the JS -- this project has hit multiple bugs that only showed up
live (mobile viewport CSS collapse, a dialog's status message landing
behind its own backdrop, a JSON/form-urlencoded body mismatch). Spin up
`uvicorn app.main:app` against a scratch `DATABASE_URL`/`CONFIG_PATH`
sqlite file, don't touch the real deployment's data.
- **Deploy**: Gitea Actions at `git.thumeit.com/tfaour/espresso_frame`
(`.gitea/workflows/server-docker-build.yml`: `test` -> `build-and-push`
-> `deploy` on any push to `main` touching `server/**`; `deploy` SSHes
into the host as `espressoframe_deployer` and runs `docker compose pull
&& docker compose up -d`). A separate workflow
(`firmware-release-build.yml`) builds+publishes firmware binaries as
Gitea release assets when `firmware/version.txt` changes. Poll CI status
with `curl https://git.thumeit.com/api/v1/repos/tfaour/espresso_frame/actions/tasks`
rather than asking the user to check.
- **Device-facing paths are frozen.** `/frame/image`, `/frame/advance`,
`/frame/back`, `/frame/config`, `/frame/battery`, `/frame/firmware` and
their exact JSON key names (`refresh_interval_s`, `firmware_version`,
etc.) are baked into deployed firmware -- never rename or restructure
these without a firmware-side migration story to match.
+3 -2
View File
@@ -24,9 +24,10 @@ time in deep sleep.
- ESP32-C6 dev board (8MB flash)
- [Waveshare 7.3" E Ink Spectra 6 (E6)](https://www.waveshare.com/7.3inch-e-paper-hat-e.htm) panel -- 800x480, 6-color, SPI
See [`docs/hardware.md`](docs/hardware.md) for wiring and
See [`docs/hardware.md`](docs/hardware.md) for wiring,
[`docs/architecture.md`](docs/architecture.md) for how the two halves talk
to each other.
to each other, and [`docs/widgets.md`](docs/widgets.md) for the server's
placeable photos/calendar/whiteboard widget system.
## Getting started
+16 -5
View File
@@ -22,17 +22,17 @@ sequenceDiagram
Frame->>Frame: Connect to home WiFi
alt next-photo button pressed
Frame->>Server: POST /frame/advance
Server->>Server: Force-advance to next queued photo, reset interval clock
Server->>Server: Run every action assigned to NEXT, in order<br/>(may span several widgets -- see docs/widgets.md)
else back-photo button pressed
Frame->>Server: POST /frame/back
Server->>Server: Return to previously-current photo (bounded history),<br/>reset interval clock
Server->>Server: Run every action assigned to BACK, in order
else normal wake
Frame->>Server: GET /frame/image
Server->>Server: Advance only if refresh_interval_s has elapsed<br/>since the current photo was set -- otherwise a no-op
Server->>Server: Render every widget on the panel into its own region<br/>(each independently idempotent -- a photo widget only<br/>actually advances once its own refresh_interval_s has elapsed)
end
Server->>Immich: List album assets / download preview / faces
Server->>Immich: List album assets / download preview / faces<br/>(once per photo widget on the panel)
Immich-->>Server: JPEG + face bounding boxes
Server->>Server: Crop (face-aware) + quantize (dither) + pack 4bpp
Server->>Server: Composite every widget's region onto one canvas,<br/>then enhance/overlay/quantize (dither)/pack 4bpp once
Server-->>Frame: 192,000 raw bytes, streamed
Frame->>Frame: Write to panel SPI buffer, compute CRC32
alt CRC unchanged since last physical refresh
@@ -45,6 +45,17 @@ sequenceDiagram
Frame->>Frame: Deep sleep (server-configured interval, or a short<br/>retry interval on any failure)
```
The device-facing endpoints above (`/frame/image`, `/frame/advance`,
`/frame/back`, `/frame/config`) are frozen -- baked into deployed firmware
-- and unchanged by any of this. What *does* change server-side: a frame's
panel isn't a single fixed "mode" anymore, it holds an arbitrary
arrangement of independently placed/sized widgets (photos/calendar/
whiteboard, including several of the same type), each rendered into its
own region and composited together, with NEXT/BACK each mapped to their
own ordered list of per-widget actions rather than one fixed meaning. See
[`docs/widgets.md`](widgets.md) for the widget system's data model,
placement grid, and button-action dispatch.
## Firmware boot flow
1. **No stored config** (first boot, or NVS erased): bring up the display,
+149
View File
@@ -0,0 +1,149 @@
# Widget system
A frame's panel isn't one fixed "mode" anymore -- it holds N independently
placed/sized widgets (photos/calendar/whiteboard), like arranging icons on
an Android home screen. A frame can hold several widgets of the same type
(e.g. two photo widgets pointed at different Immich albums side by side).
This replaced an earlier design where `Frame.mode` picked exactly one
full-panel renderer; that column (and the other now-dead per-mode `Frame`
columns it left behind -- `album_id`, `calendar_*`, `whiteboard_*`, etc.)
is still physically present but unused, pending a final cleanup migration
(see "Known gaps" below).
The device-facing contract is unchanged by any of this: `GET /frame/image`,
`POST /frame/advance`, `POST /frame/back` are the same frozen paths
firmware has always called (see `docs/architecture.md`) -- what changed is
entirely server-side, in how those endpoints decide what to render and what
a button press does.
## Data model
- `Widget` (`server/app/models.py`): `id`, `frame_id`, `widget_type`
(`"photos"` | `"calendar"` | `"whiteboard"`), `x`/`y`/`w`/`h` (grid
cells), `sort_order`. Widgets never overlap (enforced server-side in
`routers/api_widgets.py`, re-validated regardless of what the client
already checked) -- that's what keeps compositing simple: no z-order,
no blending, just N independent regions pasted onto one shared canvas.
- Per-type 1:1 extension tables -- `PhotoWidgetConfig`,
`CalendarWidgetConfig`, `WhiteboardWidgetConfig`, each keyed by
`widget_id` with `ondelete="CASCADE"` -- rather than one wide table with
every type's mostly-irrelevant columns. `PhotoWidgetConfig` mirrors
`app/photo_queue.py`'s attribute names exactly, so that module's
advance/back/queue logic ports across widget instances unchanged.
- `FrameCalendar` is keyed by `widget_id` (not `frame_id`) since a frame
can now have more than one independent calendar widget, each with its
own set of included calendars.
- `FrameButtonAction` (`id`, `frame_id`, `button` [`"next"`|`"back"`],
`widget_id`, `action`, `sort_order`) -- see "Button actions" below.
## Placement: a grid, not freeform pixels
`app/grid.py` is pure grid math, no I/O. The grid is `GRID_LONG=8` x
`GRID_SHORT=5` cells, defined relative to the panel's long/short axis
(not "landscape" specifically) so it stays valid across
`image_pipeline.logical_render_size(orientation)`'s genuine width/height
swap for portrait -- landscape orientations are 8 cols x 5 rows, portrait
are 5 cols x 8 rows, same cell size either way. **Changing a frame's
orientation invalidates its existing layout** (an 8x5 arrangement isn't
valid on a 5x8 grid) -- the server resets to one full-panel widget on an
orientation change rather than trying to remap coordinates.
Each widget type has a minimum grid footprint (`grid.MIN_FOOTPRINT`):
photos 1x1, calendar 3x2 (a crammed calendar is illegible regardless of
size-tier scaling), whiteboard 2x2. Enforced both client-side (UX, in the
Layout tab's drag/resize canvas -- `static/frame_layout.js`) and
server-side (`routers/api_widgets.py`) -- the client is never trusted
alone.
## Rendering: one shared compositor
`app/widgets/` is the render/action registry -- one module per
`widget_type` (`photos.py`, `calendar.py`, `whiteboard.py`), each
exposing:
- `render(db, frame, widget, target_w, target_h, is_normal_wake) -> Image`:
an unquantized RGB image exactly `target_w x target_h`, the widget's
content composed into its own region. Never raises for a foreseeable
failure (an Immich hiccup, an unconfigured widget) -- falls back to a
small placeholder within its own region instead, so one widget having a
bad moment doesn't blank the whole panel.
- `ACTIONS: dict[str, Callable]` -- named button actions this type
supports (`"advance"`/`"back"` for photos and calendar, `"check_now"`
for whiteboard).
- `ACTION_LABELS: dict[str, str]` -- human labels for the button-
assignment UI.
`routers/device.py`'s `_render_widgets` loads every `Widget` row for the
frame, maps each one's grid rect to pixels (`grid.cell_to_pixels`), calls
its module's `render()`, and hands the whole list of `(rect, image)`
regions to `image_pipeline.render_panel` -- which pastes every region onto
one shared canvas, then runs enhance/manage-overlay/quantize/dither/pack
**once** over the composited result. Quantizing the whole canvas together
(not each region separately before pasting) is what keeps the 6-color
e-ink dithering pattern consistent across a widget boundary instead of a
visible seam at the edge.
Calendar widgets pick from discrete size tiers (`calendar_render.py`'s
`_SIZE_TIERS`) for font size/margins/row heights based on their actual
grid footprint, rather than continuously scaling constants tuned for a
full ~800x480 canvas -- falls back to agenda view if a widget is too small
for month view to stay legible.
## Button actions
Each physical button (NEXT/BACK) maps to an **ordered list** of
`(widget, action)` bindings, not a fixed meaning -- e.g. NEXT can be
"photo widget A: advance" *and* "calendar widget B: advance" together, or
even a mismatched combination on purpose. On a press,
`routers/device.py`'s `_run_button_actions` runs every assigned action for
that button in order (each in its own `widget_locked` span -- never nested,
since the underlying per-frame lock isn't reentrant), catching and
logging any single action's failure without blocking the rest, then
re-renders and returns the whole composed panel once at the end regardless
of which actions succeeded.
The web UI for this is the "Button assignments" card on a frame's
Configuration tab (`static/frame_config.js`, `GET`/`PUT
/api/frames/{id}/buttons`) -- add/remove/reorder, autosaved. Two widgets of
the same type would otherwise both just say "Photos" in the assignment
dropdowns; the UI disambiguates using each widget's grid position (e.g.
"Photos 1 (left)" / "Photos 2 (right)"), the same way you'd tell them
apart by eye on the Layout canvas.
A newly-created widget (including the one auto-migrated from a frame's old
`mode` on upgrade) gets a sensible default binding reproducing its old
button behavior -- see `migration.py`'s `_default_button_actions`.
## Per-widget config UI
Each widget has a gear-icon button on the Layout canvas that opens a
`<dialog>` with that widget's own settings (album, calendar inclusion,
whiteboard source, etc.) -- not a per-frame tab, since a frame can now
have several widgets of the same type with independent settings. The
dialog HTML is injected server-rendered (`routers/frame_pages.py`'s
`widget_dialog`, dispatching on `widget.widget_type`); its JS is a
top-level, always-loaded file (`static/widget_dialog_*.js`) exposing
`init<Type>Dialog()`/`close<Type>Dialog()`, since dynamically-injected
HTML can't carry executable `<script>` tags. While a dialog is open,
`window.FRAME_API` is repointed at that widget's own API base
(`/api/frames/{id}/widgets/{widget_id}`) and restored on close;
`window.FRAME_BASE_API` stays pointed at the frame-level base throughout
for the always-present header/status-bar JS.
## Known gaps (Phase 6, not yet done)
The original 8-phase rollout plan's last phase is still open:
- Legacy per-mode `Frame` columns (`mode`, `album_id`,
`current_asset_id`, all `calendar_*`, all `whiteboard_*`, etc.) are
still physically present in the schema but no longer read or written
anywhere -- they need a dedicated final migration to drop them. Left in
place deliberately through the widget-system rollout (a much larger
blast radius cutover than this project's usual same-migration-drop
convention) but there's no reason to keep carrying them now that every
phase has shipped.
- `server/README.md` still describes photos/calendar/whiteboard as
per-frame "modes" in several places rather than widgets -- needs a pass
once the column drop above is safely deployed.
- Whiteboard rendering is tagged **(alpha)** in the UI -- not fully
reliable yet, treat it as experimental if extending it.