espresso_frame/docs/widgets.md

Widget system

A frame's panel isn't one fixed "mode" anymore -- it holds N independently placed/sized widgets (photos/calendar/whiteboard/tasks/static image/text), like arranging icons on an Android home screen. A frame can hold several widgets of the same type (e.g. two photo widgets pointed at different Immich albums side by side). This replaced an earlier design where Frame.mode picked exactly one full-panel renderer; that column (and the other now-dead per-mode Frame columns it left behind -- album_id, calendar_*, whiteboard_*, etc.) is still physically present but unused, pending a final cleanup migration (see "Known gaps" below).

The device-facing contract is unchanged by any of this: GET /frame/image, POST /frame/advance, POST /frame/back are the same frozen paths firmware has always called (see docs/architecture.md) -- what changed is entirely server-side, in how those endpoints decide what to render and what a button press does.

Data model

• Widget (server/app/models.py): id, frame_id, widget_type ("photos" | "calendar" | "whiteboard" | "tasks" | "static" | "text"), x/y/w/h (grid cells), sort_order. Widgets never overlap (enforced server-side in routers/api_widgets.py, re-validated regardless of what the client already checked) -- that's what keeps compositing simple: no z-order, no blending, just N independent regions pasted onto one shared canvas.
• Per-type 1:1 extension tables -- PhotoWidgetConfig, CalendarWidgetConfig, WhiteboardWidgetConfig, TaskWidgetConfig, StaticWidgetConfig, TextWidgetConfig, each keyed by widget_id with ondelete="CASCADE" -- rather than one wide table with every type's mostly-irrelevant columns. TextWidgetConfig.content is parsed rich text (paragraphs of styled runs), never raw HTML -- see server/app/text_content.py's module docstring for why that parse step is the widget's actual stored-XSS sanitization boundary. PhotoWidgetConfig mirrors app/photo_queue.py's attribute names exactly, so that module's advance/back/queue logic ports across widget instances unchanged. TaskWidgetConfig used to be a handful of tasks_* columns bolted onto CalendarWidgetConfig (a week-view-only, single-list task list); split into its own widget type (migration 17) so a task list can be placed and sized independent of any calendar's view/footprint, then (migration 18) given the same multi-source shape a calendar widget already has.
• FrameCalendar/FrameTaskList are keyed by widget_id (not frame_id) since a frame can now have more than one independent calendar/tasks widget, each with its own included set. Identical shape and permission model (owner-added, anyone-linked-can-mute, see "Per-widget config UI" below) -- FrameTaskList just has no "ics" calendar_key variant, since a plain ICS subscription has no VTODO (task) collection.
• FrameButtonAction (id, frame_id, button ["next"|"back"], widget_id, action, sort_order) -- see "Button actions" below.

Placement: a grid, not freeform pixels

app/grid.py is pure grid math, no I/O. The grid is GRID_LONG=8 x GRID_SHORT=5 cells, defined relative to the panel's long/short axis (not "landscape" specifically) so it stays valid across image_pipeline.logical_render_size(orientation)'s genuine width/height swap for portrait -- landscape orientations are 8 cols x 5 rows, portrait are 5 cols x 8 rows, same cell size either way. Changing a frame's orientation invalidates its existing layout (an 8x5 arrangement isn't valid on a 5x8 grid) -- the server resets to one full-panel widget on an orientation change rather than trying to remap coordinates.

Each widget type has a minimum grid footprint (grid.MIN_FOOTPRINT): photos 1x1, calendar 3x2 (a crammed calendar is illegible regardless of size-tier scaling), whiteboard 2x2, tasks 2x2, static image 1x1, text 2x1. Enforced both client-side (UX, in the Layout tab's drag/resize canvas -- static/frame_layout.js) and server-side (routers/api_widgets.py) -- the client is never trusted alone.

Rendering: one shared compositor

app/widgets/ is the render/action registry -- one module per widget_type (photos.py, calendar.py, whiteboard.py, tasks.py, static_image.py, text.py), each exposing:

• render(db, frame, widget, target_w, target_h, is_normal_wake) -> Image: an unquantized RGB image exactly target_w x target_h, the widget's content composed into its own region. Never raises for a foreseeable failure (an Immich hiccup, an unconfigured widget) -- falls back to a small placeholder within its own region instead, so one widget having a bad moment doesn't blank the whole panel.
• ACTIONS: dict[str, Callable] -- named button actions this type supports ("advance"/"back" for photos and calendar, "check_now" for whiteboard). Empty for tasks, static image, and text -- nothing to advance/back/force for a passive checklist, a fixed uploaded image, or a fixed block of authored text.
• ACTION_LABELS: dict[str, str] -- human labels for the button- assignment UI.

routers/device.py's _render_widgets loads every Widget row for the frame, maps each one's grid rect to pixels (grid.cell_to_pixels), calls its module's render(), and hands the whole list of (rect, image) regions to image_pipeline.render_panel -- which pastes every region onto one shared canvas, then runs enhance/manage-overlay/quantize/dither/pack once over the composited result. Quantizing the whole canvas together (not each region separately before pasting) is what keeps the 6-color e-ink dithering pattern consistent across a widget boundary instead of a visible seam at the edge.

Calendar widgets pick from discrete size tiers (calendar_render.py's _SIZE_TIERS) for font size/margins/row heights based on their actual grid footprint, rather than continuously scaling constants tuned for a full ~800x480 canvas -- falls back to agenda view if a widget is too small for month view to stay legible.

Button actions

Each physical button (NEXT/BACK) maps to an ordered list of (widget, action) bindings, not a fixed meaning -- e.g. NEXT can be "photo widget A: advance" and "calendar widget B: advance" together, or even a mismatched combination on purpose. On a press, routers/device.py's _run_button_actions runs every assigned action for that button in order (each in its own widget_locked span -- never nested, since the underlying per-frame lock isn't reentrant), catching and logging any single action's failure without blocking the rest, then re-renders and returns the whole composed panel once at the end regardless of which actions succeeded.

The web UI for this is the "Button assignments" card on a frame's Configuration tab (static/frame_config.js, GET/PUT /api/frames/{id}/buttons) -- add/remove/reorder, autosaved. Two widgets of the same type would otherwise both just say "Photos" in the assignment dropdowns; the UI disambiguates using each widget's grid position (e.g. "Photos 1 (left)" / "Photos 2 (right)"), the same way you'd tell them apart by eye on the Layout canvas.

A newly-created widget (including the one auto-migrated from a frame's old mode on upgrade) gets a sensible default binding reproducing its old button behavior -- see migration.py's _default_button_actions.

Per-widget config UI

Each widget has a gear-icon button on the Layout canvas that opens a <dialog> with that widget's own settings (album, calendar/task-list inclusion, whiteboard source, etc.) -- not a per-frame tab, since a frame can now have several widgets of the same type with independent settings. The dialog HTML is injected server-rendered (routers/frame_pages.py's widget_dialog, dispatching on widget.widget_type); its JS is a top-level, always-loaded file (static/widget_dialog_*.js) exposing init<Type>Dialog()/close<Type>Dialog(), since dynamically-injected HTML can't carry executable <script> tags. While a dialog is open, window.FRAME_API is repointed at that widget's own API base (/api/frames/{id}/widgets/{widget_id}) and restored on close; window.FRAME_BASE_API stays pointed at the frame-level base throughout for the always-present header/status-bar JS.

Saved layouts

A user can snapshot a frame's whole widget arrangement -- every widget's type/placement/settings, calendar/task sources, and button-action bindings -- under a name (SavedLayout + SavedLayoutWidget + SavedLayoutSource + SavedLayoutButtonAction, server/app/models.py), then switch back to it later, or apply it to a different frame. Saved layouts are owned by the user, not any one frame -- the same set shows up (with a per-frame compatible flag) on every frame that user controls whose grid matches (grid.grid_dims(orientation)'s cols/rows, landscape-class 8x5 vs. portrait-class 5x8), not just the frame it was captured from.

Saving only captures an authored setting, never runtime/cache state -- a photo widget's current queue position, a calendar's fetch cache, a whiteboard's rendered-image cache, etc. are deliberately left out (see routers/api_layouts.py's LAYOUT_CONFIG_FIELDS allowlist per widget_type), so applying a layout feels like a fresh widget of that type with its settings pre-filled, not a resurrection of stale state from whenever it was saved. A static-image widget's uploaded bytes are the one exception carried through verbatim (SavedLayoutWidget.image). Saving again under a name the user already has overwrites that layout's snapshot in place rather than erroring or creating a duplicate -- SavedLayout's own docstring.

Applying a layout to a frame (api_layout_apply, require_frame_control) deletes every widget currently on that frame and recreates the saved arrangement from scratch, remapping calendar/task sources and button bindings onto the newly-created widget ids -- same "act unconditionally on the server, confirm on the client" posture as the Layout tab's own "Clear all". A source whose owning user account no longer exists is silently dropped rather than left dangling (config is JSON, not FK-checked, so nothing else would catch that).

The web UI lives in the Layout tab's "Saved layouts" card (static/saved_layouts.js, GET/POST /api/frames/{id}/layouts, PATCH/DELETE /api/layouts/{id}, POST /api/frames/{id}/layouts/{id}/apply) -- name + Save, then a list of saved layouts each with Apply/rename/delete, incompatible ones shown greyed-out with a "different orientation" badge rather than hidden.

Known gaps (Phase 6, not yet done)

The original 8-phase rollout plan's last phase is still open:

• Legacy per-mode Frame columns (mode, album_id, current_asset_id, all calendar_*, all whiteboard_*, etc.) are still physically present in the schema but no longer read or written anywhere -- they need a dedicated final migration to drop them. Left in place deliberately through the widget-system rollout (a much larger blast radius cutover than this project's usual same-migration-drop convention) but there's no reason to keep carrying them now that every phase has shipped.
• server/README.md still describes photos/calendar/whiteboard as per-frame "modes" in several places rather than widgets -- needs a pass once the column drop above is safely deployed.
• Whiteboard rendering is tagged (alpha) in the UI -- not fully reliable yet, treat it as experimental if extending it.