espresso_frame/docs/widgets.md

Widget system

A frame's panel isn't one fixed "mode" anymore -- it holds N independently placed/sized widgets (photos/calendar/whiteboard/tasks/static image/text/ weather/battery), like arranging icons on an Android home screen. A frame can hold several widgets of the same type (e.g. two photo widgets pointed at different Immich albums side by side). This replaced an earlier design where Frame.mode picked exactly one full-panel renderer; that column (and the other now-dead per-mode Frame columns it left behind -- album_id, calendar_*, whiteboard_*, etc.) is still physically present but unused, pending a final cleanup migration (see "Known gaps" below).

The device-facing contract is unchanged by any of this: GET /frame/image, POST /frame/advance, POST /frame/back are the same frozen paths firmware has always called (see docs/architecture.md) -- what changed is entirely server-side, in how those endpoints decide what to render and what a button press does.

Data model

• Widget (server/app/models.py): id, frame_id, widget_type ("photos" | "calendar" | "whiteboard" | "tasks" | "static" | "text" | "weather" | "battery"), x/y/w/h (grid cells), sort_order. Widgets never overlap (enforced server-side in routers/api_widgets.py, re-validated regardless of what the client already checked) -- that's what keeps compositing simple: no z-order, no blending, just N independent regions pasted onto one shared canvas. Also carries an optional per-widget border (border_style -- "none" | "solid" | "dashed" | "dotted" | "fancy", border_thickness, border_color_index, an index into the frame's palette so a border always renders as one of the panel's exact 6 ink colors) directly on Widget itself rather than a per-type config table, since every widget type can have one regardless of widget_type. Drawn by image_pipeline.draw_widget_border onto each widget's own region in routers/device.py's _render_widgets, before that region is pasted onto the shared canvas -- one central integration point instead of every app/widgets/*.py module needing to know about it. Set via the gear-icon dialog's shared "Border" card (_widget_border_fields.html, included by every _widget_dialog_*.html template) and POST .../widgets/{id}/border, its own endpoint (not folded into api_widget_config_save) since that endpoint's per-type dispatch is keyed on a config row via widget_locked, and border fields live on Widget itself, not any per-type config table.
• Per-type 1:1 extension tables -- PhotoWidgetConfig, CalendarWidgetConfig, WhiteboardWidgetConfig, TaskWidgetConfig, StaticWidgetConfig, TextWidgetConfig, WeatherWidgetConfig, BatteryWidgetConfig, each keyed by widget_id with ondelete="CASCADE" -- rather than one wide table with every type's mostly-irrelevant columns. TextWidgetConfig.content is parsed rich text (paragraphs of styled runs), never raw HTML -- see server/app/text_content.py's module docstring for why that parse step is the widget's actual stored-XSS sanitization boundary. PhotoWidgetConfig mirrors app/photo_queue.py's attribute names exactly, so that module's advance/back/queue logic ports across widget instances unchanged. PhotoWidgetConfig.locked (migration 27) freezes current_asset_id against both the timer-elapsed auto-advance (photo_queue.get_current) and the advance/back button actions (app/widgets/photos.py's ACTIONS) until unlocked -- toggled via a "Lock this photo" button in the widget's own dialog (POST .../widgets/{id}/lock), shown as a lock badge on the widget's box on the Layout tab canvas. TaskWidgetConfig used to be a handful of tasks_* columns bolted onto CalendarWidgetConfig (a week-view-only, single-list task list); split into its own widget type (migration 17) so a task list can be placed and sized independent of any calendar's view/footprint, then (migration 18) given the same multi-source shape a calendar widget already has. WeatherWidgetConfig similarly lifts CalendarWidgetConfig's embedded weather strip (still present and unchanged, weather_* columns) out into its own placeable widget type (migration 24) -- see "Weather widget" below. BatteryWidgetConfig (migration 25) is the odd one out -- its actual content (Frame.battery_percent/battery_as_of) isn't in this table at all, already existing frame-level state set by routers/device.py's frame_battery regardless of whether a battery widget is even placed; the config row only holds a display-mode setting ("compact" | "detailed").
• FrameCalendar/FrameTaskList are keyed by widget_id (not frame_id) since a frame can now have more than one independent calendar/tasks widget, each with its own included set. Identical shape and permission model (owner-added, anyone-linked-can-mute, see "Per-widget config UI" below) -- FrameTaskList just has no "ics" calendar_key variant, since a plain ICS subscription has no VTODO (task) collection.
• FrameButtonAction (id, frame_id, button ["next"|"back"], widget_id, action, sort_order) -- see "Button actions" below.

Placement: a grid, not freeform pixels

app/grid.py is pure grid math, no I/O. The grid is GRID_LONG=8 x GRID_SHORT=5 cells, defined relative to the panel's long/short axis (not "landscape" specifically) so it stays valid across image_pipeline.logical_render_size(orientation)'s genuine width/height swap for portrait -- landscape orientations are 8 cols x 5 rows, portrait are 5 cols x 8 rows, same cell size either way. Changing a frame's orientation invalidates its existing layout (an 8x5 arrangement isn't valid on a 5x8 grid) -- the server resets to one full-panel widget on an orientation change rather than trying to remap coordinates.

Each widget type has a minimum grid footprint (grid.MIN_FOOTPRINT): photos 1x1, calendar 3x2 (a crammed calendar is illegible regardless of size-tier scaling), whiteboard 2x2, tasks 2x2, static image 1x1, text 2x1, weather 2x2 (its hourly/daily strips need the room; current/multi_city modes would tolerate smaller, but every mode shares one footprint value), battery 1x1 (just an icon + a percent, legible even at a single cell, like photos/static -- though see MIN_FOOTPRINT's own comment in grid.py on a mobile-width gear-icon click-target gap at that size, already pre-existing for photos/static too). Enforced both client-side (UX, in the Layout tab's drag/resize canvas -- static/frame_layout.js) and server-side (routers/api_widgets.py) -- the client is never trusted alone.

Rendering: one shared compositor

app/widgets/ is the render/action registry -- one module per widget_type (photos.py, calendar.py, whiteboard.py, tasks.py, static_image.py, text.py, weather.py, battery.py), each exposing:

• render(db, frame, widget, target_w, target_h, is_normal_wake) -> Image: an unquantized RGB image exactly target_w x target_h, the widget's content composed into its own region. Never raises for a foreseeable failure (an Immich hiccup, an unconfigured widget) -- falls back to a small placeholder within its own region instead, so one widget having a bad moment doesn't blank the whole panel.
• ACTIONS: dict[str, Callable] -- named button actions this type supports ("advance"/"back" for photos and calendar, "check_now" for whiteboard and weather -- both throttled external fetches with a forced-refetch action). Empty for tasks, static image, text, and battery -- nothing to advance/back/force for a passive checklist, a fixed uploaded image, a fixed block of authored text, or a number the device itself pushes on every wake.
• ACTION_LABELS: dict[str, str] -- human labels for the button- assignment UI.

routers/device.py's _render_widgets loads every Widget row for the frame, maps each one's grid rect to pixels (grid.cell_to_pixels), calls its module's render(), and hands the whole list of (rect, image) regions to image_pipeline.render_panel -- which pastes every region onto one shared canvas, then runs enhance/manage-overlay/quantize/dither/pack once over the composited result. Quantizing the whole canvas together (not each region separately before pasting) is what keeps the 6-color e-ink dithering pattern consistent across a widget boundary instead of a visible seam at the edge.

Calendar widgets pick from discrete size tiers (calendar_render.py's _SIZE_TIERS) for font size/margins/row heights based on their actual grid footprint, rather than continuously scaling constants tuned for a full ~800x480 canvas -- falls back to agenda view if a widget is too small for month view to stay legible.

"Modern" render style (experimental)

Every widget type except photos has a render_style column ("classic" default | "modern") that swaps its hand-drawn PIL primitives for an HTML/CSS render: a Jinja2 template (app/templates/widget_html/) drawn through a persistent headless-Chromium browser (app/html_render.py, Playwright) instead of ImageDraw -- gradients, shadows, and soft icon shading PIL can't easily do. Calendar's own modern-style builders (all four view modes) live in app/calendar_html_render.py rather than html_render.py itself, mirroring calendar_render.py's own separation from the simpler widget types.

Every modern-style builder runs its own ordered_dither (Bayer/ordered, not Floyd-Steinberg) before returning, committing the widget to exact palette colors before compositing -- safe to mix with photo/other classic-rendered widgets on the same frame without a Floyd-Steinberg seam at the boundary, because ordered dithering has no cross-pixel error term the way Floyd-Steinberg's diffusion does (see html_render.py's module docstring). No Frame-level dithering setting was needed to make this work.

Not offered for the photos widget -- a real photograph isn't a synthesized dashboard card, and photos has a different concern instead: its own independent palette/dithering strength (Frame.photo_palette_rgb / photo_dither_strength, a second "Photos configuration" card in Advanced Configuration, separate from the main palette_rgb/ dither_strength every other widget uses). widgets/photos.py's render() quantizes itself against these before returning, so a frame can tune the rest of its widgets' look (e.g. a calibrated palette for modern-style dashboard widgets) independently of what actually looks best for real photographs, with no render_panel changes needed -- see that module's own docstring for the one small, accepted edge case (a border on a photos widget whose palette genuinely diverges from the frame's main one).

Playwright/Chromium is a real, heavyweight runtime dependency imported lazily only when a widget actually uses modern style. Its browser binary is fetched by start.sh at container startup rather than baked into the image (see server/Dockerfile's own comment) -- a single ~181MB chrome-headless-shell binary can't be split across Docker layers the way this project's pip/npm installs were, and confirmed-failed to push to the registry as a build-time layer; cached on the /data volume (PLAYWRIGHT_BROWSERS_PATH) so only the very first boot on a fresh volume actually downloads it. Still real-panel-unverified -- treat every "modern" style as experimental regardless of deploy status.

Per-widget-type notes:

• weather: current/daily modes only -- hourly/multi_city always render classic regardless of this setting (see the Weather widget section below).
• calendar: all four view modes (agenda/today_tomorrow/week/month) have a modern builder -- the only widget type with full modern-style coverage from the start, rather than a partial rollout like weather's. Month view's "falls back to agenda below a size threshold" behavior (_month_view_fits) is honored identically in both styles.
• battery/text/tasks: full coverage (both battery modes; text reuses its own _fit() shrink-to-fit sizing logic, only the drawing differs).
• static image/whiteboard: modern style is the first visual chrome either widget type has ever had (classic draws the image with zero frame/card at all) -- a rounded-corner, shadowed card (framed_image.html.jinja, shared between the two) wrapping the already-composed image.

Button actions

Each physical button (NEXT/BACK) runs the (widget, action) binding of every widget on the frame that has one -- at most one binding per widget per button (a widget can't be bound to two different actions on the same button). On a press, routers/device.py's _run_button_actions runs every widget's assigned action for that button (each in its own widget_locked span -- never nested, since the underlying per-frame lock isn't reentrant), catching and logging any single action's failure without blocking the rest, then re-renders and returns the whole composed panel once at the end regardless of which actions succeeded. Which widget's action runs first never matters -- each only touches its own state, and the shared re-render happens once, after all of them finish.

The UI for this lives in each widget's own gear-icon config dialog (the "Button actions" card, templates/_widget_button_fields.html + static/widget_dialog_button_actions.js, POST /api/frames/{id}/widgets/{widget_id}/button-actions) -- not a frame-level tab, since assigning a widget's next/back behavior is naturally part of configuring that widget. The card only renders for widget types with a non-empty ACTIONS (photos, calendar, whiteboard, weather); tasks/ static/text/battery have nothing to bind so the card is omitted for them. An empty selection ("(none)") clears that button's binding for the widget.

A newly-created widget (including the one auto-migrated from a frame's old mode on upgrade) gets a sensible default binding reproducing its old button behavior -- see widgets.default_button_actions (called from both migration.py's backfill and api_widgets.py's api_widget_create), so a widget is never left with nothing bound until someone deliberately reassigns it.

Hold-for-global-action

Holding NEXT or BACK past a configurable duration (Frame.hold_duration_ms, minimum 3000ms, set on the Configuration tab) triggers a global action instead of the per-widget one -- not scoped to any widget, e.g. cycling through the user's saved layouts. See app/global_actions.py's GLOBAL_ACTIONS/GLOBAL_ACTION_LABELS registry and routers/device.py's /frame/global-next//frame/global-back (the device calls these instead of /frame/advance//frame/back once it detects a long press -- see firmware/main/next_button.c/back_button.c). Frame.next_hold_action/back_hold_action pick which registry entry (if any) each button's hold triggers; unset is a silent no-op, same convention as an unbound short-press button.

Per-widget config UI

Each widget has a gear-icon button on the Layout canvas that opens a <dialog> with that widget's own settings (album, calendar/task-list inclusion, whiteboard source, etc.) -- not a per-frame tab, since a frame can now have several widgets of the same type with independent settings. The dialog HTML is injected server-rendered (routers/frame_pages.py's widget_dialog, dispatching on widget.widget_type); its JS is a top-level, always-loaded file (static/widget_dialog_*.js) exposing init<Type>Dialog()/close<Type>Dialog(), since dynamically-injected HTML can't carry executable <script> tags. While a dialog is open, window.FRAME_API is repointed at that widget's own API base (/api/frames/{id}/widgets/{widget_id}) and restored on close; window.FRAME_BASE_API stays pointed at the frame-level base throughout for the always-present header/status-bar JS.

Saved layouts

A user can snapshot a frame's whole widget arrangement -- every widget's type/placement/settings, calendar/task sources, and button-action bindings -- under a name (SavedLayout + SavedLayoutWidget + SavedLayoutSource + SavedLayoutButtonAction, server/app/models.py), then switch back to it later, or apply it to a different frame. Saved layouts are owned by the user, not any one frame -- the same set shows up (with a per-frame compatible flag) on every frame that user controls whose grid matches (grid.grid_dims(orientation)'s cols/rows, landscape-class 8x5 vs. portrait-class 5x8), not just the frame it was captured from.

Saving only captures an authored setting, never runtime/cache state -- a photo widget's current queue position, a calendar's fetch cache, a whiteboard's rendered-image cache, etc. are deliberately left out (see routers/api_layouts.py's LAYOUT_CONFIG_FIELDS allowlist per widget_type), so applying a layout feels like a fresh widget of that type with its settings pre-filled, not a resurrection of stale state from whenever it was saved. A static-image widget's uploaded bytes are the one exception carried through verbatim (SavedLayoutWidget.image). Saving again under a name the user already has overwrites that layout's snapshot in place rather than erroring or creating a duplicate -- SavedLayout's own docstring.

Applying a layout to a frame (api_layout_apply, require_frame_control) deletes every widget currently on that frame and recreates the saved arrangement from scratch, remapping calendar/task sources and button bindings onto the newly-created widget ids -- same "act unconditionally on the server, confirm on the client" posture as the Layout tab's own "Clear all". A source whose owning user account no longer exists is silently dropped rather than left dangling (config is JSON, not FK-checked, so nothing else would catch that).

The web UI lives in the Layout tab's "Saved layouts" card (static/saved_layouts.js, GET/POST /api/frames/{id}/layouts, PATCH/DELETE /api/layouts/{id}, POST /api/frames/{id}/layouts/{id}/apply) -- name + Save, then a list of saved layouts each with Apply/rename/delete, incompatible ones shown greyed-out with a "different orientation" badge rather than hidden.

Weather widget

A standalone widget type (models.WeatherWidgetConfig, app/widgets/ weather.py) -- distinct from, and unrelated in code to, CalendarWidgetConfig's own embedded weather strip (still present, still Open-Meteo-only, still working exactly as before). Four display modes (WeatherWidgetConfig.mode, switchable in the widget's dialog like calendar_view):

• current -- one city's current temp + a condition icon.
• hourly -- one city, a row of ticks across the day at a configurable interval (hourly_interval_hours: 3/4/6/12).
• daily -- one city, a multi-day strip (daily_days, 1-14).
• multi_city -- several cities' current-day high/low/icon side by side -- the calendar widget's embedded strip, as a standalone widget's whole content instead of a strip above an agenda day.

Render style (WeatherWidgetConfig.render_style, "classic" default | "modern", experimental) -- see "Modern render style" above; weather's own modern coverage is current/daily only, hourly/multi_city always render classic regardless of this setting.

current/hourly/daily share one configured location (city_label/city_latitude/city_longitude, set via POST .../ weather-location, geocoded through weather.geocode_city); multi_city has its own list (cities, add/remove via POST .../weather-widget- cities/add|remove -- named to avoid colliding with the calendar widget's own, differently-scoped weather-cities/add|remove routes, which share the same {widget_id}-parameterized path shape).

Providers (app/weather/, a dispatch registry over pluggable implementations mirroring app/widgets/ itself): WeatherWidgetConfig. provider selects which of app/weather.PROVIDERS actually fetches -- "open_meteo" (worldwide, no API key), "nws" (api.weather.gov, US only, no API key, approximates "current" with the first hourly forecast period rather than a real station observation), or "ec" (Environment Canada, api.weather.gc.ca's MSC GeoMet OGC API, Canada only, no API key). Every provider function returns already-normalized {"category": ...} entries (one of clear/partly_cloudy/cloudy/fog/rain/snow/ thunderstorm) so app/weather_render.py's drawing code never needs to know which provider supplied an entry. geocode_city (name -> lat/lon) always goes through Open-Meteo's free geocoder regardless of which provider is chosen to fetch with the result.

EC's citypageweather-realtime collection is only queryable by bounding box (OGC API - Features), not a direct by-coordinate endpoint -- unlike Open-Meteo/NWS's simple lat/lon REST, app/weather/ec.py's _nearest_site widens the box progressively and picks the closest site by straight-line distance, rejecting anything beyond 300 km (calibrated against a real bug caught in development: an unconditional "nearest site, however far" matched a Miami, FL query to a site in Ontario, 1824 km away, once the box widened enough to cover the whole country).

app/weather_render.py holds every weather-related drawing primitive: draw_weather_icon/draw_weather_row (extracted out of calendar_render.py, which still imports draw_weather_row for its own embedded strip, unchanged) plus this widget's own build_current/ build_hourly/build_daily/build_multi_city, dispatched by build() -- the weather analogue of calendar_render.py's own _build_tasks/ render_tasks_preview_png relationship. Icons are hand-drawn (no custom font/icon asset), styled after Environment Canada's own icon set (pointed sun rays, a puffy cloud, teardrop rain, dendrite snowflakes, a zigzag bolt) but filled with the panel's exact ink RGB values rather than an arbitrary bitmap's anti-aliased colors -- a flat fill that's already a palette color quantizes with zero dithering error to diffuse, where a fetched/vendored icon's colors (almost never an exact match) dither into a visible speckle at these small on-panel sizes (confirmed by actually running one through the real quantize pass during development). Used for every provider's rendering, not just when EC is selected as the provider.

Battery widget

The simplest widget type (models.BatteryWidgetConfig, app/widgets/ battery.py): shows this frame's own last-reported battery level. Unlike every other widget type, there's no live upstream to poll and nothing to cache -- the content is Frame.battery_percent/battery_as_of, set by routers/device.py's frame_battery on every device wake-on-battery report, which already existed for the Device panel's own history chart regardless of whether a battery widget is placed anywhere. The widget's own config is just a display mode: "compact" (icon + percent) or "detailed" (default, adds routers/common.py's existing battery_estimate_s time-remaining estimate and the last report's age). render() falls back to a "No reports yet" placeholder for a frame that has never reported (never run on battery, or not yet claimed by a device) rather than showing a stale or fabricated number. The battery icon fill color (red/yellow/green by percent) uses the same exact-panel- ink-RGB approach as the weather icons above and manage_overlay.py's own battery glyph on the "scan to manage" overlay -- a separate, unrelated piece of code with its own fixed small size, not shared with this widget, but drawing from the same thresholds/colors so a battery glyph reads the same wherever one shows up on a panel.

Known gaps (Phase 6, not yet done)

The original 8-phase rollout plan's last phase is still open:

• Legacy per-mode Frame columns (mode, album_id, current_asset_id, all calendar_*, all whiteboard_*, etc.) are still physically present in the schema but no longer read or written anywhere -- they need a dedicated final migration to drop them. Left in place deliberately through the widget-system rollout (a much larger blast radius cutover than this project's usual same-migration-drop convention) but there's no reason to keep carrying them now that every phase has shipped.
• server/README.md still describes photos/calendar/whiteboard as per-frame "modes" in several places rather than widgets -- needs a pass once the column drop above is safely deployed.
• Whiteboard rendering is tagged (alpha) in the UI -- not fully reliable yet, treat it as experimental if extending it.