Serial Protocol (v14)

USB CDC serial at 115200 baud, newline-terminated lines.
No custom kernel driver — companion app opens the CDC port (Web Serial or Tauri).

Firmware 0.14.x speaks protocol 14 (Rev E sensors, LEDs, haptics). Older decks still speak the v0.12 command set below; unknown commands are ignored.

Pages (v0.10+, dynamic since v0.11)

A deck has 1–8 pages of 6 keys, resized at runtime with SET_PAGES (persisted in NVS; profiles carry their own page count). A slot index is page*6 + position; every indexed command (SET_KEY, SET_IMAGE, SET_ANIM, ANIM, DRAW, …) takes a slot (0–47). The six screens show the current page; commands addressing slots on other pages update config and SD media without drawing. Shrinking the page list resets the dropped slots to defaults.

Reserved HID codes make keys switch pages on-device (standalone and under the companion alike):

Device → Host (unsolicited)

Boot:

{"event":"info","name":"Open Screen Deck","fw":"0.14.5","proto":14,"keys":6,"pages":2,"page":0,"orient":0,"mode":"hid","sd":true,"psram":8388608,"leds":8,"imu":true,"als":true,"haptic":true,"lux":120.0,"autodim":true,"bright":255,"clickbeep":false}

Key press / release (index is the global slot; taps from v0.12):

{"event":"key","index":0,"action":"press","taps":1}
{"event":"key","index":0,"action":"release"}

Multi-tap (v0.12+)

Each key has three HID bindings: hid (single), h2 (double), h3 (triple) — set via SET_KEY, 0 = unbound. Latency rule: a key with no h2/h3 resolves on the first press instantly. A key with multi-tap bindings waits up to 300 ms between taps, and resolves early the moment the highest bound level is reached. Press events report the resolved taps count.

HID codes 224–229 are silent sentinels: the firmware never types them. The companion sets h2/h3 to 229 (TAP_ARM) when a double/triple slot holds a host-only action (launch, hotkey, …) so the firmware still counts taps for it. Page codes (230–239) work at any tap level, standalone.

Page changed (SET_PAGE or an on-device page key):

{"event":"page","page":1}

Host → Device

Companion mode (v0.7+)

While the desktop companion app is connected it sends MODE COMPANION followed by PING every 2 s. In this mode the firmware does not type HID keys — it only emits key events, and the companion executes the configured action (launch app, hotkey, mic mute, …). If no heartbeat arrives for 6 s (app quit, cable pulled), the firmware falls back to plain HID F13–F24 automatically. The deck always works standalone.

SET_KEY

SET_KEY {"index":0,"label":"MUTE","sublabel":"Mic","hid":104,"bg":19049}

hid = Arduino HID code (KEY_F13 = 240 on the ESP32 core; use value from GET_KEYS). bg = RGB565 as decimal. Persisted to NVS — survives reboot.

String fields are optional: omit label/sublabel to keep the current value, send an explicit empty string ("label":"") to clear it.

SET_IMAGE

SET_IMAGE {"index":0,"len":32768}

Device replies {"event":"send_data"}; host then sends exactly 32 768 raw RGB565 bytes (big-endian per Adafruit GFX). Frame is drawn immediately and saved to SD as /osd/keys/0/icon.rgb565 when a card is mounted.

Transparency (v0.9.1+): icon pixels with the sentinel value 0x0821 (bytes 08 21) are replaced by the key's current bg color at draw time — recoloring a key never requires re-uploading its image. The companion app writes this sentinel for transparent regions of logos and PNGs.

SET_FACE (v0.10+)

SET_FACE {"index":0,"len":32768}

Identical handshake to SET_IMAGE, but the frame is only drawn — nothing is written to SD and the stored icon is untouched. The companion streams live tiles (clock, timer, CPU meter, volume, now playing, OBS scene) through this at 0.3–2 fps per tile. Slots on a hidden page ack without drawing.

SET_ANIM (v0.5+)

SET_ANIM {"index":0,"frame":1,"len":32768}

Same handshake as SET_IMAGE: device replies {"event":"send_data"}, host sends 32 768 raw RGB565 bytes. The frame is written to /osd/keys/0/anim/0001.rgb565 (frame numbers are 1-based, max 999). Requires a mounted SD card — errors with bad_anim_header otherwise.

Typical upload sequence from the companion app:

ANIM_CLEAR 0                              → ok
SET_ANIM {"index":0,"frame":1,...} + data → ok
SET_ANIM {"index":0,"frame":2,...} + data → ok
…
ANIM 0 10                                 → anim event, playback starts

USB CDC on the ESP32-S3 is full-speed (~1 MB/s real-world), so a typical 20-frame clip (640 KB) uploads in about a second.

ANIM_CLEAR (v0.5+)

ANIM_CLEAR 0

Stops playback if key 0 is animating and deletes /osd/keys/0/anim/*.rgb565. Responds {"event":"ok","cmd":"ANIM_CLEAR","index":0,"removed":20}.

SD_LS (v0.6+)

SD_LS /osd/keys/0/anim

One sd_entry line per item, then a terminator:

{"event":"sd_entry","name":"0001.rgb565","dir":false,"size":32768}
{"event":"sd_ls_done","path":"/osd/keys/0/anim","count":1}

SD_LS with no path lists the card root. Retries the SD mount if a card was inserted after boot.

SD_RM (v0.6+)

SD_RM /osd/keys/0/anim/0001.rgb565

Removes a file (or an empty directory). Responds {"event":"ok","cmd":"SD_RM","path":"..."} or rm_failed.

GET_KEYS response lines

One line per global slot (6 × current page count):

{"event":"key_state","index":0,"page":0,"label":"MUTE","sublabel":"Toggle","hid":240,"bg":19049,"ov":0}

(hid is Arduino KEY_F13 etc.)

Storage & media

On-board microSD holds offline icons and animation frames:

/osd/keys/0/icon.rgb565
/osd/keys/0/anim/0001.rgb565
/osd/profiles/streaming.json

Animations survive reboot — replay with ANIM n fps any time.

See Architecture for video limits on dual SPI.

Animation rules

• One key animates at a time (ANIM n fps); 1–30 fps
• Frames stream from SD: /osd/keys/n/anim/0001.rgb565, 0002 …
• All six keys cannot run video simultaneously — see Architecture for the bandwidth math

Rev E hardware (protocol 14+)

INFO includes leds, imu, als, haptic, lux, autodim, bright, and clickbeep. The companion's Settings → Deck hardware card uses these commands. Logical LED index 0–5 follows keys J1–J6; 6 is the rear USB-link indicator; 7 is SD status. Physical SK6812 wiring order is in hardware/pinout.py (LED_CHAIN_REFS).

Quick test (Chrome console via Web Serial)

1. Connect device with USB CDC enabled
2. Open serial monitor or Web Serial
3. Send PING → expect pong
4. Press a key → expect key event JSON