Skip to content

Handover notes

Where the work stands and how to pick it up: for the next session, human or agent.

Status as of 2026-10-10, commit ed802fd, CI and GPU runner green.

The machines

zentrale bazzite
Role Host (is remote-controlled), development machine Client
GPU AMD RX 7800 XT (VAAPI) NVIDIA GTX 1080, driver 580 (NVENC/NVDEC)
Network Wired, 192.168.178.87 Wi-Fi
System Bazzite 44, KDE Wayland, two monitors 2560×1440@165 (DP-2 left, DP-1 right) Bazzite
Repo ~/fernsicht ~/Projects/fernsicht
  • Building: in the distrobox fernsicht (image: dev/Containerfile).
  • Running: The programs built in the box run directly on Bazzite. Bazzite ships FFmpeg 8, libva and WebKitGTK 4.1.
  • Self-hosted runner: Both machines have one for the GPU tests (.github/workflows/gpu.yml, gpu-runner.md).
  • bazzite's GTX 1080 throws GPU errors. The kernel log shows NVIDIA Xid 13 ("Graphics Exception: EXTRA/MISSING_MACRO_DATA", always at the same offsets of the 3D class) and Xid 32 (invalid push buffer), also from Brave, VS Code, the screen locker and vulkaninfo, right after boot too. Work in the faulted channel never finishes: a test waiting for it times out or hangs (in FFmpeg's own waits for ever, until CI's step limit). Since the runner started (2026-10-10) every GPU run on it collects a few; most tests survive them. Not the clocks: with the GPU held at P0 under full load they came as often (tried 2026-10-12). No PCIe errors (AER counters 0, x16). A red or hung NVIDIA run is worth a rerun before blaming the code; journalctl -k | grep Xid tells whether one came with it. Possible remedies: another 580 driver, another card. Fernsicht itself gives such a GPU up after its wait times out and ends the session with gpu-lost instead of hanging; an NVENC call that hangs inside the driver ends the session after 5 s and restarts the host (video; never seen live). Once (2026-10-12, Vulkan Video HEVC end to end) an Xid 13 made the test process abort (SIGABRT) with no message of ours or FFmpeg's, so inside the driver; 60 runs of the same test under gdb afterwards brought neither an Xid nor an abort. The Xids come in clusters while the whole GPU job runs.

What is done

Goal of the last stage: use Fernsicht like a normal user. All five steps are built, tested and pushed:

  1. Pairing and encryption.
  2. One-time, with a 6-digit PIN (SPAKE2).
  3. Every session uses a Noise IK handshake, after which everything is sealed.
  4. Only paired devices get in. Code: crates/secure.
  5. Host as a service.
  6. systemd unit packaging/fernsicht-host-agent.service.
  7. Control socket /run/fernsicht/control.sock with the commands fernsicht-host-agent pair | status | unpair.
  8. Encoder chosen to match the GPU (--encoder auto).
  9. Code: apps/host-agent/src/control.rs.
  10. Device discovery.
  11. fernsicht-client discover: broadcast over the stream port 47800. Deliberately no mDNS: the firewall on the zentrale blocks 5353.
  12. Hosts answer with name, key, OS, GPU, whether pairing is open, and whether they are busy.
  13. Code: apps/client/src/discover.rs, packets Discover/Announce in crates/proto.
  14. The app.
  15. Tauri around the client UI, in apps/desktop. This is a separate Cargo workspace with its own CI job.
  16. It lists machines, pairs and starts sessions.
  17. The picture runs in the native window of fernsicht-client --app. The client prints the overlay as JSON and exits when stdin closes.
  18. “Dieser Rechner” (this computer) opens pairing on the local host and shows the PIN.
  19. The UI (web/client-ui) uses the real commands in Tauri, and demo data in the browser.
  20. Installation.
  21. packaging/build.sh builds everything in the box.
  22. packaging/install-app.sh installs the app and client to ~/.local, including a start menu entry, without sudo.
  23. packaging/install-host.sh sets up the service and needs sudo.
  24. Instructions: installation.md.

Added after that:

  1. Web viewer on the LAN.
  2. The host serves the page itself (--web, default 0.0.0.0:47800 TCP; --web-root, installed to /usr/local/share/fernsicht/viewer).
  3. Access with the pairing PIN, once.
  4. WebRTC with str0m: H.264 and Opus, plus a data channel (mouse cursor as protocol packets and statistics to the browser, input as JSON to the host).
  5. Code: apps/host-agent/src/web/, web/viewer/src/lib/host.ts, web/viewer/src/routes/live.tsx.

  6. Tested: with headless Chrome (in the box) against a real host with a VAAPI test pattern, 1920×1080 at 60 fps, glass-to-glass according to the overlay ≈ 15 ms on localhost.

  7. Gaming mode, gamepads, bitrate adaptation, app settings.

  8. Native client: --gaming (a click captures the pointer, relative motion), Ctrl+Alt+Shift+M captures and releases.
  9. Gamepads: The client reads controllers through SDL3, loaded at runtime (crates/input/src/sdl.rs), else directly via evdev (crates/input/src/gamepad.rs). The browser uses the Gamepad API. The host creates one virtual Xbox 360 controller per player (crates/input/src/uinput.rs), for a PlayStation pad a DualSense through uhid with gyro and touchpad (crates/input/src/uhid.rs). Buttons map by position (A at the bottom, Y at the top); the X/Y swap of the xpad driver is compensated on both sides.
  10. Bitrate: crates/net/src/rate.rs. Down on frames that are lost despite FEC, on an overloaded sender, on more than 20 % loss, and before any loss: on frames waiting 15 ms or more in a queue, on ECN CE marks (D3). Up after 5 s of clean network. The host paces at the delivery rate the client measures. The running encoder takes the new rate (NVENC in place, VAAPI with a new codec context on the same GPU objects; one keyframe). Tested with VAAPI and 10 % artificial loss.

  11. App: page “Einstellungen” (settings) (resolution, frame rate, bitrate), “Gerät vergessen” (forget device); sound and mode during the session go to the client over stdin (mute, unmute, gaming, desktop).

  12. One AppImage for everything.

  13. packaging/appimage.sh builds locally: app, stream window, host and web viewer in one file. The GPU libraries (libva, libdrm, Vulkan loader) are left out; appimagetool takes care of that.
  14. In the app: “Diesen Rechner freigeben” (share this computer) (apps/desktop/src/share.rs) extracts the AppImage via pkexec to /opt/fernsicht/app and sets up fernsicht-host-agent.service. Plus “Freigabe beenden” (stop sharing) and “Host aktualisieren” (update host).
  15. Release as in the other repos: Bump version (firsttris/workflows), then release.yml (checks, AppImage on Ubuntu 24.04, GitHub release).
  16. Tested locally: The programs from the AppImage stream on Bazzite with the system drivers (VAAPI H.264). The app window shows “Diesen Rechner freigeben”.

  17. Not tested live yet: the setup with the password dialog.

  18. AV1 and HEVC. Each connection takes the best codec both sides can do in hardware (AV1, HEVC, H.264); the web viewer uses AV1 if the browser offers it, otherwise H.264. Setting “Videoformat” (video format) in the app, --codec on the client. Details and measurements in performance.md. Not seen live yet:

  19. App on the bazzite → zentrale: the overlay should show “HEVC”.
  20. Chrome/Firefox on the bazzite → zentrale: “AV1” (in the web viewer's overlay; in Chrome also under chrome://webrtc-internals).

Measured latency (earlier sessions):

Path Glass-to-glass
Loopback ≈ 9.4 ms
zentrale → bazzite, 1080p 11.5 ms
zentrale → bazzite, 1440p 16.3 ms (encode 10 ms, because the GPU clocks down between frames)

What nobody has seen live yet

These items are tested automatically but have never run on real hardware. They need the user at the machine.

Confirmed live (2026-10-10, according to the user):

  • Web viewer, also with Firefox and on a phone: picture, mouse and keyboard.
  • bazzite (NVIDIA, client) → zentrale (AMD, host): picture, mouse and keyboard; the app window shows something under NVIDIA (item 3 done).

  • Sound was not checked in either case, neither in the web viewer nor in the app.

  • Service on the zentrale.

  • First stop the old host that the user started in the root box fernsicht-root. It holds UDP 47800, otherwise the service does not start.
  • Then install: ./packaging/build.sh && sudo ./packaging/install-host.sh.
  • Check:
    • Picture via KMS as root without a sudo session.
    • Sound from the logged-in user (pulse as root via /run/user/1000/pulse/native).
    • Monitor layout (KWin config of the desktop user).
    • Input via uinput.
    • GPU clocks raised during a session and back afterwards.
    • These with the hardened unit (review B10: capability bounding set, ProtectSystem=strict): sound through the user's socket and the GPU's sysfs clocks are the likeliest to need something more.
    • Log: journalctl -u fernsicht-host-agent -f.
  • App on both machines.
  • Install: ./packaging/build.sh && ./packaging/install-app.sh.
  • On the zentrale: “Gerät koppeln” (pair a device).
  • On the bazzite: zentrale in the list, “Koppeln” (pair), enter the PIN, “Desktop”.
  • Check:
    • List, online status, PIN countdown.
    • Overlay in the app.
    • Disconnect.
    • Closing the app ends the session.
  • ~~App on NVIDIA (bazzite).~~ Done, see above. (The app sets WEBKIT_DISABLE_DMABUF_RENDERER=1 if /proc/driver/nvidia exists; without it the windows stay empty, tauri#9394.)
  • Web viewer for real. On the zentrale, after the service installation, open http://192.168.178.87:47800 from another device, enter the PIN, then check KMS picture, sound, mouse and keyboard. Also with Firefox and a phone.
  • AppImage sharing: Start the AppImage from the first release on the zentrale and choose “Diesen Rechner freigeben” (KDE asks for the password). Then check: Is fernsicht-host-agent running (systemctl status fernsicht-host-agent)? Is there picture, sound and input? Then “Freigabe beenden” and share again. Since review B11 the AppImage is unpacked without running it (unsquashfs, else a loop mount of its squashfs): check that /opt/fernsicht/app belongs to root.
  • Gaming mode and gamepad. Choose “Gaming” in the app. A click into the window captures the pointer, Ctrl+Alt+Shift+M releases it. Check a game with mouse control. A controller on the client shows up on the host as “Fernsicht X-Box 360 pad 1”: check in Steam or evtest that A is at the bottom and Y at the top. With Xbox and PlayStation controllers.
  • Bitrate adaptation over Wi-Fi. Connect on the bazzite, look for bitrate … Mbit/s (… queue … ms, … CE) in the host's log. Normal Wi-Fi losses and Wi-Fi's own jitter (a few ms of queue) leave the bitrate alone. Frames lost despite FEC, an overloaded sender, a queue of 15 ms and more, or CE marks lower it. NACK (2026-10-12): losses FEC cannot repair should no longer cost a frame in the LAN; the client's summary at the end says how many packets were “sent again”. CE marks need a router with an AQM that marks (fq_codel or CAKE with ECN), which the home network probably lacks.
  • Device discovery over Wi-Fi. Does the broadcast from the bazzite reach the zentrale, and the answer come back? The answer goes to an ephemeral port of the client. Fedora's zone FedoraWorkstation allows UDP 1025–65535; check on the bazzite (firewall-cmd --list-all). As a fallback the app also queries paired hosts directly.
  • System keys to the host. Built, never seen live:
  • App: In fullscreen or with a captured pointer, the window asks the desktop to pass its shortcuts through (Wayland keyboard-shortcuts-inhibit, X11 keyboard grab; apps/client/src/shortcuts.rs). KDE asks the first time. Check: Meta, Meta+W, Alt+Tab and Ctrl+Alt+Del reach the host; in a normal window Alt+Tab stays local; Ctrl+Alt+Shift+F/M/Q still work.
  • Web viewer: fullscreen button in the toolbar; Chrome/Edge then lock the keyboard (Keyboard Lock, hold Esc to leave).
  • Both: menu “Tasten senden” (send keys) (keyboard icon in the toolbar) with Ctrl+Alt+Del, Windows key, Windows+W, Alt+Tab, Alt+F4, Print.
  • Switching monitors. Built, never seen live. The host reports its monitors (packet Monitors, JSON in the browser), the client chooses (SelectMonitor). The capture thread then opens the other monitor on the same card and sends a keyframe; the mouse mapping follows (FollowScreen). Controls: “Bildschirm wählen” (choose screen) in the toolbar (from two monitors up), in the app window Ctrl+Alt+Shift+←/→. Check on the zentrale (DP-2 left, DP-1 right): switching in both directions, the mouse lands on the monitor shown, the pointer is in the right place. A monitor of a different size gets its own stream size (fitted into the size the client asked for, aspect ratio kept): the host rebuilds the encoder, the keyframe carries the new size, and the client's picture and mouse mapping follow it. Check this with monitors of different shapes, too.

  • Web viewer on a phone. Built, never seen live: gestures (web/viewer/src/lib/touch.ts), touchpad mode, local zoom, on-screen keyboard (lib/textkeys.ts, layout from the browser language), gaming mode on a phone without pointer capture. Check on a real phone (Android Chrome, iPhone Safari): tap, drag, long press for right click, two-finger scrolling, zoom and reset, typing with Gboard/iOS keyboard (including word suggestions), rotation.

  • 10 bit and HDR (D4). Built, the codecs tested on both GPUs, never seen live. On zentrale (KDE, AB30) the stream should come in 10 bit: the overlay says “HEVC 10 Bit” (or “AV1 10 Bit”) on a client that decodes it. Then switch KDE to HDR (Display settings, “HDR”; needs an HDR monitor): the host's log says “the captured monitor now gets (Bt2020, Pq)”, the overlay “HDR”, and the picture on an SDR client should look like the desktop, not washed out. An HDR monitor on the client still shows it tone-mapped to SDR.

  • Rumble (D9). Built, tested against SDL's virtual pad and (in CI) a uinput pad, never felt live. On a client with SDL3 (the AppImage carries it, packaging/sdl3.sh; never run from a release yet) the log says “gamepads through SDL”; a game on the host that rumbles (or fftest /dev/input/eventN on the virtual pad, from linuxconsoletools) should rumble the client's pad.

  • Clipboard. Built, tested against KWin (virtual) and headless Sway, never used live. In a session from bazzite to zentrale (KDE on both): copy text on one, paste on the other, both ways; what bazzite held when the session started should be on zentrale. The host's log says “clipboard shared”.

  • DualSense (D9). Built, tested against a virtual SDL pad and (in CI) the kernel's hid-playstation, never played live. With a PS4 or PS5 pad on the client: the host's log says “gamepad 1 connected (DualSense)”, Steam on the host shows a DualSense, and a game with gyro aiming or touchpad controls (or evtest on “Fernsicht DualSense pad 1 Motion Sensors”) follows the pad's movements.

Open tasks, by importance

The findings of the code review (bugs, security, performance, clean-up) are listed with their status in code-review.md; they are worked through from top to bottom.

The performance items (AV1/HEVC, immediate display, slices, intra refresh, congestion control, 4:4:4) with their benefit and effort are collected in performance.md.

  1. The live tests above, then fix the bugs that turn up.
  2. Measure GPU clock boost. It is built: during a session the host sets power_dpm_force_performance_level to high (apps/host-agent/src/power.rs, switch in the app under “Einstellungen”). Encode at 1440p used to be 10 ms. Now measure on the zentrale with the host as a service (root), once with and once without (--no-gpu-boost or the switch), and compare the encode time in the overlay. If high does not help enough, try pp_power_profile_mode (e.g. the VR profile).
  3. Check sharpness at native resolution (the user found the picture at 1080p over Wi-Fi not quite sharp). Bitrate and keyframe quality.
  4. Reference measurement with Sunshine/Moonlight (latency-baseline.md).
  5. Games, remaining: rumble in the browser, more than 4 controllers, the DualSense's adaptive triggers and lightbar.
  6. NVIDIA as host: check live. Screen capture with NVENC is built. The path: KMS DMA-BUF → Vulkan compute (RGB → NV12, BT.709, scaled) → memory imported by CUDA → NVENC. Code: crates/gpu/src/convert.rs, crates/codec/src/{nvidia,cuda}.rs.
  7. Tested on the bazzite with test patterns in the NVIDIA tiling layout: 2.5 ms per 1080p frame, no Xid messages.
  8. Never run live: real KMS on the bazzite (HDMI-A-1). The user starts sudo ./target/release/fernsicht-host-agent --capture kms --encoder nvenc --pair on the bazzite, and the zentrale connects as the client.
  9. Small things in the app: file transfer (the clipboard works for text).
  10. Bitrate adaptation in the web viewer: built (TWCC estimate, tested behind a simulated 3 Mbit/s link); not seen live on a phone over mobile data yet.
  11. Later: internet (NAT, rendezvous server), PipeWire capture. Web viewer over the internet (rendezvous, TURN), remembering the browser instead of asking for a PIN every time, a test with real Chrome in CI.

Rules for working

The rules for working on the project (what never to do on the user's desktop, root commands, pushing, language, code style, the checks before every push) are in CLAUDE.md. The test setup is described in testing.md (fuzzing, coverage threshold 90 %, GPU tests).

Viewing the app without a real desktop

This keeps the user's screen free: virtual X display, no Wayland, sound goes nowhere.

Xvfb :99 -screen 0 1280x800x24 &
env -u WAYLAND_DISPLAY DISPLAY=:99 GDK_BACKEND=x11 WEBKIT_DISABLE_COMPOSITING_MODE=1 \
  LIBGL_ALWAYS_SOFTWARE=1 PULSE_SERVER=unix:/nonexistent \
  XDG_CONFIG_HOME=$(mktemp -d) FERNSICHT_CLIENT=$PWD/target/debug/fernsicht-client \
  apps/desktop/target/debug/fernsicht &
sleep 8; xwd -root -display :99 | magick xwd:- target/shots/app.png

Important here:

  • Test host: A test host on a free port (e.g. --bind 127.0.0.1:47801 --pair) is not found by broadcast. Pair it first with the same XDG_CONFIG_HOME via fernsicht-client pair 127.0.0.1:47801 PIN; then the app queries it directly.
  • Clicking: DISPLAY=:99 xdotool mousemove X Y click 1.
  • Storage: Put screenshots in target/shots. The /tmp folder of the VS Code Flatpak session is not visible on the host.