CG Virtual Monitor

This demo showcases how an fx-CG calculator can be used as an external monitor by forwarding VNC framebuffers through USB. It was made as a test of the new USB driver features introduced in gint 2.10. The program also supports multiple parallel monitors and can transmit key presses on calculators back to the PC.

• Demo video with explanations
• Another video of the setup by Tituya

How it works

In this demo, the desktop manager (sway) has two virtual outputs not connected to any monitor. The calculators don't act as a physical monitors since there is no standard device for that in USB 2 (it's too slow). Instead, sway's outputs are exposed by a VNC server running locally (wayvnc).

The VNC client in this repository's vnc-client folder is connected to the VNC server, while at the same time using the communications library libfxlink from the fxSDK to discover calculators and exchange messages. It sends framebuffer updates and receives keyboard updates (which are, unsurprisingly, forwarded to the VNC server).

The calculators themselves run the addin in the cgvm-addin folder, which is based on gint 2.10 with newly-added support for host → calculator USB communication. They "simply" show framebuffers they receive and send back keyboard updates so that the calculator keyboards can control the PC. Obviously the USB driver is not that simple... but this is invisible from the application code.

The entire setup runs at about 30 FPS, which each frame taking about:

• 20 ms to receive framebuffers via USB into the VRAM, including
  • 14 ms of raw reading from the USB buffer;
  • 6 ms of waiting for data transfer from the host.
• 11 ms to send the VRAM update to the display.

There are many ways to optimize this further if the need ever arises... I'd be surprised if it couldn't run at 60 FPS with double-buffer mode, frame streaming and/or parallelizing both bottlenecks by reading directly to the display.

Download

You can download the calculator add-in here: CGVirtMo.g3a

You'll need to build the VNC client, though.

Building

Here are the dependencies:

• The fxSDK. The fxSDK repository alone is needed for the VNC client, while the entire tool suite including gint and the cross-compiler is needed to build the add-in.
• libvncserver for the RFB protocol.
• SDL2 for debug display on the PC.

Build the VNC client with:

% cd vnc-client
% cmake -B build # options...
% make -C build

If you installed the fxSDK tools outside of the default $HOME/.local, add -DFXSDK_PATH=<the path> to CMake options. (GiteaPC uses the default path.)

Build the virtual monitor add-in with:

% cd cgvm-addin
% fxsdk build-cg
# Send .g3a file to calculator using fxlink and the LINK app:
% fxsdk build-cg -s

Using virtual monitors

Step 1: Creating the virtual output

This step depends on the desktop manager. I'm using sway, which has a hidden create_output command to create and configure headless outputs:

# Creating outputs under sway specifically
% swaymsg create_output
% swaymsg output "HEADLESS-1" resolution 396x224 # position 1920 0

# For a second monitor:
% swaymsg create_output
% swaymsg output "HEADLESS-2" resolution 396x224 # position 1920 224

Other Wayland desktops might have their own way of doing this, though there doesn't seem to be a unified method.

On X, Xvfb seems to allow this (but I'm not sure whether it can run alongside the normal display). Or you can use a section of your primary display (see below).

Step 2: Exposing the virtual output on a VNC server

The VNC client will connect to VNC servers at addresses 127.0.0.1:5900 and 127.0.0.1:5910, whichever is up. This is because wayvnc only exposes a single output, so I run two instances to get both monitors.

% wayvnc -o "HEADLESS-1" 127.0.0.1 5900

# For a second monitor:
% wayvnc -o "HEADLESS-2" -S /tmp/second-wayvnc.socket 127.0.0.1 5910

As the name suggests, wayvnc is for Wayland. There are alternatives on X, such as TigerVNC. The following setup (courtesy of Tituya) grabs the top-left corner of the primary display:

% vncpasswd
% x0vncserver -rfbauth ~/.vnc/passwd -rfbport 5910 -Geometry 394x224+0+0

Step 3: Run the VNC client and calculator add-in

Transfer the add-in to the calculator and run it from the main menu. It will show "USB connected!" as soon as the PC establishes a connection. Then, start the VNC client with:

% ./cgvm_vnc --calc # --sdl

You can add --sdl to have a copy of the outputs in SDL windows for debugging. The VNC client will automatically find calculators exposing the proper USB interface and send them the frames.

Step 4: Stop servers and unplug calculators

To take the setup down, interrupt the VNC client and close the add-in on the calculator with EXIT or MENU (in any order). Then stop the VNC servers, and finally remove the outputs.

# Removing the outputs under sway specifically
% swaymsg output "HEADLESS-1" unplug

# For a second monitor:
% swaymsg output "HEADLESS-2" unplug

Known issues

• The VNC client seems to lose connection with the calculator sometimes (libusb returning "No such device"). Not sure why yet, as it's hard to reproduce. In that case restart the VNC client.