cg-virtual-monitor/README.md

# CG Virtual Monitor

This demo showcases how an fx-CG calculator can be used as an external monitor
by forwarding VNC framebuffers through USB. It was made as a test of the new
USB driver features introduced in gint 2.10. The program also supports multiple
parallel monitors and can transmit key presses on calculators back to the PC.

* [Demo video with explanations](https://www.youtube.com/watch?v=rXAcUgodzik)
* [Another video of the setup](https://www.youtube.com/watch?v=of1vAH6FpIs)
  by Tituya

![](/images/triple-monitor-setup.jpg)

## How it works

In this demo, the desktop manager ([sway](https://swaywm.org/)) has two virtual
outputs not connected to any monitor. The calculators don't act as a physical
monitors since there is no standard device for that in USB 2 (it's too slow).
Instead, sway's outputs are exposed by a VNC server running locally
([wayvnc](https://github.com/any1/wayvnc)).

The VNC client in this repository's `vnc-client` folder is connected to the
VNC server, while at the same time using the communications library libfxlink
from the [fxSDK](https://gitea.planet-casio.com/Lephenixnoir/fxsdk) to discover
calculators and exchange messages. It sends framebuffer updates and receives
keyboard updates (which are, unsurprisingly, forwarded to the VNC server).

The calculators themselves run the addin in the `cgvm-addin` folder, which is
based on [gint](https://gitea.planet-casio.com/Lephenixnoir/gint) 2.10 with
newly-added support for host → calculator USB communication. They "simply" show
framebuffers they receive and send back keyboard updates so that the calculator
keyboards can control the PC. Obviously the USB driver is not that simple...
but this is invisible from the application code.

The entire setup runs at about 30 FPS, which each frame taking about:

- 20 ms to receive framebuffers via USB into the VRAM, including
    * 14 ms of raw reading from the USB buffer;
    * 6 ms of waiting for data transfer from the host.
- 11 ms to send the VRAM update to the display.

There are many ways to optimize this further if the need ever arises... I'd be
surprised if it couldn't run at 60 FPS with double-buffer mode, frame streaming
and/or parallelizing both bottlenecks by reading directly to the display.

## Download

You can download the calculator add-in here: [CGVirtMo.g3a](https://gitea.planet-casio.com/attachments/dfb57302-f77f-4dab-bedc-922fc1f088b8)

You'll need to build the VNC client, though.

## Building

Here are the dependencies:

- The [fxSDK](https://gitea.planet-casio.com/Lephenixnoir/fxsdk). The fxSDK
  repository alone is needed for the VNC client, while the entire tool suite
  including gint and the cross-compiler is needed to build the add-in.
- [libvncserver](https://github.com/LibVNC/libvncserver) for the RFB protocol.
- SDL2 for debug display on the PC.

Build the VNC client with:

```bash
% cd vnc-client
% cmake -B build # options...
% make -C build
```

If you installed the fxSDK tools outside of the default `$HOME/.local`, add
`-DFXSDK_PATH=<the path>` to CMake options. (GiteaPC uses the default path.)

Build the virtual monitor add-in with:

```bash
% cd cgvm-addin
% fxsdk build-cg
# Send .g3a file to calculator using fxlink and the LINK app:
% fxsdk build-cg -s
```

## Using virtual monitors

**Step 1: Creating the virtual output**

This step depends on the desktop manager. I'm using sway, which has a hidden
[`create_output`](https://github.com/swaywm/sway/blob/master/sway/commands/create_output.c)
command to create and configure headless outputs:

```bash
# Creating outputs under sway specifically
% swaymsg create_output
% swaymsg output "HEADLESS-1" resolution 396x224 # position 1920 0

# For a second monitor:
% swaymsg create_output
% swaymsg output "HEADLESS-2" resolution 396x224 # position 1920 224
```

Other Wayland desktops might have their own way of doing this, though there
doesn't seem to be a unified method.

On X, [Xvfb](https://www.x.org/releases/X11R7.6/doc/man/man1/Xvfb.1.xhtml)
seems to allow this (but I'm not sure whether it can run alongside the normal
display). Or you can use a section of your primary display (see below).

**Step 2: Exposing the virtual output on a VNC server**

The VNC client will connect to VNC servers at addresses `127.0.0.1:5900` and
`127.0.0.1:5910`, whichever is up. This is because wayvnc only exposes a single
output, so I run two instances to get both monitors.

```bash
% wayvnc -o "HEADLESS-1" 127.0.0.1 5900

# For a second monitor:
% wayvnc -o "HEADLESS-2" -S /tmp/second-wayvnc.socket 127.0.0.1 5910
```

As the name suggests, wayvnc is for Wayland. There are alternatives on X, such
as [TigerVNC](https://tigervnc.org/). The following setup
([courtesy of Tituya](https://www.planet-casio.com/Fr/forums/topic17303-1-teaser-usb-utiliser-une-calto-comme-ecran-pour-son-pc.html#191111))
grabs the top-left corner of the primary display:

```bash
% vncpasswd
% x0vncserver -rfbauth ~/.vnc/passwd -rfbport 5910 -Geometry 394x224+0+0
```

**Step 3: Run the VNC client and calculator add-in**

Transfer the add-in to the calculator and run it from the main menu. It will
show "USB connected!" as soon as the PC establishes a connection. Then, start
the VNC client with:

```bash
% ./cgvm_vnc --calc # --sdl
```

You can add `--sdl` to have a copy of the outputs in SDL windows for debugging.
The VNC client will automatically find calculators exposing the proper USB
interface and send them the frames.

**Step 4: Stop servers and unplug calculators**

To take the setup down, interrupt the VNC client and close the add-in on the
calculator with EXIT or MENU (in any order). Then stop the VNC servers, and
finally remove the outputs.

```bash
# Removing the outputs under sway specifically
% swaymsg output "HEADLESS-1" unplug

# For a second monitor:
% swaymsg output "HEADLESS-2" unplug
```

## Known issues

- The VNC client seems to lose connection with the calculator sometimes (libusb
  returning "No such device"). Not sure why yet, as it's hard to reproduce. In
  that case restart the VNC client.