Current Status

The project was very successful in its goal of providing a text input method for xrdesktop independent of SteamVR’s virtual keyboard. The keyboard is usable for people in 56 languages and it works with KDE. You can try it out by compiling the ‘next’ branches of the various libraries. See below for detailed instructions.

How to Run

You will need to compile and run it for KDE. Follow this how-to guide to install xrdesktop from source. However, you need to use different branches in a few cases:

• gulkan: gulkan:next
• gxr: gxr:next
• xrdesktop: remcokranenburg:keyboard-internationalization
• libinputsynth: remcokranenburg:keyvals-input
• kwin-effect-xrdesktop: remcokranenburg:virtual-keyboard-support

Future Work

During our feedback sessions, my mentors and I decided to focus first on making the basic keyboard experience complete, by implementing it for a desktop environment and providing keyboard layouts for a lot of languages. In the revised planning, the prediction mode was postponed.

A start was made on a predictive mode that would work like the swipe keyboard mode on Android. Post-GSoC, I’d like to continue this work. When finished, I’d also like to investigate whether the same technology can be used to add a swipe mode to GNOME Shell’s on-screen keyboard.

The third and most experimental mode was dropped for the time being. It entailed a Dasher-like interface adapted for a virtual reality environment. Since VR has such a different way of interacting, a keyboard input method might not make the most use of your abilities. The hypothesis is that a Dasher-like method might be faster when used with six-degrees-of-freedom controllers.

While implementing keyboard layouts, I learned that languages with non-alphabetic writing systems, such as Chinese, need a different way of handling keyboard input. They need to be able to write tentative words, that can be committed by additional key presses. This feature could potentially be based on the prediction system, because that also predicts tentative words, which can be committed by choosing one of the proposals.

Merge Requests

Some warm-up bugfixes / improvements:

• MERGED xrdesktop!24 Improve menu position when attached to controller
• MERGED xrdesktop!27 Bump minimum meson version
• MERGED xrdesktop!28 Client to shell example
• MERGED xrdesktop!30 g3k-button: correct calculation of background circle radius
• MERGED gulkan!15 Add pango example
• MERGED libinputsynth!4 gitignore: ignore build directory

The main merge requests, implementing the G3kKeyboard and making it work in KDE:

• MERGED xrdesktop!31 Add virtual keyboard
• OPEN kwin-effect-xrdesktop!4 Support G3kKeyboard

Implementing multiple keyboard layouts:

• DECLINED libintputsynth!5 Keysequence input
• OPEN libinputsynth!6 Keyvals input
• OPEN xrdesktop!33 Keyboard internationalization

Blog Posts

This project also prompted me to finally start a blog:

• Getting Started with GLib
• Introduction to xrdesktop
• Keyboard Event Handling
• Emoji Mode ✨
• KDE in VR with Keyboard Support
• Keyboard Layouts

Final Words

I loved working on this project. I’ve always been interested in computer graphics and virtual reality, so this was the perfect fit for a start in open source development.

I would like to thank my mentors Christoph Haag and Lubosz Sarnecki for their fantastic support, and Google and Collabora for providing the opportunity to work on such a cool project. We will see each other again, I hope!

The Unicode Common Locale Data Repository specifies many keyboard layouts, so we convert this data to our internal JSON format.

Currently, we only support languages with alphabetic writing systems, such as English, Arabic or Russian. Languages with logographic writing systems, such as Chinese, Japanese or Korean, are not supported yet. They require a more complicated interaction model, because a character is built by multiple keystrokes before being committed.

KDE’s xrdesktop support comes in the form of a KWin ‘desktop effects’ plugin. It is in this plugin that windows are mirrored in the VR environment. This is also the place where I needed to add support for input through the virtual keyboard.

Luckily, there was existing code that worked with SteamVR’s keyboard. Adding xrdesktop’s own keyboard support was a matter of connecting to its key-pressed-event event with a callback, which synthesizes a key press using libinputsynth.

The modes are specified in JSON format and loaded as a GResource. The following shows an abbreviated example of how each key is specified. Each key is part of a mode and has a label, x and y coordinates and a width. An optional target makes a key switch to a different mode.

{
  "name": "Small",
  "modes": [
    {
      "name": "Main",
      "keys": [
        { "label": "q", "x": 0.0, "y": 0.0, "w": 1.0 },
        { "label": "w", "x": 1.0, "y": 0.0, "w": 1.0 },
        { "label": "e", "x": 2.0, "y": 0.0, "w": 1.0 },
        { "label": "r", "x": 3.0, "y": 0.0, "w": 1.0 },
        { "label": "t", "x": 4.0, "y": 0.0, "w": 1.0 },
        { "label": "y", "x": 5.0, "y": 0.0, "w": 1.0 },
        ...
        { "label": "⇧", "x": 0.0, "y": 2.0, "w": 1.5, "target": "Capitals" },
        { "label": "z", "x": 1.5, "y": 2.0, "w": 1.0 },
        { "label": "x", "x": 2.5, "y": 2.0, "w": 1.0 },
        { "label": "c", "x": 3.5, "y": 2.0, "w": 1.0 },
        { "label": "v", "x": 4.5, "y": 2.0, "w": 1.0 },
        { "label": "b", "x": 5.5, "y": 2.0, "w": 1.0 },
        { "label": "n", "x": 6.5, "y": 2.0, "w": 1.0 },
        ...
        { "label": "⌫", "x": 8.5, "y": 2.0, "w": 1.5 },
        { "label": "?123", "x": 0.0, "y": 3.0, "w": 1.5, "target": "Numbers" },
        { "label": ",", "x": 1.5, "y": 3.0, "w": 1.0 },
        { "label": "☺", "x": 2.5, "y": 3.0, "w": 1.0, "target": "Emoji" },
        { "label": " ", "x": 3.5, "y": 3.0, "w": 4.0 },
        { "label": ".", "x": 7.5, "y": 3.0, "w": 1.0 },
        { "label": "↵", "x": 8.5, "y": 3.0, "w": 1.5 }
      ]
    },
    {
      "name": "Capitals",
      "keys": [
        { "label": "Q", "x": 0.0, "y": 0.0, "w": 1.0 },
        { "label": "W", "x": 1.0, "y": 0.0, "w": 1.0 },
        { "label": "E", "x": 2.0, "y": 0.0, "w": 1.0 },
        { "label": "R", "x": 3.0, "y": 0.0, "w": 1.0 },
        { "label": "T", "x": 4.0, "y": 0.0, "w": 1.0 },
        { "label": "Y", "x": 5.0, "y": 0.0, "w": 1.0 },
        ...
        { "label": "⇧", "x": 0.0, "y": 2.0, "w": 1.5, "target": "Main" },
        { "label": "Z", "x": 1.5, "y": 2.0, "w": 1.0 },
        { "label": "X", "x": 2.5, "y": 2.0, "w": 1.0 },
        { "label": "C", "x": 3.5, "y": 2.0, "w": 1.0 },
        { "label": "V", "x": 4.5, "y": 2.0, "w": 1.0 },
        { "label": "B", "x": 5.5, "y": 2.0, "w": 1.0 },
        { "label": "N", "x": 6.5, "y": 2.0, "w": 1.0 },
        ...
        { "label": "⌫", "x": 8.5, "y": 2.0, "w": 1.5 },
        { "label": "?123", "x": 0.0, "y": 3.0, "w": 1.5, "target": "Numbers" },
        { "label": ",", "x": 1.5, "y": 3.0, "w": 1.0 },
        { "label": "☺", "x": 2.5, "y": 3.0, "w": 1.0, "target": "Emoji" },
        { "label": " ", "x": 3.5, "y": 3.0, "w": 4.0 },
        { "label": ".", "x": 7.5, "y": 3.0, "w": 1.0 },
        { "label": "↵", "x": 8.5, "y": 3.0, "w": 1.5 }
      ]
    },
    ...
  ]
}

Numeric mode:

Special character mode is accessible from the numeric mode.

And we have support for emojis!

The emojis are specified literally in the JSON file:

{
  "name": "Emoji",
  "keys": [
    { "label": "😀", "x": 0.0, "y": 0.0, "w": 1.0 },
    { "label": "😃", "x": 1.0, "y": 0.0, "w": 1.0 },
    { "label": "😄", "x": 2.0, "y": 0.0, "w": 1.0 },
    { "label": "😁", "x": 3.0, "y": 0.0, "w": 1.0 },
    { "label": "😆", "x": 4.0, "y": 0.0, "w": 1.0 },
    { "label": "😅", "x": 5.0, "y": 0.0, "w": 1.0 },
    ...
  ]
}

To get it all working, I had to add Pango as a dependency, because it supports font fallbacks. Most fonts do not support all possible characters, so when you want to draw text in many languages, you will need to use several backup fonts.

I added a small pango example to gulkan, showing Arabic, Chinese and Hindi text plus emojis:

That’s it for now! A future post will showcase Kwin support for xrdesktop with a virtual keyboard.

Event Handling

In the past few days of working on the virtual keyboard for xrdesktop, I’ve been trying to make the keyboard behave like a keyboard. That is, it should notice when one of its keys is clicked by the user, and it should generate the proper key event when this happens. This is the result:

In my previous blog post, you could already see the keys react to clicks, but this is not enough for a fully functioning keyboard system. For one, you don’t want to handle the events for each button separately; you would have to connect more than 100 events for a standard keyboard! You just want to connect a single key-press-event to your active window, which would contain all the information of the pressed key.

The second reason for handling the click events inside the keyboard, is that we need to keep track of the modifier keys (Shift, Ctrl, Alt, things like that). The keyboard should maintain the state of these keys (on or off) internally, so the key-press-event would send the key capitalized when Shift is active, or ‘Ctrl + X’ when Ctrl is active, etc.

So how is this done technically? We need to register a key-press-event signal for G3kKeyboard, which we can then emit at the right time. The user of the keyboard will write an event callback, and will connect the signal to it.

What is the right time to emit the key-press-event? Well, when a button has been clicked. So, we need to connect the button’s grab-start-event to a callback that emits the key-press-event.

You might be familiar with event handling in other languages, but in GLib style this is a fairly verbose affair. Once I understand the GObject system a bit better, I will write that part 2 of ‘Getting Started with GLib’ with a short tutorial on how to create objects with it. Find the raw event handling code below:

g3k_keyboard.c:

static void
g3k_keyboard_class_init(G3kKeyboardClass *klass)
{
  keyboard_signals[KEY_PRESS_EVENT] =
    g_signal_new ("key-press-event",
                  G_TYPE_FROM_CLASS (klass),
                  G_SIGNAL_RUN_LAST,
                  0, NULL, NULL, NULL, G_TYPE_NONE,
                  1, G_TYPE_POINTER | G_SIGNAL_TYPE_STATIC_SCOPE);
}

void
g3k_keyboard_click_cb (XrdWindow     *window,
                       GxrController *controller,
                       gpointer       user_data)
{
  G3kKeyboard *keyboard = G3K_KEYBOARD (user_data);

  g_autofree gchar *title = NULL;
  g_object_get (window, "title", &title, NULL);

  G3kKeyEvent event = { .key = title[0] };

  g_signal_emit (keyboard, keyboard_signals[KEY_PRESS_EVENT], 0, &event);
}

Example usage:

void
keyboard_pressed_cb(G3kKeyboard *keyboard,
                    G3kKeyEvent *event,
                    gpointer     user_data)
{
  g_print ("Yay, key %c was pressed!\n", event->key);
}

static void
_init_keyboard(XrdShell *shell)
{
  G3kContext *context = xrd_shell_get_g3k (shell);
  G3kKeyboard *keyboard = g3k_keyboard_new (context);

  G3kContainer *container = g3k_keyboard_get_container(keyboard);
  GSList *buttons = g3k_container_get_windows (container);

  for (GSList *button = buttons; button != NULL; button = button->next)
    {
      g_signal_connect (button->data, "grab-start-event",
                        (GCallback) g3k_keyboard_click_cb, keyboard);
    }

  g_signal_connect (keyboard, "key-press-event",
                    G_CALLBACK (keyboard_pressed_cb), NULL);
}

That’s it for now. Up next, let’s start on the Swype prediction mode! For this, we need to show the path that the controller traced over the keyboard’s keys and we need to calculate the word that was most likely meant to be typed.

Predictive Text Input for Virtual Reality

xrdesktop is a project that brings your Linux desktop into a VR space, with support for KDE and GNOME. It is written in the form of a GLib library. You could easily write your own VR window manager, in theory in any language (though language bindings are currently not finished).

One area in which xrdesktop is lacking, is a virtual keyboard. SteamVR provides one, but it is not available when you are in a virtual environment without SteamVR, such as when you are using a standalone headset, or when you are using the fully open source OpenXR implementation Monado. Since xrdesktop is intended to work in all such environments, we need our own keyboard implementation.

During GSoC 2021, that is exactly what I will implement. The project is divided into three broad milestones:

1. a basic QWERTY keyboard implementation
2. a Swype-like predictive mode
3. a Dasher-like predictive mode

The first milestone is the important one, since it enables text input in xrdesktop at all. Any of the other modes are nice-to-have, but those are the modes I’m personally most excited about!

First Week

It was quite a rush to be able to make that first screenshot! In fact, I was so excited that I forgot to make a ‘C’ key.

Let’s look a bit into the structure of the xrdesktop project! xrdesktop is in the middle of a major refactoring currently, with much of the code that is not desktop-specific being moved out to a separate namespace called G3k. It should result in a very easy to learn system, but right now it is a bit of a mess.

The project uses Vulkan to draw graphics and OpenXR to interface with VR hardware. Instead of writing directly against these APIs, two GLib wrappers were created: Gulkan and GXR. GXR used to have support for both OpenVR (SteamVR, basically) and OpenXR, but the refactoring simplifies this to just OpenXR. We also have a vector math library called graphene, which provides your basic matrix operations. Then there is G3k, which you can think of as Gtk for VR. It should provide things like buttons, containers, stuff like that. I say ‘should’, because it is only a few days old at this point! Finally, there is xrdesktop, which contains the code that positions your desktop windows in VR.

At first, I created a class XrdKeyboard (more on how to create classes with GLib in an upcoming part 2 of Getting Started with GLib), but it seemed like this was the wrong level of abstraction, so I renamed it G3kKeyboard in the end. But because the refactoring is not complete yet, it still depends on XrdClient for now. This is also true for G3kButton, which is implemented as an XrdWindow at the moment. The buttons of G3kKeyboard are currently XrdWindows, but that will surely change in the future.

Currently, when you click on a key, it prints a message to stdout. Next up, I’ll read up a bit more about GObject signals and properties, so we can connect the keyboard events to the active window. Let’s end with another screenshot showing our amazing VR stdout printer!

What is GLib?

First things first, GLib is a C library designed with two goals:

1. making C programmers’ lives easier by adding high-level language constructs to C
2. adding an interoperability layer to C libraries so they can automatically be used in other programming languages

Initially, we’ll only be concerned with the first goal. GLib extends the standard C environment in many ways, such as safer string handling, abstract data types, and a fully-featured object-oriented type system.

A Simple Program

As an example program, let’s look at a solution for the first puzzle of the wonderful Advent of Code 2019 puzzles. If you want to do these puzzles yourself, you might want to stop reading further now, although the first exercise is just an introduction of course!

#include <math.h>
#include <stdio.h>
#include <string.h>

int calculate_fuel(int weight, gboolean include_fuel) {
    int additional_weight = fmax(weight / 3 - 2, 0);

    if(include_fuel && additional_weight > 0) {
        additional_weight += calculate_fuel(additional_weight, include_fuel);
    }

    return additional_weight;
}

int main(int argc, char *argv[]) {
    _Bool include_fuel = 0;

    for(int i = 0; i < argc; i++) {
        if(strcmp(argv[i], "--include-fuel") == 0) {
            include_fuel = 1;
        }
    }

    int total_fuel_required = 0;

    for(;;) {
        int mass = 0;
        int tokens_scanned = scanf("%d", &mass);

        if(tokens_scanned <= 0) {
            break;
        }

        int fuel_required = calculate_fuel(mass, include_fuel);
        total_fuel_required += fuel_required;
    }

    printf("%d\n", total_fuel_required);
    return 0;
}

This program must be compiled with gcc -lm main.c, to link C’s math library to our program.

The first thing we need to do to convert this to a GLib-based program, is to add the glib.h include to the top of the file:

#include <glib.h>

We will also need to add some more options to the compiler. You can find out which ones we need exactly by running pkg-config --cflags --libs glib-2.0. On my system, this gives the following output:

-I/usr/include/glib-2.0
-I/usr/lib64/glib-2.0/include
-I/usr/include/sysprof-4
-pthread
-lglib-2.0

You see a few include directories that tell the compiler where to find GLib’s header files, as well as -pthread which enables thread support, and -lglib-2.0, which links the GLib library to our program.

GLib Conversion

Now that we have GLib, we can start thinking about improving the code.

Math

One thing that comes to mind immediately is that we need to include math.h and link C’s math library only for the fmax() function. If we are using GLib anyway, we can use its MAX() macro instead.

Booleans

Another peculiarity of C is its boolean logic. It is essentially integer comparison: 0 means false and any other value means true. Traditional C code would look like this:

int done = 0;

while(!done) {
    do_cool_things();

    if(is_done()) {
        done = 1;
    }
}

This is fairly weak from a type system perspective, since it is now easily possible to mix integer arithmetic and boolean logic. For example, the values 1 and 42 are both true, but they do not equal each other, and it is possible to turn a false value into true by adding to it value += 1. A _Bool type was added to the language at a relatively late stage, and it is even possible to use bool, true and false if you include stdbool.h. However, this does not change the fact that it is all integer logic underneath.

GLib has its own gboolean type with TRUE and FALSE values, so we will use that here.

Other Basic Types

GLib has a bunch of other types, such as types with explicit sizes (gint32, gint64), easier alternatives to some types (guint instead of unsigned integer, gconstpointer instead of const void *), and a few cosmetic types that just exist for completeness (gint, gfloat).

Safer Strings

Nul-terminated strings in C are notoriously difficult to use safely, because a string’s length is not stored in the object. It is simply an array of bytes that ends in an all-zeroes byte (a nul). All string-related functions that need to know the length will search the array until they find such a nul byte. Often, this byte is accidentally replaced with another character and the string functions will start operating on memory that is unrelated to the string! This usually leads to code that is vulnerable to attack.

GLib includes a GString type that makes it easier to use strings correctly. It does this by storing the length alongside the character array. You can use it like this:

void print_hello_world() {
    GString *message = g_string_new("hello");
    g_string_append(message, ", world!");
    printf("%s\n", message->str);
    g_string_free(message);
}

This makes it much easier to do string operations like inserting and removing parts without making memory management mistakes. Note that you must still remember to free the memory you allocated with g_string_new().

Automatic Memory Management

Way back in the 1990s, Java was developed as an alternative to C that was not only easier to use, but also safer. One of these safety features is automatic memory management. Instead of manually marking an area of memory as being used by an object in our program, the runtime would decide by itself whether an object and its memory was still reachable in the current scope of the program.

void printBookTitles() {
    List<Book> books = get_books();

    for(Book book : books) {
        System.out.println(book.title);
    }
}

In the above snippet, a list of Book objects is retrieved from the get_books() function. Each book title is printed and then the function ends. As programmers we don’t need to know whether references to books still exist after the function ends: if this was the last reference, the Java runtime will automatically free its memory, but if other references exist, the runtime will keep the memory marked as being in use. This makes two undesirable things impossible: using the memory after it has been freed, and keeping the memory marked as used even though it is unreachable by the program.

GLib has a type g_autoptr, which does something similar: the system keeps track of all references to the object, and if the last reference is removed, the memory is freed. An example:

void print_hello_world() {
    g_autoptr(GString) message = g_string_new("hello");
    g_string_append(message, ", world!");
    printf("%s\n", message->str);
}

Note that the memory behind message is never explicitly freed. GLib will automatically free the memory at the end of the function, because message is the only reference and it is removed at the end of the function.

The Result

Combining all of the above, we get the following program:

#include <glib.h>
#include <stdio.h>

gint calculate_fuel(gint weight, gboolean include_fuel) {
    gint additional_weight = MAX(weight / 3 - 2, 0);

    if(include_fuel && additional_weight > 0) {
        additional_weight += calculate_fuel(additional_weight, include_fuel);
    }

    return additional_weight;
}

gint main(gint argc, gchar *argv[]) {
    gboolean include_fuel = FALSE;

    g_autoptr(GString) include_fuel_argument = g_string_new("--include-fuel");

    for(gint i = 0; i < argc; i++) {
        g_autoptr(GString) argument = g_string_new(argv[i]);

        if(g_string_equal(argument, include_fuel_argument)) {
            include_fuel = TRUE;
        }
    }

    gint total_fuel_required = 0;

    for(;;) {
        gint mass = 0;
        gint tokens_scanned = scanf("%d", &mass);

        if(tokens_scanned <= 0) {
            break;
        }

        gint fuel_required = calculate_fuel(mass, include_fuel);
        total_fuel_required += fuel_required;
    }

    printf("%d\n", total_fuel_required);
    return 0;
}

You might have noticed that I still use scanf() and printf(), even though GLib provides alternatives such as g_printf() and GUnixInputStream. It should be possible to use these constructs, but it requires a different compiler configuration because we then need to use a Unix-specific version of GLib.

With such a simple example, the conversion to a GLib-based program is maybe not so useful. However, you will quickly see the benefit when you start working with more complicated data types and their memory management.

You can find the code for my solution here. You might see an interesting file called meson.build there, which is an increasingly popular way of building your programs. But that is a topic for another post! Let’s see what the next puzzle brings.