Pinning VM CPUs in Proxmox

Proxmox 4 doesn't seem to support pinning a VM's CPUs to specific host CPUs. It also doesn't support VM startup hooks, so there's no straightforward way to run taskset on the newly created VM. However, when the QEMU process is created, it writes its PID to the file /var/run/qemu-server/$id.pid, where $id is the VM ID. By watching writes to this file, e.g. through inotifywait, it's actually pretty easy to create a startup hook to run taskset or perform any other tasks.

I wrote a systemd service that watches the /var/run/qemu-server directory, and automatically calls taskset on newly created VMs processes based on configuration files.

/etc/systemd/system/autotaskset.service
[Unit]
Description = Auto taskset service
 
[Service]
Type = simple
ExecStart = /bin/bash -c " \
        conf=/etc/autotaskset; \
        dir=/var/run/qemu-server; \
        mkdir -p \"$$dir\"; \
        /usr/bin/inotifywait -mq -e modify --format %%f \"$$dir\" | \
        while read pid; do \
                [ -f \"$$conf\"/\"$$pid\" ] && \
                /usr/bin/taskset $$(< \"$$conf\"/\"$$pid\") $$(< \"$$dir\"/\"$$pid\"); \
        done"
 
[Install]
WantedBy = multi-user.target

To enable the service, install the inotify-tools package, then run systemctl enable autotaskset.

To configure CPU pinning for each VM, create a file /etc/autotaskset/$id.pid, where $id is the VM ID, that contains all arguments to taskset. For example, for a VM with ID 100, if I want to pin the VM's 4 CPUs on the host's every other CPU (to skip the second hyper-threaded CPU on each physical core for example), I would use the following conf file.

/etc/autotaskset/100.pid
-cp 0,2,4,6

The -c option enables specifying the host CPU by its ID, and the -p option is required because we're operating on an existing process given its PID.

Passing GTX 1060 Through To Windows VM

I'm building a homelab server running Proxmox with a Windows 10 VM for daily usage, so I wanted to use PCI passthrough to let the VM access the GTX 1060 graphics card that's installed in the server. Here's what I had to do to get past the infamous “code 43” error from the Nvidia driver when you try to pass through consumer-grade cards to Windows VMs. Credit for the first three points goes to this post on the vfio-users mailing list.

  1. Include “kvm=off” in your KVM “-cpu” option
    • In Proxmox, this is done automatically when you specify the “x-vga=1” option for your PCI passthrough line in the VM config (e.g. “hostpci0: 03:00,pcie=1,x-vga=1”; see an explanation for “pcie=1” below).
    • You can also explicitly add the “hidden=1” option to your “cpu” config line (e.g. “cpu: host,hidden=1”).
  2. Rename the hypervisor vendor ID
    • In Proxmox, this is also done automatically when you specify “x-vga=1”. “proxmox” is used as the new vendor ID.
  3. Enable MSI (Message Signaled Interrupts)
    • You can follow this popular guide to add the “MSISupported” entries to the registry.
    • MSI needs to be enabled for both the graphics device and the audio device. For the audio device, don't use the “Device Instance Path” parameter to locate the registry key, because that parameter is under “HDAUDIO\…”; instead, you can usually find the “PCI\…” key for the audio device right next to the one for the graphics device, with the same “VEN” vendor ID.
    • In Proxmox, in order to truly enable MSI, I had to switch the VM machine type to “q35” (by adding a “machine: q35” line to the VM config file) and enable PCI-e mode for the passthrough (by using the “pcie=1” option mentioned above).
    • Once MSI is truly enabled, you can see the negative-numbered interrupts in the Windows Device Manager, and also see “MSI: Enable+” in the host's “lspci -v” output.
  4. Disable memory ballooning
    • The memory ballooning feature and/or the vfio-balloon Windows driver doesn't seem to play well with the Nvidia driver, so disable ballooning in the VM configuration.
    • It may also help to uninstall the Windows vfio-balloon device/driver completely, and then restart the VM a couple times.

With these steps, I was able to finally have a working passed-through graphics card in the Windows VM. Note that every time you update the graphics driver in the VM, you may have to repeat the steps to re-enable MSI, because new drivers create new keys in the registry that don't have the “MSISupported” entries.

Mounting Lenovo LI2264d monitor on VESA mount

The Lenovo LI2264d (or LI2364d) monitor doesn't support VESA mounts directly, but here's one way to mount it on a VESA mount. All the parts can be bought for < $10 at stores like Home Depot, and takes a couple hours at most to put together.

First, take apart the vertical part of the stand that came with the monitor. There is a black plastic cover that can be taken off by removing the screws, revealing the assembly inside.

Stand assembly

We want to reuse the bare metal part that fits into the monitor body. I used a 5/16“ wrench to reach into the assembly and unscrew the hex nuts.

5/16" wench

After disassembly, you should also end up with two screws and a bunch of washers/spacers. The washers are important because we'll later use them to match the width of our bracket to the width of the monitor mount.

Disassembled stand

To make the bracket that attaches the VESA mount to the monitor mount, I used two corner braces. The 3” ones from Home Depot worked for me because, 1) the pre-drilled holes are big enough for the screws that fit into the monitor mount, and 2) the distance between the edge and the inner holes, plus the width of the monitor mount, is close to the 75mm distance on the VESA mount.

3" corner braces

First, I attached the braces to the VESA mount plate that came with the VIVO monitor stand that I'll be using. The assembly consists of M4 screws, hex nuts, and a couple washers

Bracket/VESA mount plate parts

On the other end of the bracket, line up the bracket with the monitor mount from the monitor stand assembly. Use the original screws, hex nuts, washers, and spacers to secure the parts. It may take a little effort to fit them together. Here's the assembled bracket with VESA mount plate on one end and the monitor mount on the other end. This particular assembly actually didn't work because it was too wide for the recess in the monitor body that you fit the monitor mount into. I had to take this apart and move the outside spacers to the inside for it to fit.

Bracket/VESA mount plate/monitor mount

At this point I thought I was done, but of course when I tried to fit the monitor mount onto the monitor body, it wouldn't fit because part of the bracket was pushing into top wall of the recess on the monitor body. Ugh! I had to cut two grooves into that wall to let the bracket through.

Monitor body with cut grooves

And with that… the mount finally fits!

Monitor with VESA mount assembly

It's working pretty well with the monitor stand. The monitor doesn't weigh that much, so I don't expect any problems for regular landscape usage. Actually, the mount feels sturdy enough sideways that you might even get away with using it in portrait.

Monitor on stand

Edit: I've since switched the corner braces to smaller braces from Menards. These braces also fit the 75mm VESA mounts, but require some additional spacers to give enough clearance to the back connectors. One benefit of the smaller braces is they give a greater degree of tilt adjustment.

New bracket made from smaller braces

Fennec LogView add-on now supports Android 5+

The Fennec LogView add-on has been updated to version 1.2, and it now supports Android Lollipop, Marshmallow, and above.

The previous version read directly from the Android logger device located at /dev/log/main, which worked well on Android 4.x. However, starting with Android 5.0, apps no longer have read permission to /dev/log/main. Fortunately, Android 5.0 also added several new APIs in the liblog.so library specifically for reading logs. LogView 1.2 uses these new APIs, when available, through js-ctypes, and this approach should continue to work on future versions of Android as well.

Recent Fennec platform changes

There has been a series of recent changes to the Fennec platform code (under widget/android). Most of the changes was refactoring in preparation for supporting multiple GeckoViews.

Currently, only one GeckoView is supported at a time in an Android app. This is the case for Fennec, where all tabs are shown within one GeckoView in the main activity. However, we'd like to eventually support having multiple GeckoView's at the same time, which would not only make GeckoView more usable and make more features possible, but also reduce a lot of technical debt that we have accumulated over the years.

The simplest way to support multiple GeckoViews is to open multiple nsWindows on the platform side, and associate each GeckoView with a new nsWindow. Right now, we open a new nsWindow in our command line handler (CLH) during startup, and never worry about having to open another window again. In fact, we quit Fennec by closing our only window. This assumption of having only one window will change for multiple GeckoView support.

Next, we needed a way of associating a Java GeckoView with a C++ nsWindow. For example, if a GeckoView sends a request to perform an operation, Gecko would need to know which nsWindow corresponds to that GeckoView. However, Java and platform would need to coordinate GeckoView and nsWindow creation somehow so that a match can be made.

Lastly, existing messaging systems would need to change. Over the years, GeckoAppShell has been the go-to place for platform-to-Java calls, and GeckoEvent has been the go-to for Java-to-platform calls. Over time, the two classes became a big mess of unrelated code stuffed together. Having multiple GeckoViews would make it even harder to maintain these two classes.

But there's hope! The recent refactoring introduced a new mechanism of implementing Java native methods using C++ class members 1). Using the new mechanism, calls on a Java object instance are automatically forwarded to calls on a C++ object instance, and everything in-between is auto-generated. This new mechanism provides a powerful tool to solve the problems mentioned above. Association between GeckoView and nsWindow is now a built-in part of the auto-generated code – a native call on a GeckoView instance can now be transparently forwarded to a call on an nsWindow instance, without writing extra code. In addition, events in GeckoEvent can now be implemented as native methods. For example, preference events can become native methods inside PrefHelper, and the goal is to eventually eliminate GeckoEvent altogether 2).

Effort is underway to move away from using the CLH to open nsWindows, which doesn't give an easy way to establish an association between a GeckoView and an nsWindow 3). Instead, nsWindow creation would move into a native method inside GeckoView that is called during GeckoView creation. As part of moving away from using the CLH, making a speculative connection was moved out of the CLH into its own native method inside GeckoThread 4). That also had the benefit of letting us make the speculative connection much earlier in the startup process.

This post provides some background on the on-going work in Fennec platform code. I plan to write another follow-up post that will include more of the technical details behind the new mechanism to implement native calls.

1) Bug 1178850 (Direct native Java method calls to C++ classes), bug 1186530 (Implement per-instance forwarding of native Java methods), bug 1187552 (Support direct ownership of C++ objects by Java objects), bug 1191083 (Add mechanism to handle native calls before Gecko is loaded), bug 1192043 (Add mechanism to proxy native calls to Gecko thread)
2) Bug1188959 ([meta] Convert GeckoEvent to native methods)
3) Bug 1197957 (Let GeckoView control nsWindow creation)
4) Bug 1195496 (Start speculative connection earlier in startup)

Post Fennec logs to Pastebin with LogView add-on

The LogView add-on for Fennec now lets you copy the logcat to clipboard or post the logcat to pastebin.mozilla.org. Simply go to the about:logs page from Menu → Tools → Logs and tap on “Copy” or “Pastebin”. This feature is very useful if you encounter a bug and need the logs, but you are not next to a computer or don't have the Android SDK installed.

Copy to clipboard Posting to Pastebin Posted to Pastebin


Back from leave

Back in January, I left on a two-month-long leave from Mozilla, in order to do some traveling in China and Japan. Now I'm finally back! I was in China for 1.5 months and in Japan for 2 weeks, and it was amazing! I made a short video highlighting parts of my trip:

Being a mobile developer, I naturally paid some attention to mobile phone usage in China, and how it's different from what I'm used to in the U.S. The cellular infrastructure was impressive. It was fairly cheap, and I was getting full 3G/4G service in small villages and along high-speed rail routes. It seemed like everyone had a smartphone, too. I would see grandmas standing on the side of the road checking their phones.

I never use QR codes in the U.S., but I actually used them quite often in China. For example, you would scan another person's QR code to add them as friends on Wechat. In some places, you could scan a merchant's QR code to pay that merchant using Alipay, a wallet app. Many types of tickets like train tickets and movie tickets also use QR codes over there.

Everyone used Wechat, a messaging app that's “way better than anything else in the U.S.” according to my American friend living in China. It's more than just a messaging app though – you have a “friend circle” that you can post to, a la Facebook; you can also follow “public accounts”, a la Twitter. The app has integrated wallet functionality: I paid for a train ticket and topped up my phone using the app; during Chinese New Year, people were sending each other cash gifts through it.

For some reasons, you see a lot of these “all-in-one” apps in China. I used Baidu Maps during my travel, which does maps and navigation. However, you can also call taxis from within the app or hire a “private car”, a la Uber. You can use the app like Yelp to find nearby restaurants by type and reviews. While you're at it, the app lets you find “group buy” discounts to these restaurants, a la Groupon. I have to say it was super convenient. After I came back to the States, I wasn't used to using Google Maps anymore because it didn't do as much.

Of course, on the flip side, these apps probably would be less popular without the Internet censorship that's so prevalent over there. By creating a barrier for foreign companies to enter the Chinese market, it provided opportunities for domestic companies to create and adapt copycat products. I found it amusing that Android is so prevalent in the Chinese smartphone market, but everything Google is blocked. As a result, you have all these third-party markets that may or may not be legitimate. Mobile malware seems to be a much larger issue in China than in the U.S., because people have to find their apps off of random markets/websites. It was strange to see an apps market promising “safe, no malware” with every download link. Also amusingly, every larger app I saw came with its own updater, again because these apps could not count on having a market to provide update service.

Overall, the trip was quite eye-opening, to see China's tremendous development from multiple angles. I loved Japan, too; I felt it was a lot different from both China and the U.S. Maybe I'll write about Japan in another post.

Smart JNI reference classes for Fennec

Historically, JNI code in Fennec has mostly used raw JNI types like jobject and jstring. However, the need to manage object lifetimes and the lack of strong typing make this practice error-prone. We do have some helper classes like AutoLocalJNIFrame, RefCountedJavaObject, WrappedJavaObject, and AutoGlobalWrappedJavaObject, but I've found them to be inconvenient to use.

Bug 1116868 is introducing several new “smart” classes to improve dealing with JNI references. As a start, instead of using raw jobject types, there are now different smart types for different usages:

Type When to use
Object::LocalRef To replace local jobject references; e.g. local variables
Object::GlobalRef To replace global jobject references; e.g. instance members
Object::Param To replace jobject function parameters

Note that these new classes are under the mozilla::jni namespace, so your code should include it first,

using namespace mozilla::jni; // then use Object::LocalRef
namespace jni = mozilla::jni; // then use jni::Object::LocalRef

These classes make managing lifetimes very easy. Previously, it was easy to make mistakes like this,

jobject obj = GetObject();
// use obj
obj = GetAnotherObject(); // oops! first obj was leaked

This mistake could happen to both local and global references. Now with the smart classes, these errors are eliminated,

Object::LocalRef obj = GetObject();
// use obj
obj = GetAnotherObject(); // first obj was automatically deleted

The new classes also make it easy to convert between local and global references,

// Object::GlobalRef mObj;
Object::LocalRef obj = mObj; // automatic conversion from global to local ref
obj = GetNewObject();
mObj = obj; // automatic conversion from local to global ref

For function parameters, you can pass either a LocalRef or a GlobalRef to a Param parameter,

void SetObject(Object::Param obj);
 
Object::LocalRef obj;
SetObject(obj); // pass in LocalRef
// Object::GlobalRef mObj;
SetObject(mObj); // pass in GlobalRef

For function return values, you should return a LocalRef per JNI convention,

Object::LocalRef GetObject() {
  Object::LocalRef ret = MakeObject();
  return ret;
}

Note that in the above example, only one local reference is ever created because of return value optimization performed by the compiler.

Both LocalRef and GlobalRef support move semantics, so the following example still creates only one local reference overall,

Object::LocalRef GetObject();
 
Object::LocalRef foo;
foo = GetObject();

It also means you can use LocalRef and GlobalRef with container classes like mozilla::Vector without worrying about performance impact.

LocalRef and GlobalRef can be used like pointers/Java references,

Object::GlobalRef ref = nullptr;
ref = GetRef();
if (ref) {
  Object::LocalRef ref2 = ref;
  // compare underlying objects, so a LocalRef
  // and a GlobalRef of the same object are equal
  MOZ_ASSERT(ref == ref2);
}

Other types

jobject is not the only wrapped type; other JNI types correspond to different smart types,

JNI type Use these smart types
jstring String::LocalRef String::GlobalRef String::Param
jclass ClassObject::LocalRef ClassObject::GlobalRef ClassObject::Param
jthrowable Throwable::LocalRef Throwable::GlobalRef Throwable::Param
jbooleanArray BooleanArray::LocalRef BooleanArray::GlobalRef BooleanArray::Param
jbyteArray ByteArray::LocalRef ByteArray::GlobalRef ByteArray::Param
jobjectArray ObjectArray::LocalRef ObjectArray::GlobalRef ObjectArray::Param

And if you use auto-generated classes from widget/android/GeneratedJNIWrappers.h, each class is being updated in bug 1116589 to have its own reference types. For example, mozilla::widget::ViewTransform::LocalRef is a local reference to a Java ViewTransform instance,

ViewTransform::LocalRef vt = ViewTransform::New();
float x = vt->OffsetX(); // get offsetX field
vt->OffsetX(1.0f); // set offsetX field

Using separate classes ensures better type-safety, because type-checking is done by the compiler,

Foo::LocalRef foo;
Bar::LocalRef bar = foo; // error: invalid conversion

However, the auto-generated classes often only accept Object::Param parameters or return Object::LocalRef values. Therefore, as a special case, any LocalRef or GlobalRef can automatically convert to Object::Param, and Object::LocalRef can automatically convert to any other LocalRef.

Object::LocalRef Foo(Object::Param foo);
 
Bar::LocalRef bar;
bar = Foo(bar); // bar is converted to Object::Param
// then return value is converted to Bar::LocalRef

String types

The String types have some custom behavior in addition to the standard behavior. A String::LocalRef or String::GlobalRef can automatically convert to a nsString or a nsCString,

String::LocalRef GetString();
 
nsString str = nsString(GetString());
nsCString cstr = nsCString(GetString());

Conversely, a nsAString or a nsACString can automatically convert to a String::Param,

void SetString(String::Param param);
 
nsString str;
SetString(str);
SetString(NS_LITERAL_CSTRING("text"));
 
String::LocalRef ref;
SetString(ref); // okay too

A String::LocalRef, String::GlobalRef, or String::Param also has a Length method,

size_t GetStringLength(String::Param param) {
  return param.Length();
}
 
MOZ_ASSERT(GetStringLength(NS_LITERAL_STRING("text")) == 4);

Adapting to raw JNI usage

The goal of these new classes is to make using raw JNI types obsolete. However, until all the refactoring is done, there are still cases where raw JNI values are needed, for example to call JNIEnv functions.

To get a raw JNI reference from any LocalRef or GlobalRef, call its Get method,

void Foo(jobject param);
 
Object::LocalRef obj;
Foo(obj.Get());

To turn a raw JNI reference into any Param, call the Foo::Ref::From method,

void Foo(Object::Param param);
 
jobject obj;
Foo(Object::Ref::From(obj));

To return a raw JNI reference from any LocalRef or GlobalRef, call its Forget method,

jobject GetRawRef() {
  Object::LocalRef ref = GetRef();
  return ref.Forget();
}

Use LocalRef::Adopt to manage a returned raw local reference,

jobject GetRawRef();
 
Object::LocalRef ref = Object::LocalRef::Adopt(GetRawRef());

LocalRef also has an Env method that returns a cached JNIEnv pointer,

Object::LocalRef ref = GetRef();
auto cls = ClassObject::LocalRef::Adopt(
  ref.Env()->GetObjectClass(ref.Get()));

LogView add-on for Fennec

We use the adb logcat functionality a lot when developing or debugging Fennec. For example, outside of remote debugging, the quickest way to see JavaScript warnings and errors is to check the logcat, which the JS console redirects to. Sometimes, we catch a Java exception (e.g. JSONException) and log it, but we otherwise ignore the exception. Unless you are actively looking at the logcat, it's easy to miss messages like these. In other cases, we simply want a way to check the logcat when away from a computer, or when a user is not familiar with adb or remote debugging.

The LogView add-on, available now on AMO, solves some of these problems. It continuously records the logcat output and monitors it. When it sees an error in the logcat, the error is displayed as a toast for visibility.

Error toastError details

You can also access the current logs through the new about:logs page.

about:logs

The add-on only supports Jelly Bean (4.1) and above, and only Fennec logs are included rather than logs for all apps. Check out the source code or contribute on Github.

Feature suggestions are also welcome! I think the next version will have the ability to filter logs in about:logs. It will also allow you to copy logs to the clipboard and/or post logs as a pastebin link.

Let's not protest protesters

Jon Foreman (of the band Switchfoot) recently wrote a post called “Why I Refuse To Protest Protestors”. In it, he talked about his experience dealing with protests at his band's concerts. A few sentences he wrote struck me the most,

All at once I had an epiphany: these puzzling creatures that are yelling at you are human souls – as unpredictable, perplexing and unpredictable as I am. Here's the shocker: this guy with the bullhorn could be my cousin! he could be a friend of mine! Better yet: this guy could be me! If our lives were swapped, who can say that I would be any different? I put nothing below me. Who can say what I would do if I had his reality? Compassion makes you realize what you have in common with the rest of humanity.

For me as a Mozillian, these past few days have been confusing and unsettling. First it was the protests against Brendan's appointment. For an organization you are a part of, for a pioneer you look up to, for them to be rallied against, it was hard to understand. Then it was Brendan's resignation. That was unexpected and felt like things falling apart. Now it is another wave of protests against Brendan's departure. In the midst of this whirlwind, I want to feel anguish. I want to shout. Why couldn't Brendan just recant? Why couldn't the protesters just understand? Why couldn't the media just set the story straight? Why couldn't Mozilla just handle everything a little better? I want to protest.

But that's not what Mozilla is about. Like Katharina said, Mozilla is great at not only shipping great products but also at shipping love. It was love, love for the web, that first brought Mozilla together. It is love that continues to drive our mission now. Like Jon wrote,

On my best days, I want to stand for love conquering a multitude of wrongs. I want to stand for forgiveness, for mercy, for beauty, for grace. I stand for you, sir and madame. Whether you are holding a megaphone or not. Even when you refuse to shake my hand I love you. Whether you insult me or not, drunk or sober; I honestly love you! I love your passion, your fervor, your dedication. I want to know you better. I want to find out what makes you tick. I want to know why you believe what you believe. I want to learn from you. I am for you, emphatically for you!

Let's do what we do best, at showing love. Show love to Brendan; thank him for all that he's done and wish him the best. Show love to the activists; empathize with them. Show love to people who think Mozilla is not inclusive. Show love to the journalists, to companies protesting us. Show love to the Board, to your peers, to the community. Let's show our love to Mozilla.