The Fennec LogView add-on has been updated to version 1.2, and it now supports Android Lollipop, Marshmallow, and above.

The previous version read directly from the Android logger device located at /dev/log/main, which worked well on Android 4.x. However, starting with Android 5.0, apps no longer have read permission to /dev/log/main. Fortunately, Android 5.0 also added several new APIs in the liblog.so library specifically for reading logs. LogView 1.2 uses these new APIs, when available, through js-ctypes, and this approach should continue to work on future versions of Android as well.

There has been a series of recent changes to the Fennec platform code (under widget/android). Most of the changes was refactoring in preparation for supporting multiple GeckoViews.

Currently, only one GeckoView is supported at a time in an Android app. This is the case for Fennec, where all tabs are shown within one GeckoView in the main activity. However, we'd like to eventually support having multiple GeckoView's at the same time, which would not only make GeckoView more usable and make more features possible, but also reduce a lot of technical debt that we have accumulated over the years.

The simplest way to support multiple GeckoViews is to open multiple nsWindows on the platform side, and associate each GeckoView with a new nsWindow. Right now, we open a new nsWindow in our command line handler (CLH) during startup, and never worry about having to open another window again. In fact, we quit Fennec by closing our only window. This assumption of having only one window will change for multiple GeckoView support.

Next, we needed a way of associating a Java GeckoView with a C++ nsWindow. For example, if a GeckoView sends a request to perform an operation, Gecko would need to know which nsWindow corresponds to that GeckoView. However, Java and platform would need to coordinate GeckoView and nsWindow creation somehow so that a match can be made.

Lastly, existing messaging systems would need to change. Over the years, GeckoAppShell has been the go-to place for platform-to-Java calls, and GeckoEvent has been the go-to for Java-to-platform calls. Over time, the two classes became a big mess of unrelated code stuffed together. Having multiple GeckoViews would make it even harder to maintain these two classes.

But there's hope! The recent refactoring introduced a new mechanism of implementing Java native methods using C++ class members ¹⁾. Using the new mechanism, calls on a Java object instance are automatically forwarded to calls on a C++ object instance, and everything in-between is auto-generated. This new mechanism provides a powerful tool to solve the problems mentioned above. Association between GeckoView and nsWindow is now a built-in part of the auto-generated code – a native call on a GeckoView instance can now be transparently forwarded to a call on an nsWindow instance, without writing extra code. In addition, events in GeckoEvent can now be implemented as native methods. For example, preference events can become native methods inside PrefHelper, and the goal is to eventually eliminate GeckoEvent altogether ²⁾.

Effort is underway to move away from using the CLH to open nsWindows, which doesn't give an easy way to establish an association between a GeckoView and an nsWindow ³⁾. Instead, nsWindow creation would move into a native method inside GeckoView that is called during GeckoView creation. As part of moving away from using the CLH, making a speculative connection was moved out of the CLH into its own native method inside GeckoThread ⁴⁾. That also had the benefit of letting us make the speculative connection much earlier in the startup process.

This post provides some background on the on-going work in Fennec platform code. I plan to write another follow-up post that will include more of the technical details behind the new mechanism to implement native calls.

The LogView add-on for Fennec now lets you copy the logcat to clipboard or post the logcat to pastebin.mozilla.org. Simply go to the about:logs page from Menu → Tools → Logs and tap on “Copy” or “Pastebin”. This feature is very useful if you encounter a bug and need the logs, but you are not next to a computer or don't have the Android SDK installed.

Back in January, I left on a two-month-long leave from Mozilla, in order to do some traveling in China and Japan. Now I'm finally back! I was in China for 1.5 months and in Japan for 2 weeks, and it was amazing! I made a short video highlighting parts of my trip:

Being a mobile developer, I naturally paid some attention to mobile phone usage in China, and how it's different from what I'm used to in the U.S. The cellular infrastructure was impressive. It was fairly cheap, and I was getting full 3G/4G service in small villages and along high-speed rail routes. It seemed like everyone had a smartphone, too. I would see grandmas standing on the side of the road checking their phones.

I never use QR codes in the U.S., but I actually used them quite often in China. For example, you would scan another person's QR code to add them as friends on Wechat. In some places, you could scan a merchant's QR code to pay that merchant using Alipay, a wallet app. Many types of tickets like train tickets and movie tickets also use QR codes over there.

Everyone used Wechat, a messaging app that's “way better than anything else in the U.S.” according to my American friend living in China. It's more than just a messaging app though – you have a “friend circle” that you can post to, a la Facebook; you can also follow “public accounts”, a la Twitter. The app has integrated wallet functionality: I paid for a train ticket and topped up my phone using the app; during Chinese New Year, people were sending each other cash gifts through it.

For some reasons, you see a lot of these “all-in-one” apps in China. I used Baidu Maps during my travel, which does maps and navigation. However, you can also call taxis from within the app or hire a “private car”, a la Uber. You can use the app like Yelp to find nearby restaurants by type and reviews. While you're at it, the app lets you find “group buy” discounts to these restaurants, a la Groupon. I have to say it was super convenient. After I came back to the States, I wasn't used to using Google Maps anymore because it didn't do as much.

Of course, on the flip side, these apps probably would be less popular without the Internet censorship that's so prevalent over there. By creating a barrier for foreign companies to enter the Chinese market, it provided opportunities for domestic companies to create and adapt copycat products. I found it amusing that Android is so prevalent in the Chinese smartphone market, but everything Google is blocked. As a result, you have all these third-party markets that may or may not be legitimate. Mobile malware seems to be a much larger issue in China than in the U.S., because people have to find their apps off of random markets/websites. It was strange to see an apps market promising “safe, no malware” with every download link. Also amusingly, every larger app I saw came with its own updater, again because these apps could not count on having a market to provide update service.

Overall, the trip was quite eye-opening, to see China's tremendous development from multiple angles. I loved Japan, too; I felt it was a lot different from both China and the U.S. Maybe I'll write about Japan in another post.

Historically, JNI code in Fennec has mostly used raw JNI types like jobject and jstring. However, the need to manage object lifetimes and the lack of strong typing make this practice error-prone. We do have some helper classes like AutoLocalJNIFrame, RefCountedJavaObject, WrappedJavaObject, and AutoGlobalWrappedJavaObject, but I've found them to be inconvenient to use.

Bug 1116868 is introducing several new “smart” classes to improve dealing with JNI references. As a start, instead of using raw jobject types, there are now different smart types for different usages:

Note that these new classes are under the mozilla::jni namespace, so your code should include it first,

using namespace mozilla::jni; // then use Object::LocalRef
namespace jni = mozilla::jni; // then use jni::Object::LocalRef

These classes make managing lifetimes very easy. Previously, it was easy to make mistakes like this,

jobject obj = GetObject();
// use obj
obj = GetAnotherObject(); // oops! first obj was leaked

This mistake could happen to both local and global references. Now with the smart classes, these errors are eliminated,

Object::LocalRef obj = GetObject();
// use obj
obj = GetAnotherObject(); // first obj was automatically deleted

The new classes also make it easy to convert between local and global references,

// Object::GlobalRef mObj;
Object::LocalRef obj = mObj; // automatic conversion from global to local ref
obj = GetNewObject();
mObj = obj; // automatic conversion from local to global ref

For function parameters, you can pass either a LocalRef or a GlobalRef to a Param parameter,

void SetObject(Object::Param obj);
 
Object::LocalRef obj;
SetObject(obj); // pass in LocalRef
// Object::GlobalRef mObj;
SetObject(mObj); // pass in GlobalRef

For function return values, you should return a LocalRef per JNI convention,

Object::LocalRef GetObject() {
  Object::LocalRef ret = MakeObject();
  return ret;
}

Note that in the above example, only one local reference is ever created because of return value optimization performed by the compiler.

Both LocalRef and GlobalRef support move semantics, so the following example still creates only one local reference overall,

Object::LocalRef GetObject();
 
Object::LocalRef foo;
foo = GetObject();

It also means you can use LocalRef and GlobalRef with container classes like mozilla::Vector without worrying about performance impact.

LocalRef and GlobalRef can be used like pointers/Java references,

Object::GlobalRef ref = nullptr;
ref = GetRef();
if (ref) {
  Object::LocalRef ref2 = ref;
  // compare underlying objects, so a LocalRef
  // and a GlobalRef of the same object are equal
  MOZ_ASSERT(ref == ref2);
}

jobject is not the only wrapped type; other JNI types correspond to different smart types,

And if you use auto-generated classes from widget/android/GeneratedJNIWrappers.h, each class is being updated in bug 1116589 to have its own reference types. For example, mozilla::widget::ViewTransform::LocalRef is a local reference to a Java ViewTransform instance,

ViewTransform::LocalRef vt = ViewTransform::New();
float x = vt->OffsetX(); // get offsetX field
vt->OffsetX(1.0f); // set offsetX field

Using separate classes ensures better type-safety, because type-checking is done by the compiler,

Foo::LocalRef foo;
Bar::LocalRef bar = foo; // error: invalid conversion

However, the auto-generated classes often only accept Object::Param parameters or return Object::LocalRef values. Therefore, as a special case, any LocalRef or GlobalRef can automatically convert to Object::Param, and Object::LocalRef can automatically convert to any other LocalRef.

Object::LocalRef Foo(Object::Param foo);
 
Bar::LocalRef bar;
bar = Foo(bar); // bar is converted to Object::Param
// then return value is converted to Bar::LocalRef

The String types have some custom behavior in addition to the standard behavior. A String::LocalRef or String::GlobalRef can automatically convert to a nsString or a nsCString,

String::LocalRef GetString();
 
nsString str = nsString(GetString());
nsCString cstr = nsCString(GetString());

Conversely, a nsAString or a nsACString can automatically convert to a String::Param,

void SetString(String::Param param);
 
nsString str;
SetString(str);
SetString(NS_LITERAL_CSTRING("text"));
 
String::LocalRef ref;
SetString(ref); // okay too

A String::LocalRef, String::GlobalRef, or String::Param also has a Length method,

size_t GetStringLength(String::Param param) {
  return param.Length();
}
 
MOZ_ASSERT(GetStringLength(NS_LITERAL_STRING("text")) == 4);

The goal of these new classes is to make using raw JNI types obsolete. However, until all the refactoring is done, there are still cases where raw JNI values are needed, for example to call JNIEnv functions.

To get a raw JNI reference from any LocalRef or GlobalRef, call its Get method,

void Foo(jobject param);
 
Object::LocalRef obj;
Foo(obj.Get());

To turn a raw JNI reference into any Param, call the Foo::Ref::From method,

void Foo(Object::Param param);
 
jobject obj;
Foo(Object::Ref::From(obj));

To return a raw JNI reference from any LocalRef or GlobalRef, call its Forget method,

jobject GetRawRef() {
  Object::LocalRef ref = GetRef();
  return ref.Forget();
}

Use LocalRef::Adopt to manage a returned raw local reference,

jobject GetRawRef();
 
Object::LocalRef ref = Object::LocalRef::Adopt(GetRawRef());

LocalRef also has an Env method that returns a cached JNIEnv pointer,

Object::LocalRef ref = GetRef();
auto cls = ClassObject::LocalRef::Adopt(
  ref.Env()->GetObjectClass(ref.Get()));

We use the adb logcat functionality a lot when developing or debugging Fennec. For example, outside of remote debugging, the quickest way to see JavaScript warnings and errors is to check the logcat, which the JS console redirects to. Sometimes, we catch a Java exception (e.g. JSONException) and log it, but we otherwise ignore the exception. Unless you are actively looking at the logcat, it's easy to miss messages like these. In other cases, we simply want a way to check the logcat when away from a computer, or when a user is not familiar with adb or remote debugging.

The LogView add-on, available now on AMO, solves some of these problems. It continuously records the logcat output and monitors it. When it sees an error in the logcat, the error is displayed as a toast for visibility.

You can also access the current logs through the new about:logs page.

The add-on only supports Jelly Bean (4.1) and above, and only Fennec logs are included rather than logs for all apps. Check out the source code or contribute on Github.

Feature suggestions are also welcome! I think the next version will have the ability to filter logs in about:logs. It will also allow you to copy logs to the clipboard and/or post logs as a pastebin link.

Jon Foreman (of the band Switchfoot) recently wrote a post called “Why I Refuse To Protest Protestors”. In it, he talked about his experience dealing with protests at his band's concerts. A few sentences he wrote struck me the most,

All at once I had an epiphany: these puzzling creatures that are yelling at you are human souls – as unpredictable, perplexing and unpredictable as I am. Here's the shocker: this guy with the bullhorn could be my cousin! he could be a friend of mine! Better yet: this guy could be me! If our lives were swapped, who can say that I would be any different? I put nothing below me. Who can say what I would do if I had his reality? Compassion makes you realize what you have in common with the rest of humanity.

For me as a Mozillian, these past few days have been confusing and unsettling. First it was the protests against Brendan's appointment. For an organization you are a part of, for a pioneer you look up to, for them to be rallied against, it was hard to understand. Then it was Brendan's resignation. That was unexpected and felt like things falling apart. Now it is another wave of protests against Brendan's departure. In the midst of this whirlwind, I want to feel anguish. I want to shout. Why couldn't Brendan just recant? Why couldn't the protesters just understand? Why couldn't the media just set the story straight? Why couldn't Mozilla just handle everything a little better? I want to protest.

But that's not what Mozilla is about. Like Katharina said, Mozilla is great at not only shipping great products but also at shipping love. It was love, love for the web, that first brought Mozilla together. It is love that continues to drive our mission now. Like Jon wrote,

On my best days, I want to stand for love conquering a multitude of wrongs. I want to stand for forgiveness, for mercy, for beauty, for grace. I stand for you, sir and madame. Whether you are holding a megaphone or not. Even when you refuse to shake my hand I love you. Whether you insult me or not, drunk or sober; I honestly love you! I love your passion, your fervor, your dedication. I want to know you better. I want to find out what makes you tick. I want to know why you believe what you believe. I want to learn from you. I am for you, emphatically for you!

Let's do what we do best, at showing love. Show love to Brendan; thank him for all that he's done and wish him the best. Show love to the activists; empathize with them. Show love to people who think Mozilla is not inclusive. Show love to the journalists, to companies protesting us. Show love to the Board, to your peers, to the community. Let's show our love to Mozilla.

Oftentimes I make a big commit that I later realize should be split into two commits/patches. I think the usual way to split a Git commit is the reset/“add -p”/commit sequence, like this,

# assuming HEAD is the commit to split,
git reset HEAD^ # first undo the commit
git add -p      # then choose bits to separate out
git commit      # commit separated changes
git commit -a   # commit other changes

But lately I've been using another way that involves a “commit –fixup”/revert/“rebase -i” sequence, like this,

# assuming HEAD is the commit to split,
# first, in Your Favorite Editor, delete everything that you want to separate out
git commit -a --fixup HEAD        # then commit the changes you removed as a fixup
git revert -e HEAD                # revert the fixup; this will become the split commit
git rebase -i --autosquash HEAD~3 # combine the original commit and the fixup

One thing I like about this second method is that it lets me pick out the changes to separate out in an editor with full context. I think this is a lot better than git add -p, which involves editing the hunks directly, and sometimes provides too little context to let me determine if a particular hunk should be separated out or not.

Over the last few months, I've been working on an improved App Not Responding (ANR) dashboard for Fennec, which is now hosted at telemetry.mozilla.org/hang/anr. With the help of many people, I'm glad to say that the dashboard is now mature enough to be a useful tool for Fennec developers.

The idea of ANR/hang reporting is similar to crash reporting — every time the Fennec UI becomes unresponsive for more than five seconds, Android would show an “App Not Responding” dialog; the ANR Reporter detects this condition and collects these information about the hang:

• Stacks for Java threads in Fennec
• Stacks for Gecko threads (C++ stacks and profiler pseudo-stacks)
• System information listed in about:telemetry
• Fennec logs to help debug the hang

The ANR Reporter is enabled on Nightly and Aurora builds only, and if the user has not opted out of telemetry, the collected information is sent back to Mozilla, where the data are aggregated and presented through the ANR Dashboard. Because the debug logs may contain private information, they are not processed and are only available internally, within Mozilla.

The ANR Dashboard presents weekly aggregated data collected through the ANR reporter. Use the drop-down list at the top of the page to choose a week to display.

Data for each week are then grouped by certain parameters from ANR reports. The default grouping is “appName”, and because ANR reports are specific to Fennec, you only see one column in the top hangs chart labeled “Fennec”. However, if you choose to group by, for example, “memsize”, you will see many columns in the chart, with each column representing a different device memory size seen from ANR reports.

Each column in the top hangs chart shows the number of hangs, and each column is further divided into blocks, each representing a different hang. Hover over the blocks to see the hang stack and the number of hangs. This example shows 8 hangs with that signature occurred on devices with 768MB of memory over the past week.

Colors are preserved across columns, so the same colored blocks all represent the same hang. The blue blocks at the bottom represent all hangs outside of the top 10 list.

To the right of the top hangs chart is the distributions chart. It shows how different parameters are distributed for all hangs. Hover over each block to see details. This example shows 36% of all hangs occurred on devices running Android API level 15 (corresponding to Android 4.0.3-4.0.4 Ice Cream Sandwich) over the past week.

The distributions chart can also be narrowed down to specific groups. This would let us find out, for example, on devices having 1GB of memory, what is the percentage of hangs occurring on the Nightly update channel.

Clicking on a block in the top hangs chart bring up a Hang Report. The hang report is specific to the column that you clicked on. For example, if you are grouping by “memsize”, clicking on a hang in the “1G” column will give you one hang report and clicking on the same hang in the “2G” column will give you a different hang report. Switch grouping to “appName” if you want to ignore groups — in that case there is only one column, “Fennec”.

The hang report also contains a distributions chart specific to the hang. The example above shows that 14% of this hang occurred on Nexus 7 devices.

In addition, the hang report contains a builds chart that shows the frequency of occurrence for different builds. This example shows there was one hang from build 20140224030203 on the 30.0a1 branch over the past week. The chart can be very useful when verifying that a hang has been fixed in newer builds.

Last but not least, the hang report contains stacks from the hang. The stacks in the hang report are more detailed than the stack shown on the main page. You can also look at stacks from other threads — useful for finding deadlocks!

When comparing the volume of hangs, a higher number can mean two things — the side with higher number is more likely to hang, or the side with higher number has more usage. For example, if we are comparing hangs between devices A and B, and A has a higher number of hangs. It is possible that A is more prone to hanging; however, it is also possible that A simply has more users and therefore more chances for hangs to occur.

To provide better comparisons, the ANR Dashboard has a normalization feature that tries to account for usage. Once “Normalize” is enabled at the top of the dashboard, all hang numbers in the dashboard will be divided by usage as measured by reported uptime. Instead of displaying the raw number of hangs, the top hangs chart will display the number of hangs per one thousand user-hours. For example, 10 hangs per 1k user-hour means, on average, 1000 users each using Fennec for one hour will experience 10 hangs combined; or equivalently, one user using Fennec for 1000 hours will experience 10 hangs total. The distributions chart is also updated to reflect usage.

As a demonstration, the image below shows un-normalized hangs grouped by device memory size. There is no clear trend among the different values.

The image below shows normalized hangs based on the same data. In this case, it is clear that, once usage is accounted for, higher device memory size generally corresponds to lower number of hangs. Note that the “unknown” column became hidden because there is not enough usage data for devices with “unknown” memory size.

At the moment, I think uptime is the best available measurement for usage. Hopefully there will be a better metric in the future to provide more accurate results. Or let me know if it already exists!

The standard library function setvbuf() can be used to set unbuffered mode for a stream. However, fread() on Android has the behavior that, in unbuffered mode, it can sometimes return EOF, or (size_t)(−1), instead of the number of elements actually read. This unexpected behavior was the cause for bug 935831.

I took a glance at the few other places where Gecko uses unbuffered mode, and fortunately, it appears none of them apply to a regular Android run.