The Benefits

The work on the Places branch brings us a number of benefits. In general, we’ve parallelized work, and made it substantially less likely that we’ll block on the GUI thread. Some of the important fixes we have landed are:

• Faster Location Bar
  The location bar is faster because other database work no longer blocks us from searching, and the queries are much simpler.
• Asynchronous Bookmark Notifications
  Indicating if the current page is bookmarked in the location bar (with the star) is now an asynchronous operation that does not block the page load.
• Faster Bookmarks & History Management/Searches
  Simpler queries and other improvements should make this all work faster.
• Faster Link Coloring
  Link coloring is now executed on a separate database connection so it cannot block other database work.
• Expiration Work
  Less work at startup, less work at shutdown, and less work when we run expiration.
• Less Data Stored
  Embedded pages are now tracked only in memory and never hit the disk.
• Better Battery Management
  Much less work during idle time, which will improve our power consumption behaviors.
• Fixes 29 blockers and 18 other issues

A bit of History

Way back in the days leading up to Firefox 3.5, we moved from storing all of our history and location data in disk tables to in-memory tables that we’d flush out to disk every two minutes off of the GUI thread. The benefit of this was two-fold:

1. No longer performing the vast majority of our disk writes on the GUI thread
2. No longer performing the vast majority of our fsyncs/Flushes on the GUI thread

More details about how we came up with this solution can be found in a series of blog posts.

The Problem

This solution has worked out pretty well for us for a while, but recently, especially on OS X, it has not been. The short story is that our architecture did not scale well due to lock contention between our GUI thread and our background I/O thread. While the common case access case may be fine, the failure case (when we hit lock contention) is pretty terrible. The problem is so terrible that I once described it like this:

the failure case makes us fall on our faces, skid about 100 feet, and then fall off a cliff without a parachute.

Ultimately, the only way we can avoid this situation is to not do any database work on separate threads with the same database connection. It was not an issue in the past because we just did not do enough work on the I/O thread, but as we have added to the workload of that thread, we increase the likelihood of it holding the lock, which means there is a higher probability that the GUI thread will not be able to instantly acquire the lock and do whatever it needs to do. This essentially leaves us with two options:

1. Move the rest of our database work off of the GUI thread.
2. Move database work from the I/O thread back to the GUI thread.

The Solution

The second choice is not actually a viable option. Disk I/O completes in a non-deterministic amount of time, which is why we have been moving it from the GUI thread to an I/O thread since Firefox 3.5. The first choice is not entirely viable either due to schedule constraints either (we have tons of API calls that are not used heavily but still synchronous). A hybrid solution exists, however. We can reduce the amount of work we do on the I/O thread by using additional I/O threads. Additionally, we can move the remaining synchronous operations during browsing to an I/O thread. In the end, Places ends up with one read/write thread, and multiple read-only threads.

This wasn’t really an option back in the Firefox 3.5 days because in SQLite readers and writers blocked each other. However, the SQLite developers recently devised a new journaling method called WAL that lets readers not block writers, and writers not block readers. When the Places branch merges into mozilla-central, we will end up with three read-only I/O threads and our original read-write I/O thread. The three read-only threads are used for location bar searches, visited checks (is a given hyperlink visited), and some bookmark operations. Each I/O thread has its own connection to the database, allowing operations to happen in parallel (SQLite is only threadsafe because it serializes all access on each connection object, which is why we had the lock contention in the first place).

Performance Test Issues

One of the things that made this work especially difficult is seemingly random changes in performance numbers. We often had regressions suddenly appear (according to talos) on changesets that would have zero impact on performance, and then backing out the change would cause an additional regression. Other times, when we would merge mozilla-central into Places, we would suddenly get new regressions when comparing to mozilla-central. This could be indicative of a bad interaction with our code and the changes on mozilla-central, however after looking at the changes on mozilla-central that landed with the merge, that appeared to be highly unlikely.

I’m also quite certain that some of our performance tests do not actually test/measure what we actually want to test/measure. I’ll leave that discussion to a future blog posts, however.

Background

This fix wouldn’t be in our hands if it was for the recent CrashKill effort here at Mozilla. Sam Sidler gave me a link to a list of crashes that were reported to us during a 24 hour period in SQLite and mozStorage code. I immediately started adding links to crash reports and filing bugs to some of the more frequent crashes so we could start tracking these issues. I also informed the SQLite team about this page in case they found the data useful, and they sure did. They started to look into this particular crash. It turns out that there are two other signatures that are actually the same crash, but show up differently due to compiler optimizations causing differences in the generated program.

Fix Details

SQLite has a virtual machine that processes queries. The virtual machine is register-based, which is different from the usual stack-based virtual machines you might encounter with Java. This particular crash would surface when a SQL query used an IN operator with more than ~32 thousand entries on its right-hand side and the EXISTS expression. If either of these conditions occurred on their own, SQLite could handle this just fine. The IN operator stores each entry in a register, and when processing the EXISTS expression, SQLite would store the number of the register in which an expression was written into into a 16-bit integer on an Expr object. However, once you try to write a number greater than 32,767 to a 16-bit integer, you get overflow and you will likely end up with a negative number. This negative number was then used to index into an array, which is all sorts of bad (and led to the crash).

The workaround fix is to just use a 32-bit integer for now. This means this issue will not happen unless you have over 2,147,483,647 entries on the right-hand side of the IN operator. You’d hit other limitations of SQLite before that would ever happen, so overflow is no longer a concern. The SQLite team is going to do a more complicated fix which stores the value separately from Expr object it was previously stored in. The Expr object is used a lot, so they don’t want a long-term solution that increases memory usage.

You might be surprised that such a small and trivial seeming bug got through. This type of bug is more common than you might think, and overflow bugs have even caused the destruction of an Ariane 5 rocket. It’s an unfortunate reality of software that sometimes these edge cases get missed, but we must always try to remain vigilant.

Special thanks to the SQLite team for investigating this fix, and providing information for this blog post.
Provide anonymous feedback on this post with Rypple.

Background

We’ve been getting reports for some time about the location bar hanging the application for some users when they are typing in it. This wasn’t a problem that was reproducible on every machine, and even on machines that saw it, it wasn’t always 100% reproducible. Clearly, this behavior is not desirable, so we set out to fix it.

I had a theory to the cause almost a year ago and filed a bug that I was hoping we could work on and fix for Firefox 3.5. We knew that reading data off a disk can be slow (and certainly would complete in a non-deterministic amount of time). Since SQLite uses blocking read calls (no more code can execute until the data is read from disk), this could certainly be the cause of the slowdown our users were seeing. Some simple profiling showed that this was largely the cause of the hanging. Work began on the project, but it was clear that enough issues were cropping up that we were not going to be able to safely take this change for Firefox 3.5, and resources were diverted elsewhere.

Process and Solution

This section is a bit technical, so feel free to skip it. The short answer is “do not block the main thread while reading from the hard drive.”

In order to not block the main thread while reading from disk we either need to make SQLite use non-blocking read system calls, or call into SQLite off of the main thread. Changing the SQLite code isn’t something we want to do, so that solution was out of the question. Luckily, we had solved a similar problem with writes and fsyncs earlier in the Firefox 3.5 development with the asynchronous Storage API.

The first implementation that we tried essentially did the same thing that the old code did. We would execute a query, but this time asynchronously, and then process the results and see if they match. There were two issues with this approach, however. The first issue was that we were filtering every history and bookmark entry on the main thread for a given search. That could be a lot of work we end up doing, and with the additional overhead of moving data across threads, the common case would see no win. The second issue was that once we selected a result in the location bar, and a search was not yet complete, there would be a hang as the main thread processed a bunch of events that Storage had posted to it containing results.

At this point, we realized we needed to do the filtering on a thread other than the main thread. After some thought, we was figured that the easiest way to do that would be to use a SQL function that we define in the WHERE clause of our autocomplete queries. This way, all the filtering is done on a background thread, and the code that runs on the main thread only deals with results we will actually use. This solution exposed some things in the Storage backend like lock contention and a few other subtle issues, but nothing major came up.

For more details on how the location bar search results are generated, see my explanation here.

If you weren’t having a problem before, chances are you won’t notice any difference at all.

Luckily, SQLite exposes a handy function that tells us this information. In general, you want to use a index in your ORDER BY clause so SQLite can use it to generate the order the results a given back. If you do not use an index, SQLite has to first get all the results in memory, then it sorts them, and then it starts to return results. If you expect a lot of results, this can get very expensive.

There were suggestions on the newsgroups to also add an API so debuggers such as ChromeBug or SQLite Manager could listen for these types of errors. I’ll be filing follow-up bugs for some of these suggestions later on.

Once I had those logs, I needed some way to parse the files for data so I can use it in a reasonable way. I wrote a python script to parse the strace logs, and then insert them into a sqlite database file (26.8 MB) so I could run some interesting queries on the data.

With that data, I decided to generate some graphs to easily see what was going on. All of these graphs compose the data from the six runs of Firefox that talos ran – the data is all summed up. All the graphs have larger versions available if you click on them.

I figured that the most useful graph for investigating this Ts regression would be execution time:

Note that that is six runs of Firefox, which is why it is as long as it is. Next, I looked at the average execution time for each function call:

And finally, I looked at the number of calls of each of these functions:

We are clearly seeing an increase in the number of fsync calls, and we know that on Linux those can be more painful than they are on other operating systems. My next step is to see if we also see this increase on OS X. If we do, I’m going to assume we see it on windows as well, and get backtraces of every single fsync call to determine why we’ve double the number of calls by upgrading.

I’ll make a new post as more data comes in.