The Tests

• Clean profile. This is just the standard profile that we normally run Ts with on tinderbox. This is basically used a baseline for best possible performance.
• Dirty profile. This is actually my daily profile, with eight tabs that will open through session restore during startup. Because of how talos works, these tabs don’t all have to load for the number to be generated. Even so, you’ll notice a substantial slowdown. Sadly, I fear I modified the profile I was using in a bad way because I can no longer reproduce the numbers I got (but the numbers recorded were reproduced four times before I moved on to the rest of the tests initially).
• Bookmarks toolbar disabled. This is a variation on the dirty profile test that just disables the bookmarks toolbar.
• No places. This is a variation on the dirty profile test that removes places files from the profile.
• No sessionstore.js. This is a variation on the dirty profile test that removes sessionstore.js from the profile. This has the side effect of also not making the eight tabs load at startup.
• No urlclassifier. This is a variation on the dirty profile test that removes the urlclassifier related files from the profile.
• No cookies.sqlite. This is a variation on the dirty profile test that removes cookies.sqlite from the profile.
• No extensions. This is a variation on the dirty profile test that removes all add-on manager bits in the profile.
• No formhistory.sqlite. This is a variation on the dirty profile test that removes formhistory.sqlite from the profile.
• No downloads.sqlite. This is a variation on the dirty profile test that removes downloads.sqlite from the profile.
• No content-prefs.sqlite. This is a variation on the ditry profile test that removes content-prefs.sqlite from the profile.

Results

I’m going to let some graphs do the talking here. The first shows the raw test run data (which isn’t terribly interesting). The second compares the reported startup time for each test. You will probably want to click to zoom in.

Conclusions

It looks like the best wins that we can get are related to fixing session restore to not scale linearly with the number of tabs it is restoring, and reduce the startup time costs of loading places files and cookies.sqlite. It should be noted that this test was not measuring the load time for each tab, so something like BarTab would not help in this case. The other good news is that we already have work underway to make cookies.sqlite load time not hurt us so much during startup.

There are still some apparent issues with the REST API that the add-on is using, and I’ll likely include some workaround in upcoming versions. 0.2.0 is available on addons.mozilla.org and is a recommended upgrade. Current users will have to update since sandboxed add-ons do not automatically update.

Luckily, the fix was trivial. Now we handle the enter keypress immediately when the user hits it, and not later. Problem solved :)

Background

Session restore writes out to disk very frequently – every ten seconds, in fact. This behavior is controllable by the preference browser.sessionstore.interval for those who want to reduce that, but then you run the risk of not having all your data saved if you crash. We really don’t want to reduce that time for our users.

The amount of data that is written out to disk by session restore scales linearly with the number of tabs and windows you have open. The more you have, the more data has to be written out to disk, and the longer it is going to take.

As we learned in the past with Places, writing to disk and calling fsync can be painfully slow. In session restore code, we are doing this very often and on the main thread. Clearly, this is a bad thing.

Process and Solution

This section is a bit technical, so feel free to skip it. The short answer is “do not block the main thread while writing and flushing data to the hard drive.”

We wanted to address this problem as much as we could for Firefox 3.6. In order to actually reduce the number of writes and fsync calls, we would have to heavily modify how session restore manages and writes its data. That is a big change that we were not comfortable doing this late in the 3.6 cycle. On top of that, we do not really have the manpower to do that change since the people who know that code well are working on other performance improvements for this release. The simple solution for now then is to move our write and fsync calls off of the main thread.

Luckily, Boris Zbarsky had recently written a new API for JS consumers to asynchronously copy an input stream to an output stream. This API would work great for session restore! We had to fix one minor issue with the underlying code not properly handling nsISafeOutputStreams (which make sure we fsync properly), but once that was done, the fix was incredibly simple.

Background

We’ve been getting reports for some time about the location bar hanging the application for some users when they are typing in it. This wasn’t a problem that was reproducible on every machine, and even on machines that saw it, it wasn’t always 100% reproducible. Clearly, this behavior is not desirable, so we set out to fix it.

I had a theory to the cause almost a year ago and filed a bug that I was hoping we could work on and fix for Firefox 3.5. We knew that reading data off a disk can be slow (and certainly would complete in a non-deterministic amount of time). Since SQLite uses blocking read calls (no more code can execute until the data is read from disk), this could certainly be the cause of the slowdown our users were seeing. Some simple profiling showed that this was largely the cause of the hanging. Work began on the project, but it was clear that enough issues were cropping up that we were not going to be able to safely take this change for Firefox 3.5, and resources were diverted elsewhere.

Process and Solution

This section is a bit technical, so feel free to skip it. The short answer is “do not block the main thread while reading from the hard drive.”

In order to not block the main thread while reading from disk we either need to make SQLite use non-blocking read system calls, or call into SQLite off of the main thread. Changing the SQLite code isn’t something we want to do, so that solution was out of the question. Luckily, we had solved a similar problem with writes and fsyncs earlier in the Firefox 3.5 development with the asynchronous Storage API.

The first implementation that we tried essentially did the same thing that the old code did. We would execute a query, but this time asynchronously, and then process the results and see if they match. There were two issues with this approach, however. The first issue was that we were filtering every history and bookmark entry on the main thread for a given search. That could be a lot of work we end up doing, and with the additional overhead of moving data across threads, the common case would see no win. The second issue was that once we selected a result in the location bar, and a search was not yet complete, there would be a hang as the main thread processed a bunch of events that Storage had posted to it containing results.

At this point, we realized we needed to do the filtering on a thread other than the main thread. After some thought, we was figured that the easiest way to do that would be to use a SQL function that we define in the WHERE clause of our autocomplete queries. This way, all the filtering is done on a background thread, and the code that runs on the main thread only deals with results we will actually use. This solution exposed some things in the Storage backend like lock contention and a few other subtle issues, but nothing major came up.

For more details on how the location bar search results are generated, see my explanation here.

If you weren’t having a problem before, chances are you won’t notice any difference at all.

With my work on the asynchronous location bar, I wanted to speed up the call to executeAsync to make it as fast as possible so autocomplete look ups would be quick. To accomplish this, we now cache the asynchronous sqlite3_stmt object so we don’t recreate it on every call to executeAsync. This amortizes the cost that was causing the asynchronous API to be slower than the synchronous one.

This solution is not sufficient, however, if you bind parameters to the statement before calling executeAsync. This is because binding parameters stores state on the underlying sqlite3_stmt object, and we can’t store that information on it because it could be in use on the background thread. To get around this issue, I reuse a code from the recent asynchronous storage API addition to store the bound values in memory and then bind them at the time of execution on the background thread. Andrew Sutherland and I were both very happy at how small this patched ended up being as a result of good API design in the past.

The bonus with this two step solution is that we’ve also completely removed the possibility of lock contention inside of the SQLite library on the calling thread of executeAsync. If you have a long running sqlite3_step command on the background thread, this could block your call to one of the binding calls or executeAsync, which is undesirable.

This code is checked in on mozilla-central, and will be a part of Gecko 1.9.2.

1. There is a lot of flickering when new results show up. This is being tracked in bug 393902.
2. Your computer will hang for a period of time (it will become responsive again) if you continue to type once no results are found. This is being tracked in bug 503701.

This is built off of a “stable” point of mozilla-central, so it’s like using a 3.6 nightly. All the normal warnings apply about using it. I’m told this greatly increases the speed at which results obtained by many people. If you experience any issues (other than the two listed here), please let me know! The builds can be found here.

Your feedback is appreciated!

Luckily, SQLite exposes a handy function that tells us this information. In general, you want to use a index in your ORDER BY clause so SQLite can use it to generate the order the results a given back. If you do not use an index, SQLite has to first get all the results in memory, then it sorts them, and then it starts to return results. If you expect a lot of results, this can get very expensive.

There were suggestions on the newsgroups to also add an API so debuggers such as ChromeBug or SQLite Manager could listen for these types of errors. I’ll be filing follow-up bugs for some of these suggestions later on.

The Firefox team has been focusing on code sprints to get some small well scoped things done for Firefox 3.1 since we’ve got a bit more time. My latest sprint can be found over in bug 480211, where I’ve removed a write and fsync that we used to do after every page visited. If we had enough pages in history that were old enough, we would remove them from history. We now do this off of the main thread, asynchronously at the same time we flush data from our temporary tables to our permanent ones. The net result is the same number of writes and one less fsync. Additionally, the write is no longer done on the main thread.

Sadly, I couldn’t measure any real-world performance gains with my DTrace scripts – in fact I saw no change during several different runs of Tp3 with various places.sqlite files. It’s quite possible I did not have the conditions setup correctly to have pages expiring, and I could have spent a few more hours generating just the right places.sqlite file to demonstrate wins in the real world, but the theory behind the patch is pretty simple. The gain is pretty obvious.

Just another drop in the bucket of performance wins for Firefox. Stay tuned, as there is more to come!