Done:

• Work on bug 461199
• Work on bug 542592 (learning oh so much about content/)
• CMU career fair

Done:

• Work on bug 461199

Done:

• Hammered away on getting tests passing on bug 461199
• Created "New Tabs at the End" add-on for the support team
• Created a control flow diagram of how mozilla::IHistory and mozilla::dom::Link interact to help other folks understand

Done:

• Q1 I/O Reduction goal page up and running (https://wiki.mozilla.org/Firefox/Goals/2010Q1/IO_Reduction) with numbers and bugs
• Review queue neutralized
• Bug 461199 (async visited checks) nearly ready. Working on fixing last remaining failing tests. Two more issues remain to be fixed (other than tests).

• NetUtil.newURI can take a string (plus optional character set and base URI) or an nsIFile.
• A new method for creating channels has been created. NetUtil.newChannel can take an nsIURI, a string (plus optional character set and base URI), or an nsIFile.
• NetUtil.asyncFetch can take an nsIChannel, an nsIURI, a string (plus optional character set and base URI), or an nsIFile.

This means, among other things, that it now requires less code to read a file asynchronously than it does synchronously. The old way to do this asynchronously can be seen here on MDC. This would give the consumer a byte array of the data in the file. Compared to the synchronous case, which can be seen here. Both are pretty verbose and clunky to use. The new way looks like this:

NetUtil.asyncFetch(file, function(aInputStream, aResult) {
  if (!Components.isSuccessCode(aResult)) {
    // Handle Error
    return;
  }
  // Consume input stream
});

One function call, with a callback passed in. There is a slight difference from the old asynchronous method, however. NetUtil.asyncFetch gives the consumer an nsIInputStream instead of a byte array. The input stream is a bit more useful than a raw byte array, although it can be painful to use in JavaScript at times (maybe we need an easy method to convert an input stream to a string?). I look forward to patches using this method to read files instead of doing it synchronously.

• Use version 0.3 of the REST API.
• Enable spell checking in the comment field.
• Allow replying to New: bugmail.
• Offline support.

Found a bug? E-mail me or leave a comment.

Directory structure

In my home directory, I created a new folder called opt.
mkdir opt
You’ll also want to add the following lines to your .bashrc file:
export CFLAGS="-I$HOME/opt/include -L$HOME/opt/lib $CFLAGS"
export CXXFLAGS="-I$HOME/opt/include -L$HOME/opt/lib $CXXFLAGS"
I also created a sources directory, where I’ll be putting all of my source files in.
mkdir sources
We’ll want to be in the sources directory as we install everything.
cd sources

Installation Fun

Install python

wget http://www.python.org/ftp/python/2.6.4/Python-2.6.4.tgz
tar xvfz Python-2.6.4.tgz
cd Python-2.6.4
./configure --prefix=/home/sdwilsh/opt/
make
make install


Install OpenSSL

After you get the source from a mirror and untar it, enter that directory, and run these commands:
./config --prefix=/home/sdwilsh/opt --openssldir=/home/sdwilsh/opt/openssl shared
make
make install

Install Erlang

Note: I had to bump my memory on my PS up to 304 MB (from 150 MB) in order for this to compile. I just doubled it, so you may be able to get by with less. This also takes a while to install.
wget http://erlang.org/download/otp_src_R13B02-1.tar.gz
tar xvfz otp_src_R13B02-1.tar.gz
cd otp_src_R13B02-1
./configure --prefix=/home/sdwilsh/opt/ --with-ssl=/home/sdwilsh/opt/
make
make install

Install ICU

wget http://download.icu-project.org/files/icu4c/4.2.1/icu4c-4_2_1-src.tgz
tar xvfz icu4c-4_2_1-src.tgz
cd icu/
chmod +x runConfigureICU configure install-sh
./runConfigureICU Linux --prefix=/home/sdwilsh/opt
make
make check
make install

Install autoconf-2.13

wget http://ftp.gnu.org/gnu/autoconf/autoconf-2.13.tar.gz
tar xvfz autoconf-2.13.tar.gz
./configure --prefix=/home/sdwilsh/opt/
make
make install

Install Spidermonkey

Note: You many need to install Mercurial. I used easy_install to do.
hg clone http://hg.mozilla.org/releases/mozilla-1.9.2/
cd mozilla-1.9.2/
hg update -r FIREFOX_3_6b3_RELEASE
cd js/src/
autoconf
mkdir build-release
cd build-release
../configure --prefix=/home/sdwilsh/opt/
make
make install

Install libssh2

wget http://www.libssh2.org/download/libssh2-1.2.2.tar.gz
tar xvfz libssh2-1.2.2.tar.gz
cd libssh2-1.2.2
./configure --prefix=/home/sdwilsh/opt/ --with-openssl
make
make install

Install curl

After you get the source from a mirror and untar it, enter that directory, and run these commands:
cd curl-7.19.7
./configure --prefix=/home/sdwilsh/opt/ --with-libssh2=/home/sdwilsh/opt --with-ssl=/home/sdwilsh/opt
make
make install

Install couchdb

After you get the source from a mirror and untar it, enter that directory, and run these commands:
./configure --prefix=/home/sdwilsh/opt/ --with-erlang=/home/sdwilsh/sources/otp_src_R13B02-1/include --with-js-include=/home/sdwilsh/opt/include --with-js-lib=/home/sdwilsh/opt/lib --with-erlang=/home/sdwilsh/opt/lib/erlang/usr/include
make
make install
Note: I feel like I may have messed up how I installed erlang given the strange place it put its header files, but I wasn’t about to recompile it.

Change Permissions & Run

CouchDB suggests you create a new user for to run it is, but this is hard to do it seems, so I skipped it. I did change the permissions, however.
chmod -R 0770 /home/sdwilsh/opt/etc/couchdb
chmod -R 0770 /home/sdwilsh/opt/var/lib/couchdb
chmod -R 0770 /home/sdwilsh/opt/var/log/couchdb
chmod -R 0770 /home/sdwilsh/opt/var/run/couchdb

And now to run CouchDB:
couchdb -b
Sadly, this was taking something close to 500MB of memory. This is far to high for my server to sustain, so I haven’t gotten any further. If anyone has any ideas, I’d be glad to hear them.

There are still some apparent issues with the REST API that the add-on is using, and I’ll likely include some workaround in upcoming versions. 0.2.0 is available on addons.mozilla.org and is a recommended upgrade. Current users will have to update since sandboxed add-ons do not automatically update.

Background

This fix wouldn’t be in our hands if it was for the recent CrashKill effort here at Mozilla. Sam Sidler gave me a link to a list of crashes that were reported to us during a 24 hour period in SQLite and mozStorage code. I immediately started adding links to crash reports and filing bugs to some of the more frequent crashes so we could start tracking these issues. I also informed the SQLite team about this page in case they found the data useful, and they sure did. They started to look into this particular crash. It turns out that there are two other signatures that are actually the same crash, but show up differently due to compiler optimizations causing differences in the generated program.

Fix Details

SQLite has a virtual machine that processes queries. The virtual machine is register-based, which is different from the usual stack-based virtual machines you might encounter with Java. This particular crash would surface when a SQL query used an IN operator with more than ~32 thousand entries on its right-hand side and the EXISTS expression. If either of these conditions occurred on their own, SQLite could handle this just fine. The IN operator stores each entry in a register, and when processing the EXISTS expression, SQLite would store the number of the register in which an expression was written into into a 16-bit integer on an Expr object. However, once you try to write a number greater than 32,767 to a 16-bit integer, you get overflow and you will likely end up with a negative number. This negative number was then used to index into an array, which is all sorts of bad (and led to the crash).

The workaround fix is to just use a 32-bit integer for now. This means this issue will not happen unless you have over 2,147,483,647 entries on the right-hand side of the IN operator. You’d hit other limitations of SQLite before that would ever happen, so overflow is no longer a concern. The SQLite team is going to do a more complicated fix which stores the value separately from Expr object it was previously stored in. The Expr object is used a lot, so they don’t want a long-term solution that increases memory usage.

You might be surprised that such a small and trivial seeming bug got through. This type of bug is more common than you might think, and overflow bugs have even caused the destruction of an Ariane 5 rocket. It’s an unfortunate reality of software that sometimes these edge cases get missed, but we must always try to remain vigilant.

Special thanks to the SQLite team for investigating this fix, and providing information for this blog post.
Provide anonymous feedback on this post with Rypple.

Please think twice before publishing/committing something. Thanks!