The default behavior of SQLite is to only hold file locks for the minimum amount of time necessary. So when a process writes to the database, it first obtains a write lock on the file, makes its change, then immediately releases the lock so that other processes can also have a turn writing.

There is an alternative behavior called “exclusive locking mode” which FF uses (if I am not badly mistaken.) In exclusive locking mode, once SQLite obtains a write lock on the database, it holds that lock until the database connection is closed. This means that other processes (or even other database connections within the same process) are unable to modify the database; concurrency takes a big hit. But the way FF uses SQLite, only one process is accessing the database at a time anyhow, so that doesn’t really matter. And by holding the lock indefinitely, the process that owns the lock does not have to bother to reacquire and rerelease the lock for each transaction, nor does it have to check to see if another process has modified the database between transactions. There are speed advantages there. (The speed advantages are not that great on a workstation running Linux, but they can be significant for embedded devices with less powerful filesystems.)

A change occurred in the way transactions commit in exclusive locking mode in between SQLite version 3.5.4 and 3.5.9. In exclusive locking mode in version 3.5.4, a transaction commits by truncating the rollback journal to zero length. In SQLite 3.5.9, we changed commit to zero out the header of the rollback journal and then fsync(). We found that on many systems, increasing the length of a file (such as the rollback journal) is much more expensive than overwriting an existing segment of a file. By zeroing the rollback journal header, the rollback journal is still non-zero length at the start of the next transaction and writes to the rollback journal are therefore faster. This is a big performance win on many systems, especially embedded systems.

To summarize, 3.5.4 does a single ftruncate() call whereas 3.6.2 does a small write() followed by an fsync().

One could argue that the ftruncate() call in 3.5.4 should also have been followed by an fsync(). I suppose whether or not it is safe to do an ftruncate() that is not followed by an fsync() is filesystem dependent. Probably the fsync() is not required on ext3, but it might be required on other older filesystems. I’m not sure. But it seems that a small write() followed by an fsync() as is done in 3.5.9 and later is a safer approach.

On the other hand, the new approach does generate extra fsync() calls.

There is no option in the latest SQLite that allows one to revert from the write()/fsync() commit back to the older ftruncate() commit. However, taking SQLite out of exclusive locking mode will cause it to use unlink() for commits. Perhaps an unlink() is faster than a write()/fsync() on ext3. It seems like it is worth a try.

We will also investigate compile-time or run-time options to SQLite that allow one to choose the ftruncate() commit algorithm instead of the newer write()/fsync() commit.