eTLD Meet mozStorage?

While talking with Edward last night about searching in the download manager, we came across an interesting idea. Instead of trying to guess a good substring of the URI to do a LIKE comparison for searching, we could process it with nsIEffectiveTLDService (there’s no devmo page for this sadly) after we query the DB. We then realized that that is not the most efficient way to do this, so we thought, “Hey, we can add functions to the database! We can just hook up eTLD that way!” Then it dawned on us on just how bad that could get, especially if we implemented that in JS like the download manager’s UI is in. We’d be crossing xpconnect quite a bit, and searches are already kinda slow. As a result, that idea got canned. We started thinking about how it could be done in C++ with the download manager component, which wouldn’t be as bad. That’s when I started to really think about this.

Places does some search stuff with the Awesomebar. I’m not sure if they do some guessing like we were thinking about doing, or if they post process it, or something entirely different. But, what if mozStorage automagically provided functions for you to perform eTLD things on strings in the DB? Storing URI’s in their string form seems to be fairly common in many of our sqlite databases. So, I’m looking to see what people think about providing some SQL functions that make people’s lives easier:

SELECT eTLD_GetPublicSuffix(referrer) AS suffix FROM moz_downloads
Where referrer is a column in the DB that stores an nsIURI. I don’t find this one so useful, but…

SELECT eTLD_GetBaseDomain(referrer, 1) AS referrer_search FROM moz_downloads WHERE referrer_search LIKE ?1
Again, with referrer being a column in the DB that stores an nsIURI. In this case, if the uri were “”, it’d return, and that is what would be searched by the LIKE expression. I can see this being potentially useful to many consumers, so I’m putting out feelers to see what people’s thoughts are. What do you think?

By Shawn Wilsher

The man behind the site.

5 replies on “eTLD Meet mozStorage?”

Figured this was as good a time as any to start contributing to DevMo. nsIEffectiveTLDService. It’s a start, at least. Now to write some Python that can grok the .idl file and auto-generate a doc stub, ’cause copy/paste is fun for all of 2 seconds.

Well… it maybe to late but…

wouldn’t be much faster and easier if subdomain, eTLD and the path were different fields on the table, instead of processing a string?

Well… I don’t know SQLite very well, but this way would be nice we could make URI a type and querying it’s parts just like we do with timestamps.

With different fields or a type, would be much quicker to search for all occurrences of a domain…


That’s actually a fairly good point. Ideally anywhere that we store URI’s that we need to parse for certain things, we should probably be storing those parts separately.

The reality of the situation though is that there isn’t much of a perf impact regardless (now, if this were a client server model, this would probably be an issue).

Comments are closed.