Recently I’ve seen a large number of add-ons get nominated for public on addons.mozilla.org (AMO) that didn’t meet some of the basic requirements that an author is supposed to meet. I thought it might be a good idea to let add-on developers know what it is that I look for when I take a look at the public nomination queue. Hopefully they’ll stop nominating things that don’t meet the requirements.
Does the add-on have any reviews from its users?
AMO requires an add-on to have reviews from its users for it to even be considered for public nomination. Without reviews, we editors cannot effectively do our job. This is the most common reason that I have seen for an add-on being denied in the public nomination queue.
Are the reviews for the add-on simple, or are they more like a technical review?
Reviews such as “This is a great extension!” or “This theme looks really nice” are not very helpful. These type of reviews do give me an idea on how useful people think your add-on might be, but that is not enough for public nomination. If an add-on only has a few reviews, and they are like those two examples, I’ll often be inclined to retain the add-on in the sandbox.
The reviews that I really look for are people who install the add-on, test it, and report their findings. A report on a broken feature, with the platform (Windows, OS X, Linux), application (Firefox, Thunderbird) and its version, and the version of the extension is an awesome review. Reviews that go over what the reviewer tested and what they liked and didn’t like (with some details) are just as useful. If there are a few reports about a broken feature, I’ll be inclined to retain it in the sandbox and indicate that to the author. However, if there are enough favorable technical reviews, I’ll continue the process.
Is the author responsive?
If there are discussion posts associated with the add-on, I like to check and see if the author is responding, or is essentially ignoring them. If it has been several weeks since a post was posted, and still no response from the author, I’ll be inclined to retain an add-on in the sandbox, as per the AMO policy. If there are no discussion posts, I’ll look to see if there is another place where the author distributes the add-on and see if there are posts there. Same guidelines apply. If the author seems responsive, or there are no posts to be found, I move on to the next step.
Is the add-on clearly and accurately described?
I’m not going to deny you for a typo, but if your description is false or too vague, you’ll have to fix that and then re-nominate. If you add-on has had a few versions released, I’ll look at version notes and see how well you update them. I won’t deny it again for poor version notes, but I will make a note of it in the comment, and expect you to fix it. Editors will look at it next time you submit a version and ensure that you are doing as you are told by other editors.
Are all privacy and security concerns clearly spelled out?
Is the add-on free of unlicensed trademarks or copyrights?
The policy page is pretty clear about this. AMO doesn’t want to host anything that violates anybody’s trademarks or copyrights, so please don’t nominate it.
Do the add-on and add-on author both treat the user respectfully?
If you add-on starts installing unnecessary software, forces its toolbar to be always visible, or other things that are not respectful for the user, it will be kept in the sandbox. Your extension should be a good citizen in the application you target and provide the user with options to turn off features if they choose to do so (this is highly subjective and not always necessary).
Has the add-on been well-tested, and is it free of obvious or serious defects?
This is probably the most time consuming part of the editing process for me. After I install the add-on and restart my browser (or Thunderbird if I’m testing an add-on for it), the first thing I check is for any errors or warnings in the Error Console. I have the Console² extension which allows you to enabled strict warnings and filter errors and warnings so you only see the ones from chrome code. I strongly recommend every add-on author to check their work with this before nominating it for public. Warnings are generally OK, but we expect the number of warnings to go down with each release (not up, nor staying constant). Errors all depend on what the error is, and if it breaks functionality. Those, however, I expect to be fixed in the next release of your extension.
Now I’ll start to play with the functionality of the add-on, keeping the error console visible so I can see if normal use triggers any warnings or errors. If things are working as advertised, this add-on is one step closer to being approved. If not, I evaluate each issue on a case by case basis as to whether or not this issue is something that should keep the add-on in the sandbox.
Finally, I’ll start to look at the source code of the add-on. What I am looking for here is basically glaring mistakes in the code, potential conflicts with other extensions, and privacy and security issues. Any one of these issues is enough to keep the add-on in the sandbox, but I always make sure to state how to go about fixing the code so the author can do so quickly and renominate.
If the add-on makes it all the way through that, it will be approved, and I get to move on to the next one.
11 replies on “Life of an Editor on AMO”
My Aquatint and Aquabird Redone themes keep getting retained in Sandbox but they meet all requirements.
The Rules state,
“Has the add-on been well-tested, and is it free of obvious or serious defects?
One important thing that we look for when considering an add-on for public access is whether its sandbox reviews indicate that it has received thorough testing, and that it doesn’t have serious problems or negative impacts on the browser. If reviewers report problems such as major performance issues, crashes, frequent problems using the functions of the add-on, or spamming of messages to the error console, you should take those reports to heart, and re-nominate your add-on after you’ve addressed them as best you can. We don’t expect you to perfectly optimize or have zero bugs — Firefox itself undergoes constant improvement in these areas — but we do want you to take reasonable efforts to minimize downsides, and to clearly call out cases where users may be surprised by those that remain.
If your add-on has been tested outside of the AMO Sandbox process, such as by a group of users of your service or an in-house QA team, you should indicate that in your nomination message. It certainly helps us establish what the level of testing has been, and can help get your add-on up on the site.”
My theme has been well tested and reviewed since it is based on well tested and reviewed themes. The only difference is that it has different icons so my theme does meet all criteria as stated in the rules.
Just being based on an existing theme tells us editors nothing useful. Based on could mean that it looks like the other themes, or uses the same back-end. Without any indication, we editors really can’t use that information as a factor.
In your e-mails, you indicated that it was the same back-end code. That information may have changed the situation, but at the time your comments to editors/reviewers did not indicate this.
Who exactly are the users? once it’s on the sandbox is it accessible to people for download? My add-on is the Scribd Slurper, for Scribd.com.
It’s been on the sandbox for a coule of days but no one seems to have looked at it yet. Please let me know if I need to do something extra to have people look at it.
Where are these reviews supposed to come from? Should I tell my friends to review the thing? I don’t think they would be very trustworthy, in that case. But I can’t simply spam my extension, either.
I posted the sandbox link to MozillaZine and waited a week, but apparently nobody is interested in logging into the sandbox. I was hoping addons.m.o would be a central repository for extensions (not to mention the added security) but this seems like a major barrier.
Anybody can sign up to be a sandbox user. It generally takes some time to get reviews. This isn’t supposed to be a lightning fast process. :)
I really don’t think it’s that nobody is logging into the sandbox, I’m sorry to say.
Thanks for replying to Tim and Avery — it helps a bit, but I’m still confused.
I’m in the same boat as Tim and Avery (a sandboxed extension with no reviews). Do I understand that the sandbox is the only source of reviewers? If so, what control does the author have over obtaining reviews? If not, what other review sources to you accept?
Please: a bit more detail on getting through this part of the process.
[…] wagon! Been trying to drum up a few reviews from the Mozillazine crowd to get us out of the AMO Sandbox, that’s also a priority. On the whole they seem to really like it but not many of them have […]
[…] pundits on our news page failed to satisfy the user review requirement, because they’re really more interested in people saying things like “I’ve been using it for a couple days on Firefox 220.127.116.11 […]
I’m writing up a post about it now that will be out tonight/tomorrow that should hopefully explain it in more detail.
[…] last post keeps getting comments, and I’m seeing this brought up on the amo-editors mailing list a lot. […]
Just curious, do you know how many amo addon editors there are?
One thing that would be really cool on the amo site is an estimation of when your addon would be reviewed. So knowing no. of editors and the average review rate would be useful :)
I was going to apply to be an editor, until one of my extensions got a public review and was kept in the sandbox. The reasons given were very clear (and very correct) and linked to a deeper explanation and solution. Since I’d thought my addon was ready and it wasn’t clearly I’m not ready to be an editor! You guys do a great job, thanks :)