Random result about automatic signing, same code, sometimes pass sometimes fail

Add-ons addons.mozilla.org
#1

Hi There,
We have an add-on that is built with automatic signing with our daily build system, and recently the automatic signing result is quite unstable. There were no code changes and just new version triggered by the build routine. However, the add-on built with the same code may pass the signing check or not on random basis. Would you please help review and suggest?

Here I am sharing two validation results:

passed one
https://addons.mozilla.org/en-US/developers/addon/hpe-functional-testing-agent/file/594542/validation

failed one
https://addons.mozilla.org/en-US/developers/upload/c0a710228319403ea7280bfad9ac43e6

#2

The difference I see in the second one are some warnings about Possible attempt to override new tab page, which shouldn’t cause the validation to fail. Are both validations for the exact same file? Small code changes could cause some things to be flagged that weren’t flagged before.

If this is causing you problem when submitting, please file a bug report here: https://github.com/mozilla/amo-validator/

#3

For the two I provided they are with same code, just version changed. And I want to add more information:

  • We create WebExtensions with the same extension ID, with some code change
  • We have another branch which will build the same extension, with different code change

Should that be a concern? Following please find the validation results accordingly.

WebExtensions version
https://addons.mozilla.org/en-US/developers/addon/hpe-functional-testing-agent/file/596070/validation

Addon SDK version (always passing, code is a little different from the original two validations, but code is same for the mentioned “Possible attempt to override new tab page,”)
https://addons.mozilla.org/en-US/developers/addon/hpe-functional-testing-agent/file/596232/validation

#4

WebExtensions do use a different validator than other extensions, so different results should not be surprising.

As long as this isn’t preventing you from submitting your add-ons, I don’t think you should worry about it.

#5

I just tried to build the same code into an extension (Addon SDK) with different ID, it passed with no problem.

With my existing extensions, the WebExtensions version can always pass signing, but not the Addon SDK one. Would you please suggest what to do next?

#6

Follow up discussion at https://github.com/mozilla/amo-validator/issues/501