Test Pilot, Experiments, and the SDK

Archives Test Pilot/Development
#1

The add-on SDK is on its way out the door, but is still used extensively by
Test Pilot. Following is a rough strategy to get Test Pilot off of the SDK.

First, let’s get rid of the easy stuff: We don’t need to talk about Snooze
Tabs or Page Shot as they both are full web extensions now. Activity
Stream uses the SDK, but that team has their own plan to get off of it so
we can skip that here as well. Similarly, the Cliqz team has a proof of
concept web extension already which, I’m guessing, will go live in Q3.

Tab Center is going through a Shield study right now as a precursor to
fully graduating. It should be through Test Pilot in late Q2 or early Q3.
Additionally, the containers experiment is entering into a Shield study in
June, so it should be finished with Test Pilot in Q3.

Pulse, as it exists today, is built to provide performance data for the
Quantum project. We can retire the current Pulse experiment alongside the
launch of the Quantum project with Firefox 57 in November.

The upcoming experiments are currently all Web Extensions

Which brings us to the trouble makers: the Test Pilot add-on and the Min
Vid experiment both use the SDK and both are expected to live until next
year, at least. The good news is, it looks like all the APIs except one
already exist as web extensions. We’ll work on dropping the SDK from Test
Pilot and Min Vid over the remainder of Q2 and beginning of Q3. See
trackers and bugs below:

Replacing Test Pilot APIs is being tracked at

Replacing Min Vid APIs is being tracked at

Supporting sdk/window/utils from a Web Extension is being tracked at
https://bugzilla.mozilla.org/show_bug.cgi?id=1363132

Thanks!

Wil

#2

We’ve had a follow up meeting to discuss more specifics. From that meeting:

Decisions:

  • As of Fx 57 experiments may use only GA. Experiments wanting to process PII or do more complex analysis should use Ping Centre.
  • Experiments may use Mozilla’s GA account credentials to record data.

Test Pilot Plan:

  • Become a Bootstrapped Mozilla Extension.  Continue to support our current experiments which depend on us (heartbeat, telemetry stuff)

    • Replace existing SDK modules with xpcom stuff

  • At some point in the future (6 months?) reconsider evolving past a bootstrapped add-on

Extensions Plan:

  • New extensions are strongly encouraged to be Web Extensions. If they need to get to hairy things, they may be Bootstrapped Mozilla Extensions.

We are aware that the SDK is being turned off in Nightly in about 7 weeks. We expect to be prepared for that.