2017Q1 Test Pilot OKR mid-quarter review

Hi everyone,

Q1 is halfway over. Let’s check in on our OKRs!

O1KR1: Two former Test Pilot experiments in active SHIELD studies
Activity Stream was already in Shield and has finished. There isn’t publicly available data from that study (yet?). The second study will likely be around Page Shot but it’s not scheduled yet. You could quibble over the word ‘former’ here, but let’s just call this 50% done and on track.

O2KR1: Test 3 contributor value hypotheses and publish the results
We’ve had a couple of meetings (big thanks to George for facilitating). This is On Track.

O2KR2: Publish conclusions from three experiments as they graduate from Test Pilot
We graduated Tracking Protection (https://testpilot.firefox.com/experiments/tracking-protection) and No More 404s (https://testpilot.firefox.com/experiments/no-more-404s). We might get a third one, or we might not, but we’re not prioritizing it over, say, Page Shot in 54. Call it on track for now, but we know it’s a bit soft.

O3KR3: Create a draft curriculum for an upcoming class with the Open Innovation team
There have been several meetings and emails about this and it is progressing. On track.

O3KR1: At the end of Q1 double the percentage of new sessions enabling an experiment on intial Test Pilot install
Running the numbers on this in Q1 it actually dropped to 5.5%. The percentage of users installing Test Pilot almost doubled (12.4% -> 21.4%) but the number of users who installed an experiment after that is way down (47% -> 31.4%). We should verify that our new one-click install buttons are reporting installing Test Pilot and installing an add-on correctly.

O3KR2: Improve Smoothed Test Pilot Engagement Ratio by 6%
We started at 53% and our target is 59%. Quarter-to-date we’re at 55.3% so this is moving in the right direction. On track.

O3KR3: Plan a mobile GTM for 2017
We have had one meeting, but time is flying. I’d call this at risk.

O3KR4: Create 4 feature cards ready for UX inspired by user testing reports
Sharon is conducting interviews next week. Calling this on track.

3 Likes

Thanks for being open about your goals! Love it, keep on the good work! :tada:

Q1 is already over. We had a lot of successes, but they weren’t all represented by our goals. We spent a lot of time getting Firefox Screenshots ready to land in 54 (the good news: it seems like it’s still on track to do so).

Thanks to everyone who participated in our review. The final numbers and some analysis is at https://wiki.mozilla.org/Test_Pilot/2017Q1

Cheers,

Wil