Role
- Product Management
- Service Design
Problem
At Prenetics, we had a team of people reviewing the photos of Lateral Flow Test Cassettes uploaded through the app to verify the Positive, Negative or Inconclusive result.
The guaranteed review turnaround time was 12 hours - a duration that dissatisfied customers but ensured accuracy at scale.
Opportunity
We believed we could get the review time down to seconds using Image Recognition, so we engaged with a third party who built an API that could receive images and respond with a review code immediately. The Image Recognition service had been trained on 10,000 images of Lateral Flow Tests provided by NHS England. We gave them an additional 10,000 of our images to enhance the system and then embarked on a rollout process.
Release strategy
Despite the promising credentials of the Image Recognition service, I emphasised to the business that we need to integrate the system safely. Providing an inaccurate response to a customer would be costly, so I designed a phased rollout.
Don't be a Daily Mail headline.
UAT Phase
Goal
Test the image recognition at scale without risking incorrect result release.
Solution
Compare MagnifEye reviews with Prenetics manual reviews
Approach
Run the below for one week.
- Pass all negative and inconclusive result uploads through MagnifEye.
- Receive the analysis response from MagnifEye without automatically releasing the result.
- Assign a "yes" against an "automatic review "column in the LFA review data.
- Manually review the negative and inconclusive result uploads through EP.
- At the end of the week, query the number of matches between the MagnifEye and human analyses.
KPI
Result match % between human and automated reviews. If 99% match is achieved, we could progress to Rollout Phase.
Rollout phase
Goal
Test the auto-release process without service failure.
Solution
Gradually ramp-up use of MagnifEye for auto-releasing negative and inconclusive results
Approach
- Expose uploads to MagnifEye to 5% of uploads (or activations) and auto-release results for three days.
- Manually spot-check x50 per day
- Report any issues
- If there are no issues, increase the exposure
- If issues are found, decrease exposure to 0%
Ramp-up progression
5%, 10%, 30%, 50%, 100%
Outcome
- Reduced review time to 2 seconds from 12 hours.
- Safely transitioned to the automated service with no risk to service.