What would you do with unflagged assets?
Compare AI flags with the asset team’s labels before you change the review queue.
The question
You review an image model for a utility in Great Britain. The asset team independently labelled all 40 distinct assets in the first sample. The programme asks whether it can stop manual review of unflagged assets.
| Assets | Confirmed high priority | |
|---|---|---|
| Flagged | 16 | 12 |
| Unflagged | 24 | 0 |
Precision is the share of flags that are high priority. Recall is the share of high-priority assets the model flags.
“What do the two measures show? What evidence would you ask for before changing the queue?”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Measure the flagged queue
What would you do here? Open the move you’d make.
Use 12 of 16 to measure the flagged queue.
The strong move. Precision divides the confirmed flags by the total flags.
Use 12 of 40 to report 30% precision.
Tempting, but it misses something. All 40 assets form the sample. The precision denominator is the 16 flags.
See how a strong candidate says it
“The asset team confirmed 12 of 16 flags, so precision is 75% in this sample. Four flags were lower priority.”
Why it works: Precision divides the confirmed flags by the total flags.
Step 2: Check the unflagged records
What would you do here? Open the move you’d make.
Report 100% recall on these records and check how the sample was chosen.
The strong move. The first sample has no high-priority misses. Its coverage still matters before a process change.
Remove manual review of unflagged assets based on the first sample.
Tempting, but it misses something. The sample result gives no evidence that all image conditions were covered.
See how a strong candidate says it
“In this sample the model caught 12 of 12 confirmed high-priority assets. Recall is 100%. I’d ask whether the unflagged group represents the conditions we see in the wider queue.”
Why it works: The first sample has no high-priority misses. Its coverage still matters before a process change.
Step 3: Add the second group
The asset team separately reviews 6 new, unflagged assets with older low-light images. It confirms 4 as high priority. The records are distinct from the first 40.
What would you do here? Open the move you’d make.
Include the four confirmed misses and report both rates on the reviewed sample.
The strong move. The four misses change the recall denominator. The flagged queue has the same entries.
Keep 100% recall because none of the new records were flagged.
Tempting, but it misses something. The asset team confirmed four high-priority cases in the unflagged group. Recall must count them.
See how a strong candidate says it
“Now there are 46 reviewed assets and 16 confirmed high-priority cases. The model caught 12 of 16, so recall is 75%. Precision stays 75% because 12 of its 16 flags are still confirmed.”
Why it works: The four misses change the recall denominator. The flagged queue has the same entries.
Step 4: Give the misses to the owner
What would you do here? Open the move you’d make.
Keep the manual review and ask the asset owner to validate across image conditions.
The strong move. The asset owner can act on the confirmed cases. The small follow-up group gives a specific condition to investigate.
Apply the combined 75% recall to every asset in the network.
Tempting, but it misses something. The follow-up deliberately targeted a weakly represented condition. Its sample rates do not describe the whole network.
See how a strong candidate says it
“I’d keep the existing review for unflagged assets and pass the four misses to the asset team. I’d report how each sample was chosen and ask for validation across image conditions before we change the queue.”
Why it works: The asset owner can act on the confirmed cases. The small follow-up group gives a specific condition to investigate.
The answer, in short
- Decision
- Keep the unflagged review and give the confirmed misses to the asset owner.
- Because
- Four high-priority cases appeared in the second unflagged group. Sample recall fell to 75%.
- What would change it
- Validation across image conditions and the asset owner’s approval could support a change to the queue.
Common traps
- Measuring only flagged assets.
- Treating results from a targeted sample as a fleet-wide rate.
Check your answer
- I gave the two denominators.
- I checked unflagged assets.
- I included the new misses in recall.
- I gave the asset owner the confirmed cases and sample limits.
Last reviewed Nov 11, 2026