Benchmark Methodology

9,147 Frames Not Used for Training

The evaluation set contained 9,147 frames: 4,657 positive frames and 4,490 negative frames. None of these evaluation frames were used to train the model.

These are totals for the evaluation set, not a per-detection-head breakdown. The published detection rates were measured without per-site training and before the alarm engine was applied.

Frame-Level Detection Rate

A positive frame counts as detected when the model correctly detects more than 50% of the actual leak area. The threshold is strictly greater than 50%, not greater than or equal to 50%.

Detection rate = qualifying detected positive frames ÷ evaluated positive frames. This measures whether enough of the leak was found in a frame; it is not the same as pixel-level recall.

Frame-Level False Positive Rate

A negative frame counts as a false positive when the model marks more than 10% of the entire frame as a detection. The threshold is strictly greater than 10%.

False positive rate = qualifying false-positive negative frames ÷ evaluated negative frames. Smaller erroneous detections do not count under this frame-level definition, but their pixels still contribute to pixel-level false positives where a human mask is available.

This percentage is not the share of alarms that are false, nor a false-alarm count per hour or per camera.

Pixel-Level Precision and Recall

Model predictions are compared with human-drawn masks. True positives (TP) are pixels marked as liquid by both model and human. False positives (FP) are pixels marked by the model but not the human. False negatives (FN) are pixels marked by the human but missed by the model.

Precision = TP / (TP + FP). Of the pixels the model highlighted, how many were also marked by the human?

Recall = TP / (TP + FN). Of the pixels the human marked, how many did the model find?

F-Score: F1 / Dice

F1 = 2 × (precision × recall) / (precision + recall) = 2 × TP / (2 × TP + FP + FN). This is the Dice coefficient.

The harmonic mean balances precision and recall: highlighting too much introduces false positives, while missing leak pixels introduces false negatives. A strong score requires both accurate and complete segmentation.

TP, FP and FN are accumulated across every evaluation frame with a human mask for the relevant detection head, then one F1 score is calculated from those totals. This is not an average of per-frame F1 scores. Pixels contribute according to their area, so larger regions contribute more pixels than smaller ones.

Reading the Published Results

Puddle: 97.1% frame-level detection rate, 0.8% frame-level false positive rate and 0.889 pixel-level F1/Dice. Spray and visible vapor combined: 98.1% frame-level detection rate, 3.7% frame-level false positive rate and 0.923 pixel-level F1/Dice.

Spray and visible vapor share a combined benchmark; these are not separate spray-only and vapor-only results. The 50% and 10% frame-level thresholds are not the rules for calculating F1.

These model measurements do not measure the effect of alarm progression, rain inputs or other alarm rules. They do not substantiate a separate percentage reduction in operational false or nuisance alarms, or a separate nighttime/NIR performance score.

Separate Customer Result: Alarm Reduction

In a customer deployment with an operator in the Permian Basin, VisionAery verified a 90% reduction in nuisance and false alarms compared with the previous version of the application over one month.

This is a deployment-specific operational result, separate from the 9,147-frame model evaluation. It is not a 90% improvement in detection accuracy, an independently audited result, or a guarantee of the same reduction at other sites.