# Blind reference-only evaluation protocol v1

## Roles

- VLM: semantic cause, region description, detector query, uncertainty, optional coarse fallback box.
- Grounding DINO: convert the query into scored target-image boxes.
- SAM 2.1: convert boxes into raw pixel masks.
- Evaluator: after inference is frozen, use paired reference images to build pseudo-GT and calculate metrics.

## Leakage controls

1. Copy inputs into `blind_inputs`; do not symlink.
2. Replace filenames with opaque IDs. The blind manifest may contain only ID, image path, dimensions, hash, and context IDs.
3. Choose four context images using camera pose only. Do not use reference images or labels for context selection.
4. The inference prompt must not contain dataset, scene, condition, object, reference, or oracle names.
5. Freeze and hash raw VLM JSONL before the evaluator is allowed to read the private ID map or references.
6. Never write GT fields back into the blind manifest or inference files.

## Fixed v1 settings

- Context count: 4.
- Grounding DINO box threshold: 0.25.
- Grounding DINO text threshold: 0.20.
- NMS IoU: 0.50.
- Maximum boxes per query: 10.
- SAM 2.1 multimask choice: highest predicted IoU for each box; union boxes within each region.
- VLM normalized box: use only when grounding returns no box; record source as `vlm_fallback`.
- Routing mask: raw SAM mask dilated by 5 px; evaluate raw and routing masks separately.

## Pseudo-GT

Paired-reference masks are evaluation-only pseudo-GT, not human semantic GT. Generate them after inference is locked with weak RGB max-difference 0.04, strong difference 0.10, minimum connected area `max(512, 0.00035 * H * W)`, closing kernel 15, and dilation kernel 9. Record a ±5 px boundary uncertainty band.

## Metrics

- Frame detection recall on positive train observations.
- False-positive frame rate and predicted area on clean test observations.
- Box GT-pixel coverage and box area precision.
- Mask IoU, Dice, pixel precision, recall/SafetyRecall, Leakage, Overmask, and Boundary-F at 5 px.
- Report mean, median, 10th percentile, per-frame failures, and bootstrap 95% confidence intervals.

This first run is descriptive pilot evaluation. Do not tune prompt or thresholds on the locked evaluation results and then present them as held-out performance.