gMAD Select Video Clips

What counts as one attack task?

How does gMAD work? First, each pooling rule scores the candidate videos. The rules then take turns as “defender” and “attacker.” The attacker seeks two videos that the defender rates almost equally but the attacker rates very differently. People must compare the selected videos to determine which rule better matches perceived quality; that human evaluation has not yet been done.

Method reference: K. Ma et al., “Group maximum differentiation competition: Model comparison with few samples,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no. 4, pp. 851–864, 2020, doi: 10.1109/TPAMI.2018.2889948.

A task is specified by one scene, one defender rule, a different attacker rule, and one quality bin.

For example: “In the middle-quality bin of bistro, let median defend and mean attack.” The program looks for two videos that median rates almost equally but mean rates as differently as possible. A task succeeds only if the pair meets the predefined defender and attacker gap criteria. A task with no valid pair still counts as an attempted task.

6 scenes × 15 defender rules × 14 other attacker rules × 3 quality bins = 3,780 tasks

Why is K=3 different from searching all videos once? The three bins together do contain every candidate video. Without bins, however, each defender–attacker combination would select just the single pair with the largest disagreement, possibly concentrating selections in one score range. With K=3, the attacker searches separately within the defender's low-, middle-, and high-score bins and can obtain up to three pairs. Both videos in a pair must belong to the same bin, and the defender and attacker gap thresholds still apply. For example, if the largest attacker gaps in the three bins are 1.0, 0.8, and 0.3, a search without bins would select only the 1.0 pair; binning attempts to select one pair from each bin. K=3 makes every bin eligible for an attempt, but does not guarantee equal numbers of pilot pairs from each bin.

3,414 tasks yielded a pair. Different tasks can select the same two videos; after deduplication, there are 1,225 distinct pairs.

insufficient_attack_gap: 222 means a pair was close enough under the defender, but the largest available score difference under the attacker fell below the required threshold. Those 222 tasks had no valid attack; this does not establish that the defender wins. Rules with identical predictions cannot attack each other effectively either.

Why do some rules give identical scores?

The comparison uses 15 pooling rules in the code. Under the current parameters and video duration, some differently named rules produce the same result for every candidate:

How long are the time windows?

The original study's 25 fps durations are converted to 120 Hz: 100 frames, or 4 seconds, becomes 480 frames; 125 frames, or 5 seconds, becomes 600 frames.

worst_window slides a 4-second window across the entire 4.5-second video. exp_weighted_minkowski uses a 4-second weighting decay scale. The 5-second last_frames_mean window covers the whole video, so it equals mean. hysteresis explicitly uses 120 Hz and a 2-second memory.

Four position-weighting rules

Split each 4.5-second video into three 1.5-second chunks, average the scores within each chunk, and combine the three means using these weights.

How many frame scores were negative?

The 108 chunk files contain 19,440 frame scores. Of these, 18 scores are below zero, spread across 6 files. The table lists every negative record in the CSV. Frame indices start at zero; “position in full video” is where that frame would fall after assembling the 4.5-second video.

These are per-frame proxy scores exported from CVVDP video-mode features. This run excludes the 18 negative records from the pooling inputs. It neither replaces them with zero nor adds a positive offset. The lowest retained raw score is 0.145. As a result, 216 candidate score sequences have fewer than 540 frames, although the videos themselves remain 4.5 seconds long. These are screening scores based on shortened sequences; the assembled videos still need full-video verification before use.

How close must the defender's scores be?

For each scene and defender rule, take its scores for the 216 candidate videos. The 95th percentile minus the 5th percentile is the width of the usual score range, excluding the extremes. The maximum permitted difference between the two videos is the smaller of 5% of that width and 0.10.

Defender score difference ≤ min(0.10, 5% × [defender p95 − p5])

Actual example: for mean on bistro, p5 is about 8.10 and p95 about 9.86. The usual range is about 9.86 − 8.10 = 1.76 points; 5% of it is about 0.088 points. Thus, when mean defends, a pair's scores may differ by at most about 0.088 points. This is a selection threshold, not evidence from human judgments that viewers cannot tell the videos apart.

How were the 84 pilot pairs chosen?

Each scene contributes 10 gMAD pairs + 2 random pairs + 2 diagnostic pairs, giving 84 pairs across 6 scenes. A pair linked to several attack tasks is shown only once.

1. 60 gMAD pairs: Choose 10 per scene from the full list of distinct gMAD pairs. Prefer pairs that cover more different defender–attacker–quality-bin combinations; break ties by the larger attacker score gap. This is not a random sample of 60 pairs.
2. 12 random pairs: Two per scene. Sample one from pairs where both videos keep fixed settings throughout, and one from pairs where at least one video changes settings. These pairs need not satisfy the gMAD gap criteria.
3. 12 diagnostic pairs: Two per scene. One compares reversed orders of quality changes. The other compares fixed settings against changing settings, with similar mean scores. The reversal pair does not have to have similar mean scores.

What does “cover a task” mean? If an attack task selected a video pair and that pair enters the pilot, the task is covered. For example, suppose a chosen pair already covers tasks 1 and 2. A candidate linked only to tasks 1 and 2 adds little, while a candidate that also covers task 3 is preferred. The selection counts newly covered defender–attacker–quality-bin combinations, rather than simply how often a pair appears. Each pair is shown only once.

Each video lasts 4.5 seconds, so one A/B viewing takes 9 seconds. Allowing roughly 20 seconds per pair for viewing and response gives about 28 minutes of trials for 84 pairs. Add practice, replays, and breaks when planning the full session. Randomize A/B presentation order. No human ratings are available yet, so these results do not rank the pooling rules by perceptual accuracy.

Pilot pair list

A and B are only list labels here; their playback order must be randomized for participants.

What is the difference between the two pair lists?

All distinct gMAD pairs: Every pair selected by a successful attack task, with duplicate video combinations removed; this run has 1,225 pairs. Pilot pair list: The 84 pairs for a small-scale study: 60 from the full gMAD list, plus 12 random and 12 diagnostic pairs.

This is a first screening based on concatenated chunk-level score sequences. Before using the selected items, assemble the chunks into actual 4.5-second videos and verify CVVDP scores at the transitions using the full videos. Also check that chunk content is continuous in the rogue scene.