Video Verification Pipeline

The previous section covered how images are stripped, hashed, encrypted, and stored. Video follows the same high-level pipeline — C2PA extraction, ZK proof generation, metadata stripping, encryption — but adds three video-specific challenges: extracting C2PA manifests from video containers, fingerprinting across keyframes, and generating temporal fingerprints for cross-codec deduplication.

C2PA in video containers

Video files use different container formats than images, but the C2PA manifest inside is the same JUMBF structure. The difference is where the JUMBF bytes live.

The container parser uses mp4box.js to parse ISO BMFF containers. It searches for a top-level uuid box matching one of two known C2PA UUIDs — the c2pa-rs UUID (most common) or the C2PA specification reference UUID. If direct property access fails, the parser falls back to recursive tree search, then to a content heuristic scanning for "manifest" strings in UUID box payloads.

Once the UUID box is found, the raw JUMBF bytes are extracted and fed into the same parser used for images. From this point forward, the C2PA extraction pipeline is identical — COSE signature parsing, certificate chain extraction, claim hash computation, and circuit input building all work the same way regardless of whether the source is a JPEG or an MP4.

Keyframe extraction

Video fingerprinting requires extracting individual frames from the video. Apertrue uses the browser's WebCodecs API for hardware-accelerated video decoding — frames are decoded on the GPU without downloading the entire video into JavaScript memory.

Decoding pipeline

1. Container demuxing. mp4box.js extracts all samples from the first video track, including per-sample metadata: composition timestamp, duration, and sync flag (whether the sample is a keyframe).
2. Codec configuration. The codec description bytes are serialised from the sample's description box — avcC for H.264, hvcC for H.265/HEVC, av1C for AV1. These configure the hardware decoder.
3. Keyframe filtering. Only sync samples (keyframes) are decoded. A minimum interval of 0.5 seconds between keyframes prevents over-sampling rapid-fire IDR frames. Default extraction limit: 32 keyframes.
4. Hardware decoding. Each keyframe is fed to a VideoDecoder as an EncodedVideoChunk. The decoder outputs VideoFrame objects backed by GPU memory.
5. Frame rendering. Each VideoFrame is drawn to an OffscreenCanvas, converting from YUV to RGB. The pixel data is extracted via getImageData() and the alpha channel is dropped — output is a 3-channel RGB byte array with width, height, and timestamp.
6. GPU memory release. Each VideoFrame is .close()d immediately after rendering to release GPU memory. Failing to close frames causes GPU memory leaks that crash the tab.

Per-keyframe PDQ hashing

Each extracted keyframe is perceptually hashed using pdq-wasm — the same WASM module used for image PDQ hashing. The hash function takes the RGB pixel data, width, height, and channel count, and produces a 256-bit hash plus a quality score (0-100).

A typical 1-hour video at 24 fps contains roughly 90 keyframes. The default extraction limit of 32 selects keyframes evenly distributed across the video's duration. Each keyframe hash is stored as a 64-character hex string alongside its frame number, quality score, and timestamp.

vPDQ — video perceptual matching

vPDQ (Video Perceptual Distance Quality) is a video matching algorithm published by Meta as part of ThreatExchange. Apertrue uses a clean-room TypeScript implementation of the published algorithm. vPDQ treats a video as a bag of frame hashes and compares two videos by matching frames pairwise.

Matching algorithm

1. Quality filtering. Frames with a PDQ quality score below 50 are discarded. Low-quality frames (dark scenes, heavy motion blur) produce unreliable hashes.
2. Nearest-neighbor search. For each query frame, the algorithm computes the Hamming distance against every candidate frame and finds the closest match. Hamming distance counts the number of differing bits between two 256-bit hashes (range: 0-256). If the distance is 0 (perfect match), the search short-circuits.
3. Match threshold. A frame pair counts as a match if the Hamming distance is at most 31 bits (default). This allows for minor visual differences from re-encoding, color correction, or cropping.
4. Overall score. The match percentage is the fraction of query frames that found a match in the candidate set. Two videos are considered a match if at least 80% of query frames match (default).

vPDQ is order-agnostic — frame order does not affect the match. A video that has been trimmed, re-ordered, or had segments removed will still match on the frames that remain unchanged. This makes vPDQ robust against common video editing operations.

TMK — temporal media fingerprint

vPDQ compares videos frame-by-frame, but it misses temporal patterns — a video played at 2x speed has different frame timestamps but the same visual content. TMK (Temporal Match Kernel) solves this by generating a fixed-size fingerprint that captures the temporal structure of the video using Fourier analysis.

TMK is also published by Meta as TMK+PDQF. Apertrue implements accumulation in TypeScript (browser-side) and scoring in both TypeScript and Rust (backend).

Fingerprint generation

1. Temporal resampling. The video's native framerate is resampled to a fixed 15 FPS. Higher framerates (30, 60) skip frames; lower framerates (10, 12) duplicate frames. This normalises the temporal axis across different source formats.
2. PDQ to float vectors. Each frame's 256-bit PDQ hash is converted to a 256-dimensional float vector: set bits become +1.0, clear bits become -1.0. This preserves the sign pattern of the DCT output while enabling floating-point arithmetic.
3. Pure average. An unnormalised running sum of all frame feature vectors, divided by the frame count at finalisation. This captures the "average appearance" of the video.
4. Fourier accumulation. For each frame at time t, the L2-normalised feature vector is weighted by cosine and sine terms at four different periods and 32 coefficients per period. The periods are 2731, 4391, 9767, and 14653 frames at 15 FPS — roughly 3, 5, 11, and 16 minutes. This captures how the visual content changes over time at multiple temporal scales.
5. Finalisation. Each Fourier feature is L2-normalised and scaled by the square root of its Poullot coefficient (Bessel-derived weighting that emphasizes low frequencies). The result is serialised as raw float32 values.

Fingerprint structure

Matching

TMK matching runs in two levels:

1. Level 1 — fast pre-filter. Cosine similarity of the pure average vectors. This is a single dot product — O(256) — that quickly rejects obviously different videos. Threshold: 0.7 (default).
2. Level 2 — temporal alignment. For each of the four Fourier periods, the algorithm tries all possible temporal offsets and finds the alignment that maximizes the score. This handles videos that are the same content but start at different points. The maximum across all offsets and periods is normalised to produce a 0-1 score. Threshold: 0.7 (default).

A match requires both levels to exceed their thresholds. Level 1 catches obvious duplicates cheaply. Level 2 catches temporally shifted or speed-changed duplicates that Level 1 might miss.

How video differs from images

The key insight is that video verification is image verification plus fingerprinting. The C2PA cryptographic chain (signature, certificate, trust list, hash chain) is verified identically. The additional complexity is in generating fingerprints that enable near-duplicate detection across re-encodes, crops, and temporal edits.

Video in the upload flow

1. C2PA extraction. The video container is parsed, the C2PA UUID box is located, and the JUMBF manifest is extracted. From here, the standard C2PA pipeline runs — signature parsing, certificate extraction, trust tier determination.
2. ZK proof generation. If C2PA is present, the split-proof pipeline runs identically to images — ProofA verifies the certificate chain, ProofB verifies the COSE signature, and the aggregation tree produces a root commitment.
3. Keyframe extraction and hashing. In parallel with proof generation, the WebCodecs pipeline decodes up to 32 keyframes and computes PDQ hashes for each. These become the video's vPDQ feature set.
4. TMK fingerprint generation. Also in parallel, the TMK accumulator processes the full video at 15 FPS. Unlike keyframe extraction (which only decodes sync samples), TMK processes every frame to capture the complete temporal structure. The 257 KB fingerprint is included in the upload metadata.
5. Thumbnail capture. A single frame is captured at the 1-second mark (or at 0 for clips shorter than 1 second) by seeking a <video> element and rendering to canvas. The JPEG is stored as a data URL in the media index.
6. Encryption and upload. The full video file is encrypted with AES-256-GCM (same as images) and uploaded as an encrypted blob. Video decryption uses a dedicated web worker to avoid blocking the main thread during playback.

Data sizes

The next section covers the verification and share flow — how viewers verify an image's provenance and how verified content is shared with its trust signals intact.