← Video to Sheet Music

How it works

The hard part is not reading the staff — it is already a clean, high-contrast render. The hard part is that a coloured playhead sits on top of the music, and any single frame you grab has the cursor smeared across it.

The pipeline solves this with timing rather than repair.

1. Find the staff

A narrow strip at the left edge of the frame is checked for brightness. Bright means we are on white paper rather than a title card; the sharpest vertical brightness change inside that strip is the bottom edge of the sheet music. Everything below it — falling notes, keyboard, lyrics — is discarded.

Three frames across the video's fade-in with the inspected strip and corner marked, above a plot of corner brightness over time and the brightness profile down the strip.
This video opens on a title card and fades the paper in, so you can watch the check flip: the corner reads 225 at frame 288 and 232 at frame 289, and the frame that crosses the threshold is the frame the detector wakes up on.

2. Track the playhead

The staff is greyscale, so the cursor is the only strongly saturated thing in the crop. Converting to HSV and taking the column with peak mean saturation locates it in one pass, with no template matching and no colour to tune per video.

A frame, its saturation channel whole and cropped, and the mean saturation of each column with the cursor standing out as a single peak.
The falling-note area saturates all the way to 255 — which is why the crop comes first — while above the boundary the median is 0 and the cursor stands alone.

3. Capture each half at the right moment

As the playhead sweeps left to right, the right half of the staff is saved once the cursor passes 25% of the frame width, and the left half once it passes 85%. Neither capture has the cursor in it. Stitching the two halves reconstructs one complete, cursor-free staff line. This is the whole trick.

The playhead's position through one sweep with the two thresholds marked, the two frames at the moments of capture with the captured half highlighted, and the stitched result.
The right half is grabbed at frame 446 with the cursor at x=487, the left at frame 1020 with it at 1677, and the stitch of the two is a staff line the cursor was never in.

4. Lay out pages

Staff lines are scaled to A4 width at 300 DPI and stacked until the next one would overflow the page, then a new page starts. The lines are spread evenly rather than crammed, so a page never ends with a gap that looks like a line failed to fit.

Three pages of the finished score under each layout mode with the margins marked, beside a chart of how many staff lines land on each page.
Fifteen staff lines over three pages, and the difference balancing makes: 5/5/5 rather than 6/6/3, for the same three sheets of paper.

Where it breaks

The technique rests on one assumption: the staff does not move during a sweep. When that holds, the seam between the two halves is exact — they come from the same rendering of the same music. When it does not, the halves come from different music and the stitch is visibly wrong. Videos that scroll continuously rather than a screenful at a time need an entirely different approach.

For the reasoning behind the design — why inpainting and frame compositing both fail here, why the two thresholds are asymmetric, and what the whole thing costs — see Why split capture.