It reads the notation
the world forgot.
Then it sings it back.
One engine reads Western staff, Jianpu (简谱), and Sargam — the numbered and solfège systems that general vision models can't — collapses them into a single symbolic score, and turns that score into sound. Pick a page below, watch it be understood, then press Sing.
then the playhead sweeps it while it sings.
The same OMRDSL target appears no matter which notation you pick — that invariant is the product: three ways of writing music, one thing the machine understands.
No longer a mockup — the model reads them itself
A ~26M-param image→sequence model, trained overnight on a Mac mini (M4) I already had, on thousands of synthetic pages. Everything below is its genuine prediction on held-out pages it never saw — image in, symbolic score out. Correct tokens are green, mistakes red.
In plain terms: we show the model a page of music it has never seen and ask it to write out the score; then we check how much it got right. Two numbers below. Token accuracy counts everything it wrote — including the easy bits every score has (clef, time signature, bar-lines), which it almost always nails. Content-only counts just the hard part — the actual notes. The gap between them is the honest story: it gets the layout of a page right, and the real music only partially — about one note in ten, so far. Hover any number for a plain-English explanation.
The encouraging part is the balance: it reads Jianpu and Sargam about as well as Western staff — the engine generalizes across notations, not just the familiar one. It reliably recovers structure and reads content partially; predicted lengths now match the targets closely. Still an early model (this is one overnight run at level-0 difficulty), but a real, honest one — these are held-out results, not cherry-picked beyond the per-notation numbers above. More steps and higher resolution are the obvious next lift.
What the trained model will understand
Bounded honestly by what the engine can already generate perfect labels for. This is the melodic core — deep and multi-notation, not yet wide.
| Notations | Western staff · Jianpu (简谱) · Sargam — one shared target |
| Image quality | Clean engraving → manuscript-like: warp, ink-bleed, wavy & broken staff lines, jitter (4 curriculum levels) |
| Clef / range | Treble · about two octaves (G3–B5) |
| Rhythm | 16th → dotted-half, seven durations, rests |
| Keys / meter | up to ±4 sharps or flats · 2/4, 3/4, 4/4, 6/8 |
| Texture | Monophonic, up to two voices |
| Not yet | Full polyphony, multiple clefs, dynamics, ornaments, slurs — and real historical manuscripts (the funded domain step). |
Two honest answers
♪ It sings the melody
Recognized notes → MusicXML → MIDI → a synthesizer. That's exactly the audio above — music box, flute, or a voice on a vowel. Read a Sargam page, hear the tune. A better instrument only means a better synth, not more training.
- howOMRDSL → MusicXML (
predict.py --musicxml) → synth - this pagebrowser Web-Audio formant voice, live
✎ Singing the words
To sing lyrics needs syllables in the notation and the DSL, plus a singing-voice synthesizer (pitch + duration + phonemes → vocal audio). Expression — dynamics, phrasing — comes with it.
- needslyric tokens + vocal-synth model
- thenit sings the piece, not just the tune
We write the answer first, then draw the question
There's no labeled dataset of real Jianpu or Sargam pages — so we don't hunt for one. We run the labeling backwards and manufacture unlimited perfect pairs for free. The whole trick is where the label sits.
The label sits at the end — locked behind slow, costly human transcription that, for Jianpu & Sargam, nobody has done at scale. Dead end.
The label sits at the start — we chose it, so the answer key is automatic. Unlimited pairs, no humans, perfect ground truth.
Synthetic gets the model most of the way for free; a small, affordable batch of real pages bridges the last gap to messy real-world scans. That hand-labeling is the only labeling anyone ever does — and it's tiny, because the model already learned the notation on the synthetic flood.
What's built, and what your investment buys
Render-forward data
Three notations from one symbolic record, perfect labels, unlimited pairs, curriculum degradation. Public on GitHub.
Read & play, end-to-end
Tokenizer, data loader, model + train.py, predict.py inference, MusicXML round-trip, this demo.
Trained overnight on a Mac mini
A ~26M model trained on level-0 synthetic pages, MPS, $0 — reads held-out pages (see results above). Early but real.
Scale + real manuscripts
More steps/curriculum for Jianpu & Sargam, then adapt to real handwritten and non-Western sources; add lyrics & voice.