Why the crop is centred
The clever version of this feature is a demo. The boring version is what ships.
In short
Generated Worlds crops horizontal video to vertical with a fixed centred window rather than a window that follows the subject. Face-aware reframing held the subject a few points better in our tests, but it drifts visibly while the viewer holds the phone still, and it cuts through burnt-in captions, which sit in the middle of the frame.
Turning a horizontal picture into a vertical one means throwing away two thirds of the frame, and the obvious question is which two thirds. The obvious answer is: whichever ones the subject is not in. Detect the faces, follow them, keep the speaker in shot.
We built it and measured it. Following the subject does hold the subject in frame slightly more often than a centred window. It also looks wrong, for a reason that has nothing to do with the detector's accuracy: the picture moves while the viewer is holding the phone perfectly still. A pan the audience did not ask for reads as a wobble, not as camerawork.
01The thing that settled it was the captions
A lot of short drama arrives with captions burnt into the picture, centred. A window that leaves the centre cuts them in half in the overwhelming majority of frames, against a small minority for a centred window. There is no amount of subject-holding that pays for unreadable dialogue.
02Where the detector still runs
- Per shot rather than per frame: the window is chosen once for a stretch of picture and then holds still, which keeps the benefit and loses the drift.
- For choosing which of several speakers a clip is about, where a static crop has nothing to say.
- Never as a live effect during playback. The crop is decided when the file is made, not while somebody is watching it.
The general lesson is worth more than the specific setting: a model that is measurably better on its own metric can be worse for the product, because the product's metric includes things the model was never scored on — like whether the picture sits still, or whether the words are still readable.
What it does not do
A centred crop loses a subject standing at the edge of the frame, and on wide two-shots it will cut somebody out entirely. For material without burnt-in captions — a talking head, a piece shot for the purpose — subject tracking is the better answer, and we use a per-shot version of it there rather than pretending otherwise.
Questions
- How do you convert horizontal video to vertical without ruining it?
- Use a fixed centred window unless you have a reason not to, and choose any moving window once per shot rather than per frame. Continuous tracking drifts under a still viewer, and if the source has burnt-in captions a window that leaves the centre destroys them.
- Does Generated Worlds crop landscape uploads automatically?
- The player fits landscape material to a phone at display time, and creators can set an explicit crop per episode or across a whole series from Creator Studio.
Keep reading
- Subtitles: we run the model ourselves
Transcription runs on faster-whisper rather than a hosted API, with voice-activity detection on and the model kept in memory between episodes. What that choice fixes, and when it is the wrong one.
- Never ask a language model what time it is
Our episode analysis asks a language model which line of dialogue a clip should end on, and looks the time up itself. Asking the model for the timestamp produced numbers that fit nothing.
All posts from the Generated Worlds Tech Blog.