Why we stopped asking open questions of video models
Open, single-shot VQA changes its answer every time you ask. Without a fixed schema the model narrates, invents products, and cannot be rolled up. A harness that demands mechanical evidence is what made outputs stable.
Open visual question answering on a raw LLM is not a measurement system. Ask the same clip twice and you can get a different SKU, sentiment, or claim. The model narrates, invents a product on the desk, and spends tokens on lighting. If the record moves when you refresh, you have a chat window, not intelligence. We stopped asking open questions. Extraction is normalised into fixed JSON fields — including 65 behavioural signals — so the model must say whether a physical fact is present, with evidence, instead of how the video felt.
Hallucination is a harness problem
Leave the reasoning open and two things happen: token usage goes through the roof, and your team starts prompt-hacking to suppress vibes that keep coming back. Each patch fixes one hallucination class and opens another. The output also becomes incomparable across videos, which is fatal if you want to roll up a category.
What the VQA harness forces
- Closed fields, not a paragraph: hook, friction, application, wear, shade check, competitor in frame.
- Each signal is yes/no or a constrained enum with evidence, not a vibe.
- The same schema on every video so category rollups are comparable.
- Refuse to ship an API that returns a different SKU on refresh.
Why 65 behavioural signals
The number is the schema, not a slogan. We needed a set large enough to describe creator behaviour that actually changes a brief, and small enough that a model cannot hide in prose. Poetic guesses are expensive and unusable. Forced evidence is the only shape of VQA we will put behind an API.
How we know this
These notes come from building Noodle4’s review and listening pipelines, not from a published academic sample. Cost curves, intent splits, and ASR collisions are from internal R&D and client-shaped tests. They will not match every category. We include them because the failure mode is structural — not because one campaign is universal.
Alex Gray is the founder of Noodle4 and a former technical AI product owner. He works with agencies to apply multimodal AI to creator screening, draft approval, and post-publication monitoring.
Frequently asked questions
Why does AI give different answers about the same video?
Open, single-shot VQA is underspecified, so the model narrates instead of measuring. Without a fixed schema, answers drift between runs and cannot be rolled up.
What is a VQA harness?
A VQA harness forces video models to fill fixed JSON fields with mechanical evidence — such as Noodle4’s 65 behavioural signals — instead of free-form description.