Why we abandoned OpenAI for comment analysis
Most social listening tools do not read every comment. Frontier LLMs are too expensive at thread scale, so they sample. The allergic reactions and product hacks live in the outliers that sampling throws away.
Most social listening tools do not analyse every comment. A viral beauty video can carry 4,000 to 14,000 comments. Sending that volume through a frontier LLM will bankrupt a SaaS company if you do it honestly, so the rational move is to sample a hundred lines and guess the rest. Sampling estimates a mean. It deletes the tails: allergic reactions, the three-week dry out, the shade that oxidises. We stopped sending the firehose to a general model. Dedicated smaller classifiers now read 100 percent of comments into Intent, Complaint, and Praise lanes. The expensive model only sees the slice that still needs it.
The token-budget survival problem
The first comment pipeline we shipped sent threads to a frontier model. Quality on a hundred-comment demo was excellent. Unit economics on a real campaign were not. That is the gap the market papers over with “we use frontier LLMs to analyse your community’s sentiment.”
What sampling keeps versus what brands need
Why a 100-comment sample is the wrong estimator| Signal | Likely in a liked/random sample | Where it actually lives |
|---|
| “Love this” / emoji | Yes | The bulk of the thread |
| Allergic reaction | Rare | A handful of nested replies |
| Three-week dry out | Rare | Outlier complaints |
| Shade oxidises on one skin tone | Rare | Specific reply chains |
| Product hack | Rare | Low-like comments from buyers |
Lane-gating instead of a one-shot LLM
- Read every comment with a small dedicated classifier — not a sampled subset.
- Route into Intent, Complaint, and Praise lanes.
- Send only the ambiguous or SKU-specific slice to a larger model.
- Keep the full thread as the record, not a sentiment score on a sample.
We tried cheaper frontier tiers, shorter prompts, batching, and truncation. Every path either reintroduced sampling or produced a bill that could not sit under a monthly workspace price. A generalised LLM is a general reasoner being used as a classifier on a firehose.
How we know this
These notes come from building Noodle4’s review and listening pipelines, not from a published academic sample. Cost curves, intent splits, and ASR collisions are from internal R&D and client-shaped tests. They will not match every category. We include them because the failure mode is structural — not because one campaign is universal.
Alex Gray is the founder of Noodle4 and a former technical AI product owner. He works with agencies to apply multimodal AI to creator screening, draft approval, and post-publication monitoring.
Frequently asked questions
Do social listening tools analyse every comment?
Usually not. Sending every comment through a frontier LLM is too expensive, so many tools sample a subset and guess the rest. The outliers that contain real product insight are the first to go.
What is lane-gating in comment analysis?
Lane-gating uses smaller dedicated models to classify 100 percent of comments into lanes such as intent, complaint, and praise, instead of sampling a thread into a general LLM.