Turning editorial judgment into a system users could trust

How I helped Switchful turn messy data, competing user priorities, and human expertise into a scalable ratings methodology.

Switchful helped consumers compare TV, internet, and home security providers. The business earned affiliate revenue when readers chose a provider through the site, but editorial integrity was critical to both the user experience and the business.

For TV services, we wanted to give readers something familiar and easy to understand: a five-star rating that let them quickly compare providers. But creating a rating that was actually useful was surprisingly complicated.

The “best” TV service depends on what you care about. A sports fan may prioritize completely different channels than a family with young children. Price matters, but promotional rates expire. Equipment, contracts, DVR features, installation, and customer service all affect the experience, too.

And while some of those things can be measured objectively, others can't.

Our challenge was to make the decision simpler for consumers without making the methodology overly simplistic.

Our four-person editorial team started with consumer interviews our founders had conducted, search data that showed what prospective customers were asking, and our own accumulated knowledge from researching and reviewing dozens of providers.

We asked a basic question:

What does someone actually need to know to choose the right TV service for them?

From there, we identified and weighted four major criteria: content, value, equipment and features, and customer experience. Then we broke those categories down further based on how people actually make decisions.

For example, we didn't assume that more channels meant better content. We evaluated channel lineups across different use cases, including sports, news, and family programming

We wanted to quantify as much as we could. I led the efforts to research possible data sources and evaluate whether they met three requirements: Were they credible? Were they available across providers? And did they measure providers consistently enough for a fair comparison?

Sometimes the answer was yes. Often it wasn't.

That created a recurring tension for our team. We knew inconsistent human judgment could create biased ratings. But relying only on available quantitative data could create something just as dangerous: a rating that looked objective without accurately reflecting the customer experience.

At times, the conversation veered toward, “Maybe it's impossible to do this right.”

I saw a false dichotomy.

We didn't need a perfect methodology. We needed one that made people's decisions easier and more informed than they'd be if they used less rigorous comparison sites—or tried to research every available provider themselves.

And we didn't have to choose between structure and judgment. We needed a system that told us when to rely on structure and when to apply judgment.

The framework we developed used three layers: primary considerations, secondary considerations, and editorial considerations. Wherever reliable data existed, we created consistent scoring rules. Where numbers couldn't tell the whole story, we defined the factors writers should consider when applying their expertise.

For example, price-per-channel could be calculated using a consistent formula. But evaluating how easy it was to find something to watch involved factors like navigation, channel-guide usability, parental controls, DVR organization, and the distinction between free and premium content. Those required structured editorial judgment.

We built that guidance into the system. Then we tested it.

Each team member independently rated providers and compared the results. When our scores differed, we investigated why. That led us to another safeguard: writers had to explain the reasoning behind judgment calls, and editors independently reviewed the rating. If they disagreed, they discussed the evidence.

We created an escalation process in case they couldn't agree, but we never had to use it.

The final TV methodology was 12 pages long, supported by a spreadsheet that automated many of the calculations. A new writer could use it to research a provider, understand the criteria, calculate scores, and know where and how to apply editorial judgment.

But the best part was: We'd created a process for building methodologies.

The TV framework took our whole team roughly a week of intensive collaboration, followed by testing and refinement. When we developed our internet methodology, we moved through the process in about two-thirds of the time.

Later, Switchful launched a home-security vertical. An editor who hadn't participated in our original methodology sessions was able to use the patterns we'd established to develop a new ratings system largely on his own. The broader team helped test it, but he no longer needed four people in a room to figure out how the system should work.

We'd taken knowledge that once lived in individual editors' heads and turned it into something transparent, repeatable, and scalable, without removing the human judgment that made it genuinely useful.

View the full TV ratings methodology →