Switch to two-point rating scales to reduce racism in performance reviews, research suggests

The plumber has just left after fixing that leaky basement pipe. Ping—a phone alert asks you to rate their service. Hmm—if it wasn’t an outright terrible job, do you give them three, four or five stars? New research from the University of Toronto’s Rotman School of Management shows that a multi-point system like that is prone to subtle, often unconscious, racial bias—yet with significant financial consequences for non-white workers.

Using data from a real-life online home maintenance matching service, researchers showed white workers got higher average ratings than non-white workers. They also got top marks 86.9% of the time compared to only 83.4% of the time for workers who weren’t white. The results are published in Nature.

Those ratings cut into workers’ earnings. Under the service’s pay system, lower average ratings reduced the slice of total revenue a worker made from subsequent jobs. Under a five-point system, non-white workers ended up with 91 cents for every dollar earned by white workers, the researchers estimated.

“While the objective difference, on average, between white and non-white worker ratings is very small, it matters because of the impact it has on income, highlighting the importance of structure and organizational design for racial equality at work,” said Katherine DeCelles, a professor of organizational behavior at the University of Toronto’s Rotman School of Management who was among the four-member research team.

Unlike overt racism, where a customer might refuse to work with a non-white contractor, subtle racism tends to be harder to pin down and can be expressed even when a person doesn’t believe they’re being racist.

A home repair service might not be able to change people’s attitudes, but the researchers found that when the service being studied switched to a two-choice system—customers were only asked if they would use the contractor again, thumbs-up or thumbs-down—racial gaps in who got top ratings virtually disappeared. And new workers who joined after the switch saw no racial differences in earnings for the same job.

Follow-up experiments using online participants and artificial scenarios again showed reduced racial disparities in two-point performance ratings among participants with subtle racial bias. Participants also told the researchers that their opinions and biases were less likely to influence their performance evaluations when there were only two options.

“People can more clearly evaluate whether someone’s work was good versus not, instead of ‘how good was it?’ which is relatively more subjective and ambiguous—that’s where we’d expect a larger problem with racial bias in evaluations,” said Prof. DeCelles.

Given that rating systems are increasingly common in many digital platform-based services, the researchers recommend keeping them simple and focusing evaluators on whether something was good or bad to avoid the influence of bias. Platforms can also regularly audit their systems to identify systematic variations in evaluations that may be bias-related and give customers other ways to provide more detailed feedback on top of the basic evaluation, without consequences for worker pay, suggested Prof. DeCelles.