The Influence of Status on Evaluations: Evidence from Online Coding Contests

SKU
16178

Received: April 2, 2019

Revised: December 11, 2019; September 4, 2020; June 8, 2021; February 16, 2022

Accepted: February 16, 2022

Published Online as Accepted Author Version: Forthcoming

Published Online as Articles in Advance: Forthcoming

Published in Issue: Forthcoming

https://doi.org/10.25300/MISQ/2022/16178

$0.00
Abstract

Online contest platforms rely on contestants to ensure submission quality. This scalable evaluation mechanism offers a collective benefit. However, contestants may also leverage it to achieve personal, competitive benefits. Our study examines this tension from a status-theoretic perspective, suggesting that the conflict between competitive and collective benefits, and the net implication for evaluation efficacy, is influenced by contestants' status. On the one hand, contestants of lower status may be viewed as less skilled and hence more likely to make mistakes. Therefore, low-status contestants may attract more evaluations if said evaluations are driven predominantly by an interest in collective-benefits. On the other hand, if evaluations are driven largely by an interest in personal, competitive-benefits, a low-status contestant makes for a less attractive target and hence may attract fewer evaluations. We empirically test these competing possibilities using a dataset of coding contests from Codeforces. The platform allows contestants to assess others' submissions and improve evaluations (a collective benefit) by devising test cases (hacks) in addition to those defined by the contest organizer. If a submission is successfully hacked, the hacker earns additional points, and the target submission is eliminated from the contest (a competitive benefit). We begin providing qualitative evidence based on semi-structured interviews conducted with contestants spanning the status spectrum at Codeforces. Next, we present quantitative evidence exploiting a structural change at Codeforces wherein many contestants experienced an arbitrary status reduction as a result of sudden changes to the platform's color-coding system around contestant ratings. We show that status-loser contestants received systematically more evaluations from other contestants, absent changes in their short-run submission quality. Finally, we show that the excess evaluations allocated toward affected contestants were less effective, indicating status-driven evaluations as potentially less efficacious. We discuss the implications of our findings for managing evaluation processes in online contests.

Additional Details
Author Swanand J. Deodhar, Yash Babar, and Gordon Burtch
Year 0
Volume Forthcoming
Issue Forthcoming
Keywords Evaluation, Arbitrary Status Change, Competitive Coding, Online Contests
Page Numbers
Copyright © 2022 MISQ. All rights reserved.