Measuring polarization in high-dimensional data: Method and application to Congressional speech
This project studies the partisanship of Congressional speech from 1873 to 2009. Partisanship is defined as the ease with which an observer could infer a congressperson’s party from a fixed amount of speech, and it is estimated using a structural choice model and methods from machine learning. The estimates reveal that partisanship is far greater today than at any point in the past. Partisanship was low and roughly constant from 1873 to the early 1990s, then increased dramatically in subsequent years. Evidence suggests innovation in political persuasion beginning with the Contract with America, possibly reinforced by changes in the media environment, as a likely cause. Naive estimates of partisanship are subject to a severe finite-sample bias and imply substantially different conclusions.