What Is Judicial Ideology, and How Should We Measure It?
UVA Law Faculty Affiliations
Scholars have for decades sought to measure judicial ideology and its impact on judicial behavior. However, they have not always taken care to identify the phenomenon that they are measuring in clear and convincing terms. Nor have they made a habit of evaluating different measurement approaches for their suitability to the task at hand. In this Article, we attempt to remedy the recurring weaknesses of the existing empirical literature in three ways. First, we identify the major conceptual and methodological obstacles to the empirical measurement of judicial ideology. Second, we compare the performance of several popular approaches to the measurement of judicial ideology. Third, we propose measurement approaches of our own that, for certain applications, offer significant improvements over some of the most popular alternatives.
At a conceptual level, empirical scholars confront the difficulty of defining ideology and the related challenge of disentangling ideological and nonideological preferences. As a methodological matter, scholars face the challenge that ideology is an intangible phenomenon that cannot be directly observed. From this fact stems the further problem of observational equivalence: the observable behavior of judges is open to multiple interpretations and may be attributable to both ideological and nonideological motivations. A further complication is the possibility that judicial ideology may be multidimensional across different areas of law.
It is crucial that scholars give explicit attention to the strengths and weaknesses of different measurement approaches, and their suitability to different types of research questions. To that end, we identify and contrast three broad families of approaches - namely, those that rely upon some easily observable proxy for a judge's ideology, those that require assessment of a judge's actual behavior in a particular context, and those that involve transplanting ideology estimates from one context to another. We also discuss issues surrounding the coding of cases for empirical analysis, with particular attention to the choice between unidimensional, multidimensional, and agnostic coding schemes.
Finally, we perform head-to-head evaluations of several popular measures of judicial ideology to see how well they predict voting behavior on the federal courts of appeals and the Supreme Court. For the courts of appeals, we test two widely used proxy measures - namely, party of appointing president and the judicial common space scores - against a novel approach that entails analyzing past voting behavior to estimate not only each judge's ideology, but also the impact of collegiality concerns and dissent costs on judicial voting. As between the two proxy measures, we find that the common space scores perform only slightly better than party of appointing president at predicting actual votes. Moreover, both proxy measures yield almost identical estimates of the impact of ideology on actual judicial voting. By contrast, the novel approach that we employ boasts greater predictive power than either of the proxy measures, and it further reveals that ideology has a much greater impact on judicial voting than could be discovered using either of the proxy measures.
For the Supreme Court, we test party of appointing president, the Martin-Quinn scores, the Segal-Cover scores, and our own measure of judicial ideology computed from the voting behavior of the justices. The performance of the Martin-Quinn scores at predicting judicial votes is comparable on the whole to that of our customized measure and noticeably superior to that of the Segal-Cover scores, which in turn outperform party of appointing president by a substantial margin.