Government and Opposition, Vol. 44, No. 4, pp. 471–475, 2009 doi:10.1111/j.1477-7053.2009.01299.x
Tim Veen and Jonathan Sullivan News Sources and Decision-Making in the EU Council: A Rejoinder goop_1299
471..475
RESPONDING TO A RECENT REVIEW ARTICLE IN THIS JOURNAL,1
Selck, Yardımcı and Kathan argue that in order to improve our understanding of decision-making processes in the EU Council, we need to improve the quality of the data we collect and analyse.2 The authors also suggest that instead of further analysis of existing data sets, we should expand our horizons to search out new forms of information on the Council. On these two points we are in complete agreement; indeed this argument forms the crux of our original review article, where we discuss the potential exploitation of increasingly abundant primary data made available by the Council. While we acknowledge Selck et al.’s argument that the information contained within these primary sources is sometimes limited, we question their suggestion that newspaper analysis offers a potential improvement. In this short note we briefly develop three points. First, the data that it is possible to derive from newspapers simply does not meet the exacting requirements of the bargaining models that Selck et al. argue it can improve, even in combination with alternative data sources. Second, given the nature of Council decision-making and issues of bias that affect all research using newspaper coverage, we question the assertion that ‘newspapers [are a] proxy of what really happened during the bargaining process.’3 Third, Selck et al. appear unaware of the methodological challenges involved in generating requisite data from newspapers. 1
Jonathan Sullivan and Tim Veen, ‘The EU Council: Shedding Light on an Opaque Institution,’ Government and Opposition, 44: 1 (2009) pp. 113–23. 2 Torsten J. Selck, S, ebnem Yardımcı and Constanze Kathan, ‘Still an Opaque Institution? Explaining Decision-Making in the EU Council using Newspaper Information: A Reply to Sullivan and Veen’, Government and Opposition, 44: 4 (2009), pp. 463–70. 3 Ibid., pp. 464–5. © The Authors 2009. Journal compilation © 2009 Government and Opposition Ltd Published by Blackwell Publishing, 9600 Garsington Road, Oxford OX4 2DQ, UK and 350 Main Street, Malden, MA 02148, USA.
472
GOVERNMENT AND OPPOSITION
Although sympathetic to Selck et al.’s proposition that ‘the data is out there; all we have to do is use it’,4 we are not convinced that newspapers are appropriate for the task. Selck et al. argue that ‘properly applied’ newspaper analysis can provide ‘the type of valid and reliable data that is needed to improve model-testing’.5 We question whether this is the case, given that the various model types discussed in our review article require specific numerical estimates of actor positions at various stages of the decision-making process, estimates of the salience that actors attach to each issue, actors’ relative capabilities to influence negotiations etc.6 Furthermore, evaluating the predictive power of decision-making models also requires numerical estimates of the location of the status quo and the bargaining outcome.7 Although prior research demonstrates that it is possible to derive numerical estimates of policy positions from political texts,8 we are unaware of any research that uses newspaper sources to generate detailed numerical estimates on actor positions and the other requisite variables. Although Selck et al. imply that newspaper data represents a less costly and more sustainable source of information than the current reliance on experts, it is unclear by which method they propose to ‘improve data quality’ using newspapers.9 Our original argument about the potential utility of Council documentation was
4
Ibid., p. 470. Ibid., p. 468. 6 Christopher H. Achen, ‘Institutional Realism and Bargaining Models’, in Robert Thomson, Frans N. Stokman, Christopher H. Achen and Thomas König (eds), The European Union Decides, Cambridge, Cambridge University Press, 2006, pp. 86–123. 7 Ibid. 8 Edward E. Azar, ‘Conflict Escalation and Conflict Reduction in an International Crisis: Suez, 1956’, Journal of Conflict Resolution, 16: 2 (1972), pp. 183–201; Gary King and Will Lowe, ‘An Automated Information Extraction Tool for International Conflict Data with Performance as Good as Human Coders: A Rare Events Evaluation Design’, International Organization, 57: 3 (2003), pp. 617–42; Wouter van Atteveldt, Jan Kleinnijenhuis and Nel Ruigrok, ‘Parsing, Semantic Networks, and Political Authority using Syntactic Analysis to Extract Semantic Relations from Dutch Newspaper Articles’, Political Analysis, 16: 4 (2008), pp. 428–46; Wouter van Atteveldt, Jan Kleinnijenhuis, Nel Ruigrok and Stefan Schlobach, ‘Good News or Bad News? Conducting Sentiment Analysis on Dutch Text to Distinguish between Positive and Negative Relations’, Journal of Information Technology and Politics, 5: 1 (2008), pp. 73–95. 9 Selck et al., ‘Still an Opaque Institution?’, p. 468. 5
© The Authors 2009. Journal compilation © 2009 Government and Opposition Ltd
NEWS SOURCES AND DECISION-MAKING IN THE EU COUNCIL
473
based on the observation that preliminary empirical applications demonstrate how it can deliver appropriate and reliable data comparable to expert estimates.10 The same cannot be said for newspaper analysis. Our second concern relates to the extent to which newspaper coverage reveals what ‘really happened’ in the bargaining process, particularly as Council negotiations are notoriously secretive and specialized. The claim raises the issue of selection bias and reliability that affects all research using media coverage as a data source. Selck et al. advocate collecting newspaper articles from multiple sources, claiming that any biases in reporting would be offset, but one factor that would not be offset is the so-called newsworthiness of an event.11 Several variables influence the probability that an event is covered, the extent to which it is covered over time and the degree of detail that is reported. Legislation with far-reaching consequences, particularly when it incurs high levels of contestation among the more influential states, may indeed receive comparatively extensive coverage.12 However, the majority of EU legislation does not fit this description, which makes us wonder how much newspapers reveal about the ‘day-to-day decision-making process’ of the EU,13 particularly with the Council’s well-known predilection for masking conflict between its members. Notwithstanding Selck et al.’s observation that the Financial Times published over 300 articles on the crucial and strongly contested EU Services Directive (over a period that we calculate to be at least five years), we suggest that media coverage of ‘day-to-day decision-making’ in the EU that Selck et al. want to
10 Robert Thomson, ‘Appendix II: Comparison of Expert Judgements with Each Other and with Information from Council Documentation’, in Thomson et al., The European Union Decides, pp. 329–47. 11 John T. Woolley, ‘Using Media-Based Data in Studies of Politics’, American Journal of Political Science, 44: 1 (2000), pp. 156–73, p. 158. For an empirical example see Daniel J. Myers and Beth Schaefer Caniglia, ‘All the Rioting That’s Fit to Print: Selection Effects in National Newspaper Coverage of Civil Disorders, 1968–1969’, American Sociological Review, 69: 4 (2004), pp. 519–43. 12 Hans-Jörg Trenz, ‘Media Coverage of European Governance: Exploring the European Public Sphere in National Quality Newspapers’, European Journal of Communication, 19: 3 (2004), pp. 291–319; Christina Holtz-Bacha, Medienpolitik für Europa, Wiesbaden, VS-Verlag für Sozialwissenschaften, 2006. 13 Selck et al., ‘Still an Opaque Institution?’, p. 467.
© The Authors 2009. Journal compilation © 2009 Government and Opposition Ltd
474
GOVERNMENT AND OPPOSITION
illuminate is too limited to make anything but the most tentative and rudimentary inferences.14 The final issue to mention here concerns methods. Although Selck et al. mention ‘electronic content analysis’15 in passing, there is no indication of how the authors propose to generate the required quantitative data from the newspaper sources they promote. And though they provide an example of analysing a text manually, which they claim ‘reveals substantial pieces of information on the bargaining situation’,16 this leaves us none the wiser as to how to apply extant computer-assisted content analysis (CCA) techniques, or whether indeed the authors have developed an alternative method. Surveying the techniques currently available, we find several that may be applicable. Wordscores and Wordfish are well-known CCA methods that are able to generate positional estimates of the author of the text.17 These methods are thus usually employed to analyse political speeches or party manifestos, where the de facto author – a politician or party – is clearly identified. A newspaper article, on the other hand, does not generally explicate the journalist’s policy preference, but instead reports the interactions of political actors. For this reason, event data research, which typically focuses on conflict and provides measures of the type and intensity of actor interactions, is one of the few techniques for which newspaper articles are well suited, but is of limited use in the context of bargaining and decision-making models. Although promising advances in the field of semantic network analysis has successfully yielded estimates of party policy positions and salience during recent Dutch elections,18 estimating actor positions 14
David G. Ortiz, Daniel J. Myers, Eugene N. Walls and Maria-Elena D. Diaz, ‘Where Do We Stand with Newspaper Data?’, Mobilization: An International Journal, 19: 3 (2005), pp. 397–419. 15 Selck et al., ‘Still an Opaque Institution?’, p. 467. We take this to mean computer-assisted content analysis. 16 Ibid., p. 469. In reality, however, the information is insufficient in scope and detail to be of much use in model-testing. 17 Michael Laver, Ken Benoit and John Garry, ‘Extracting Policy Positions from Political Texts using Words as Data’, American Political Science Review, 97: 2 (2003), pp. 311–31; Will Lowe, ‘Understanding Wordscores’, Political Analysis, 16: 4 (2008), pp. 356–71; Jonathan Slapin and Sven-Oliver Proksch, ‘A Scaling Model for Estimating Time Series Policy Positions from Texts’, American Journal of Political Science, 52: 3 (2008), pp. 705–22. 18 van Atteveldt et al., ‘Good News or Bad News?’. © The Authors 2009. Journal compilation © 2009 Government and Opposition Ltd
NEWS SOURCES AND DECISION-MAKING IN THE EU COUNCIL
475
from non-Dutch-language texts is currently out of reach for existing software. In areas where textual data appropriate to the research purpose is available in sufficient quantities, CCA can deliver reliable quantitative data. But in the context of improving models of decision-making in the EU Council, newspapers contain insufficient information and current methods are not developed enough to generate the specific data we need. Nonetheless, we concluded our original article by noting the timeliness of exploring new sources of information on the decision-making processes of the EU Council. To this end, Selck et al.’s contribution is to be encouraged. And while we have voiced scepticism about their projected use of newspaper sources, we look forward to future empirical applications where Selck and colleagues show our scepticism to be misplaced.
© The Authors 2009. Journal compilation © 2009 Government and Opposition Ltd
