AQR: Design of Social Research

SOC-GA 1301 – Fall 2017

Robert Max Jackson

Assisting: Christina Nelson

Christina's office hours: Wed 1-3pm,
Office 4177 (Puck Building)



Notes: Critical Questions about Initial Project Ideas

What should we ask about the initial project ideas?  As they were limited to one, single-spaced page, we do not expect them to provide details, to consider complex design issues, or to ponder tough research questions.  We do, however, want them to get the basic ideas across clearly and to avoid ambiguity, vagueness, and confusion.  We want them to recognize obvious big issues with the project ideas and to avoid obvious design errors.  Here are a set of straightforward questions to help guide your critical assessment of your own and others' initial project ideas.

  1. Is the research question clear?
    1. Does it avoid ambiguity and vagueness?  We don’t expect it to be precise here, but we should not wonder what the author means, or be able to think of two or more questions that it might imply.
    2. Do you have to infer the question because it doesn’t seem to make sense as written?
  2. Is the characterization of the relevant literature sufficient that you could find that literature? 
  3. How effective are the initial possible causal interpretations? 
    1. Is the logic of the causes clear? 
    2. Can you identify which are the relevant outcomes (dependent variable(s)), major causes (independent variables), possible intervening or mediating causes? 
    3. Is it clear how the causal conditions or actions are supposed to bring about the outcomes? 
    4. Do the causal statements take into account obvious issues of other causes demanding analytical control because they are likely to confuse the causal relationships in the data? 
  4. Does the report describe the proposed data and how it will be used sufficiently that you can follow what kinds of variables correspond to the elements of the causal process and how those variable might be analyzed?
    1. Does the report indicate how the sampling procedure for the data is adequate for the research problem?
    2. Does the report consider obvious potential issues about the samples available in the data, particularly including possible selection biases?  (E.g., if we want to consider how much being in a charter school rather than a regular public school affects poor children’s achievements, we have to consider that the kids who go to charter schools are different than those going to the regular schools.)
    3. Do you see any possible issues with the proposed data for the specific projects?  For example, does it seem to have the right measures, are the units of analysis appropriate?
  5. Does the discussion of why the research is worth doing provide a clear statement of what the research hopes to discover – usually the validity of some proposed relationship or the size of some effect – and indicate who would find this worthwhile (either as a contribution to science or to social policy)