AQR: Design of Social Research

SOC-GA 1301 – Fall 2017

Robert Max Jackson

Assisting: Christina Nelson

Christina's office hours: Wed 1-3pm,
Office 4177 (Puck Building)

      In this course we will investigate how to design and carry out effective research projects aimed at social science problems.  The course focuses on methodological issues, such as causality, identification of mechanisms, the logic of argument, objectivity, reliability and validity, problems of measurement, representation, conceptualization, inference, and sampling.  The course seeks to illuminate: (1) how to choose a research project that is worthwhile and practical; (2) how to choose and use methods adequate for a research project; (3) how to evaluate the research, evidence, and arguments offered in support of theoretical claims.

Table of Contents ...
click on a topic to go to that section
Overview  ·······  Scope, Organization, and Access  Read this first!
Topic 1  ·······  Introduction.
Topic 2  ·······  Elements of research design
Topic 3  ·······  Causality - What are causes, mechanisms, and the like?
Topic 4  ·······  Causality 2 - Actors, Conditions, Processes, Structures & the role of "counterfactuals"?
Topic 5  ·······  Surveys and experiments: Archetypes for Research Design?
Topic 6  ·······  Sampling
Topic 7  ·······  Observation and gathering data
Topic 8  ·······  Assembling the pieces of a research project. Good aspirations and causal concerns.
Topic 9  ·······  Measurement
Topic 10  ·······  How to Prepare a Good Literature Review.
Topic 11  ·······  Writing, diagramming, and Modeling Multivariate Relationships.
Topic 12  ·······  The Art of a Critical Manuscript Review.
Topic 13  ·······  Sources of Error. Where do we go wrong? How do we minimize, reveal, and overcome errors?
Topic 14  ·······  Research is an ethical issue. (& Class Presentations of Research Proposals.)
Table of Contents ...
click on a topic to go to that section
Overview  ·······  Scope  Read this first!
Topic 1  ·······  Introduction.
Topic 2  ·······  Elements
Topic 3  ·······  Causality
Topic 4  ·······  Causality 2
Topic 5  ·······  Surveys & Experiments
Topic 6  ·······  Sampling
Topic 7  ·······  Gathering Data
Topic 8  ·······  Integrating
Topic 9  ·······  Measurement
Topic 10  ·······  Literature Review
Topic 11  ·······  Multivariate Relationships
Topic 12  ·······  Manuscript Review
Topic 13  ·······  Sources of Error
Topic 14  ·······  Ethics & Proposals

Table of Contents

Description – Scope, Organization, and Access:

The Scope of the Topics and Materials. The class will be organized around issues of research development and design.  We will use a good, standard textbook by Singleton & Straits that provides a reasonable overview of the knowledge commonly expected in a basic graduate, social science methodology class.  We will augment this with a highly-regarded book, The Craft of Research, that aims at the conceptual level of the processes by which we go from disconnected ideas, to research questions, to transforming our research into arguments, and to making those arguments into effective papers.  Along the way, we will also read a range of articles and other materials that provide further illustrative examples and allow us to examine some important topics that are too often neglected.   These other materials will be available on line, through the class web site.

The class organization and goals. We will run all class meetings as discussions. Every student must come prepared to every class and participate.  Each week, a written task analysis will be due.  A term paper in the form of a full research proposal will be due at the end of the semester.

Written assignments will include short papers every week.  On occasion these will be group efforts of several students, otherwise individual efforts.  In the weekly writing tasks, students will attempt to solve research design issues related to the week's readings.  In the second half of the semester, these weekly efforts will also be aimed at the research proposal that is the culminating work for the class.  All assignments must be handed in on time.  I will not give incompletes.

Generally, all students' weekly writing assignments will be circulated among all class members before our class meeting, so that we can discuss them. 

The research proposal produced as the final project of this class is intended to be the MA thesis proposal for AQR students.  This is a daunting goal for all of us.

Academic integrity.  In this class, the AQR Program, and at NYU more broadly, we expect and require students to adhere to the highest standards of academic conduct.  Students who engage in academic dishonesty will be subject to review and the possible imposition of penalties in accordance with the standards, practices, and procedures of NYU and its college and schools. Violations may result in a failure on a particular assignment, a lower course grade, failure in a course, suspension or expulsion from the University, or other penalties.

Students are often encouraged to seek outside assistance from tutors, writing coaches, and online resources. Such behavior is permitted when the intellectual contribution of completed work is that of the student, and when outside assistance is appropriately acknowledged. Outside assistance must never contribute to intellectual content. Assignments, research papers, and other materials submitted for evaluation or review must be a student’s own in their entirety.

Further detail on NYU policies and procedures related to academic integrity are available online.

Books Recommended for Purchase:

Author
Title
Publisher
Royce Singleton & Bruce Straits Approaches to Social Research (6th ed) Oxford
Wayne Booth, et. al. The Craft of Research (6th ed) Chicago
Jane Miller Writing About Multivariate Analysis Chicago

The Topics

1. Introduction. 

Let us start by thinking about the task ahead of us.  

2. Elements of research design.

Research design, not statistical expertise, largely decides if a research project succeeds.  What does it take to go from a question about how things work in the world to an answer that can be defended with evidence and sound logic?  It takes knowledge about the specific issue, about the relevant social processes, and about how to do good research; it requires a thoughtful plan that balances scientific aspirations with practical possibilities; it takes a lot of work over time; and it takes a nimble responsiveness to the unexpected.  To do it really well also requires a disciplined willingness to recognize and respond to the limits of the research and to that which very few researchers can abide: evidence that we are wrong.

3. Causality - What are causes, mechanisms, and the like?

We casually refer to causes and effects in normal interactions all the time.  We all conduct our lives – choosing actions, making decisions, trying to influence others – based on theories about why and how things happen in the world.  From the early stages of childhood we attribute causes, building a vision of the social (and physical) world that makes it understandable.  Every action, every choice about what to do, is based on our anticipation of its effects, our understandings of consequences.  Analytical and scientific reasoning has a similar form, but requires that we approach causation more systematically and self-consciously. 

4. Causality 2 - Actors, Conditions, Processes, Structures & the role of "counterfactuals"?

Without becoming philosophers of science, effective social science researchers must have a reasonable grip on thinking about causality.  Even most social research not aimed at questions about social causation usually relies on critical assumptions about causation and can only be used in arguments or policies that overrun with causal thinking.  Unfortunately, causal thinking is difficult and fads guide causal argument choices as much as (and often more than) rigorous logic.  For some time the framework of counterfactuals has held considerable influence in social science, and scholars steeped in its rhetoric often hold themselves superior to those who do not adhere to its argument style.  Truthfully, one would be hard pressed to identify research or theoretical development in social science that have gone further using a counterfactual language than they would have without it, but lacking that rhetorical skill puts one at a disadvantage.  So here we want to get the basic ideas clear.  At the same time, for a contrast, it is worth looking carefully at the practical strategy for assessing causality put forth by  the epidemiologist and statistician, Sir Austin Bradford Hill a half-century ago that had profound and lasting influence on real-world assessments of causality and public health. This should remind us that in science and in life, causality is ultimately a practical problem, not a philosophical one.

5.   Surveys and experiments: Archetypes for Research Design

The experiment is the classic research design by which all others are measured.  Physicists, medical researchers, biologists and others all use experiments when possible.  Experiments are exercises in control that directly match our understanding of causation.  The researcher divides subjects (whether they be people, rocks, events, or something else) into two or more groups by random assignment or some analytically controlled selection rule, then the subjects receive differential "treatments," and finally the subjects are observed to discern if any consistent differences between the experimental groups suggest that exposure to the treatment has a causal impact.  In sociology and related disciplines, we are unable to use experiments to discover the impact of most causal influences.  We must rely on observation of "natural" variations.  From participant observation of ongoing activities to counting artifacts (such as wills) to sorting through government records (such as death certificates), social scientists mine the social world for data.  Here the sample survey substitutes for the experiment as the archetypal research design.   If we fully understand the issues of survey research design, we are generally able to handle any research design in the social sciences.

6. Sampling

Selecting a subset of the target population for research is fundamental to all social science research.  Even research that seems to observe every member of a population generally depends on sampling ideas.  A population census, for example, has to be sampled to produce the public use samples made available to scholars for research.  Experimental designs often neglect sampling concerns, relying on random assignment to comparison groups; yet, such randomization occurs within the sample of subjects (people or otherwise) available for research.  Generalization to the full population or beyond depends on the implicit sampling that has occurred.  Research using organizations, nations, or other collective entities faces significant issues establishing the population being studied, often reversing the logical sequence by trying to define the population based on the available sample.  In short, all social research must contend with sampling as a crucial facet of research design and defense of inferences.  Since acquiring data involves time, effort, and expense, and may be curtailed by many obstacles, sampling is both a practical and a conceptual problem for research.

7.  Observation and gathering data

Data have to be collected and processed.  Even if we use existing data from past censuses, surveys, or government records, our research depends crucially on the processes that produced our data.  If we treat data as simple, easily interpreted, unambiguous, valid indicators of what we want to know, our plan is sunk before we leave the dock.  To have a good idea what we need to look for and worry about, we need to understand how data comes into being, what good practices are, and what are the many reasons that our data might not represent what we want.

8. Assembling the pieces of a research project.  Good aspirations and causal concerns. 

Midway through our journey, let us pause, stand back, and consider the research project as a whole, given our understandings up to this point. 

9.  Measurement

All research depends on the assumption that we can assess the characteristics of actors, relationships among actors, and events accurately and realistically enough to allow us to describe them and analyze how the influence each other.  Our capacity to do effective analyses depend crucially on the processes by which we categorize and count social phenomena.  These processes are prone to errors at many levels. 

10.   How to Prepare a Good Literature Review.

Preparing a good literature review is a task easily overlooked.  Yet, a weak literature review in the early stages can jeopardize one's chances of developing a good research design.  At later stages, a poor literature review can damage a paper that otherwise reflects good research.  A good literature review depends on two critical skills: (1) knowing how to find the existing research and theoretical work that is relevant to one's project and (2) knowing how to select and present the important ideas and findings in that literature.

11. Writing, diagramming, and Modeling Multivariate Relationships.

Good research design commonly involves a precarious balance between too much and too little.  Most social phenomena of interest involve a complex interplay between actors, processes, structures, and expectations over time.  Good theory depends on a selective abstraction from the complexities of life as experienced to create a simplified but telling model of decisive facets.  Good research must similarly pursue a selective view of reality, focused on a limited set of causal relations that are sufficient to make sense of some phenomena we wish to understand, without losing us in irrelevant details or  misleading us by leaving out things that are important.  Being able to think clearly about the relationships among multiple facets of a theoretical model is the mirror of thinking clearly about the possible relationships among multiple variables in a research design and a data analysis.  Hypothetical models of the possible relationships between social phenomena give us the goals for our research design while analyzing the patterns found in our data give us the insights to adjust our models. 

12.   The Art of a Critical Manuscript Review.

Every professional involved in social research must be prepared to give and receive critical reviews.  The standards for critical reviews are high.  A good review needs to identify the important limitations of the work under review as well as recognize its potential contribution.  The measure of weaknesses and strengths are the standards and existing knowledge in the field.  A review that overlooks significant shortcomings or fails to notice possible contributions is seriously flawed.  


13. Sources of Error. Where do we go wrong? How do we minimize, reveal, and overcome errors?

All research efforts produce mistakes.  Research is difficult and complex.  We are flawed and error prone.  A strategy that relies strictly on avoiding mistakes will fail, because it impedes recognition and correction of errors more than it prevents mistakes. 

14 & 15.  Research is an ethical issue.  (& Class Presentations of Research Proposals.)

The professional who conducts social research is a scientist, and we expect scientists to conform to ethical codes. In academic environments (and many non-profit research organizations), this is made salient to social scientists by the requirement that research must be approved by institutional review boards (IRBs), that try to ensure we do not mistreat people. Other ethical concerns, such as selectively presenting only research findings that support the researcher's argument, are all too often neglected. Only when someone is exposed as flagrantly violating rules, such as inventing the data, do our ethics get much recognition (and even then, social scientists commonly cope with the issue as one of public perception). The misuse of social science in public controversies has led many to believe that we can always manipulate the numbers to match our argument. Every time social scientists bend the rules - and this happens with embarrassing regularity - we contribute to the erosion of trust in social scientists ... by the public, by decision makers, and by other scientists. Without integrity, we become known as modern alchemists, practicing pseudo-science.

Topics in Waiting
(Click here to see Possible Sections for the Future)

Possible additional sections ...

XII.  ?

      B. 

XII.   H?

As.

To search this page, first click here [§] to reveal all the hidden text. (Refresh page to rehide)


  Qs S1 S2 S3 S4