When should we believe social science findings?

17May/12

When should we believe social science findings?

Recently, some colleagues of mine forwarded me this article from the Weekly Standard concerning the use of social science to delegitimize conservatism. There are some valid points in this article that the author uses to question specific studies. However, I think the author fails to understand the breadth of evidence that underlies most social science findings.

Social scientists deal with a far more complex subject than scientists who work with rocks or chemicals. Specifically, human beings have free will. They can decide to do or not do things in response to a stimulus. Further, because we care about human beings in a way that we don't care about rocks, we can't always design studies perfectly, as we have to respect the wishes of others. As such, all social science has problems of sampling and generalizability.

But the fact that all social science research has flaws doesn't mean you should ignore it. For example, presidential polls have flaws, even with the author's preferred sampling method, as question wording, non-response, and weighting to correct for non-response all introduce bias. While each poll is imperfect, each poll still give us some understanding of what is going on in the population. Perhaps more critically, different polls have different flaws, which means that if you aggregate across measures (e.g. see Nate Silver's five thirty eight blog), you can get something close to the truth (the same principle underlies the Wisdom of Crowds). Yes, a survey of yourmorals.org volunteers or undergraduates or mechanical turk participants or randomly selected households who will answer a survey, is imperfect. Yes, artificial experiments, neuroscience correlations, and self-report are all imperfect. But they are all imperfect in somewhat different ways, and if you find the same thing across each of these samples using a variety of different methodologies, then you can be pretty confident of your findings.

Personally, I don't believe any single study or paper, and a I wait to see if there is confirmation across research groups, methodologies, and samples before believing any research. This is true in social science and in other sciences as well. Andrew Ferguson, who wrote the Weekly Standard piece, is capitalizing on an intuition we all likely share, that so many studies out there report so many facts, many of them contradictory (e.g. is alcohol good for your health?), that we can't help but question them. And we should. Individual studies and papers are not proof, and we probably shouldn't report them as such. But much of this research that relates to liberal and conservative differences has many studies using many methodologies and samples behind them, and that is where we can be more confident. It is for this reason that I increasingly find myself drawn to computer scientists and data scientists who work on questions of aggregation, and as technology starts to pervade social science, my guess is that social science will move more towards aggregation and also place less emphasis on individual papers.

I agree with Ferguson that pathologizing the other side isn't helpful, but not because the science is wrong, but because the interpretation often is subject to bias. A lack of empathy can be thought of as an ability to make rational, competent decisions or heartlessness. Loyalty to one's family can be thought of as noble or as nepotism. Reliance on one's intuition can be thought of as indicative of common sense or of ignorance. But the fact that these things differ between liberals and conservatives are indeed facts, with as much evidence behind them as facts like cholesterol causes heart disease. The world's knowledge graph will eventually encompass not just physical facts, but facts like these as well.

- Ravi Iyer

Comments

comments

Tagged as: generalizability, liberals and conservatives, social science research, social scientists, weekly standard Leave a comment

Old Comments (2) Trackbacks (0)

Scott Wagner
June 13th, 2012 - 13:13

Your final paragraph is very well put and thought-provoking: thanks much. Mr. Ferguson’s last paragraph is the opposite- a churlish, shallow dash at a legitimate subject of inquiry (biological and evolutionary facets of ideology), with his simplistic repudiation and speculation that politics are somehow “drained” of meaning when genomic correlates are folded into the mix. But I liked much of what he had to say, especially relative to most such protests from the right: I find Mr. Mooney annoying. He’s a great, accurate reporter, so I’m frustrated, but his attitude compromises or colors his material unduly. And I have always been offended at Jost’s lack of care with language and framing, as well as his sampling, at times. To take Mr. Ferguson’s point further, I also feel that the lack of decent research on significant liberal weaknesses is a reflection of scientist predilection, not paucity of source material. He also made an explicit point multiple times I’d mirror- despite the tremendous cost and sampling convenience implications, it’s no longer appropriate to depend on students in generalized political sociological studies, due to the difficulty of adjusting for the multiple integral dimensions of bias typically introduced.

I would add to both of your arguments what I think is an important step, that we need to arrive at rationale, contingencies, and variables to better bridge the gap between, say, labeling family loyalty as either nobility or nepotism, or a closer relationship to our nonconscious mind as either common sense or ignorance. The science may not be wrong, as you said, but the interpretation is often so wrong that it may as well be. Both the gods and devils are in the details, and they pave the whole long way between the value-laden opposites. I feel that we still lack very important academic understanding of one another. The academic left has not unlocked ideological differences well yet because it’s a very complicated task, because they don’t get much help from the right, and because, as Dr. Haidt has addressed, we are allowed to drag our biases along with us while we do the research work. There was an irony in your allusion that compared our state of knowledge about ideology with the fact that cholesterol causes heart disease, because ‘cholesterol’ is now a very complicated, essentially useless notion, and there’s agreement that other much less well-known factors like insulin levels, specific antioxidant levels, and a ratio of two specific cholesterol sub-types determine risk. To say one has a closer relationship with the nonconscious, for example, suffers from the same lack of detail, to the point of deceptiveness. It sounds straight-forward in a way it isn’t: it ignores that there are asymmetric aspects of repression implied for the left and right that may be important; that there are specific tied-in groupthink and decision-making implications that play out in unintuitive ways due to personality variables; that the landscape of ‘nonconscious’ is broad enough to make the distinction deceptive when it comes to, say, intellectual achievement, or within the realm of compassionate reaction, or in the context of stress or moral disgust. To reinterpret Mr. Ferguson rather freely, the useful truths about difference are to be found at a level of analysis below these dichotomous pronouncements. You made the point early on in your note that social scientists deal with a far more complicated science than physical scientists. I look forward to the immediate future as we make headway, in a more nonpartisan way, toward a better understanding of that complexity.
Ravi
June 14th, 2012 - 14:09

Thanks for the thoughtful comment, Scott, and I agree the sentiment obviously. and thanks for correcting my ideas about cholesterol too….I tried to pick something in the physical sciences that was well settled, but some of my colleagues have pointed out that there is far more ambiguity in the physical sciences that I likely realize from a distance. Which probably helps my point about always looking for confirmation when looking at studies. Thanks again…
Ravi