J Max Wilson, calm down and go study research methods

In a recent post on what is apparently a faith-based blog, someone named J Max Wilson purports to tell Mormons “what [they] should know” about “that Mormon gender issues survey.” This survey. The blog post has many of the hallmarks of a wagon-circling, lines-in-the-sand, us-vs-them call to arms (or at least to fear):

  • A title that hints at dark secrets not apparent to the naked eye
  • Allegations of guilt by association: the parent group of the surveyors includes a scary guy, and in case you don’t know why he’s scary, J Max Wilson will tell you: because he’s a “well-known LDS dissenter and agitator.”
  • Dog-whistle terms (for conservative Latter-day Saints) are thrown around with abandon: progressiveactivistsagitation, liberaltypical agenda, academic, propaganda.

By the middle of the blog post, John Dehlin, the “agitator” who, earlier, was identified only as a member of the parent group of the people who created the survey, is now implied to be the mastermind of this project, in order to connect it to one of his previous projects (“This isn’t the first time Dehlin has used stilted academic surveys to support his agitation”). This previous project apparently (though with no citation to support this claim) “over-estimate[d] the number of unhappy women in the LDS church…”

To recap:

  1. Scary title!
  2. Scary buzzwords!
  3. This  might maybe be associated with a guy who I’m telling you is bad news.
  4. Oh wait, it is totally all done by that guy! And he did other awful things which, I’m telling you, are bad news.

J Max Whatsisface then goes on to demonstrate the “bias” in this latest survey. He presents two items as clearly “biased”:

Which statement comes closer to your own view, even if neither is exactly right?
 A. A good Latter-day Saint should obey the counsel of priesthood leaders without necessarily knowing why.
B. A good Latter-day Saint should first seek his or her own personal revelation as the motivation to obey.

He sees “bias” (a word he seems to have picked up from the internet) because “…the doctrinal truth combines the best of both positions.” From a strictly-believing LDS perspective, sure, principally because the Prophets have said so. However, the implication that this question should not be asked in this fashion is falls down in two areas.

First, numerous incidents in our history have highlighted the potential tension between these concepts (e.g., members being asked/ordered by their local leaders to canvas for Proposition 8 in California, or recent excommunications of members for advocating the ordination of women in the church). This is a question that quite nicely aims to get at Church members’ opinions about an issue many (note: especially if they have felt marginalized due to their views on gender issues) Mormons have felt personally. In fact,  most of Wilson’s objections seem to boil down to an annoyance that the survey authors have not phrased all the questions to accommodate strictly-conforming LDS responses to be dominant.

Second, this method is a reasonable method to test many kinds of hypotheses.  Wilson claims, “…because neither statement is right (by itself), forcing people to pick and choose is going to be un-interpretable, and will not represent people’s true perspectives on the issue.” (his emphasis), and he does his best to inoculate Church members against any future conclusions the researchers might make:

The survey asks elsewhere for people to self-identify themselves on a scale of conservative to liberal. I don’t know how people will answer this question (since both statements are incomplete). But if self-reported conservatives tend to choose response A, the study authors will in all likelihood conclude that conservative members are more likely to be in the “blind sheep” category, while liberal members are more likely to affirm an individuals right to seek personal revelation. Note: this would be an untrue conclusion. The question did not permit the nuances that the question itself admits are likely there. But they’ll openly trumpet it as verified by survey anyways.

And they will be right, in the way that sociology is right (which is to say, with uncertainty, a margin of error, and open to questions about methodology). This is a useful survey technique, though I don’t believe Wilson understands it. It’s called “forced choice.”

As an example, consider David Buss’s work on mating selection. He gave a forced-choice question (similar to this survey’s methods) to lots of undergrads. He asked them to choose which would be most upsetting in a romantic partner: sexual (but not emotional) infidelity, or emotional (but not sexual) infidelity. He found that women were more likely to choose the second option and men more likely to choose the first. This supported his theory about  mating strategies. Neither of those two options represented the full range of how people feel; almost nobody’s individual preferences was represented by a dichotomous interpretation of the averaged results; nevertheless, Buss learned something by using this method. The current gender issues survey will also tell its authors something (interesting, I think) by using this method.

So Wilson himself said it: if more conservative members are more likely to choose response A, then yes, logically, when forced to choose between the two alternatives (even if they don’t represent all the nuances of the issue), then the conclusion is justified. Wilson (and anyone else) is free to quibble with the methodology, as is any scientist. That’s the beauty of it. It’s open, it’s transparent (let’s hope it stays that way), and it’s available for questions. But the questions should be reasonable ones, such as “did the forced-choice method and the choice of statements really answer the questions you had?” rather than “Biased Survey!”

Wilson’s second example (apparently he only found two that seemed “biased” to him) is this (proofread by me to match the survey I see online):

Differences between of the roles of women and men in the LDS Church are:
A. Cultural
B. A mix between culture and doctrine
C. Doctrinal
D. Don’t know
E. Prefer not to respond

Predictably, Wilson claims that setting culture and doctrine opposite one another is a “false dichotomy.” I don’t have the time to look this up, but a LOT of our general authorities have invested fairly heavily in that dichotomy, contrasting our doctrine with our culture. So he’s kind of butting heads with the prophets on that statement. He suggests that God has inspired our cultural practices.

Let me just say that again: He implies that our cultural practices are inspired by God. Maybe I should put this at the top of this post-nobody-will-ever-read, because it’s clearly the biggest bombshell in his analysis.

As a Thinking, Reasoning Person, this statement seems insane to me. If we accept it at face value, then we enter a bizarre, repressive world where conformity is a virtue on par with obedience to God. Not only is it a sin to disobey the commandments, it’s also a sin to take the Sacrament with your left hand, take off your Sunday clothes after church, or dislike green Jell-O. Unless Wilson means that only some of our cultural practices are inspired by God, in which case I really hope he can tell me which ones. Because damn. And I mean damn, as in that’s what will happen to me for failing to conform on the key issues.

Wilson goes on to exhort his readers,

…we cannot and must not mistake every survey, or every social science study, as valid empirical evidence. The authors of studies and surveys have ideological biases that inform their question writing, data analysis, sampling, and study design.

This is a headdesk-obvious point, so the reason it is put there is to tell people to distrust this survey, and to distrust this evidence. Because of ideological biases. But, Mr. Wilson, ideological bias cuts every way possible. If you can write off the results of this survey because it was “pushed” by “liberal leaning groups” (whoah… liberal leaning? That’s almost as bad as actually liberal!), then isn’t your principle to disbelieve surveys created and distributed by groups with a clear ideology related to the content of the survey? That rules out all surveys by the church and by pretty much anyone except perhaps the most disinterested of academics. Actually, the claim of “bias due to ideology” is a convenient little arrow kept in one’s sling in case one can’t think of any substantive criticisms of research. By itself, it’s nearly meaningless, but by implication it can seem huge.

J Max Wilson makes the following statement in his post, no doubt to soothe the concerns of members who might be concerned about gender roles in the church:

The church does its own internal surveys to gauge member feelings on various topics, including inviting individual women and men to meet with them and discuss potential problems and propose changes. While the results are not public, there is no doubt that the church has pretty good information about the feelings of its members on a variety of subjects.

I am not terribly interested in private/secret assessments done by the church, since, if they exist, they are not available for discussion. However, I would be very interested to see what surveys the Church has done on issues like the membership’s perception of gender roles and gender-related issues. If anyone knows of any, let me know, please.