There’s an article out in an open-source journal (apparently that’s a thing) called PLOS One that caught my eye. It’s been reported in the Washington Post and The Atlantic among other places and the headlines appear shocking:
WaPo: The remarkably different answers men and women give when asked who’s the smartest in the class
The Atlantic: XY Bias: How Male Biology Students See Their Female Peers
Jezebel: Male Undergrads Automatically Overestimate Their Male Peers, Underestimate Women
I wanted to take a closer look at this article not because I’m worried about women in STEM but because it seems to challenge a piece of conventional wisdom around these parts. Namely, it appears to contradict the idea that men don’t have in-group bias towards other men the way women do with other women. So, let’s take a closer look. A couple things before we start. First, go read the damn thing yourself. Don’t take my word for anything. I’m just a guy on the internet. Second, I’m a guy on the internet who has only a passing familiarity with the sort of advanced statistics used in this study so my comments on the statistical analysis will be limited. Finally, the authors on this are a mile long. The conventional way to cite would be Grunspan et al. as Daniel Grunspan is the first author. However, from the media coverage I get the sense that he was just the guy with the PhD. The person doing the work seems to have been a graduate student named Sarah Eddy. I’m going to avoid the hornet’s nest and just say authors.
Let’s start with what this study claims and we’ll work backwards. Here’s what our authors claim to have shown in their conclusion:
“The patterns of uneven peer perceptions by gender shown in our student population suggest that future populations of academics may perpetuate the same gender stereotypes that have been illuminated among current faculty. This may not only be the case because the male students receiving high celebrity are reaffirmed in their abilities and are better able to advance through the STEM pipeline than women who do not receive this affirmation, but also because the existence of “celebrity” males and other individuals with distinction can impact and reaffirm the stereotypes held by others”
So how on earth do we reach this conclusion. Well, a few leaps and bounds. Let’s start with what they did. To give the authors credit there’s actually a lot to like about this study. The sampling is reasonably broad. They sampled multiple classes multiple times. The statistical analytical tool (with the caveats above) appears to have been well selected and properly used. They seem to have realized flaws with their methodology and improved upon it as the study progressed. However, they really shot themselves in the foot at the first hurdle and the study never recovers.
The idea here was to get students in a biology class to name students in their courses (not themselves) at various times during the course who they thought either understood the material or were going to do well (two different questions depending on the survey timing). The researchers also asked the professors to note which students were actively participating in class. They took these two data points and combined them with the final grades (in GPA) that the students received and sought to determine whether there was any bias in the student answers based on gender. What they claim to have found is that men were much more likely to think men were doing well while women sought similarly of men and women.
Again, that all sounds good but let’s go back. The biology course they selected was the second of three introductory courses in biology at the University of Washington. They claim to have chosen this course so that students would have a passing familiarity with their colleagues and thus be better able to name them. While that’s a fair point, it would have been prudent to include the previous course marks in their evaluation. Students earn reputations. If the word around the dorm is that kid x is a biology wizard and aced Bio 101, he’s going to get a lot of votes at the start of Bio 102. It’s at very least worthwhile to disprove the hypothesis. They had access to grade information. This to me is a glaring oversight. You’ll notice at theme here as we progress. This criticism doesn’t invalidate their study but it does make me question it.
Next, let’s talk about what information they actually collected. The researchers collected two very binary data points. Students were given a class list and basically asked “Is this person good at biology? Y/N”. Professors were asked “Does this student participate? Y/N”. These aren’t bad questions but they limit severely the analysis you should be doing. For instance, on the participation question the authors concede:
“Thus, a student who frequently offers an incorrect answer in class is considered equally outspoken as students who frequently offer the correct answers.”
That’s a problem. I’ve been in enough classes to know that a lot of really smart kids who get really good grades ask some of the dumbest questions. It’s how they learn. However, when you sit in the room with them all you’re thinking is “that person is stupid.” Thus, by making it a binary question your analysis is now biased with a whole bunch of junk data. Again, we don’t’ know that this impacts the gender bias shown in their results but it muddies the water significantly. As is too often the case in social science, the unrecognized variables are the problem here.
On the question the students answered there’s a different problem. The researchers do a statistical analysis to show the bias of males towards males but they then make this rather absurd claim:
“Another way to understand the magnitude of the gender bias is to compare its coefficient to that for class grade point average (GPA), our best proxy for actual mastery of course material scored on a 4 point scale. Averaged across the 11 surveys, females give a boost to fellow females relative to males that is equivalent to an increase in GPA of 0.040; i.e. they would be equally likely to nominate an outspoken female with a 3.00 and an outspoken male with a 3.04. On the other hand, males give a boost to fellow males that is equivalent to a GPA increase of 0.765; for an outspoken female to be nominated by males at the same level as an outspoken male her performance would need to be over three-quarters of a GPA point higher than the male’s. On this scale, the male nominators’ gender bias is 19 times the size of the female nominators’.”
This is just fiction and of course it’s the part that’s been picked up by all the papers. How on earth do you go from a binary “Is this person good at biology” question and turn it into a comparison within a GPA scale? You can’t do that. There’s no incremental evaluation being made here. Students weren’t asked to estimate their peers’ GPA’s. They weren’t asked to rank their peers. They were merely asked a yes/no question. They weren’t even given guidance on what is required for a yes response. Some students might perceive doing well as a 3.00 GPA others might set the bar at 3.60 some at 4.00. We don’t know because nobody asked.
The maddening thing is their own data contradicts this supposed finding. They discuss that the distribution for this study was surprisingly top heavy. Fewer students got a larger share of the yes responses than expected and more students got no yes responses than expected. They also include as part of their discussion specific scores for the top vote getters. Now, with this distribution, it would be next to impossible for that claim to be true if, as shown in their own figure, 13 out of the 14 top male vote getters (accounting for 199 of 207 votes) in 3 final surveys had GPA’s less than 0.765 from the maximum of 4.00 (the top mark given at UW). I understand that their findings are based on simulations derived from the data, and their might be some incongruence as a result but that just doesn’t pass the smell test.
Now, if we assume that this is just an anomaly, it further contradicts their conclusions. Remember, this piece is all about how men are magically buffeted by their “celebrity” status conferred upon them by other men. So, we should be looking, as feminists always are, at the top. The conclusions the authors reach are not about the people receiving one or two votes. They are about the big men in bio class. So, even if this GPA finding is statistically justifiable, it’s irrelevant to their conclusions. Let’s take a minute now to evaluate that conclusion. First, if as the authors claim, women’s perceptions are fairly accurate. The celebrity-ness of their male colleagues should have zero impact on them. They aren’t the ones overestimating the know-it-alls. They don’t underestimate their female peers either. So, how on earth would a person who has an accurate view that there are smart people of both sexes in the class (both men and women did well in this course), somehow believe that she can’t make it?
Moreover, they never demonstrate that this “affirmation” that the male students receive helps them. This isn’t a study that shows that people do better when people believe in them. There might be a study out there that does that, but this isn’t it. Furthermore, the suggestion that somehow this alleged bias has long term consequences is so far beyond of the scope of the study as to venture into the absurd. You would be equally justified in saying that men are more likely to be swindled by a fast talking salesman in the future. The only conclusion that this very flawed study has is that men are more likely to perceive other men as competent if they make some demonstration of that competence. That’s not exactly earth shattering.
5- As an aside, this is a really smart cost saving idea by the university. Dividing up the course as much as possible limits the number of students who are going to take up valuable lab time and resources in the second and third course.
6- Feminists obsessed with celebrity status. Shocking.