I was a reviewer on two of the manuscripts discussed recently in Simine Vazire’s blog by guest author Katie Corker (on behalf of the six women authors).
(What I say below won’t make sense unless you have read that first.)
My review of Ase’s Innes-Ker’s piece is below. It was early March 2017. Note: I had previously e-mailed with PoPS Editor Sternberg about making sure more voices were heard in this second round of the symposium on scholarly merit. Also, I was very conscious that the word limit for these papers was much shorter than for the original contributions. Thus, I believed that expectations for these papers should be different than for the original submissions. (You’ll see more about how I was influenced by that problem in my second review at the bottom.)
(Hmmm…. I can’t believe that I was Reviewer 2 both times.)
Comments to the Author
I think that this manuscript offers some useful / different perspectives on the AIFY symposium. As I’ve said elsewhere, I wish that the first symposium had been about merit (the alleged point) rather than about fame (what many authors addressed), but with that background I believe it is appropriate (and even necessary) to directly address problems of fame as a metric.
This manuscript does that, although it is somewhat choppy in places and not all arguments are as well supported as they could be. It relies on some philosophy of science arguments, which adds a nice angle to the discussion. I noticed that the author is a European non-native English speaker, so some turns of phrase (and punctuation) seem odd but could be fixed easily.
I like the intro and ending with the connections to Stapel and his craving for fame.
And I like that the manuscript connects back to papers from the first AIFY symposium and says that “Am I Famous Yet” is the wrong question to ask. The manuscript’s answer to what is the right question to ask appears in the following sentences. Unfortunately, the focus / language there is not consistent with the rest of the paper; they need to be made more consistent to hold the argument together across the entire paper.
I like the connection to Merton – though I suspect he did not have the sort of data that is now available on the distribution of “fame”. (So, did he “report” or “surmise”? Would some modern metascience statistics be useful there?)
The Salganic & Watts paper brings up an interesting issue — although I think that the analogy to scientific publishing needs to be made clearer. We have peer review, we have journals of different quality, and the initial “rankings” of papers probably does carry some merit. But I do agree with the overarching point that sometimes a paper becomes the paper to cite for proposition X – whether or not it actually supports it. (E.g., Nisbett & Wilson for anything having to do with not knowing one’s own preferences.) This section of the manuscript is jumpy – from the S&W study, to the critique of the h-index, to the Srull & Wyer citation investigation. (And it seems like there is something missing between the first and second sentences in “The problem of metrics”.) But, again, I do like the big point and totally agree (and have written about) the problem of how we don’t keep track of what we cite papers FOR; we just keep track of the citation count. And I like the penultimate sentence of that section – about how rewarding frequency of publication over quality of individual papers can lead to poor scientific practices (some of which should be explained in a bit more detail to the readers).
I like the connection to Hull and the Smaldino/McElreath paper. I think that those two references belong more tightly woven together as being about the evolution of science and its practices. Then that leads into the importance of community. Community is, of course, hugely important to the vetting of science, not just in the peer review process but also in the replication, falsification, and theory-advancement processes. And I agree that for a long while there had been a real problem with the ability of scientists to “expose these hypotheses to severe testing” – and to get such results published – until recently.
On the other hand, although I like the argument about the importance of community, I’m not quite sure how collaborations per se are important. Cooperation (including researchers sharing methods and data) and competition (alternative theories) = of course, but I’m not sure about “collaboration” (unless that word is being used differently from how I’m interpreting it). Hull’s “social churning” – that’s a good phrase.
So, I have trouble following the thread at the bottom of p. 5 – middle of p. 6. I do like the sentences about how the ideas need to be stress-tested not the scientists. But some of the surrounding bits don’t hang together. E.g., “other ways scientists contribute” would be a great theme for another paper but is not / cannot be fleshed out here.
In short, good ideas, but the argument needs to be more pointed and better stitched together.
Isn’t Bowie the “first author” of Fame?
I always sign my reviews,
Here is my review for Fernanda Fernanda’s manuscript. This was a couple of weeks later after I had seen a few more submissions. I saw that several important themes were emerging across papers and, in my view, were not being appropriately appreciated by other reviewers or the editor.
Comments to the Author
Review of PPS-17-136
At this point, I have read about half a dozen of the submissions for this second round of essays on merit. (Some as a reviewer, some as a friend.) There are many recurring themes; thus, a large part of the Action Editor’s job is going to be to select manuscripts that represent the diversity of common viewpoints that were not represented in the initial symposium.
This manuscript covers some ground covered by the others, and has some of the flaws of the others, but it also stands out for several reasons.
1. Balance –
This manuscript strikes a nice balance between adoring fame and demeaning fame. It describes how fame might (and should) rightfully emerge from doing both good work and good (useful to the field) deeds.
It also strikes a nice balance between engaging with (and citing) some of the papers in the initial symposium and moving beyond them.
Relatedly: I like the distinction on p 4 between the two questions of asking how someone became famous versus what one must do to become famous. And I like the points about fame as a potentially problematic heuristic (like availability).
2. My favorite part of this manuscript is the description on p. 6 of the “two particularly impressive scholars” in the distinguished speaker series. I think it nicely captures features of exemplary scientists – including how they challenged entrenched views.
I like the term “infrastructure” to describe all the “other tasks” we do to keep the field running.
Note: However, I don’t think that the author makes the best use of these examples. In the lead in to the description, she mentions “merit and quality”. After the description she talks about fame and “reputations.” And the next paragraph starts by discussing the bases of fame. I think this set of reflections could be made more coherent to really showcase the point of the examples.
I like the personal voice of this manuscript.
I worry, thought, that the manuscript might read a bit “cognitive” – e.g., prizes if they “have contributed more than most to uncovering the nature of psychological processes”; also, among the qualities of good science on pp. 6-7 it doesn’t mention doing work that could be applied, although work being “useful” is mentioned in the first sentence of the conclusion.
1. The manuscript suffers from a common weakness of all the second-round manuscripts: it makes a lot of claims without data supporting them. I believe that is a common problem due to (a) the authors wanting to make a lot of points (b) within the constraints of a tight word limit. The manuscript often appeals to how we all know people who… (e.g., bottom of p. 3 – do good work but not known and vice versa). And it makes claims about the correlations between merit and fame that might/might not be accurate.
2. The author should re-think the abstract. I don’t think focuses on the core messages of this manuscript (and, instead, mentions a lot of things that the manuscript deals with tangentially). It also seems long for such a short paper.
3. Although I generally enjoyed the writing style, I think the manuscript reads a bit “flabby”. There are sentences that could use fewer words and a couple of redundancies that could be cut. I think that would help maintain the focus of the paper.
E.g., top of p. 6 – Let me now return to a point that I made at the beginning of this article = … return to an earlier point…
Also, if there are going to be papers that focus on how the recent-past incentives have gotten us into the current replication mess, the paragraph on “The Dangers of Fame” – or at least the part re: incentives — easily could be removed from this manuscript.
I always sign my reviews,