[PHI 3000] Peer Disagreement & Peer Review

[cross-posted at The Philosophers' Cocoon]

Roughly speaking, the problem of peer disagreement is the following:

You judge that p and an epistemic peer (i.e., someone who is just as competent and well-informed) judges that not-p. Should you stick to your guns and give more weight to your own judgment? Or should you give equal weight to both judgments, and so suspend judgment until further evidence comes to light?

On the one hand, sticking to your guns in the face of peer disagreement seems unreasonable and dogmatic. On the other hand, if giving equal weight applies across the board, then it seems that you should also suspend judgment about whether or not to give equal weight to incompatible judgments made by epistemic peers, which seems incoherent.

To sum up:

1. When A and B are epistemic peers, and A judges that p, whereas B judges that not-p, A (or B) can either stick to her guns or suspend judgment.
2. If A sticks to her guns, then that seems unreasonable and dogmatic (given that B is an epistemic peer).
3. If A suspends judgment, then A should also suspend judgment about whether to give equal weight to peers’ judgments, which seems incoherent.
4. (Therefore) A is in a position in which the options seem either unreasonable or incoherent.

Now, I think that the problem of peer disagreement raises interesting questions about peer review.

• Suppose that an epistemologist submits an epistemology paper to a journal that specializes in epistemology. The editor judges that the paper is worthy of consideration and sends it off to two referees (who are epistemologists, too). The reports come back with one “publish” verdict and another “reject” verdict. Should the editor give more weight to one verdict rather than another? If so, which one? Or should the editor give them equal weight? In which case, how would the editor decide whether to publish the paper or not?
• Similar questions can be raised even if the number of referees is odd. Suppose that an epistemologist submits an epistemology paper to a journal that specializes in epistemology. An epistemologist reviews the paper and decides on rejection. However, the author of the paper, who is an epistemic peer of the referee, obviously thinks that her paper is worthy of publication (otherwise, she wouldn’t have submitted it). So, should the editor give more weight to the author’s judgment or to the referee’s judgment? Or should the editor give the two judgments equal weight? If so, how would the editor decide whether to publish the paper or not?
• Similar questions arise at the level of the editor as well. Suppose that an epistemologist submits an epistemology paper to a journal that specializes in epistemology. The editor, who is an epistemologist, judges that the paper is not worthy of consideration. But the author obviously thinks that her paper is worthy of consideration (otherwise, she wouldn’t have submitted it). The editor and the author are epistemic peers. Should the editor give more weight to her own judgment? Or should she give equal weight to her judgment and the author's judgment, since the author is her epistemic peer? If so, how would the editor decide whether to consider the paper for publication or not?

By the way, a somewhat related set of questions arises with respect to expertise quite generally.