We Need to be Better With Anonymous Refereeing

I've been doing a lot of refereeing for conferences, journals, prizes and such and a recent discussion with some of my co-referees leads me to want to post here about anonymous refereeing.  My main aim here will be to list some of the ways authors screw up in anonymizing papers and also to raise a question or two about trickier points of making a paper truly anonymous.  But I want to state out front that I think all refereed conferences should require that papers be anonymized when they are submitted. I know that in a small profession we will often have some idea who wrote papers, but I think we still ought to do the best we can within that constraint. (Continued below the fold.)

From the papers I've seen even when anonymous submission is required, I've come to think that we as a profession are not being very good about telling our younger peers what it means to make a paper anonymous.  In particular, it isn't enough to take one's name off of the paper if there are other things in the paper which make it pretty easy to figure out the author. 

One such thing that is easy to miss is meta-data attached to the electronic file that doesn't print out, but that can be viewed if you look at the document electronically.  Since many journals have web submission interfaces that strip this information out and create a new document for authors it doesn't as often cause trouble there, though not all journals use such programs.  I don't know of very many conference organizers who have the time to check and remove such identifying information before sending the papers on to referees.  So it would be good if authors looked at document properties before sending the paper along and erased any that point to authorship.

I take it that everybody knows enough to remove the typically long footnote thanking people who have made suggestions about the paper, so that people don't figure out who the author is from the information about whom they thank.  But that goes just as much for thanking people one knows for particular suggestions in the body of the paper.  Typically brackets with the word 'omitted' or 'omitted to preserve anonymity' will be sufficient.  It is often very easy to figure out what department an author is affiliated with when all the people thanked are from the same place.  I also tend to think that papers that refer too much to familiar geography can do the same kind of signalling, and I sometimes suspect it is intentional, though I don't know that.  Intentional or not, I think you should leave it out until the paper is accepted at which point add it back in if you think it makes the paper better.

Harder issues arise with citing your own work.  If you are really an unknown 'I talk about this issue in [omitted]' is probably sufficient to keep referees from guessing your identity.  But when the work is widely known and the present paper makes the topic of that work obvious, I think it is better to omit the body of the note entirely and just replace it with brackets and language about omitting the note to preserve anonymity.  I've seen papers where people cite themselves in such a way that you can't tell it is a self-citation and I think that's OK.  Things get harder when the paper builds on your own past work and it is hard to talk about the new issue without rehearsing your own prior arguments.  I'm not sure exactly how to handle that.  (Talk about yourself as though you are a different person?)

If you tend to refer to yourself by name in examples, it might be worth doing a word seach for your name.  I've seen a paper recently where I think someone meant to anonymize the paper but left his/her name in an example that they forgot to redact.

I should stress that I think for most people it is to your advantage to have the paper be truly anonymous, even if you think you might benefit from readers knowing who you are or who your friends are.  When I referee the uncertainty about who wrote the papers is enough to keep me on my toes about being fair to it.  I don't want to find out later on that I've treated someone I know and admire unfairly.  Similarly, when I find out that a paper is by a person because it hasn't been anonymized well, it isn't obvious that I succeed in compensating as opposed to overcompensating. (If it is a journal you can inform the editor and they can decide whether to send the paper to someone else, but with conferences there is not always that option.)  And referees can get offended when a paper is not properly anonymized and think less well of it or the author.  So apart from all the good reasons to anonymize having to do with tacit bias and things of that sort (which I think are sufficient in themselves to warrant anonymous refereeing) authors have self-interested reasons to want to anonymize properly, or so it seems to me.

Since this posting sounds a bit like hectoring (perhaps because it is hectoring), I should admit that I have probably messed up myself in one of these ways in the past. I'd like to hear other ideas about what people should be careful of when they are trying to anonymize a paper submission.  And I'd also be open to arguments to the effect that it isn't a good idea to referee conference submissions anonymously.

 

Comments

[Robert Seddon]

Fun fact: after you remove your name from document metadata, Word may reinsert it if you save in a different format. (There's also the matter of change tracking, if you have that active.)

[Jussi Suikkanen]

Hi Mark

thanks - this is really useful, especially the instructions for how to make the paper anonymous. I completely agree about anonymous submissions for conferences. This strikes me as an excellent idea and it should be more widespread. I'm less worried about the metadata. I take it that this is something you necessarily see as a referee unless you look for it. So, I think there is an equal duty for the referees not to check the metadata. This seems to me to be an equivalent duty as the one not to google the article title before refereeing.

I have to say often not knowing who the author is is very difficult given that people present their papers in seminars and conferences and you kind of know who is working on what. Do you think one should agree to referee a paper if one knows who has authored it?

[Eric Wiland]

Nice post, Mark.

I'd also like to hear your thoughts about whether abstracts submitted to workshops should be anonymized for referees. (This has come up locally.) I am starting to think it's a good idea.

[Mark van Roojen]

Thanks Robert, Jussi and Eric!

Robert, I'm still using WordPerfect so I didn't know that.

And Jussi, I think you're right that the metadata won't normally cause a problem if you don't go looking for it. And you're doubly right that referees shouldn't be looking for it. I think there are ways that it can be noticed inadvertently. If, for example, one is wondering if a paper is over a word limit one might look at the paper's properties and it could show up there (at least in WP, I think you hit properties to do a word count - I have no idea how it goes with Word).

Jussi's comment raises an issue that I'd meant to ask about. I've heard people suggest that to avoid illicit title Googling by referees one should submit papers with a different title than one gives them on line. I've tended to think that was needless paranoia about referees, who should with minimal self-control be able to avoid searching for the papers they are refereeing. But of late I'm starting to wonder whether there might be some reason to do this for other reasons. As Jussi notes we often find out about papers from seminars and conferences, and that can happen even when one doesn't go to the particular conference if you happen to see the program. In addition various email lists and social media are giving some of us more and more information about who is putting papers on their websites or giving a paper on a given topic. So now I'm thinking the suggested practice isn't a totally crazy idea.

Eric, I meant what I suggested to apply also to abstract refereed conferences. I know that people worry that you can't tell enough from an abstract because abstracts are promissory notes, and promises are only as good as the person who makes them. If that's a worry one could consider asking for abstracts and papers and then turn to the paper if you don't trust the promises in the abstract.

Anyway, thanks to all for the feedback!