On detection of hoaxes

| Comments (6) |
Stanley Fish riffs off the Goldstein Wine Spectator incident to muster a defense of Social Text's being taken in by the Sokal Hoax. For those of you who haven't heard about the Wine Spectator incident, Robin Goldstein invented a fake restaurant with a fake wine list and submitted it to Wine Spectator (along with a $250 fee) and received the "Wine Spectator Award of Excellence". Wine Spectator's self-defense is here. Fundamentally, there are two criticisms one might make of Wine Spectator's process in granting this award:1
  1. That their process relies completely on the claims made by the restaurant owner with no verification.
  2. That the claims made by the owner represent such a weak restaurant that to give it an "award of excellence" implies that the award is next to meaningless.

Goldstein seems to think that the former criticism is more damning, but I'm not so sure. It's certainly true that one can't review the food at a restaurant without tasting it, but it's not like you need to taste the particular wine in my cellar to know if I have a good wine selection: you know what wines are good and what are bad and barring mishandling and statistical variation, if you have a list of the wines I have, you've got a pretty good idea of what the quality is like. Now, it's true that I could be lying about what I have in my cellar, but are we really expecting WS to send around a team of experts to take inventory, verify that I don't have forged wine, and go through my books to make sure I haven't just borrowed better wine from a helpful restaurant down the street run by my brother. That all seems improbable. Obviously, there are limits to the kinds of fraud you can detect and it's not clear to me that given WS's threat model, actually going to the restaurant, having a meal, and maybe looking in the wine cellar would tell their readers much more about the quality of the wine. It's not like getting an award for a restaurant that doesn't exist is useful for defrauding your diners, after all—you pretty much need to actually have a restaurant in order to take their money. [Yes, yes, I could charge a reservation fee, but then people would do credit card chargebacks, which makes Visa unhappy with me.]

On the other hand, it seems quite clear that Wine Spectator can in principle examine a provided wine list and determine whether the wines are good or not. I don't drink wine, so I'm not equipped to come to any conclusion about the quality of the list—as far as I can make out the regular list was pedestrian but the reserve list was all wines that WS had panned. Presumably, if the wine list consisted solely of Charles Shaw and Thunderbird and WS still granted the award, we'd come to the conclusion that the award was meaningless, but as things stand, this doesn't seem to be a pure win for either Goldstein or WS.

This brings us to Fish's article. Fish writes:

Asked what the success of the hoax perpetrated on his magazine demonstrated, Matthews replied, "It has now been demonstrated that an elaborate hoax can deceive Wine Spectator."

The key word is "elaborate"; it speaks to the care with which the intention to deceive is implemented. Sokal (and those he consulted) interwove references to the theorems and experiments of famous scientists with long sentences larded with postmodern jargon, all stitched together by "therefores" that did not hold up in the face of rigorous scrutiny. The question -- one that applies to the Wine Spectator controversy -- is how rigorous a scrutiny should editors of journals and magazines be expected to conduct? In this case, the question is complicated by the fact that one of the editors of the journal Sokal deceived was a colleague of his at N.Y.U., albeit from another department. If someone down the hall or in the next building is sending you something to consider, you don't start by wondering if the submission is on the up and up; like the editors of Wine Spectator, but with even more justification, you might assume that what you have before you is bona fide.

Once again, we're faced with two possible claims, namely:

  1. The editors should have verified that Sokal wasn't lying.
  2. The editors should have determined that the material Sokal submitted was in and of itself bogus.

Obviously, it's true that the editors have no practical way of verifying that paper submissions aren't completely fraudulent. This is especially true of data-based papers where the researchers could just be making up their data. So, I sort of agree with the observation that it was reasonable for the editors to assume Sokal was acting in good faith.

However, if you're going to have a review process, part of that process surely is to try to assess whether or not the paper is at least superficially plausible/internally consistent. I've reviewed papers for conferences and journals and this is exactly what I try to do and expect my fellow referees/PC members to do as well. Otherwise, why bother to have a review process? While I haven't read the Sokal paper, my understanding is that Sokal's claim is that the paper was transparently bogus to someone with any understanding of physics.

Blackburn recalls his own experience as an editor of the journal Mind. He imagines himself receiving a paper from a "well-regarded historian" who claimed that certain issues in Thomas Hobbes's political philosophy could be clarified by "various facts about Hobbes's political experiences in Venice." He would have been able, he says, to assess the political philosophy part of the paper himself, but I "might well have taken Hobbes's presence in Venice as given" on the assumption that any credentialed historian "would not have developed the point if he hadn't gotten that bit right." And, he adds, "I would not have had the history refereed, even if I had known whom to approach."

The reason he wouldn't have thought to have the history refereed is that it was being offered to him by an historian, and was, in effect, already refereed. And even if he had sent the paper to another historian, he would have ended up, he explains, "with two things to judge rather than one," and with no more competence to judge them than he had in the first place. That is, it can't be the case that when you receive a submission from a reputable expert you check everything down to the ground, do all the basic research yourself, something you couldn't do anyway unless you were an expert on just about everything. Sooner or later you would have to rely on the judgment of some learned others. Sooner or later you would be in the position of the Social Text editors who presumed that, at least with respect to the physics part of Sokal's essay, the professional physicist knew what he was talking about. After all, Blackburn concludes - and, remember, he by and large agrees with Sokal's critique of postmodern thought - "you do trust academics to get their own subject right."

I'm not sure that this example is really that helpful. It's certainly true that you can't expect reviewers to go back to ground zero, but to the extent to which the arguments/results in the paper depend on facts from some other field that aren't represented as being well established in that field—and yes to some extent you need to rely on the author to represent that accurately—then you do need to find a reviewer to assess it. Obviously, this is a judgement call on the editor's part, but it's the editor's job to make precisely such judgements.

The argument that "Sooner or later you would have to rely on the judgment of some learned others" seems particularly odd. Editors and program chairs routinely have to decide whether to accept papers that they don't really understand themselves. That's why they send them out for multiple reviews and use those reviews as the basis for their decisions. Obviously, that's not a perfect system: the reviewers can screw up; the editors might not understand the reviews, etc. However, I don't think it's true that just because a paper contains material on a topic that the editor doesn't understand he has to throw up his hands and accept it. And if he can't find anyone to assess that portion, he should probably be asking whether the author has the right venue.2

1 These criticisms are coupled with the suggestion that because WS charges a fee for listing (and encourages you to advertise) that WS does not exactly have an incentive to be selective. I think that's a separable issue, however.

2 Yes, I know that the Social Text wasn't peer reviewed at the time. But I don't see how that absolves the editor of all responsibility to determine whether the paper is meaningful.

6 Comments

It's worth reading the actual Sokal paper. I disagree with Fish's characterization: it shouldn't have taken "rigorous scrutiny" or an advanced science degree to become suspicious after the first paragraph where Sokal rejects the idea of a physical world and science's ability to learn from it, followed immediately by a claim that physical reality itself is a social construct. It wouldn't be unreasonable to keep reading and see where the guy down the hall is going with it (NYU certainly could have a brilliant, world-changing physicist upsetting previous theories) but any serious editor should have realized that more than a cursory review was necessary - and even in the pre-Google era things like morphogenetic fields or psychoanalysis of AIDS should have merited at least a phone call or trip the library.

The appropriate XKCD cartoon for this story is this one.

That paper is hilarious no matter how many times you read it.

Note 105:
Just as liberal feminists are frequently content with a minimal agenda of legal and social equality for women and ``pro-choice'', so liberal (and even some socialist) mathematicians are often content to work within the hegemonic Zermelo-Fraenkel framework (which, reflecting its nineteenth-century liberal origins, already incorporates the axiom of equality) supplemented only by the axiom of choice. But this framework is grossly insufficient for a liberatory mathematics, as was proven long ago by Cohen (1966).

Actually, because it is ALL wines that the spetator panned in the past, for very good reasons, the reserve list (which is the one which should be checked first for such an award) really IS egg on the face of the Spectator and its "award of excellence".

Its not like the spectator doesn't have an archive of previously published material, and the wines themeselves didn't just score low, many were downright BAD, as in "only fit for the poreclan decanter in the bathroom".

Chris, the Sokal paper references a huge body of work all of which states that the physical world is a social construct, and other similar looniness. The existence of such garbage in journals, and the earnestness with which it's taken, really ought to be far more scandalous than the fact that they could be easily taken by mischaracterizations of hard science concepts.

I'd be surprised if the vast majority of restaurant patrons didn't, on seeing a WS award certificate posted at the restaurant's doorway or mention of the award in a restaurant ad, assume that WS had conducted at least a single spot-check to verify that the wine list offered to random customers met the award's standards. Otherwise, why wouldn't restaurants simply send in a wine list copied from some highly regarded restaurant, earn the award, and proclaim it an authoritative (and thus revenue-enhancing) endorsement of whatever crap they sell?

The answer, it turns out, is that they don't have to--WS will sell them the award even if they freely admit to WS that they're selling crap. And that's the point of the Goldstein hoax--not that Goldstein somehow managed to get an award for a fake restaurant, but rather that WS' standards were such that Goldstein's fake restaurant met them just as easily as any real one.

The Sokal affair is similar. Most outsiders would likely assume that an established academic journal like "Social Text" would require a fair amount of intellectual rigor and methodological care from the papers it publishes. Otherwise, why wouldn't academics submit papers based on fundamentally flawed ideas, knowing that they stood as good a chance of being accepted as more meritorious work?

The answer, it turns out, is that they don't have to--they need no ideas at all, flawed or otherwise, in order to get published in "Social Text", as long as they can superficially mimic postmodernist style. That's the whole point of the Sokal hoax--not that Sokal managed to get a meaningless parody published as a serious paper in "Social Text", but rather that "Social Text"'s standards are such that his meaningless parody met them just as easily as any seriously-intended submission.

(Disclaimer: I don't drink wine, either, and moreover I strongly suspect that the difference between "Social Text" and other academic publications is merely one of degree, rather than kind.)

Leave a comment