Yesterday’s NY Times reports on a dispute concerning the posting of the Rorschach Inkblots on Wikipedia. With all 10 inkblots posted, along with common answers, many psychologists argue the test has been compromised. Free information advocates argue that the test no longer had copyright protection, and therefore posting it is perfectly acceptable. It’s fitting that a debate about the grand-daddy of all projective tests — tests in which ambiguous stimuli provoke reactions that reveal aspects of a person’s mental state — should elicit this kind of polarized reaction.
I think most practicing professional psychologists would feel some degree of concern, knowing full well that they would not want their measures to be gamed, or to become so culturally exposed as to become irrelevant and invalid. However the advocates for posting the inkblots sound indifferent — one is quoted as laughing at the idea that a German publisher might be considering legal action, while the person who posted the Rorschach says he doesn’t care what experts think, he wants to be shown the actual damage caused by his actions.
At one point the article says that those opposed to the posting of the test feel that it is akin to posting a future version of the SAT. This is an iteresting point. Suppose we were discussing someone attempting to post next October’s SAT on Wikipedia. There would be no more laughing at the prospect of Educational Testing Services pressing legal action — Wikipedia would be doing everything in their power to absolve themselves. Presumably it would be an absolutely clear cut situation. However, there would be very little empirical evidence of the degree to which the SAT was compromised — the “no brainer” status would all be about the legal copyright. It seems like the free information advocates are failing to recognize that the hypothetical case of the compromised SAT and the actual case of the Rorschach are both worrisome to testing professionals because of the potential to undermine the integrity of the tests.
We at Criteria are also concerned about test security, as is the testing industry as a whole (Click here for a Wall Street Journal article related to test security). For some of our tests, like our neurocognitive tests, exposure isn’t really an issue. These are performance tests that show real time processing and responses. Other tests, such as our employment personality tests, are more like the Rorschach in that they do not have absolute correct and incorrect answers, although it is true that certain response profiles are considered more or less appropriate for certain jobs.
For special cases there is an interesting compromise between free information advocates and testing professionals. In some cases, it might be possible for the full range of testing materials to be freely available. Consider what ETS has done with the writing prompts for the GRE taken annually by half a million applicants to graduate schools. The prompts for which the students must write a short essay are posted online. There are hundreds of prompts, although on test day an applicant will only encounter one. If they have prepared answers to each and every one — well more power to them; they have shown a remarkable degree of perseverance which ought to count for something. If they have memorized answers from a website, their work will easily be recognized as generic (think about that the next time you play the lottery numbers from your fortune cookie, and you have to share the Powerball prize with 110 other people).
We at Criteria think that open source testing might also have a place in undergraduate education. As of now, multiple choice tests in college classes are difficult to keep secure. Too many sororities and fraternities have files with pirated versions of tests. One solution to this is to build out a universe of potential test items. Students could have access to the open source version with all the possible testing materials, and come the time for the final exam, the instructor could use a subset of the items for the test. We’re working on ways to make this possible…..we’ll keep you posted. We’re just saying that if there were 2000 Rorschach inkblots, having them up on Wikipedia wouldn’t be such a compromising event. And that guy from Saskatoon probably wouldn’t have bothered posting them in the first place, making it even less of an issue.