David Leonhardt has a piece in New York Times about prediction markets: "When the Crowd Isn't Wise".
His main argument is that the odds at the online prediction market Intrade was 75 percent for the unconstitutionality of Obamacare, while the Supreme Court ruled that it was constitutional. Leonhardt thinks that "The market — the wisdom of crowds — turned out to be wrong." But is he right? No! The 75% means that 3 out of 4 cases with that percentage should occure, while the 25% rest should not occure. If you have 100 cases of predictions each with 75% then around 75 of them should occure. If 100 of the cases occured when the prediction market had rated them at 75%, then the prediction market would be wrong! From a single sample you cannot make the argument that the prediction market is wrong. The 0.05 maniacs would laugh at a P-value of 0.25. If you want to test prediction markets you need several predictions and their outcomes. That was the idea that David Pennock used in our evaluation of Hollyword Stock Exchange and the Foresight Exchange, see the homepage of the study. The idea is to "bucket" the predictions so you, e.g., count the predictions in the interval 60%-70% that occure. In this case around 65% of them should occure.↧