Bayes Rule: It's Not Just a Good Idea--It's the Law!
Andrew Gelman writes: Hidden dangers of noninformative priors:
The simplest example yet, and my new favorite: we assign a flat noninformative prior to a continuous parameter θ. We now observe data, y ~ N(θ,1), and the observation is y=1. This is of course completely consistent with being pure noise, but the posterior probability is 84% that theta>0. I don’t believe that 84%. I think (in general) that it is too high.
But that is--virtually by definition--because you do not believe your flat noninformative prior: you don't think θ is as likely to be 10,000,000 as it is to be -1. And you probably don't think that you know that the variance is 1 with certainty either.
Start with the informative prior that θ ~ N(0,1). Then you pick one y ~ N(θ,1), and find y(1) = 1. Then your posterior is θ ~ N(0.5,0.5), yes? Your y(1) = 1 could have been all noise and θ ≤ 0 could be true, but it probably isn't. And then simply relax your prior...
