User:Kanda/ConfidenceInterval
How to take the Luck Factor out of your data[edit]
aka: how to make your and others' statistics objective
So you feel good-willed today, and think about contributing to the common knowledge by scientifically opening those 3000 Bags of Awesome Loot that you have been hoarding all month long. For good measure, you pay a financial tribute to Science and buy 2000 others, open your notebook and let the clickfest begin.
After all is said and done, you find yourself with 70 of the unanimously coveted Superior Sigils of Unanimous Covetness (well, they were when they came to be; now they dropped to a few silvers), among other stacks of random stuff. You feel ready to announce the whole (gw2-playing and reddit/wiki-aware) world that the chance of finding such a marvelous item is, indeed, 70/5000. Or 1.4%, as one can put it. Then a dawning realization puts your posting finger to a sudden freeze.
Is it true?
Well, you heard about the luck factor. You know not to trust anecdotical evidence. But hey, you opened 5000 bags. Is it not enough? Is it still possible that you have been so lucky that you got 7 where a less luck-blessed player would receive only 4? Or is your luck so ill that you should have gotten 10?
You could always put it as a line on the Wiki, of course. But the issue remains. Let's say that you did just that, and now the wiki triumphantly reads that out of 26'345 bags opened, 327 sigils were found (congratulation on your outstanding contribution, by the way). What does this 1.24% chance mean? 0.125? 0.10? 0.15? Something else?
Well, rejoice, for there is a way to know that with a quasi-certainty. Or at least, to know which range this 1.24% -or your 1.4%- actually indicates. Statistics go well beyond means and medians, so let us delve into the concept of confidence interval.
A confidence interval, simply put, is a value of uncertainty that can be used to take into account the luck factor in the result. It will turn a single number that is as subjective as one can be -1.4%, your own score- into a possible range that is as objective as one can get -here, between 1.07% and 1.73%. It depends on three simple parameters:
- Sample size: the more tries, the less range between the min and max values
- Standard deviation: the more the distribution of results strays from the ideal case where all results are the mean (which would mean that you find 0.014 sigil in each pack, yeah sure), the more range between these values
- Certainty factor (or risk factor): the chance that the whole range is erroneous; usually statisticians are happy with a 5% risk. This means that you will always have a chance for the actual value to be out of the found range: deal with it, or set the range to 0-100% for a really fail-proof (and completely useless) method.
Actually computing the confidence interval is another task. I'd recommend using a spreadsheet including such a function (Excel works fine for example, you even have the choice between several functions in the 2010 edition).
Example using the previous example and Excel (2003 and forward):
- Put a large column of 0s, the size of your sample (here, fill A1 to A5000 with 0s).
- Put as many 1s as the number of bags that contained a sigil (here, replace the 0s with 1s in A1 to A70). Whether you found them all in the first bags or they were finely disseminated among all of them is irrelevant.
- Choose an empty cell, and write '=CONFIDENCE(0.05,STDEV(A1:A5000),5000)' (for Excel 2010+, replace CONFIDENCE with CONFIDENCE.NORM and STDEV by STDEV.S, same otherwise).
- Your range is your score +/- the result of the previous step (here, 1.4% +/- 0.33%; as a bonus, the whole wiki gets 1.24 +/- 0.14%).
