SRL Archive - Discussions & Debates

Is there really such thing as probability?

I've been wondering for a long time if mathematical probability is at all a viable concept… now you're probably wondering a one of three things, Ice is crazy, of course probability is viable, or wtf who gives a dam.

Lets start with the fundamentals of probability.

Consider this:

You go to flip a coin and it's got a 50% chance (1 out of 2) or 1:1 chance of being heads or tails.

Turns up heads.

You go to flip the coin again a second time. What is the probability that you will get heads a 2nd time? Math tells us it's 25% chance. But how this possible? The coin only has two sides… is it not still the same mathematical probability of 50% chance that you will receive heads? It's a two sided coin, shouldn't it be 50%?

Now consider this, what are the odds you will flip a coin and get heads 8 times in a row? I don't think it's 50% chance…. can anyone else see the flaw in mathematical probability so far?

Lets move on to the Monty hall problem.

In this problem we have 3 doors, and your prize is behind one of them. Your goal is to get the prize, so you are offered the choice of one of the three doors. You choose door 1. The game host opens door 3 and shows you it is not your price and asks if you would still like to stay with door number 1 or to switch to door number 2. Mathematically it says you switch doors and pick door number 2. Now consider this, the game host is fully aware of the monty hall problem, and so are you. So the game host could just as easily be tricking you into switching doors or double reverse physiology you into staying with the same doors, I would argue even though mathematically you have a 66% chance switching doors, in actuality you have a 50% chance because there are two doors and you either pick the wrong or the right one and there's no way of knowing anything for sure other than victory is uncertain.

The last thing I want to point out is a paradox.

Nuances of probability theory
or: "Why you must take a probability course in order to understand pattern recognition."
Conditioning is not the same as implication

The probability statement P(A | B) = p has a very different meaning from the logical statement "B implies A with certainty p". The logical statement means that whenever B is true then A is true with certainty p. This applies regardless of any other information we may have. In other words, it is modular. But the probability statement is not modular: it applies when the only thing we know is B. If anything else is known, e.g. C, than we must refer to P(A | B, C) instead. The only exception is when we can prove that C is conditionally independent of A given B, so that P(A | B, C) = P(A | B).

To illustrate why this is important, let

* A = "It rained last night"
* B = "My grass is wet"
* C = "The sprinkler was on last night"

Given only B, it is reasonable to conclude A. But if B is deduced from C, then it is not reasonable to conclude A.

This point was made eloquently by Pearl (p57). He used it to show that logic based on "certainty factors" is not an adequate replacement for probability theory.
Query sensitivity

In the rain/sprinkler problem, it seems obvious that we need to include C. But sometimes we drop relevant information without realizing it. Consider this example:

My neighbor has two children. Assuming that the gender of a child is like a coin flip, it is most likely, a priori, that my neighbor has one boy and one girl, with probability 1/2. The other possibilities—two boys or two girls—have probabilities 1/4 and 1/4.

Suppose I ask him whether he has any boys, and he says yes. What is the probability that one child is a girl? By the above reasoning, it is twice as likely for him to have one boy and one girl than two boys, so the odds are 2:1 which means the probability is 2/3. Bayes' rule will give the same result.

Suppose instead that I happen to see one of his children run by, and it is a boy. What is the probability that the other child is a girl? Observing the outcome of one coin has no affect on the other, so the answer should be 1/2. In fact that is what Bayes' rule says in this case. If you don't believe this, draw a tree describing the possible states of the world and the possible observations, along with the probabilities of each leaf. Condition on the event observed by setting all contradictory leaf probabilities to zero and renormalizing the nonzero leaves. The two cases have two different trees and thus two different answers.

This seems like a paradox because it seems that in both cases we could condition on the fact that "at least one child is a boy." But that is not correct; you must condition on the event actually observed, not its logical implications. In the first case, the event was "He said yes to my question." In the second case, the event was "One child appeared in front of me." The generating distribution is different for the two events. Probabilities reflect the number of possible ways an event can happen, like the number of roads to a town. Logical implications are further down the road and may be reached in more ways, through different towns. The different number of ways changes the probability.

This property of probability theory, which is different from logic, is discussed at length by Pearl (p58). In logic, it does not matter how a proposition was arrived at. But in probability, the query cannot be ignored. Here is another example, based on Pearl's:

Suppose you, a Bostonian, have entered the New Hampshire lottery along with 999 people from New Hampshire. The prize will be awarded to exactly one of the 1000 people. By sheer luck, you obtain a computer printout listing 998 participants; each name is marked "no prize", and yours is not among them. Should your chances of winning increase from 1/1000 to 1/2? Under normal circumstances, yes. But suppose while poring anxiously over the list you discover the query that produced it: "Print the names of any 998 New Hampshire residents who did not win." Since you are from Boston, the list could not possibly have had you on it. Thus it is completely irrelevant to you; your probability of winning is still 1/1000. (If you are not convinced, draw a tree as before.)

What if you just have raw facts without the query that generated them? Unless you can prove that the query is irrelevant, you should average over likely queries. The only time information can safely be omitted is when it is statistically independent from the quantity of interest. This is why independence diagrams are so important for efficient probabilistic computation.

The maximum entropy principle has been proposed as a way to incorporate facts without an associated query. If you are starting from a uniform distribution, the idea is to find a distribution consistent with the facts which has maximum entropy. (If you are starting from a non-uniform distribution, you find a distribution which has minimum cross-entropy from the current one.) It is a useful approximation, but only an approximation; you can do better by knowing something about the query. The maximum entropy principle essentially assumes that any state of the world consistent with the facts is equally likely to have produced them. In the two children example, the maximum entropy distribution given "at least one child is a boy" assigns 2/3 probability to the other being a girl, which is consistent with some but not all the different ways we might have arrived at that information.
Randomness is subjective
In English, randomness is an intrinsic property of an event. For example, "Boston weather is random." But in probability theory, randomness is a function of the observer; in particular, the amount of information you have. If you look outside and see water drops falling down, the weather is not random: you can be quite sure it is raining. You might still say that tomorrow's weather is random, but a meteorologist might not.

A basic assumption of probability theory is that given enough information, the status of any event can be reduced to a certainty. Randomness is therefore the absence of information, and therefore subjective. The probability distributions we assign to events always represent our own lack of information; someone with different information would assign different probabilities. Another way to say it is that all probabilities are conditional probabilities. In many derivations, these conditions are omitted for brevity, but it is important to remind oneself that the conditions are still there.

Statisticians are often asked, "Is that the real distribution?" There is no answer to such a question, because the question presupposes that randomness is intrinsic, when it is not. A more appropriate question would be, "Does that distribution follow from the data, your stated assumptions, and the axioms of probability theory?" Distributions encode the information available to the practitioner; nothing more.

Probability theory is not about absolute truth. It is about inference consistent with certain axioms. It cannot tell you how often an event will actually occur in practice; that is an objective quantity that you can only approximate, by acquiring more and more information about the random process.

A common, but flawed, rebuttal to the subjectivist argument is that the success of quantum physics "proves" that some things are intrinsically random. But quantum theory does not prove intrinsic randomness any more than the fact that coin flipping, despite being in the realm of Newton's laws, is best described statistically, or that random number algorithms, which are completely deterministic, may pass statistical tests. The convenience of a mental model does not prove that the model is correct.

This is a favorite topic of Edwin Jaynes, who focused especially on the subjectivity of entropy in physics. For example, see "Clearing up Mysteries - The Original Goal" and "Probability in Quantum Theory".

Statistical independence is not causal independence
Webster's dictionary defines x and y to be independent when

* x is not subject to control by y, and
* they do not both have a larger controlling entity

Notice that this definition of independence, which probabilists call "causal independence", is an intrinsic property of x and y. Statistical independence, on the other hand, is not intrinsic, and does not imply nor is implied by causal independence. Since statistical independence is determined by probabilities, it is subjective. People with different information, even when they are acting perfectly logically, may come to different conclusions about whether x and y are statistically independent.

For example, suppose a, b, and s are all independent Gaussian random variables. Define x = a + s and y = b + s. Given only this information, x and y are dependent (make sure you understand why this is true).

Suppose I now tell you the value of s. Conditional on this information, x and y are independent. Suppose I also tell you that the product xy is positive. This makes x and y dependent again. In addition to all this information, suppose I now tell you that both x and y are positive. This makes them independent again.

Improbable data does not imply a bad theory
Let's say you propose a statistical model of some phenomenon. If you collect some data which turns out to have low probability under the model, does that support or refute your model? The answer is neither. Data can only support one model vs. another; it can say nothing about one model in isolation. If the other model assigns higher probability to the data, then the data refutes your model. If the other model assigns lower probability, then the data supports your model. It does not matter what the actual probability value is, and it should not, because improbable data is a fact of life.

Unfortunately, orthodox model selection via p-values is based on flawed reasoning of this sort. Note that the p-value is not the probability of the hypothesis. The frequentists gave it the mysterious name "p-value" because it is a mysterious quantity that means very little objectively.

A bad argument that I found in the New York Times illustrates the problem. All of today's climate models, when started with the Earth's conditions a few million years ago, give low probability to the favorable conditions for life that we see today. Therefore, the article concludes, these models must be flawed, and we shouldn't believe what they say about global warming and the like. The problem is that we know from studies of the universe that favorable conditions for life are very rare. So following the logic in the article, the meteorologists on any planet must necessarily have bad models of climate.

Fulfilled predictions carry information
Suppose you have been hired to build a classifier to distinguish bowling balls from ping-pong balls by weight. Since bowling balls weigh more than ping-pong balls, the classifier can simply compare the weight to a threshold and say "ping-pong ball" if it is below the threshold and "bowling ball" if above. Even the lightest bowling ball is heavier than a ping-pong ball, so if the threshold is set right, 100% accuracy is possible. From prior knowledge you know that the true threshold is somewhere between 1 and 9, with equal probability. Given this information alone, your best guess is to use a threshold of 5.

To test the system, you place an object on the scale; it weighs 7 and so you classify it as a bowling ball. An expert on bowling balls inspects the object and lets you know that it is indeed a bowling ball. Does this give you any useful information? Yes, because it eliminates any threshold above 7. Together with your prior knowledge, this means that the true threshold is between 1 and 7, with equal probability. And so your best guess is now to use a threshold of 4.

This phenomenon suggests that the error-driven training procedure used for neural nets, where only erroneous predictions can alter the classifier, is incomplete. Error-driven training does not average over all classifiers that are consistent with the data, which is necessary for making optimal inferences. Newer techniques like the support vector machine and Bayes-point machine are not error driven and come closer to true probabilistic averaging.

The effect of fulfilled predictions can be even more extreme than described here. The optimal classifer can change in an arbitrary way, with new decision boundaries appearing or disappearing, when a prediction is confirmed. The set of consistent classifiers is a convex polytope in the space of classifiers. The optimal classifier is the polytope's center of mass. New information cuts away pieces of the polytope, thus moving the center of mass.

So in theory… probability is just something that is relative… like logic… but even less reliable than anyone has imagined.

I didn't read that whole thing, but I saw your comment on the coin being 25% the second flip.

The thing is, if you are going to flip a coin twice, there is a 25% chance of it coming up heads twice. Before the 2 flips, would you say that it is more likely to come up heads both times? You probably wouldn't bet money on this. However, after the first flip is made, it is a 50% chance again.

I'm trying to put this the right way. Lets say, to make it more extreme, 10 flips are going to come up heads. Would you say that generally half the time (50% chance), they will all come up heads? Probably not. Each individual flip has a 50% chance, but combined, it is a less percent chance.

Every coin flip has a 50% chance of coming up heads, and 50% chance for tails. Looking at a certain amount of flips from the future though, we can determine the probabability. Probablility is simply the most accurate and consistent method of guessing. If I had a coin and we were going to flip it 2 times, would you rather bet that it was going to come up heads twice, or that it would be any other combination? It is an obvious decision, because the likelihood of it coming up heads twice is less likely.

It's really hard to explain… Looking at the 2 flips from the future, we can determine the probablility is less than 50%. The second flip does not actually have a 25% chance of coming up heads or tails, it's still 50 50.

program New;
var i: Integer;
begin
  for i := 1 to 20 do
  begin
    if(random(2) = 0) and (random(2) = 0)then
      Writeln('Came up heads both times');
  end;
end.

Try running that. I ran it 3 times, and it came up writelning 5 the first time, 5 the second time, and 4 the third.

Probability assumes absolute perfection.

From: IceFire908

Now consider this, what are the odds you will flip a coin and get heads 8 times in a row? I don't think it's 50% chance…. can anyone else see the flaw in mathematical probability so far?

No it's not 50% because you said 'in a row'. When you add those three little words it changes the question, changes the calculations, and changes the answer.

From: IceFire908

Lets move on to the Monty hall problem.

In this problem we have 3 doors, and your prize is behind one of them. Your goal is to get the prize, so you are offered the choice of one of the three doors. You choose door 1. The game host opens door 3 and shows you it is not your price and asks if you would still like to stay with door number 1 or to switch to door number 2. Mathematically it says you switch doors and pick door number 2. Now consider this, the game host is fully aware of the monty hall problem, and so are you. So the game host could just as easily be tricking you into switching doors or double reverse physiology you into staying with the same doors, I would argue even though mathematically you have a 66% chance switching doors, in actuality you have a 50% chance because there are two doors and you either pick the wrong or the right one and there's no way of knowing anything for sure other than victory is uncertain.

Again, you have to use accurate wording. You are not just picking one of two choices. You are reversing a choice that was already made, or keeping that choice. Sometimes, those two types of wording lead to the same calculations, and you would be right with 50%. Those situations are when the choice that was already made is just as likely to be made as the one that wasn't. However this is not one of those situations, because the choice that was already made was more likely to be wrong than right, so reversing is more likely to be right than wrong. You don't know anything for sure, and you can still lose when switching, but you do know that your original choice was probably a goat, because it was 2 out of 3, so knowing that information, you do have a better chance than a shot in the dark.

From: R0b0t1
Probability assumes absolute perfection.

Beautifully put, sir.

The way that you're thinking of, Ice, is in the future probabilities. You were saying that there is a 50% chance even on the 10th flip, and although this is true, you can only say that DURING the 10th flip. The odds of a coin landing on heads 10 times in a row are much less than 50% because you can't see into the future.

If you wanna know what is the chance to get the next "n" tosses Heads, the propability is (0.5^n)*100%.
Examples:

What is the chance of getting 8 Tails in a row?
n = 8
(0.5^8)*100% = ((5^8)/(10^8))*100%
=0.00390625 *100% (propability can also be given as a decimal, not percentage)
= 0.390625% (percentage just looks neat)
Answer: 0.40%

What is the chance of getting at least 1 Heads on 8 tosses?
Propability 1 means 100% sure.
What is the opposite of "at least 1 Heads?" It's no "Heads at all", ofcourse.
p(no Heads at all) means the propability to get 0 Heads on 8 tosses.

p(at least 1 Heads) = 1 - p(no Heads at all) [formula 1]
p(no Heads at all) = p(8 Tails in a row) (see former example)
Let's add this all to the [formula 1]

p(at least 1 Heads) = (1 - 0.00390625)*100%
p(at least 1 Heads) = 0.99609375*100%
p(at least 1 Heads) = 99.6%
Answer: 99.6%


I love calculating propabilities. It's very interesting, because 8 tosses is not that much but see how big/small those chances are? The most facinating thing is that even if you toss the coin a billion times, it doesn't prove anything :)

Damn you guys are good at math ~_~

You should see calculus probabilities… I haven't done it in a couple years but it gets ridiculous. Like the probability of getting A, K, Q, J & 10 of the same suit dealt to you and such. But just getting them is one thing, you can also do probability of getting them dealt to you in a certain order etc.

It all has to do with possible options.

If you flip a coin you can get

heads
tails
(1/2)

If you flip a coin twice, you can get

heads + heads
tails + tails
heads + tails
tails + heads

(1/2 * 1/2 = (1/2)^2 = 1/4)

And it goes on like that.

program New;
var i, c, h, t: Integer;
begin
  Changereportwidth(600);
  for i := 1 to 10000 do
  begin
    c := random(2);
    if c = 0 then inc(h);
    if c = 1 then inc(t);
    wait(5);
    ClearReport;
    AddToReport('Heads := ' + IntToStr(h));
    AddToReport('Tails := ' + IntToStr(t));
  end;
end.

[php]
Heads := 4995
Tails := 5005
[/php]

I know it doesnt solve the probability issue, maybe I will add some percentages and such and make it a neato coin flipper script.

From: 3Garrett3

program New;
var i, c, h, t: Integer;
begin
  Changereportwidth(600);
  for i := 1 to 10000 do
  begin
    c := random(2);
    if c = 0 then inc(h);
    if c = 1 then inc(t);
    wait(5);
    ClearReport;
    AddToReport('Heads := ' + IntToStr(h));
    AddToReport('Tails := ' + IntToStr(t));
  end;
end.

[php]
Heads := 4995
Tails := 5005
[/php]

I know it doesnt solve the probability issue, maybe I will add some percentages and such and make it a neato coin flipper script.

Your relying on the random function that is programed not exactly "real" if you think about it. The universe isn't programed.

From: Narcle
Your relying on the random function that is programed not exactly "real" if you think about it. The universe isn't programed.

I know. I thought it would be neat though. Random still does the same thing as pretending to flip a coin in probability.

The function is seeded with the time, that's just as good.


But, like I said: Probability assumes absolute perfection. You can not assume it will happen certain ways.

From: JAD
I didn't read that whole thing, but I saw your comment on the coin being 25% the second flip.

The thing is, if you are going to flip a coin twice, there is a 25% chance of it coming up heads twice. Before the 2 flips, would you say that it is more likely to come up heads both times? You probably wouldn't bet money on this. However, after the first flip is made, it is a 50% chance again.

I'm trying to put this the right way. Lets say, to make it more extreme, 10 flips are going to come up heads. Would you say that generally half the time (50% chance), they will all come up heads? Probably not. Each individual flip has a 50% chance, but combined, it is a less percent chance.

Every coin flip has a 50% chance of coming up heads, and 50% chance for tails. Looking at a certain amount of flips from the future though, we can determine the probabability. Probablility is simply the most accurate and consistent method of guessing. If I had a coin and we were going to flip it 2 times, would you rather bet that it was going to come up heads twice, or that it would be any other combination? It is an obvious decision, because the likelihood of it coming up heads twice is less likely.

It's really hard to explain… Looking at the 2 flips from the future, we can determine the probablility is less than 50%. The second flip does not actually have a 25% chance of coming up heads or tails, it's still 50 50.

program New;
var i: Integer;
begin
  for i := 1 to 20 do
  begin
    if(random(2) = 0) and (random(2) = 0) then
      Writeln('Came up heads both times');
  end;
end.

Try running that. I ran it 3 times, and it came up writelning 5 the first time, 5 the second time, and 4 the third.

May I suggest a bit of a code modification?

I would change:

if (random(2) = 0) and (random(2) = 0) then

to:

if (RBool = 0) and (RBool = 0) then

That's just how I would do it :)

~ Craig` aka. Mathematics.

flipping a coin is not 50/50


its more like 49.95/49.95

the .10 goes to the chance that it can land on its side standing stright up

[php]
Heads := 499311
Tails := 500689
Longest in a row is 29 Heads
Percents := 50% Heads, 50% Tails
Percent chance of having the longest in a row := 0.0028999971000029%
Took 4 Hours, 11 Minutes and 12 Seconds
[/php]

Just left that going on for 1 000 000 coin flips. The probability of getting 29 heads in a row is 0.0029%. :p.

It took a long time too XD