Back to : Text, Bytes and Videotape Main Page

Text, Bytes and Videotape

Answers to Task 3

formula for entropy, reads entropy equals the sum from s=1 to s=r of minus p(s) times the log to base 2 of p(s)

Use the additional notes for Task 3 if you want help with understanding this formula.

The log function finds the power to which 10 has to be raised to give the value whose log you are finding, so log10 = 1, log100 = 2, log1000 = 3, log10 000 = 4, log1 = 0, log0.1 = -1, log0.01 = -2, log0.001 = -3, and so on. If a function is labelled just 'log', it means finding logs to the base of 10, ie. powers to which 10 has to be raised to give a certain value.
'log₂' means finding logs to the base of 2, ie. powers to which 2 has to be raised to give a certain value. So log₂2 = 1, log₂4 = 2, log₂8 = 3, log₂16 = 4, log₂32 = 5, log₂1 = 0, log₂1/2 = -1, log₂1/4 = -2, log₂1/8 = -3.
The entropy (or degree of uncertainty) of a coin toss is given by:

$\begin{displaymath}H = -(p(H) log_2p(H) + p(T) log_2p(T)) \end{displaymath}$

$\begin{displaymath}So H = -(1/2 x log_2(1/2) + 1/2 x log_2(1/2)) = -(1/2 x -1 + 1/2 x -1) = 1 \end{displaymath}$
$\begin{displaymath}H = -(p(A) log_2p(A) + p(B) log_2p(B) + p(C) log_2p(C) + p(D) log_2p(D) + p(E) log_2p(E)) \end{displaymath}$

$\begin{displaymath}So H = -(1/2 x log_2(1/2) + 4 x 1/8 x log_2(1/8) ) \end{displaymath}$

$\begin{displaymath}= -(1/2 x -1 + 4 x 1/8 x -3) = 2 \end{displaymath}$
Using a spreadsheet, with values of x spread at intervals of 0.01, gives log₂2/3 = -1.58 and log₂1/3 = -0.58, approximately.
Using the formula for changing base of a log:

$\begin{displaymath}x = log_2y = (log_{10} y)/(log_{10} 2) \end{displaymath}$

$\begin{displaymath}x = log_2(2/3) = (log_{10} (2/3))/(log_{10} 2) = -0.58496 to 5dp \end{displaymath}$

and

$\begin{displaymath}x = log_2(1/3) = (log_{10} (1/3))/(log_{10} 2) = --1.58496 to 5dp \end{displaymath}$

The entropy of the two outcomes, occurring with probability 2/3 and 1/3, is then given by:

$\begin{displaymath}H = -(p(A) log_2p(A) + p(B) log_2p(B)) \end{displaymath}$

$\begin{displaymath}H = -(2/3 x -1.58 + 1/3 x -0.58) = 1.25 approximately.\end{displaymath}$

If you thought about the extra item at the end of Task 3, which asked you to justify why log₂2/3 - log₂2/3 might be equal to 1, here is one way you could do this:

$\begin{displaymath}Let x = log_2(2/3) and y = log_2(1/3) \end{displaymath}$

$\begin{displaymath}Then 2^x = 2/3 and 2^y = 1/3 \end{displaymath}$

$\begin{displaymath}So 2^x/2^y = (2/3)/(1/3) \end{displaymath}$

$\begin{displaymath}and 2^{x-y} = 2 \end{displaymath}$

$\begin{displaymath}taking logs, log_22^{x-y} = log_22 \end{displaymath}$

$\begin{displaymath}and x - y = 1 as required \end{displaymath}$
Football odds

Wolverhampton Wanderers v Chelsea: Wolves win 5/1; draw 5/2; lose 8/15.
Manchester United v Arsenal : Man U win 6/5; draw 9/4; lose 21/10.

The odds for the Man Utd v Arsenal game are closer, so this game is more uncertain, and therefore has higher entropy.
Probability that Wolves win is 1/6; that they draw is 2/7; and that they lose is 15/23.
Probability that Man Utd win is 5/11; that they draw is 4/13; and that they lose is 10/31.
Sum of the probabilities is approximately 1.1 in each case. As you can see, bookmakers' odds aren't fair, since they have to include the bookmakers' profit margin.

$\begin{displaymath}H = -(p(W) log_2p(W) + p(D) log_2p(D) + p(L) log_2p(L)) \end{displaymath}$

Wolves v Chelsea:

$\begin{displaymath}H = -(1/6 x -2.585 + 2/7 x -1.807 + 15/23 x -0.617) = 1.35 approximately.\end{displaymath}$

Man U v Arsenal:

$\begin{displaymath}H = -(5/11 x -1.137 + 4/13 x -1.701 + 10/31 x --1.618) = 1.56 approximately.\end{displaymath}$

So the game between Manchester United and Arsenal is more uncertain, as you would expect.

Return to Olly's talk

Return to Text, Bytes, and Videotape homepage.

top of page

Back to : Text, Bytes and Videotape Main Page

Text, Bytes and Videotape

Answers to Task 3

Football odds