Text, Bytes and Videotape

Task 3

In Olly's talk, you were introduced to Shannon's formula for entropy:

The first thing you need to do in this task is to make some sense of this formula. Use the additional notes for Task 3 if you want help with this. Then start by working out what the log term means.

1. Use your calculator to find out what the following are:
   log10, log100, log1000, log10,000, log1, log0.1, log0.01, log0.001.
   What do you think the 'log' function means?



2. Now use your answer to question 1 to think about what 'log₂' means. You can't use your calculator for this question, as it does not have logs to the base of 2, but you should be able to work these out if you compare what log₂ means with what log₁₀ means:
   log₂2, log₂4, log₂8, log₂16, log₂32, log₂1, log₂1/2, log₂1/4, log₂1/8.



3. Use what you have learnt about log₂ to work out the entropy (or degree of uncertainty) of a coin toss. Remember there are just two outcomes, heads and tails, with a probability of 1/2 each. So the formula becomes
   
   
   
   




4. The next part of this task is to work out the entropy for the code you worked on earlier where the letters A, B, C, D, E occur with probabilities p(A) = 1/2, p(B) = 1/8, p(C) = 1/8, p(D) = 1/8 and p(E) = 1/8.



5. Suppose two outcomes occur with probability 2/3 and 1/3. What is the entropy of this situation?
   With the two examples you have worked through so far, it was possible to work out the log₂ terms exactly, but this is not possible in this example. There are two ways you might go about working about log₂2/3 and log₂1/3, one exact and one using a spreadsheet. Go to the additional notes for help with these methods.




6. Football odds*

   Consider two footbal games, where the bookmakers offer the following odds.
   Wolverhampton Wanderers v Chelsea: Wolves win 5/1; draw 5/2; lose 8/15.
   Manchester United v Arsenal : Man U win 6/5; draw 9/4; lose 21/10.

   Start by thinking about which game is more uncertain, or, equivalently, which game is more certain. Which will therefore have the higher entropy?
   Now work out the probabilities of each outcome happening. (If you don't know how to convert bookmakers' odds to probabilities, go to the additional notes for Task 3). Why do you think the probabilities add up to more than 1?

   *[If you are short of time, use these figures for the probabilities, rather than working them out from scratch:
   Wolves win: p = 1/6; draw: p = 2/7; Chelsea win: p = 15/23
   Man U win: p = 5/11; draw: p = 4/13; Arsenal win: p = 10/31]

   Once you've got the probabilities worked out, you can work out the entropy (or degree of uncertainty) of each game. Were you right about which one it was?

7. If you've got time (or you could think about this later on), it looks as if log₂2/3 - log₂2/3 = 1. Can you justify this result?

Check your answers to this task
Return to Olly's talk

top of page