Support ESC Insight on Patreon

Voting Insight: How Strong Is Jury Voter Collusion? Written by on July 7, 2014 | 3 Comments

At the end of Voting Insight: Episode Three we highlighted that we believed the working together of the jury groups was a bigger factor than any split we could do on each individual jury member based on their personal background, occupation, age, or gender.  

The similarity of voting patterns between jury members from the same country has also concerned the European Broadcasting Union (EBU) and current voting partners Digame, especially noticed when the results of the Georgian jury results were disqualified due to the ‘statistically impossible’ results that were found after their voting in the Grand Final.

Ben Robertson investigates how significant the voting correlations are between juries of different countries and why this may occur.  

The Georgian Judgement

One trend that has been discussed in the aftermath of the contest is that there has been evidence of juries colluding together, and the result of the Georgian jury was annulled because it was ‘statistically impossible’ for each jury member to have the same top eight songs in the same order.

For interest, we calculate that the likelihood of this happening (assuming the chance of liking a song is random) is 6.352 x 10-44, or in other terms we would expect this to happen once every 42 500 000 000 000 000 000 000 000 000 000 000 000 000 years in Eurovision Song Contest voting in the Grand Final. So yes, it does appear quite unlikely.

Although last year one officially unnamed country was removed from the results of Semi Final Two, it was a brave and unprecedented decision from the EBU to disqualify the results of Georgia’s jury knowing it would be clear which country would be annulled and the reasoning would need to be justified. There is no specified requirement in the rules for juries to vote similarly or differently to each other. The only grounds that EBU can disqualify voting from a country is if they believe the following has taken place:

“If it appears that votes are casted only in the intent to abuse the voting system or to false the final results.”

Little specifics have as of yet been released from the European Broadcasting Union or from Georgia internally about the decision. Sietse Bakker from the EBU has been quoted saying that the need to disqualify the votes was ‘clear and immediate’, and that this only applied to Georgia.

To look into this in further detail, we decide to run some mathematics on the other countries jury voting patterns. In previous Voting Insights we have used correlation analysis to compare how strong voting patterns are linked together. What we present below are the average results of voting correlations between jury members from the same country.

Average Jury Correlations to Other Jury Members in Their Respective Country

For comparing data with twenty-six values, we are looking for correlation figures to be above 0.2598 to show 90% correlation that some voting patterns exist between members of the same country. All of the juries show positive correlation above this value. Our average value of the above data is 0.598, a staggeringly high figure which is well above the value for being 99.5% confident of correlation existing (which would be 0.4958).

Country Rank Correlation Between Jury Members
Austria 0.308
Macedonia 0.357
United Kingdom 0.358
Hungary 0.360
Ireland 0.384
Slovenia 0.440
San Marino 0.445
Malta 0.475
Switzerland 0.481
France 0.486
Sweden 0.492
Italy 0.501
Romania 0.539
Israel 0.543
Spain 0.548
Albania 0.552
Denmark 0.563
Finland 0.564
Moldova 0.590
Lithuania 0.600
Norway 0.601
Poland 0.609
Estonia 0.617
Greece 0.630
Belgium 0.674
Portugal 0.707
Armenia 0.710
Ukraine 0.719
Latvia 0.735
Iceland 0.737
Russia 0.744
The Netherlands 0.801
Germany 0.832
Belarus 0.926
Montenegro 0.966
Azerbaijan 0.979

What this mathematically represents is that we are very certain that watching the contest together, regardless of your differences in age, gender, professional background or anything you want, being a member of the same jury is a huge factor in making people vote in a similar way.

Shocking from an EBU perspective is that so many of these correlation results are very high. Azerbaijan, Belarus and Montenegro have scored clearly over 0.9 here, suggesting that the results from each jury member are almost identical. If we take the Azeri jury, the largest difference in points for any of the songs in the Grand Final is 5 (for example one jury member had Romania 2nd, and another had it ranked 6th).

The Maiden Tower, Baku

It was in Baku that the voting results most correlated from each other on the same jury

This isn’t to mean that ‘fair’ voting countries without collusion completely disagree with each other. Austria has the lowest jury correlation, and indeed the five Austrian jurors all loved Sweden’s performance seemingly equally, with their results equalling 3, 3, 2, 3, 4. However it was not like this for all the songs, with many wildly varying (the UK as an example was placed 1, 24, 12, 18, 24 by those same five members).

Similar voting does, and always will be found, across every jury if we look for it. What is shocking is the massively high agreement we find above. None of the correlations in any of our previous analysis have come close to being as strong as in this comparison which is practically mathematically certain a link exists.

Why Does This Link Exist?

One argument is of course to discuss this in terms of cultural links, styles of music which are preferred more than others in different countries, is it simply this trend that the mathematics find above? After all, we know that televoters do follow trends of demographics as well in the contest for voting for similar styles of music or the same countries each and every year, which have given rise to terms like ‘voting blocs’.

tumblr_n6qz5yf2PE1qgqbwdo1_500 (1)

J.E.M on stage at Lilla Melodifestivalen, with Elli featuring on the left of screen

This may be possible, but for it to be so certain and so clear across different countries would be a big surprise. One other alternative is that, through watching the show, juries were able to influence each other. While I was at Lilla Melodifestivalen I got to speak to Swedish jury member Elli Flemstrom who was performing in the interval act as a part of J.E.M. We spoke about what it was like to be on the jury and here are some quotes from her.

“I listened to the songs before the show, and actually I did change my mind when watching on the stage. It is different on the radio and then you see it live.”

“It was very fun, a cool gang. There was no cheating at all.”

“We were all together in a room, but we could not see each other’s papers. We were not discussing when watching and setting points. We did not want to influence each other, but during the show we did not [discuss] so much, and we discussed lots after.”

Elli made a point of telling me that the jury members could not see each others scores and were not going through bits that were good or not good. However there is still the opportunity for five faces sitting together to show their opinions to each other about the musical quality of each performance.  That they did discuss at some point could have made a subtle difference in this regard.

It is therefore impossible to establish if the positive correlation is because of cultural taste, or because of jury members influencing each other or because of more sinister reasons that have been alluded to in other media sources. If the juries are in the same room, they can talk, they will talk and I would argue they should talk. The opinions of each song that they have will have a chance to be shared and a chance to be discussed. Elli made it quite clear they they did not try and influence each other and how they voted, but then Sweden’s result was almost a full 0.1 below the average correlation across the contest. Other jury members in other countries could have had full debates about their votes and what they agreed was best and worst. This may increase their agreement to the levels we have seen because of purely honest means.

Fixing The Problem

As long as the Eurovision Song Contest has 50/50 voting, I never want to experience another sad example where a jury result needs to be disqualified. Of course it can happen that juries are put under influence, and arguably the chance of this happening is increased now the jury names are made public prior to the contest.

If we are looking for a solution to reduce this trend, our first port of call could be to have a limit the time period before release of the jury names. Even if this was a little bit later (perhaps on the day of each Semi Final) the professional integrity of the contest would still be upheld by having accountable names voting, without the potential for them to have excessive influence put on them.

Another option would be to have juries watch the show independently of each other, so they weren’t able to exert any opinions during the jury final to each other. Even if jury members are trying not to influence each other, one may notice even subtle changes in behaviour or body language as they enjoy each song differently.

Ideally this would be pushed further to have the jurors not even in the same building as each other, but on a practical level for each broadcaster (especially the smaller ones) this would be an unwelcome and potentially expensive addition to the increasingly-complicated rulebook. At the very least an option here could be that the ballot papers are filled in behind a closed area (similar to in a general election) so that any influence people may have on each other does not occur with the voting paper in their hands.

It also could be a factor to increase the amount of time that jury members have to vote, and allow them the opportunity to watch things back at their pleasure for a longer period as individuals. The juries, although voting the night before the contest, need to cast their full vote from first place to last place on the evening they watch the show. Extending this though may have a limiting factor for the Dress Rehearsal on the Saturday, where the voting order is practised with all the satellite links and the spokespeople. Delaying the submission of the Jury votes would leave even less time to plan the voting order.

The clarity of voting that we were promised has opened up the can of worms to more people and shown us exactly where many of the problems lie. It is not a response that the EBU will take lightly, and I would expect Reference Group discussions will come back to this as we approach Austria 2015. Releasing the full scores should be a benchmark for the contest to show how not just transparency, but also fairness in the contest. The regional bias, outweighing all other factors, just further highlights that the contest is still not able to shrug off those accusations of ‘bloc voting’ that have hung around its neck for far too long. More efforts and ideas to remove this from the process of selecting the best songs would be further appreciated in future editions of the Song Contest.


The fifth episode of Voting Insight will feature a bit of fun, where we create our own voting systems and analysis how Eurovision would have looked different under the different systems, feel free to suggest your own voting systems below and if we are able to calculate them we will add them in the next article.

About The Author: Ben Robertson

Ben Robertson has attended 23 National Finals in the world of Eurovision. With that experience behind him he writes for ESC Insight with his analysis and opinions about anything and everything Eurovision Song Contest that is worth telling.

Read more from this author...

You Can Support ESC Insight on Patreon

ESC Insight's Patreon page is now live; click here to see what it's all about, and how you can get involved and directly support our coverage of your Eurovision Song Contest.

Have Your Say

Leave a Reply

Your email address will not be published. Required fields are marked *