{"assets":[],"author_link":"author\/alwaysgeeky\/","author_name":"AlwaysGeeky","cat":"LD #23","categories":["LD #23"],"comments":[],"epoch":1336982340,"event":"LD23","likes":36,"metadata":{"p_key":"76959","p_author":"AlwaysGeeky","p_authorkey":"0","p_urlkey":"285233","p_title":"Are the results fair?","p_cat":"LD #23","p_event":"LD23","p_time":"1336982340","p_likes":"36","p_comments":"0","p_status":"WAYBACK","us_key":null,"us_name":null,"us_username":null,"event_start":"1334880000","event_key":"10","event_name":"LD23"},"source_url":"2012\/05\/14\/are-the-results-fair\/","text":"

Before I start this post I just wanna say that I am not writing this out of any spite or bitterness towards the results or angry about anything. I am completely proud of my entry and what I achieved during this LD (my first entry \":)\" ), so don\u2019t just dismiss this post as a bitter rant about my personal rankings.<\/p>\n

But I do want to bring up the question of how fair the results are and if the way results are currently calculated is working as intended\u2026<\/p>\n

From what I can see and assume, the results are just calculated as an average of what you given in each category. So for example if you were rated by 10 people, your score for each category is tallied up and divided by 10, to give you a score for each category. Now this seems a bit broken for a competition like this, where each entry will get wildly different number of votes\/ratings.<\/p>\n

So in theory if a game was only rated by 1 person, but that person gave the game 5\/5 for each category, then it would get a score of 5.0 for each category\u2026 This seems wrong to me and allows for wild fluctuation in the results depending on how many people played\/rated your game.<\/p>\n

Let\u2019s take a look at some examples and see if what I am saying seems to make sense:<\/p>\n

Looking at the top 25 list it doesnt take long before you can go down the list and find a game that scored *very* well, but on closer inspection seems to have only been played by a very few number of people\u2026<\/p>\n

Lonely Hated Rock<\/a> \u2013 Came 5th<\/strong> overall in the competition and had scores of #8<\/strong> in theme, #21<\/strong> in fun and #30<\/strong> in innovation #94<\/strong> in mood and #128<\/strong> in graphics. Very good scores all round! I\u2019m impressed! That is, until I Look at the spreadsheet and find that this game was only played and rated 16<\/strong> times\u2026<\/p>\n

Pocket Planet<\/a> \u2013 Came 10th<\/strong> overall and joint 1st<\/strong> in fun! Well done\u2026 20<\/strong> people rated this entry.<\/p>\n

Pale Blue Dot<\/a> \u2013 16th<\/strong> overall and #23<\/strong> in the fun category\u2026 but was only played\/rated by 20<\/strong> people.<\/p>\n

Necro Gaia<\/a> \u2013 #16<\/strong> overall, #11<\/strong> for graphics #16<\/strong> for fun and #50<\/strong> for mood. This game actually ranked in the top 100 for 6 categories! Wow.. thats amazing, how many times was that game played\/rated? 18!!<\/strong> (Whats worse\u2026 the author of this game had a coolness rating of 0\u2026 yep, according to the data, they never played\/rated any other game.<\/p>\n

Using the latest play\/rate data from https:\/\/docs.google.com\/spreadsheet\/ccc?key=0Ao74NZQqNUt5dDNvZUJ1UXVqZGkxUGVlVkxlZ3JnM2c#gid=0<\/a><\/p>\n

I am not bashing any of the entries in my example, merely using them to point out how having fewer people play your game seems to benefit your final result.<\/p>\n

It would seem that using an average score to give you a final result is flawed in a competition such as this. Average scores and using the mean is a great metric but should only be used when you can guarantee that each entry will get the same number of data points. If this is not the case you need to look to alternatives and start doing more statistics to get a fairer<\/strong> results.<\/p>\n

It really depends on your viewpoint, but as I have pointed out above, if simply using the average to calculate the final score. The competition and results are open for wild abuse and un-fair results.<\/p>\n

Maybe somebody could compile a data spreadsheet showing the number of ratings a game got, and ordered by the overall ranking of the game\u2026 would be interesting to see how high-up some games can get, by having a very few number of people rate their entry\u2026<\/p>\n

EDIT<\/strong> : I put an example in one of my reply comments below but just bringing it up to the original post for more exposure.<\/em><\/p>\n

Here is an extreme example highlighting the problem in a simple way that shows how the system doesnt work for the current dataset:<\/p>\n

With the current system a game that is played and rated by 10<\/strong> people, all giving the game 5\/5 would score a 5.0 in the final rankings\u2026 but a game that is played by 100<\/strong> people and rated 5\/5 by 50 and 4\/5 by 50 would score 4.75 in the final rankings\u2026 now just consider this example and ponder if this seems correct to you? Personally I think this highlights how the final rankings for the games doesnt correctly reflect the ratings a game got.<\/p>","time":"May 14th, 2012 7:59 am","title":"Are the results fair?"}