Oli,

"It's already fixed: http://www.vdiplomacy.com/halloffame_elo.php?gameID=12224"

It doesn't seem like it is fixed. If I go to the HoF listing, then click on my name, then scroll to the second to the bottom, it shows I won, but have -3 .

@cypeg:
This is how an elo-based system should work.
In http://www.vdiplomacy.com/halloffame_elo.php?gameID=4584 the average score of the game was 1100 and your rating was 1100 too,
but in http://www.vdiplomacy.com/halloffame_elo.php?gameID=9611 the average rating was 1010 and you had a rating of 1255 so you where much better than all the other players. Thats why you didn't get that much points.
And if you look at your stats: http://www.vdiplomacy.com/halloffame_elo.php?userID=1459
you see that the last few Defeated brought your rating down quite a lot.

Strange, if I view the point distribution in the link I gave you it's obvious that you got 3 ... I will take a look why it's wrong in the overview...

Ps: cypeg: your position on the leader boards is nearly the same no matter what rating scheme is used... :-)

@Devo: It's a visual glitch, as both games ended the exact same second... :-/ I changed the time by 1 second in the database, and I do not think this is a common problem.

Just wondering - what is the theoretical maximum limit of points on this system? Looking at the scores, I'd guess it's much lower than actual Elo, as you can go up to just over 2800 on that...

Both in Elo and this system, there is no *theoretical* maximum. However, as your rating grows, even if you maintain a perfect win rate, the gains will become rapidly diminishing unless you are able to continually find better ranked opponents to play against. Simply continuing to win against much lower rated players will only lead to very small, diminishing gains that could be easily wiped out by just one poor performance.

Thus, in practice, once ratings have somewhat stabilized, if you've managed to distance yourself from the pack, you'll have very tough time just maintaining that ranking, never-mind attempting to further grow. The upper envelope of practically achievable rankings would be determined by how and where the ratings eventually stabilize. With the FIDE Elo system for chess players, the top players are around 2700-2800. Thus, it would be very difficult to grow far beyond that unless you can maintain an unprecedented win rate against the top players.

I would expect the spread in Elo rating for Diplomacy to be much less broad than in chess. Chess is a very deep game with a wealth of strategic and tactical depth that allows for a very broad stratification of skill. Essentially, if you think about arranging chess players on a ladder, organizing each rung such that players from it can easily defeat the players in the rung beneath it but are easily beaten by the players from the rung above, then I think one can find very many rungs in broad spectrum of chess skill. With Diplomacy, I believe there is fundamentally much more variability in performance and more uncertainty due the game being non-perfect knowledge with tricky multi-player dynamics. Thus, the spread in ratings would be much narrower. Also, it's tough to judge the behavior of the rating system at this point since we are really looking at many orders of magnitude less data than has been handled in Elo systems in place for chess, and the ranking system is still quite a work in progress.

just realised a couple of banned users are in the hof lol

Thanks for clarifying that Oli, Yesterday when I had a look, it didnt seem to show me on position 62 but had a dash there.

Observation 1.So the system takes into account the rank of players to award more points?

Observation 2. Looking at playes's stats I noticed that those who win 1v1 games either get 3 or 5 . However, my win in a 843 game (3 player game) gave me 5 . That does not feel right. Or is is because of the players I played with ?

^cypeg - I believe it depends who you play against.

Observation 3. I also noticed that we start in the new system with 1000 but we do not go below 1000. So if someone lost the first 3 games will still be at 1000 but when the 4th game is a solo he will have ex. 1020?

@Kaner, it seems so, and it is interesting to see that. i.e. I played one 1v1 game with Illum and got my ass kicked, thus, Illum got 6 !

You can go below 1000... It's just that nobody with that score shows up because they're not highly ranked enough to be in the HoF.

The system looks interesting and I have just joined a Haven game with only new players... OMG ;-)

Besides silly remarks, like that the system must be wrong if Ruffhaus is on top, I also noticed that I did get punished for a number of mercy positions I took in WWIV games as a replacement player. I know it is hard to differentiate really between a replacement elimination and a from the start elimination. Still.

I am wondering how that new system will affect my tendency to join games.

a) I am more likely to join anon games
b) I am more likely to set up private games with top players
c) I dont think I will join any games as a replacement player.

hehe
well if Ruffhaus is top 1 then the system must be right at least to some. But seeing Illum above me then there is obviously a bug somewhere :P Probably because you have better draws!

Seriously now, I agrew with Illum that I will think twice to join as a replacement player and b yes we should be encouraged to play among us.

Sorry if I'm slightly off-topic, but I have a couple of trivial questions about the "normal" rating system:
1) if I'm not mistaken, the "position" you can see in player profiles (and in the HoF) is based on the available points. Shouldn't it be based on total points (i.e. account also for points in play)?
2) I believed that (total points)=(available points)+(points in play). But I see there are several players where this is not true (e.g., Ruffhaus has 739 available points 77 in play, and 871 total points). Is there a reason for this?

1) Hopefully this won't matter soon, as it will be switching to the new rating system...
2) Points stored in Anon games don't show up on the Points in Play area, so they'll make up the difference - (total points) = (available points) + (points in play) + (anon points in play).

I'm pretty sure that we'll be using both point systems at the same time.

Oli - could you confirm this?

Yes - Oli did say that - but I meant that the 'Position' tag will probably use the new rating... Although I'm not sure if that's correct...

During this thread Oli expressed the will to keep 2 rating system running in parallel:
the standard D-points system.
a new skill rating system.

@ Oli

“I'm not 100% convinced, but I changed the WTA-formula to allow only to score a win to the winner and do not count all the others.”

Due to the previous statement, I think that there’s a huge fault in the way the algorithm is currently running:
all other things being equal, the greater size is the draw you’re excluded from, the more you’ll be penalized.

I'm not sure what are the exact formulas that Oli implemented, but in my proposal, by normalizing the pairwise K factors by the number of drawers, this is not an issue.

I threw together a simple prototype (in a Perl script) of the specific system that I had proposed for rating calculations for WTA games.

You can see the code and output here: http://codepad.org/nErzh4ts

At the core is a function that implements ratings adjustments for a game given initial ratings of the players and the game results as inputs. It's just a very early prototype and there may be bugs, but hopefully it conveys the main idea.

The script runs through several examples, processing ratings adjustments for a bunch of hypothetical games. The output can be found in the above link, but I've also copied it below (with some annotations):


# Each bunch of four lines is a hypothetical ratings adjustment for a game, with a column representing a player in that game. The "Old" row are the players' ratings before the game. The "Rslt" row are the players' performance in that game. The "Delt" row are changes in rating for each player. The "New" row are the resulting ratings after the game.

# This first set of games considers 7 players each starting at 1000, and looks at how their ratings evolve for different results over 5 games.

Old: 1000.00 1000.00 1000.00 1000.00 1000.00 1000.00 1000.00
Rslt: Win Loss Loss Loss Loss Loss Loss
Delt: +13.71 -2.29 -2.29 -2.29 -2.29 -2.29 -2.29
New: 1013.71 997.71 997.71 997.71 997.71 997.71 997.71

Old: 1013.71 997.71 997.71 997.71 997.71 997.71 997.71
Rslt: 3-Draw 3-Draw 3-Draw Loss Loss Loss Loss
Delt: +2.89 +3.05 +3.05 -2.25 -2.25 -2.25 -2.25
New: 1016.61 1000.77 1000.77 995.46 995.46 995.46 995.46

Old: 1016.61 1000.77 1000.77 995.46 995.46 995.46 995.46
Rslt: Loss Loss Loss 4-Draw 4-Draw 4-Draw 4-Draw
Delt: -2.42 -2.32 -2.32 +1.77 +1.77 +1.77 +1.77
New: 1014.18 998.45 998.45 997.23 997.23 997.23 997.23

Old: 1014.18 998.45 998.45 997.23 997.23 997.23 997.23
Rslt: 2-Draw Loss Loss 2-Draw Loss Loss Loss
Delt: +5.43 -2.24 -2.24 +5.73 -2.23 -2.23 -2.23
New: 1019.62 996.21 996.21 1002.96 995.00 995.00 995.00

Old: 1019.62 996.21 996.21 1002.96 995.00 995.00 995.00
Rslt: Loss Loss Loss Loss Loss Loss Win
Delt: -2.45 -2.29 -2.29 -2.34 -2.29 -2.29 +13.94
New: 1017.17 993.92 993.92 1000.63 992.71 992.71 1008.94


# This next set looks at 5 players of differing ratings and looks at how their ratings would change in a 7 different scenarios. Note how higher rated players doing well against lower rated players (conforming to expectations) results in smaller adjustments, while lower rated players doing well against higher rated players (defying expectations) results in larger adjustments.

Old: 1200.00 1200.00 1000.00 1000.00 900.00
Rslt: Win Loss Loss Loss Loss
Delt: +7.24 -3.20 -1.54 -1.54 -0.97
New: 1207.24 1196.80 998.46 998.46 899.03

Old: 1200.00 1200.00 1000.00 1000.00 900.00
Rslt: Loss Loss Loss Loss Win
Delt: -5.43 -5.43 -4.10 -4.10 +19.06
New: 1194.57 1194.57 995.90 995.90 919.06

Old: 1200.00 1200.00 1000.00 1000.00 900.00
Rslt: 2-Draw 2-Draw Loss Loss Loss
Delt: +2.02 +2.02 -1.54 -1.54 -0.97
New: 1202.02 1202.02 998.46 998.46 899.03

Old: 1200.00 1200.00 1000.00 1000.00 900.00
Rslt: 3-Draw 3-Draw 3-Draw Loss Loss
Delt: +0.72 +0.72 +2.06 -2.09 -1.41
New: 1200.72 1200.72 1002.06 997.91 898.59

Old: 1200.00 1200.00 1000.00 1000.00 900.00
Rslt: Loss Loss 2-Draw 2-Draw Loss
Delt: -4.86 -4.86 +6.01 +6.01 -2.30
New: 1195.14 1195.14 1006.01 1006.01 897.70

Old: 1200.00 1200.00 1000.00 1000.00 900.00
Rslt: Loss Loss 3-Draw 3-Draw 3-Draw
Delt: -5.05 -5.05 +3.18 +3.18 +3.74
New: 1194.95 1194.95 1003.18 1003.18 903.74

Old: 1200.00 1200.00 1000.00 1000.00 900.00
Rslt: Loss Loss Loss 2-Draw 2-Draw
Delt: -5.15 -5.15 -3.65 +6.37 +7.57
New: 1194.85 1194.85 996.35 1006.37 907.57

Yeah, seeing the formulas would be helpful but with the most recent tweaks things seem a little more accurate.
I still see a few oddities though, and I'm thinking it's related to whether the formulas (especially for large games) are indeed zero sum.

Another nice feature (dunno how hard it is to code) would be to be able click a game and see the changes for every player in that game (kinda like how we see the D points changes currently). This would make checking the zero sum property for especially large games much easier.

Leif,
currently the system does comply with the "zero sum" property.
At least it does mathematically, conceding numerical truncation.

@tiger: that's fixed now.

@yebellz: did not apply the normalization... will add this later. Sadly I won't have much time the next few days...
As I have not much time tonight I just uploaded the code to the Wiki: http://www.vdiplomacy.com/wiki/index.php?title=Rating
If you know Perl, PHP is not that different (I have some basic Perl-knowledge too).
Now that we have WTA, would you like to give PPSC a try?

Just checked your code.
One question: What is the hardcoded DRAWEQ for?

Also I really like to have the impact of games scale for player-size in a larger scale. That's why I initially paired everybody 1on1. So a large 34-player game results in 34*33 different games. I switched this later on for the 0.75-score for the survived->Defeated for the same reason. As this proved to manipulate the outcome too much I changed the importance to 100%-player count.

As wrong as a 34-times bigger change from a 34-player game vs 1on1 is, but a mere 1.1-3-times as much for game with more than 10 players does not reason with the time and commitment these games require. Even if technically correct and even if a too large scale might lead to screwed results it might take away the fun of large games if the outcome is the same as just 3 1on1 that you can play in one evening.
So we should try another approach for that.
If we take a 8 player game as "normal" we should not only scale up (*4) for the larger games, but also scale down for the smaller games (for example *0.5 for 4-player, 025 for 2-player games). Or we should base the scaling on the smallest game-size of 2.

The DRAWEQ is to adjust the relative impact of a Draw:Draw adjustment versus the Draw:Loss adjustments for a game that was drawn. It has no effect in the

In the drawn game, each Drawer/Loser pair is adjusted such that their ratings are pushed further apart (the drawer gets a positive change, while the loser gets a negative change), while each Draw/Draw pair is adjusted such that their ratings are pulled closer together (the higher ranked player gets a negative change, while the lower ranked one gets a positive change). These second type of adjustments (between two players that both drew) creates an equalizing effect amongst the group of drawers. The DRAWEQ parameter allows one to customize the relative impact of the this equalizing effect with respect to the impact of the Draw/Loss pair adjustments. I diminished the equalizing effect to prioritize the overall adjustment based on the Draw/Loss pairs. The choice of 0.2 is somewhat arbitrary, but it corresponds to the 15 versus 3 choice that I had proposed earlier in the table of how to compute K values for each type of pair.

Due to the equalizing effect amongst a group of drawers, it's possible for a drawer to actually lose rating, if he has drawn a game along with a bunch of lower ranked players and against fewer lower ranked losers. This would happen if the equalizing effect creates more ratings loss than gained from drawing over the losing group. By using a smaller value for DRAWEQ, the equalizing effect is diminished, and hence it is less likely to see the highest ranked player in a game lose rating for a large drawing. However, as long as DRAWEQ > 0, the highest ranked player will always lose rating in an n-way draw (for a game with n players), since only the equalizing effect would play a role (as only Draw/Draw pairs would be present), although this loss would be rather small if DRAWEQ is small. If DRAWEQ = 0, then no ratings would change in the case of an n-way draw.

With regards to the scaling up and down for large and small games:

By normalizing with n (as done in my code) instead of (n-1), there is already a severe scaling down on small end of the game size scale. A 2-player game winner gets hit with a 1/2 factor, while a 3-player game winner gets hit with a 2/3 factor, while an n-player game winner gets with a (n-1)/n factor. Comparing these factors would demonstrate the relative gains for a winner in a game of each of these sizes (assuming that all else is equal).

The scaling inherent to the normalization makes winning a 2 player game worth 4/7 = (1/2)(8/7) as much as winning an 8 player game.

Of course, we could always amplify this effect for very small games. In particular, 2 player games are quite different from other diplomacy variants due to the lack of multi-player dynamics (coalitions, alliances, betrayals, meaningful negotiations), so perhaps we could scale that situation down even more.

This "normalizing by n" effect plays a diminishing effect when comparing larger games sizes to each other. For example: winning an 8 player game gets a 7/8 factor, while winning a 25 player game gets a 24/25 factor. Hence, I proposed the Big Game Boost factor, that gives a gradual up scaling (up to 3) for games larger than 10 players.

Two comments about the last two posts:
1) Oli, it's true that a large game implies a large effort; but large games are also much more subject to "random" events than small games. So, I agree that a large game should have a larger effect than a small one; but in my opinion the scaling should be less than linear. A possible suggestion is (N-1)**alpha, where N is the number of players in the game (so, N-1 is the number of "enemies"), and alpha is a parameter that needs to be chosen. I personally favour alpha=1/2, so that a 7-players game would count 2.44 times more than a 1v1 game, and a 34 -player game would count as 5.74 1v1 games. But you might prefer an higher value (for example, with alpha=2/3 a 7-player game would count as 3.30 1v1s, and a 34-player game would count as 10.28 1v1s).

2) With respect to the explanation about DRAWEQ, i believe that you shouldn't get negative points if you draw a game. I would suggest a change in the algorithm. Something to the tune of:
calculate (but don't apply) the ranking adjustments of each player
L = list of the drawing players, sorted by pre-game ranking
# the highest-ranked drawing player goes at the beginning of L,
# and the lowest-ranked drawing player goes at the end of L
foreach P in L:
delta = sum of al the adjustments for player P
if delta>=0:
break
# we exit from the foreach because players with lower
# ranking will definitely have larger values of delta
gain = sum of all the positive adjustments for player P
loss = gain-delta
#that is, loss is the absolute value of the sum of
# all the negative adjustments for player P
# since delta<0, we have loss>gain
R = gain/loss
rescale all the negative replacements (and their positive
"counterparts") by a factor R
#end [ foreach ]

I think if you draw with a high number of lower ranked players it should be possible to loose points for the draw. Most of the time the effect of this is lowered by the fact that you gain points against the Defeated. If you do not gain points from a draw you are so favored that this should not be an option at this time for you at all...

PS: How should we score the PPSC?