I just think that one of the main problems with the D-point ranking is that you can only play PPSC or WTA... The compromise would be much better for the reasons I gave above.

I can understand why you want to mimic D-points (people are used to that, and having to choose between maximising D-Point or V-Rank would be annoying) but doing so will just copy the problems with that ranking into the new system.

Anyhow, I'll get off my box now. I'm not sure repeating this will actually change anything, and I'm happy whatever system you put in - I'd like to have something other than D-points to aim for :)

Please do not forget to give some ideas how to deal with the large games.

@drano:
Any ideas to solve this?

Alas Oli, the only way I can think of to solve it is to not use the system you're using. If we were to use the system like GhostRank (where each person puts in a certain % of their points), and then have WTA mean the player takes all the points, it would end up working very similarly to GR from what I understand. That being said, a solo on a large map in WTA would make for a big boost for people, however, it rightly should since a solo against 34 players is theoretically much harder than a solo against 5 players due to the large amounts of territory needed to conquer and win.

However, that leads to its own problems of low victory conditions making solos on large maps easier than they "theoretically" should be. Perhaps very large maps should just be not included since they're causing so many issues? After all, there's only been 28 WWIV games, 10 Chaos games, 3 Chaoctopi games, and 20 Haven games played. That gets us down to 17 players. Removing a mere 36 more games gets you down to 13 players.

I categorically object to removing large maps from the stats as a huge majority of my game time on this site has been invested in large maps, WWIV specifically. Also, why is the proposal or request out there that we further limit the impact of large games anyways? Everything I've read on the subject seems to indicate that a multiplayer adaptation of an Elo system naturally penalizes larger multiplayer games.

<Apologies in advance for the questionable decipherablity of the ramblings that follow>

That being said, I don't have a good answer on how to score large player games, other than the Elo system does not necessarily appear to be useful across our wide range of variants game types for comparing players. I would contend that Elo is a useful stat within one set of game types, (Classic only WTA, OR PPSC 50SC to win anon full press WWIV), but may not be a useful stat comparing G vs I performance to WWIV 100SC's to win anon full press.




There have been only 3 WTA WWIV games, 2 of which were drawn and one of which is still active, so the majority of WWIV games are going to be scored as PPSC. Of the 28 games, 11 were solo'd, and all that were solo'd were at the 50 SC victory conditions, while the all the alternate victory condition games were drawn.

gantz is the only player with 2 solos on WWIV and so should fairly obviously be high in the standings on WWIV games, and so I'd be curious to see the standings for WWIV under the current proposals.

While we don't have this issue now, the 50SC to win WWIV game is a lot different than a 100SC to win game, thus the expected difficulty of winning a 100SC game is not taken into account even though it should be much less than winning a 50SC game.


So... where am I going with this? I dunno...

So, if a player wins a WWIV game (52 SC's), they get 34 wins, but each and every other player could only get 1 loss, else the system is no longer zero sum, and if I'm not wrong, this is where the objection seems to be?

BUT... the player who had a significant 2nd place (47 SC's) (but only a survive and not a win), would have outperformed everyone else on the map, right? and especially everyone who was defeated or resigned.
I guess I don't understand what the expected result for this player is thought to be..
Should this player go up (in the case where all players were equal to start?) or because the game was won by the player with 52SC's, all other players should go down?

Obviously the player who won should (and would) go up more than the player who took 2nd, so the differential rating would look like player B (2nd place) lost ground on player B (the winner). Likewise, Player C (3rd place) should have a ratings differential that goes down in comparison to players A and B but up in comparsion to players in 4th, etc..

The system Oli has currently in place does this (if I'm not mistaken), for PPSC, right?

More thoughts so far:

For WTA we can score
Win->Survive: 1
Win->Defeat: 1
Draw->Draw: 0.5
Draw->Defeat: 1
Survive->Survive: 0.5
Survive->Defeat: 0.75

To scale down the importance of large maps we can use 100% - players.
So a 10-player game would get a factor of -10% (count like a 9-player game),
a 20-player game would get -20% (count like a 16 player game) and a 35-player game would get -35% and count like a 23-player game.

Oli, why is there a need to scale down the importance of large variants in the scoring?

Indications I've gotten from articles I've read on the subject are that Elo naturally favors smaller matches over larger matches.

Can you explain what the need is so I can understand and thus be able to provide my recommendations appropriately?

Example:
A player rated 1000 plays a game with 34 players.
He really performs well and wins (Maybe he played at another diplomacy site). This will result in 34 single matches and his rating will go up very high, because we use his base-rating for each match. (And usually his rating will go up much too much)

If he would have played 34 single 1on1 games his rating would have been adjusted after each game, resulting in lower total score.

Also in a 34-player game you won't have played with all 34 players. Take Chaos for example. If you play London it's very unlikely that you will fight against all 3 players from Turkey.

Wouldn't the first problem be solved if you did each rating one by one rather than all at the same time? I mean, as in if a player played a WW4 game, he would be rated against one player, his rating adjusted, then with his new rating rated against the second player, and so on? This would make how much your rating changed a bit dependant on luck (which players you were rated against first) but that shouldn't come into it too much...

The problem with that is if you survive, and 29 people are eliminated, your score sky rockets almost as much as the peron who solod. Since you beat 29 people and he beat 33.

This is why I am against treating every match like multiple 1v1s between everyone in it and more in favour of the Ghost Rating Method whhere your playing everyone simultaneously.

Great work Oli. As many have already said, no system will be perfect. Not sure about the large-game calculations, but everything else you've decided upon sounds like a big improvement/addition to the points system. Thanks for all your hard work.

While this may not get to the complete crux of the matter on large map games, I think that this discussion only serves to prove while awarding a solo to someone on the WW4 map at 50 SCs is a bad idea. In the case of games that require less than 50%+1 to solo, the points awarded should be likewise lessened.

I'm not sure what the complaint/concern about large map games scoring is though. Yes, there are more centers, and more potential points, but to get them you have to play against more players in a game that could go on for a longer period of time. Is some one really suggesting that awarding the same PPC score on a large map as on the standard 34 SC map is all that unfair? I don't think it is at all. In a WW4 game there are 35 players starting out. In any properly played game (absent ethical abuse and the like) at least 20 of the nations are going to be eliminated, probably more. That's a lot of players getting nothing. And when you consider that a WW4 game properly played at 50%+1 is realistically never going to see a solo, then the real opportunity to score points is going to be by supply center accumulation.

I assume that this request “To scale down the importance of large maps” is due to the matchup algorithm definition.

If this is so, I’d suggest: take the result of the algorithm and scale it down to resultant size +/-1 , dividing by (n-1), being n the players number.

Then, it’s up to you to decide the multiplying factor to apply in front of this +/-1 number.
- A fixed number
- Another function of n (supposedly a function that increases with n)
- A function of the in-game ratings
Just a few ideas.

@CapMeme:
If we rate 1on1 and change the rating after each match it would be too random.
The first match would cause one player to loose many points, the 2nd without playing better would not loose that many points and so on as the rating adjusts. This is not fair, because it's basically 1 game and not many 1on1s and there is no way to make a fair assumption about the order on who to resolve first (and cause the most loss).

@fasces:
We will add the "fake" GR too, but I don't like the idea that 4 losses (with each putting 20% of your rating in the pot) in a row will basically cut your points in half no matter what game and opponent you played. So you can go down from 2000 to 1000 in 4 1on1 games (and your opponent can go from 2000 to 3000). This is an interesting twist but it does not represent a players skill that effective.
All ideas the treat the game as a whole has serious problems because it's hard to define any kind of elo-style ratings on a one against many algorithm.

@RUFF:
It's just that in a 34-player game you will play only 20 of the players on the board most of the time. In a 10 player game about 8-9. Thats why I think the total (global) score should be adjusted for this fact. And to flatten the rating spices caused by the 1on1 adjucation.

@DC: This is how I thought about this.
Fixed number (some Elo-systems base this on the players rating) * factor.
And the factor could be 1 - (PlayersNo/100), which would result in the above mentioned results.

Hi all, I'm kind of late to this party, but am very interested in contributing to this development. I have a very strong mathematics background and have thought quite a bit about ranking systems in the past.

I've just read over this thread. It seems that some suggestions are very similar to Ghost Rating or other Elo-like systems (http://www.diplom.org/Zine/S1998R/Nichols/ratings2.html). However, I feel that each of these systems have their shortcomings.

I think the pairwise adjustments heuristic that Oli proposed is a very clever way of handling ratings adjustments, since it breaks it down into a set of parallel 1v1 adjustments based on the relative performance between each pair of players, which are much easier to handle than adjustments considering the performance of a player against the remaining group.

For WTA games, I don't think a distinction should be made between "Survive" and "Defeat". They should both be considered a loss. A player that survives a game allowing a solo should not be considered to have done better than a player that has been defeated in that game. Perhaps, one could even argue that the survivor is more to blame for the solo since they were around to the end. Also, rewarding survivors more could create a perverse effect where a player might be less concerned about preventing a solo since they could still earn ranking off of players that were defeated.
This would also simplify it to only 4 different pairwise results to consider: win/loss, draw/loss, draw/draw, and loss/loss.

Also, I think the pairwise adjustments between two players with the same result (i.e., "loss->loss:0.5" and "draw->draw:0.5") should be reduced (by reducing the K-factor for those pairs). These adjustments have an equalizing effect, and two players both losing in a game does not necessarily indicate that they were of equal skill. Perhaps the loss->loss pairs should have no pairwise adjustment. I think the draw->draw pairs should result in a weaker adjustments that the "draw->loss" pairs.

Also, the adjustments should be normalized to prevent the effect from games with more players from having too much weight.

Basically, I propose (for WTA):
Win/Loss: 1/0, K=15/n
Draw/Loss: 1/0, K=15/(n*d)
Draw/Draw: 0.5/0.5, K=3/(n*d)
Loss/Loss: n/a, K=0 (no adjustment between these pairs)

(maybe replace "n" with (n-1) in above)

n = number of total players
d = number of players in the draw

Hi yebellz,
I think changing the K-factor based on the outcome (and player rating) is a very good idea.
Same for the non-scoring of Loss/Loss.

Once again, I'm just trying to understand why having a large rating change for a large multiplayer game is a bad idea. I can understand if the rating adjustment from soloing a 35 player game dwarfs the cumulative ratings adjustment from soloing against those same 34 players in separate 1v1 games, but is this really the case? Are they orders of magnitude apart? or is one 20% higher than the other? What sort of difference are we talking about here?

Soloing a 34 player game shouldn't have the same weight as soloing a 1v1 game, otherwise the only real way to improve your score would be play 1v1 games.
Won't this be the result of yebellz proposal about normalizing based on player count?

At the end of the day, one game is simply one data point and subject to substantial variability.

The motivation behind the normalization was to avoid putting too much weight some games over others. The fact that you get a similar benefit from winning a large game as winning a small game, is offset by the fact that you'll face a smaller penalty for losing in a large game.

Overall, the magnitude of the ratings swings need to be controlled to maintain accuracy of the system, since the philosophy is to gradually converge toward the true measure of skill through small adjustments. Large swings would perpetually keep players jumping from being overrated to underrated.

To illustrate, consider an example where all players are similarly ranked:
If you didn't normalize by n, you'd get (n-1) times more benefit from winning a large game with n players versus winning in 1v1. Without normalization, you'd also see the same penalty for losing in the large game as in a small 1v1 game.

Since large games take longer and require significant skill to solo, basically this means that by counting the large game equivalent to a 1v1, you are biasing the apparent weight of a 1v1 game higher than the large game simply by virtue of the fact that a live 1v1 game can be completed in a day, while the large game most likely won't be completed even in 3 months. The rate of change of the skill level would be hardly affected by large games and significantly affected by small games. This is why imo large games should affect the ratings change much more significantly than small games.

@Leif: IMHO The large games should not weigth the same as the 1on1, but as a 23-player game (35 * 0.65), a 23-player game would count as a 19-player game (23 * 0.74) a 10-player-game like a 9-player game (10*0.9) and so on. They would count more than a 1on1 all the time. This came up after some test-calculations with our data here (about 200 replies earlier).

That would be much better imo than equal weights. Do you have any of these test algorithms up and viewable (at least as far as looking at the resultant rankings) for comparison? That might help me see where the issues you are all referring to are. And if they aren't easily accessible, don't waste time putting them up just for me...

The discussion always push the algorithm on new directions and I have not much time to code at the moment. As the new HoF got not updated fast enough for the discussion I removed it to avoid confusion.
I will reactivate this once I have more time. Also not only the algorithm is not set in stone, the database layout too, that makes things a bit more complicated too...

Also RUFFHAUS mentioned a good point we do not need to forget:
What to do with games that ends bevore someone reaches 1/2 SCs because the game used an alternate winning condition (5SC on a classic map for example) or a small max-turns count (eg. 4 years in a classic game)

@Leif, let me help further clarify with another example. The absolute rating benefit from winning is not the only thing in play here. What's more important to look at is the ratio of the possible benefit from winning versus the penalty for losing.

Here is an example with n players, each with the same rating (just for simplicity in illustrating things).

Let's use a small tweak to the system that I proposed above (these are just choices on the parameters within the pairwise adjustment framework that Oli has described):
Win/Loss: 1/0, K=15/(n-1)
Draw/Loss: 1/0, K=15/((n-1)*d)
Draw/Draw: 0.5/0.5, K=3/((n-1)*d)
Loss/Loss: n/a, K=0 (no adjustment between these pairs)
Note: The only change is that I changed "n" to (n-1). The choices of 15 and 3 are somewhat arbitrary. If one wants faster convergence, but larger swings, these numbers can be increased. Likewise, for slower convergence, but smaller swings, these numbers would be decreased. The relative size of the choices of 3 and 15 are to deemphasize the equalizing effect of the people you share the draw with in favor of emphasizing the effect of gain from the people that were defeated.

Since everyone has the same rank, the expected result between each pair is 0.5 (i.e., the model expects them to do equally well since they have the same rank). This makes it easy to calculate what happens to a player's ranking if he wins, loses or draws.

If a player wins, he gets (n-1)*(15/(n-1))*(1-0.5) = 7.5 pts added to his rating.
If a player loses, (whether the game ends in a draw or a solo), he loses 7.5/(n-1) pts from his rating.
If a player has a d-way draw, he gains 7.5*(n-d)/(n-1) pts to his rating.

Remember, these calculations are just for illustrating with this simple example where everyone had the same rating before the game.

So, with using the (n-1) normalization, you would gain 7.5 pts to your rating for winning any size game. However, this is balanced by the fact that if you lose, you will only lose (7.5/33) = .227 pts in a 34-player game, but will lose 7.5 rating pts for a 1v1 game.

You don't have a better shot of increasing your rating by playing 1v1 games, since you risk losing more pts (because you have a better chance of winning). Assuming these equally ranked players all have equal skill, and hence should each win 1/34th of the time that the game ends with a solo, you can see that the expected gain from wins/losses balances each other out. Similarly the rating pts gained and lost in draw situations (assuming that each player would have an equally likely chance of making the draw) gets balances out as well.

A player may be able to play more 1v1 games in a shorter period of time, but that would only make their rating converge faster toward the correct rating, and not give them an unfair edge on increasing their rating.

If we remove the (n-1) normalization, then a player would stand to gain (n-1)*7.5 rating pts from winning a game, while risking the loss of the 7.5 rating pts from losing. You can see that they would still balance each other out, except that now, there is substantially more variability (larger swings) introduced for large games. In fact, a 34 player game would have 33 times the impact as a 1v1 game. The upswing from winning these games could be so large that the adjustment could easily overshoot, leaving the player temporarily over-rated. This also in essence gives larger games more weight, which I think is unjustified. From an estimation standpoint, one should not give a data point (a game) more weight simply because it took more time to play or represented more psychological effort against a larger group of players. One should only a data point more weight if it represents more meaningful information. At the end of the day, the outcome of any diplomacy game comes with substantial uncertainty despite relative skill. In fact, one could perhaps argue that larger games have more uncertainty/randomness due to the complex interplay of the increased number of players, while 1v1 matches are much more indicative of the relative skill between two players. However, I think it is reasonable to leave the weighting between games of different sizes similar.

Yet another option would be to use the "n" normalization as I originally suggested. Counter-intuitively, this would actually give larger games slightly more relative weight than smaller games. You can see that by using the n scaling factor, winning a game would earn you 7.5*(n-1)/n pts while losing would cost you 7.5/n pts. The balance between winning and losing is the same, but now, there is a (n-1)/n factor hitting the winning benefit of each game. This factor hits 1v1 (2 player) games with a 1/2 reduction factor. It hits 3-players games with a 2/3 reduction factor, and 34 player games with a 33/34 reduction factor. Thus, in the extremes, a very large game would almost be twice as much important as a 1v1 game. This isn't too ridiculous and perhaps its reasonable to give larger games slightly more weight like this since arguably larger games with the complex interaction between many players is more representative of what diplomacy should be about than the pure tactical battle of a 1v1 game.

Thus, I would perhaps recommend using the n normalization as I originally proposed:
Win/Loss: 1/0, K=15/n
Draw/Loss: 1/0, K=15/(n*d)
Draw/Draw: 0.5/0.5, K=3/((n*d)
Loss/Loss: n/a, K=0 (no adjustment between these pairs)

After reading the various proposals for large games again, I think the larger games are more challenging, as there are simply more players and x-factors to contend with, and so should be given some more weight -- just not too much. Thus, Yebellz's original proposal which results in extremely large games being given almost double weight seems like a very strong approach.

What if we remove the 1v1 games out of the ranking system? :)

and yes in a WTA a survival is a defeat. The survived players ARE to blame for the solo. I have witnessed it time and again, weak players-powers who just dont like the other player throw the game to someone else..that policy should be punished!

About large games...Chaos is the ultimate challenge. viking and Classic for example is more of a challenge that ww4.