*meant to say: "is [not] too well justified" when referring to the GR PPSC hack.

No time for posting now, but my proposal was different from Devonian's.
I don't remember the day, perhaps mine was later on.

@Decima: Oh, right, you were also the one that posted on Jan 20 proposing an Elo-like system with formulas detailed here: http://imgur.com/pBFu0Lv

As I pointed out in my first post, a very similar Elo-like proposal can be found here: http://www.diplom.org/Zine/S1998R/Nichols/ratings2.html

I think that such an approach would have some drawbacks in its attempts to calculate the expected result for each player. For the higher ranked players, they would be required to achieve at least an x-way draw to gain rating and to avoid losing rating. Also, without some sort of normalization of the expected result for PPSC games, a winner could actually lose rating if their actual result (being a fraction) is less than their expected result (which could be closer to one).

ok, I admit I didn’t think enough to Devonian’s model PPSC case.
I assume that you (and him?) are better informed than me on that one.
For all the other models the PPSC case is still incomplete/wrong, because of this I wrote “By now I would recommend to stay on the WTA case”.

Yebellz,
Let’s use the afore-mentioned example of a classic WTA game with users rated:
1400, 1200, 1140, 1250, 1460, 1050, and 1320
Is the first player under-performing if he takes a 3-way draw?
Here’re the answers from the current models:

Model from Oli:
“No he is not; a 4-way draw will result in a small gain, but a 5-way draw will be under-performing”
Model from Yebellz:
“I don’t know, I suppose not, but actually it depends on who else will be involved in the draw”
Model from Decima:
“No he is not; a 4-way draw will be borderline, but a 5-way draw will be under-performing”
Model from Devonian:
“No he is not; a 6-way draw will still result in a gain, but a 7-way draw will be under-performing”

So what’s the first player’s Expected score? Can we define it without ambiguity?
Albeit not so explicitly, Oli’s initial algorithm gives a clear answer to the question.
Decima’s and Devonian’s algorithms tell you explicitly what’s the score you’re expected to overcome.
Yebellz suggestion can’t answer the question beforehand, since changing the K pairwise values alters the overall expected score depending on the game outcome, which is kind a blasphemy to my ears.

You will also note this feature for any skill-model: any player above the average in-game skill is expected to do better than a 7-way draw, while any result except a loss will be positive for the players below average. The more you’re above the average in-game skill, the more you’re expected to perform. For example (in Oli’s model), player#5, the best, is expected to do at least a 3-way draw.

“For the higher ranked players, they would be required to achieve at least an x-way draw to gain rating and to avoid losing rating”

It’s not a drawback… it’s exactly how it’s meant to work.
Else, I misunderstood everything.

Decima,

"For the higher ranked players, they would be required to achieve at least an x-way draw to gain rating and to avoid losing rating"

is really just one philosophical approach toward making ratings adjustments. An argument against such an approach is that perhaps one should not always be punished for making it to a draw (which is generally considered a positive result). In the extreme case, for a very highly skewed game, some players may be required to 2-way draw or even win in order to avoid losing rating.

Another valid approach, which is what my system tries to do, would be:
"You always gain rating for a positive result (drawing or winning), and you always lose rating for a negative result (losing). However, the potential gains are balanced against the potential loses based on your relative skill to your opponents."

What this last sentence means is that a highly rated player will stand to gain substantially less from drawing than he risks from losing the game. As you go higher in relative rating to your opponents, you see diminishing returns from simply drawing, so it's not like a highly rated player could endlessly benefit from simply grinding out draws from lower ranked players.

Also, for my proposed system (when using a draw equalization factor of 0.2 and the Elo divisor of 400 for comparing relative ratings), in the specific example that you proposed, all of the players would stand to gain some rating when participating in 6-way draw or better. For higher rated players those potential gains become much smaller than their potential loses which creates balance in the adjustments.

I posted code for a simple prototype of what I've proposed in an earlier post, but here is the output for a variety of scenarios:

# First three players draw
Old: 1400.00 1200.00 1140.00 1250.00 1460.00 1050.00 1320.00
Rslt: 3-Draw 3-Draw 3-Draw Loss Loss Loss Loss
Delt: +1.94 +3.64 +4.13 -2.32 -3.45 -1.20 -2.73
New: 1401.94 1203.64 1144.13 1247.68 1456.55 1048.80 1317.27

# Player one draws with worst two
Old: 1400.00 1200.00 1140.00 1250.00 1460.00 1050.00 1320.00
Rslt: 3-Draw Loss 3-Draw Loss Loss 3-Draw Loss
Delt: +2.09 -2.33 +4.39 -2.61 -3.60 +5.03 -2.97
New: 1402.09 1197.67 1144.39 1247.39 1456.40 1055.03 1317.03

# Player one draws with best two
Old: 1400.00 1200.00 1140.00 1250.00 1460.00 1050.00 1320.00
Rslt: 3-Draw Loss Loss Loss 3-Draw Loss 3-Draw
Delt: +1.27 -1.15 -0.89 -1.41 +0.88 -0.58 +1.88
New: 1401.27 1198.85 1139.11 1248.59 1460.88 1049.42 1321.88

# Player one draws with best and worst
Old: 1400.00 1200.00 1140.00 1250.00 1460.00 1050.00 1320.00
Rslt: 3-Draw Loss Loss Loss 3-Draw 3-Draw Loss
Delt: +1.60 -1.72 -1.44 -1.96 +1.16 +4.69 -2.32
New: 1401.60 1198.28 1138.56 1248.04 1461.16 1054.69 1317.68

# Worst player solos
Old: 1400.00 1200.00 1140.00 1250.00 1460.00 1050.00 1320.00
Rslt: Loss Loss Loss Loss Loss Win Loss
Delt: -4.03 -3.22 -2.86 -3.47 -4.18 +21.54 -3.77
New: 1395.97 1196.78 1137.14 1246.53 1455.82 1071.54 1316.23

# Player one solos
Old: 1400.00 1200.00 1140.00 1250.00 1460.00 1050.00 1320.00
Rslt: Win Loss Loss Loss Loss Loss Loss
Delt: +8.27 -1.10 -0.84 -1.36 -2.68 -0.54 -1.77
New: 1408.27 1198.90 1139.16 1248.64 1457.32 1049.46 1318.23

# Best player solos
Old: 1400.00 1200.00 1140.00 1250.00 1460.00 1050.00 1320.00
Rslt: Loss Loss Loss Loss Win Loss Loss
Delt: -1.89 -0.84 -0.63 -1.05 +6.21 -0.39 -1.41
New: 1398.11 1199.16 1139.37 1248.95 1466.21 1049.61 1318.59

# Player one draws with worst 3
Old: 1400.00 1200.00 1140.00 1250.00 1460.00 1050.00 1320.00
Rslt: 4-Draw 4-Draw 4-Draw Loss Loss 4-Draw Loss
Delt: +1.23 +2.34 +2.64 -2.61 -3.63 +3.02 -2.99
New: 1401.23 1202.34 1142.64 1247.39 1456.37 1053.02 1317.01

# Player one draws with worst 4
Old: 1400.00 1200.00 1140.00 1250.00 1460.00 1050.00 1320.00
Rslt: 5-Draw 5-Draw 5-Draw 5-Draw Loss 5-Draw Loss
Delt: +0.68 +1.36 +1.54 +1.20 -3.61 +1.77 -2.94
New: 1400.68 1201.36 1141.54 1251.20 1456.39 1051.77 1317.06

# Player one draws with worst 5 (all but the best player)
Old: 1400.00 1200.00 1140.00 1250.00 1460.00 1050.00 1320.00
Rslt: 6-Draw 6-Draw 6-Draw 6-Draw Loss 6-Draw 6-Draw
Delt: +0.25 +0.65 +0.76 +0.56 -3.54 +0.89 +0.42
New: 1400.25 1200.65 1140.76 1250.56 1456.46 1050.89 1320.42

# All but player one share a 6-way draw
Old: 1400.00 1200.00 1140.00 1250.00 1460.00 1050.00 1320.00
Rslt: Loss 6-Draw 6-Draw 6-Draw 6-Draw 6-Draw 6-Draw
Delt: -3.19 +0.62 +0.73 +0.52 +0.08 +0.87 +0.37
New: 1396.81 1200.62 1140.73 1250.52 1460.08 1050.87 1320.37

# 7-way draw
Old: 1400.00 1200.00 1140.00 1250.00 1460.00 1050.00 1320.00
Rslt: 7-Draw 7-Draw 7-Draw 7-Draw 7-Draw 7-Draw 7-Draw
Delt: -0.16 +0.07 +0.13 +0.01 -0.21 +0.22 -0.07
New: 1399.84 1200.07 1140.13 1250.01 1459.79 1050.22 1319.93

Actually, if you take something similar to Oli's original system, but which:
1) treats defeats and survives the same (both as loses), and
2) assigns the same K factor to each pairwise adjustment, even for (loss-loss pairs), and abandoning the draw equalization factor,

then the heuristic would simplify to calculating the expected result and actual result for each player in the following manner:

Actual result: n for a win, ((n-d) + d/2) for a d-way draw, and 0 for a loss
Expected result: sum of all of the pairwise expected results for that player

Sorry, that last post isn't quite right. It should be:
Actual result: (n-1) for a win, ((n-d) + d/2) for a d-way draw, (n-d)/2 for a loss against a d-way draw, and (n-1)/2 for a loss against a solo.
Expected result: sum of all of the pairwise expected results involving that player.

"You always gain rating for a positive result (drawing or winning), and you always lose rating for a negative result (losing)"

I personally disagree with this approach. A Diplomat amongst beginners usually doesn’t feel satisfied with a 5 or 6-way draw, does he?

In any case, your system currently fails to assure that statement. The only way to make it mathematically is having your 0.2 coefficient collapsed in zero: translated, as Oli suggested on Feb 14:
“Maybe instead of pairing everyone in a draw 0.5:0.5 we could just stop to rate the drawers against each other”

We can do it, yes, but the main problem remains: metagaming.

I would be against rating the drawers against eachother. traditionally in both PPSC and WTA the draw has been divided equally, because thats what a draw is, nobody won so nobody should get the most points.

as for the current ranking system, I don't think there is enough variance between all the players, while I can only see the top 150 in terms of scores, what troubles me is that the 150th ranked player (out of 3458 accounts) has a score of 1028, only 28 a head of what a rookie would have.

I think we therefore need to increase the variance each game could give.
Thoughts?

+1 Fasces on drawing

@Decima:

In response to:
"I personally disagree with this approach. A Diplomat amongst beginners usually doesn’t feel satisfied with a 5 or 6-way draw, does he?"

I think you're missing the point. Yes, a skilled player taking a 5 or 6-way draw amongst beginners is a disappointing result, and hence my system would reward him with only a very small ratings increase, or possible even a slight ratings decrease in the extreme of a very large draw (relative to the game size) and/or a skilled player that is rated drastically higher than the rest. This tiny increase in this type of scenario is drastically offset by the much higher potential losses from losing against a group of beginners. The reason why the higher rated player gets this tiny increase is that the result is not necessarily a negative one, but instead one that is not very surprising, hence only an almost negligible increase is given since the player has not exceeded expectations.

The point of this philosophy is that the draw dynamics in Diplomacy are vastly different than those in Chess or Go. A highly rated player winding up in a draw is not necessarily a negative result that should indicate that he must lose rating since he underperformed. Often a draw becomes inevitable in the course of the game, once competent players recognize the solo threat and rush to hold the stalemate line, while putting themselves in the strong equilibrium of either staying loyal to that line to accept a draw, or facing a sure loss should they choose to betray for other gains.

In response to:
"In any case, your system currently fails to assure that statement. The only way to make it mathematically is having your 0.2 coefficient collapsed in zero: translated, as Oli suggested on Feb 14:
'Maybe instead of pairing everyone in a draw 0.5:0.5 we could just stop to rate the drawers against each other'"

It was not the intent of the system to absolutely assure that statement. That is exactly why I made that coefficient less than one but still greater than zero, in order to drastically diminish the likelihood of a player losing rating in a draw, while also still making it possible in the extreme a much higher ranked player in a very large draw. The choice of specifically 0.2 is somewhat arbitrary, but that gives us a free parameter to adjust the power of the equalizing effect shared amongst the drawers relative to the ratings adjustments made between a drawer and a loser.

In response to:
"We can do it, yes, but the main problem remains: metagaming."

As I mentioned earlier, any rating system might encourage some less ethical players to meta-game based on the relative ratings of those they are playing against. 1) they should not be doing this, 2) that shouldn't be an argument against adopting a rating system.

@fasces349: I think that's a valid observation. It looks like the overall K value may be a little too low, as player's ratings seem to be moving very slowly over time, and that some players are still trending upward even after dozens of games. Increasing the impact of all games may help with that.

Yebellz,
If that wasn't clear, we're trying to compare 4 different proposals of skill rating systems.

Metagaming (at least on my part) has never been an argument against adopting A rating system.

Metagaming is an argument against adopting YOUR rating system, since, compared to the other 3systems, yours offers way more inputs to metagame.

Decima,

Well, I think that if you're going to use the meta-gaming argument against my rating system, you have to acknowledge that meta-gaming would be a valid argument against any ratings system.

Decima,

I get that you don't like my system very much, but I feel that it's quite unproductive to constantly harp on a perceived meta-gaming concern. In reality, I don't think it would be that big of a deal since it would only happen in non-anon games with unscrupulous players. Further, I think that such behavior should be forbidden by the rules, so dealing with the cases in which it does arise should be dealt with in the same way as other forms of cheating abuses.

As far as other drawbacks and benefits to these four systems, I think there are bigger issues in terms of accuracy and volatility between these systems.

1) Oli's original system (which made equally weighted adjustments between each pair): this system would have the drawback that a loser could even gain rating for simply losing alongside much higher ranked players. Also, () a player would lose more rating against a larger draw than a smaller one or solo-er. Note: Since Oli's current system (the prototype being tested on the site and the version described in the wiki) has incorporated several of my suggestions, it's actually pretty similar to what I've proposed and these particular issues no longer exist.

2) Ghost Ranking (which for WTA games is equivalent to Devonian's proposal except for one constant being 2/35 instead of 1/22): I think there are a number of issues with Ghost Ranking, but the most substantial is its very high volatility. The amount of rating that you lose for being defeated has no dependence on who you're playing against. This is balanced by the fact that you stand to gain exceedingly more rating for a winning or drawing with higher ranked players, however, overall this results in a ton of volatility. Also, this system, when viewed from an Elo-like perspective (which is how Ghost presented the formulas originally) is essentially using a simple relative ratio to calculate expected result, and setting the K-factor as the sum of the starting ratings. The K-factor leads to games among higher ranked players being worth many times more than games among lower ranked players. The simple ratio for expected results makes points worth relatively less and less as one climbs in ranking. For example, a 100 point difference to a player ranked at 2000 has half as much impact on his expected results than a 100 point difference to a player ranked at 1000. However, this relative value of rating directly clashes with the zero-sum exchange of rating points.

3) Decima's Proposal (which is essentially equivalent to http://www.diplom.org/Zine/S1998R/Nichols/ratings2.html, save for some changes to parameters): I actually think that this is a very reasonable system. My main concern is that I'm not sure that boiling down the complexities of various possible results in diplomacy into a single number is appropriate. I feel that it's somewhat arbitrary to turn expected result into a line of "you need to earn at least a 4-way because of your relative rating". I think that better players tend to consistently win more often and draw more often than worse players, but ultimately the number of players that wind up in a draw is much trickier and less predictable matter. Sometimes you can't reduce a draw simply because they are being protected on the other side of the map. Other times, games deadlock with a lot of players simply because that's what is needed for the stalemate. The draw dynamics in Diplomacy are far different than in other games like Chess or Go, which are what Elo is more suited. Even having different types of draws (with different numbers of players and different combinations of players being involved) makes the situation much more complex. Ultimately, that's why I like the modified pairwise adjustment based approach that I proposed, since it tries to take into account who was in the draw.

“I think that such behavior should be forbidden by the rules”.
Fine.
I think that too. But what happens when someone does? We can’t punish it, cause if ever you’re able to detect such behavior, you’ll never be able to prove it, unless the guilty user is so stupid to admit it by himself.
And I don’t know how about you, but I would detest to carry on the role of the meta-gamer hunter.

I don’t care if you tell me: “we’re not supposed to metagame”.
Someone does. Full stop.
Each time there’s a global ranking list there is a potential of metagaming.
It’s not that I don’t like your system “per se”. I don’t like the consequences of your system. Consequences of FURTHER metagaming that the other 3 do not imply.
Despite being generally good at numbers handling, your system would bring consequences that are terribly bad in their potential to influence the future games dynamics.

I don’t want to target someone or to be targeted by someone because of IN-GAME (not global ranking list) points distribution convenience.
I don’t want it to happen during the early game.
I don’t want it to happen during the mid-game.
I don’t want it to happen during the end-game.
I have already pointed out the WTA solo scenario, but I could picture the draw scenario or the loss scenario as well.

Once the game is started, I demand the freedom to choose my opponents as I desire, without being concerned of the other’s performance effect on my result. I demand this freedom for my opponents too.

I repeat, your proposal is valid in terms of numbers values, but it does not grant this freedom anymore. So please don’t take it personally, but I cannot support it.

This said, I will be glad to quit with the meta-gaming argument, when you’ll have recognized objectively that IN-GAME meta-gaming is a point against your proposal, not against the others.

You can play Anon games

Yes : we could play anonymous games but what if we want or even prefer to play non-anonymous games ?
We still would want to be able to keep playing non-anonymous games without those issues.

Non-anon games always carry a huge risk of meta-gaming. Maybe that method gives one more reason to meta-game, but it can be avoided. For those who prefer non-anon games, it is not hard to start a closely knit group of people who prefer anon games that agree not to meta-game.

The discussion has now reached a point where we have some very different ideas on how a rating could/should be calculated.
For my proposal I integrated many ideas from different sources in the calculation. But we are now at a point where it's impossible to integrate some other cool ideas in a single algorithm because they work totally different.
I will implement my algorithm the next few days, and I will integrate all other ideas in separate calculations too, so we can compare them all in one big table.
Therefore I need a very detailed description of all other proposed algorithms, so I can put them in code.
I made an example with my code on the wiki:
http://www.vdiplomacy.com/wiki/index.php?title=Rating
Please add more ideas there and give as much detail as possible.

Disclaimer: My algorithm is not 100% set in stone but that's how I feel comfortable (maybe with some minor adjustments). I like the other algorithms that emerged from this thread too, but they are so different that it's impossible to merge them in one algorithm. I think we should compare the different results with the games played on this site now.

Oilver, I think that you've gone to great lengths to listen to all of us, and to incorporate as many ideas as possible. The fact that you are willing to go above and beyond your preferrred version, allowing for comparisons after digesting all of that is truly generous and absolutely unselfish.

In the end it will be impossible to please everyone completely, but I think that you'll have a fairly scientific measuring stick that addresses many facets of play. And that should be good enough to test out for some time. Perhaps you should appoint a panel of observors to study how the scoring system affects the gaming experience here over a trial period of a few months, and then assess the findings and consider tweaks.

I suggest appointing a panel of people of your choosing based on your personal evaluation of the value of their opinions to lessent the workload on you, to gain a balanced perspective from multiple sources, but also to minimize the conversation to a manageable level. In the end it strikes me that you should act as a judge having weigh the carefully presented evidence of said panel. That way there's some representation of the players without you having to wade through 100s of posts.

Hi Oli, I think your current proposal looks pretty good. Thanks for all of the work in making this happen!

I just had a question about the spec:

"Normalize the gV to the number ... winners+survivors ..."
Why is the gV being normalized by the number of winners+survivors? I guess that makes sense for a PPSC solo. However, for a WTA solo, I think there should be no normalization (i.e., divide by 1 for 1 winner).

Oli,

Thank you for going through all this effort to get as much input as possible. I agree with you when you said: "But we are now at a point where it's impossible to integrate some other cool ideas in a single algorithm because they work totally different.
I will implement my algorithm the next few days, and I will integrate all other ideas in separate calculations too, so we can compare them all in one big table."

That is why I proposed early on to have more than one rating system. I hope we can still consider having parallel systems to accommodate the differing opinions.

Everyone,

Sorry, I have been too busy to follow this thread for a while, but one point I made has gotten lost. If we are using historical data, we should have a rating system that most mirrors the way the games WERE played. This is partly why I suggested that the points be distributed the same as WTA and PPSC, with the input to the pot varying based on skill. I think this is a very different idea than the Oli's original ELO system, or yebellz's refinement, where the distribution of skill is based on after-the-fact determination of which is better.

Sorry Decima, I don't know how your system distributes the skill-points, but I think it is similar to mine.

Also, related to it, I do agree with Decima regarding the problem of metagaming. If we have a system that uses a variable result depending who shares in a draw, a systematic metagaming problem will result. Without a doubt, if a player knows that eliminating one player instead of another earns them more skill-points, some players (right or wrong) will make decisions that should not be part of the game.

I agree that all games are subject to some metagaming, but if we create a systematic reward, the problem will be amplified.

I dismissed the country-ratings, because the performance of each country is heavily depended on the player playing that country. But if you can give me some kind of calculation how you think this could work add your calculation to the wiki, and I will try to code this.

Devonian, welcome back.

In the last days I’ve been trying to carry on with your old point “we should have a rating system that most mirrors the way the games WERE played”.

There is no significant difference among your and my proposal regarding points awards (the positive part of net result, namely, the Real Result). As you did with your proposal, I wanted the criteria of points distribution of D-point system and skill system not to be conflicting with each other.

The difference among your and my proposal is the negative part of net result (called “bet” in your system, called “Expected Result” in mine).

Also, another non-negligible difference between your proposal and all the other proposals (Elo-based) is that your system uses a linear scale for ratings, the others use a logarithmic scale. Because of this, reading a Mr.Green rating of 1400 in your proposal or in the other 3 proposals takes quite a different meaning. Those numbers are not directly comparable.