Once you get to ten thousand games, I think that the stats are significant. I'm curious to consider that maybe Austria is stronger but only in the hands of players that are highly competent.

I wonder if anyone has any theories why?

The Austria/Italy comparison (and all of Diplomacy really) comes down to the smooth tongue of the full press players or the strategic acumen of the gunboat players in the pool being studied. If you study face 2 face tournaments, you are bound to see different stats over the online where anyone and the village idiot can play.

All excellent points. There is certainly a difference in how skilled players handle a game versus less experienced hands, but a large enough set of stats is predictive more generally, if not for the specific situation. I was more just amused by the anomaly in this symmetrical map where the result should be uniform across the countries over time.
In addition to the skill vs unskilled stats, we also have gunboat vs press games clumped together. In press games, you can offset the advantages a country may have in a gunboat game, especially when recognized by multiple players that see the need to offset that advantage. In gunboat, the lack of clean communication increases the advantages of superior early positioning.

One can actually quantify how reliable stats are by using probability and statistics. One can calculate the expected variance under the assumption that all countries are equally strong and compare this to the actual variance as is measured by the site. I don't have time to check it now, but it could very well be that the expected variance of the map Chromatic doesn't differ much from the measured one.

When you have tens of thousands of data points, the stats should be reliable. So, the difference between Austria and Italy performance that Enriador mentioned has most likely to do with other factors, like ScubaSteve and YouCan'tHandleTheTruth mentioned - the performance of countries differs under different settings.

Why not allow fact freaks to slice stats per variant, press mode, end condition and scoring system? That way the stats would acutally have some value once there's enough data in each relevant combination.

It varies based on the map, though. For example, in Classic, Austria is vulnerable at the start, but becomes tough to stop further on in most games. Starting, Austria is at a disadvantage. But further on, it changes.

Victory stats are also annoying when one doesn't consider the ratings of players. I would love an average rating of players next to countries in the stats.

My main point is that the environments mentioned results in a higher overall skill level playing Austria and Italy in one versus the other, thereby altering the stats. There are too many variables at play to simply look at performance of a nation across a large data set. Just as in science, you must have clean data and reduce the variables as much as possible. You simply cannot do that with internet Diplomacy or even tournament F2F Dip.

Wouldn't performance of a country over a large data set be a great indicator of that country's performance... under those conditions?

As the stats approach the long run (and the long run can be... very... very... long) the effects of player skill (i.e. more skilled players playing certain countries) would be balanced out. The variance of this variable would approach 0.

Of course, if there were a reason that better players had certain countries more often (tournament structure, for example) that would need to be examined and could affect the quality of data.

Including rating data would help the long run become shorter but, with a long enough long run, would not be necessary.

@Scuba, tournaments are more along the lines of a professional event (like an NBA game) and internet games more along the lines of a high school game. Do you really think home court advantage is going to balance out the same between the two? Tournament players will *all* be a higher caliber player where as internet games will span a broader spectrum. The strength of the country may vary among pros who are better at strategy and at diplomatic influence and can adjust their game play for the strengths and weaknesses of the country they have drawn. The amateur crowd will more accurately reflect the relative country strengths on the large data but will vary significantly from the pro crowd who can compensate for weaknesses and exploit strengths of their own country as well as weaknesses of other countries.

I agree with @Truth. There is a difference between pro-players and your average joes (nothing against playing with average joes - can still be a lot of fun). I suspect the balance of games tends to have more average player games or a mix of average and skilled players, and fewer online that consist of just high-skill players, so that an accumulation of stats probably reveal more what a game played by lesser to moderate skilled players will be like on average in the long run, not what a skilled game will be like. On the other hand, the pro-tourney stats should be good for getting that info, so maybe these online stats are good for painting that other picture :)