Baseball Articles | Index
2005 Predictions -- Keeping Score
By Tom Tippett
When we release our annual Projection Disk in the spring, we give our customers a chance to get a head start on the baseball season. With projected statistics and ratings for over 1600 established big leaguers and top minor-league prospects, plus league schedules, park factors, team rosters, projected pitching rotations, bullpen assignments, lineups and depth charts, the Projection Disk gives them everything they need to play out the new season using the Diamond Mind Baseball simulation game.
It also gives us a chance to get a head start on the season. Ever since we created the first Projection Disk in 1998, we've been publishing our projected standings along with comments on the outlook for all 30 teams. Those projected standings are based on the average of a number of full-season simulations using the Projection Disk.
Of course, nobody really knows what's going to happen when the real season starts, but we're always curious to see how our projected results compare to the real thing. And we're equally interested in seeing how our projections stack up against the predictions made by other leading baseball experts and publications. This article takes a look at those preseason predictions and identifies the folks who were closest to hitting the mark in 2005. And because anyone can get lucky and pick the winners in one season, we also look at how everyone has done over a period of years.
In addition to projecting the order of finish, our simulations provide us with projected win-loss records, projected runs for and against, and the probability that each team will make the postseason by winning its division or grabbing the wild card.
Unfortunately, most of the predictions that are published in major newspapers, magazines and web sites don't include projected win-loss records. Instead, they give the order of finish without indicating which races are expected to be hotly contested and which will be runaways. Some don't even bother to predict the order of finish, settling for the division winners and wild card teams.
As a result, we do our best to assign a meaningful score to each prediction based solely on order of finish within each division. We borrowed the scoring system from our friend Pete Palmer, co-author of Total Baseball and The Hidden Game of Baseball, who has been projecting team standings for more than 35 years.
Pete's scoring system subtracts each team's actual placement from its projected placement, squares this difference, and adds them up for all the teams. For example, if you predict a team will finish fourth and they finish second, that's a difference of two places. Square the result, and you get four points. Do this for every team and you get a total score. The lower the score, the more accurate your predictions.
We don't try to break ties. If, for example, two teams tie for first, we say that each team finished in 1.5th place for the purposes of figuring out how many places a prediction was off. Suppose a team was projected to finish third and they tied for first instead. That's a difference of 1.5 places. The square of 1.5 is 2.25, so that would be the point total for this team. That's why you'll see some fractional scores in the tables below.
Keeping things in perspective
That first year, we created a little database with our projected standings and those of fourteen national publications, and we were pleased to see that we ended the year with the best accuracy score among those fifteen forecasts. When we wrote up the results and posted them to our web site, however, we were very careful not to make any grand claims, saying:
Over time, we expanded our database to include the predictions of prominent baseball writers from major newspapers and other publications. This is easier said than done because some publications and web sites change their approach from year to year. For example, we used to track the predictions of several ESPN.com writers and editors, but they limited their picks to division winners in 2003. So the number of entries in our database can rise and fall depending on what the various publications do and whether we were able to find those predictions in our spring survey.
In the sections below, we'll show you how various prognosticators ranked in 2005 and over a period of years, with the period varying in length depending on when we added that person or publication to our database. We don't make any claims of completeness here -- there are lots of other predictions that are not in our database -- but we think you'll find that our sample is an interesting one.
For several reasons, we want to emphasize that it's important that nobody take these rankings too seriously.
First, this isn't the only scoring system one could use to rank these projections, of course. A fellow named Gerry Hamilton runs an Annual Baseball Predictions (ABP) contest (http://www.tidepool.com/~ggh1/index.html) and assigns a score based on how many games each team finished out of their predicted place in the standings.
Second, because of publishing deadlines, the predictions in some spring baseball magazines are made long before spring training started, others are prepared in early-to-mid March, and some are compiled just before opening day. Obviously, the longer you wait, the more information you have on player movement and injuries.
Third, many newspaper editors ask staff writers to make predictions so their readers have something to chew on for a couple of days. Some writers hate doing them but comply because their editors insist. Some do it even though their main beat is a different sport. Others may make off-the-wall picks just for grins or feel compelled to favor the hometown teams.
Rankings for 2005
It was a good year. In fact, it was a very good year. Our ranking using the Palmer scoring system was much lower than usual, but we're very happy with the way our projections held up this season. We'll explain why in a bit, but first, here are the rankings for 2005:
Forecaster Score Alan Schwarz 30 ABP consensus 30 Jackie McMullen, Boston Globe 31 ESPN.com power rankings 32 ESPN the magazine 32 Ken Rosenthal, Sporting News 33 Bob Ryan, Boston Globe 33 New York Daily News 34 Baseball Prospectus, PECOTA 35 Dallas Morning News 36 Boston Herald 37 Tony DeMarco, MSNBC.com 37 Peter Gammons, ESPN 39 Jay Jaffe, Baseball Prospectus 40 Los Angeles Times 40 Poll of SABR members 42 Will Carroll, Baseball Prospectus 42 Pete Palmer 42 Dayne Perry, Baseball Prospectus 43 USA Today Sports Weekly 44 Gordon Edes, Boston Globe 45 Scott Miller, CBS SportsLine 45 New York Times 45 Sports Illustrated 45 Las Vegas over-under line 45.5 Jonah Keri, Baseball Prospectus 46 Lindy's 47 Chris Snow, Boston Globe 47 Street & Smith 47 Washington Post 47 Baseball Digest 48 St. Louis Post-Dispatch 49 Seattle Times 49 2004 final standings 49 CBS SportsLine 50 David Kirsch, Baseball Prospectus 50 San Francisco Chronicle 50 Baseball America 51 Clay Davenport, Baseball Prospectus 51 USA Today 51 Rany Jazeyerli, Baseball Prospectus 52 James Click, Baseball Prospectus 53 Eric Mack, CBS SportsLine 54 Diamond Mind simulations 57 Adam Reich, CBS SportsLine 57 Nate Silver, Baseball Prospectus 57 Team payroll (per USA Today) 57 Chris Kahrl, Baseball Prospectus 59 Keith Woolner, Baseball Prospectus 60 David Lipman, ESPN.com 60 Chicago Tribune 60.5 Athlon 61 MLB Preview 61 Mike Gimbel 62 Dan Shaughnessy, Boston Globe 69 Joe Sheehan, Baseball Prospectus 72 Baseball Prospectus Today 74 Spring training results 120.5The "Diamond Mind simulations" entry is the one representing the average result of simulating the season 100 times. These simulations were done a couple of weeks before the season started.
There are a few other entries in this list that don't represent the views of an individual writer, analyst, or publication. If you predicted that the 2005 standings would be the same as in 2004, your score would have been 49. If you put together a set of standings based on the Las Vegas over-under line, you'd have racked up 45.5 points. If you thought the teams would finish in order from highest to lowest payroll, your score would have been 57.
The notion of strength in numbers has some support here. The consensus of the participants in the Annual Baseball Predictions contest -- which included most of the other entries on our list -- tied for first. A poll of SABR members finished in the top third. And the consensus of the Baseball Prospectus staff (not shown above) was in the top half, even though the individual predictions from the BP guys were all over the map.
And if you predicted that the regular season standings would match the 2005 spring training standings, your score would have been 120.5. As is the case every year, the spring training results were almost useless as a predictor of the real season.
Reviewing the season
Here's where we get back to why we're happy with our simulation results even though our ranking doesn't seem to support that view.
One reason has to do with the scoring system. Using Pete's method, it doesn't matter how close or far apart two teams were expected to be or actually were. The order of finish is the only thing that matters. You may have thought that team A would finish 12 games ahead of team B. If team A does indeed finish ahead of team B, by 20 games or 10 games or just one game, you're deemed to be correct in this method. There are no bonus points for being right, and no penalties for being wrong, when it comes to the number of wins.
I'm not saying this to suggest that the Palmer scoring system is suspect. It's not. There's only so much you can do when most predictors don't include the number of wins they expect from each team. And we've already made the argument that predictions need to be applied to a span of many years before you can start to identify the methods and the people who are consistently more accurate than others. Over a span of years, things should balance out. In some years, we've benefited from this aspect of Pete's method, while in other years it has hurt our score.
A second reason has to do with luck. Sometimes a team will win many more or many fewer games than normal given their offensive production and their ability to prevent runs. We'll see some examples of that when we review the 2005 season in a moment. This is another reason why predictors can't really be evaluated based on one season.
Our forecast for the AL East was similar that of most people. Everyone thought it would be New York first and Boston second, or vice versa. Generally speaking, the analytically oriented folks, such as Diamond Mind and Baseball Prospectus, had a more optimistic view of the Red Sox chances than did the sportswriter community. We projected 97 wins for New York and 96 for Boston, and they wound up tied with 95 despite myriad pitching problems on both teams.
Most forecasters pegged the Orioles for third place, with a handful putting Toronto in that spot. But many were fooled by the Devil Rays, who finished 4th in 2004 but didn't really have the statistical underpinning to support the idea that they had passed the Blue Jays. Our simulations put Baltimore at 80 wins, Toronto at 73, and Tampa Bay at 66.
In fact, it was Toronto that hovered around .500 and Baltimore that was several games below, largely because we didn't anticipate that Baltimore would be a little worse on both sides of the ball. For the decline in offense, you can point the finger at Sammy Sosa, and Sidney Ponson and Jorge Julio deserve most of the blame for the extra runs they allowed.
This was a fascinating division. Our simulations had Minnesota has the clear favorite, with Cleveland, Chicago and Detroit tied with 79 wins apiece, and Kansas City a long way back. The rest of the world seemed to view the division in the same way. Minnesota was picked first by almost everyone, and only two of the predictions in our sample -- Tony DeMarco and the Seattle Times -- picked the White Sox first. In fact, more people picked Cleveland to win the division than picked the White Sox.
When our pre-season simulations produced a three-way tie for second place, we chose to break the tie for the purposes of our predicted standings. On the basis of a slightly better run margin, we put Cleveland second, Chicago third, and Detroit fourth.
As it turned out, the Indians had the best statistical performance in the AL this year but finished behind the White Sox because of a huge disparity in one-run games. Chicago was 35-19 and Cleveland 22-36 in those contests. As a result, the White Sox are in the World Series and the Indians missed the playoffs altogether. I'll leave it to you to debate whether this massive swing in one-run games was due to skill or luck.
This was one of the more interesting results of our simulations. When most observers thought Oakland would have almost no chance after trading Mark Mulder and Tim Hudson, we had them eking out the division title by a game over the Angels. When many saw the Rangers as a strong challenger coming off an impressive 2004 season, we had them as a .500 team that would give up too many runs. Furthermore, we had the Mariners, bolstered by the additions of Richie Sexson and Adrian Beltra, rallying from a dismal 2004 season to finish three games ahead of the Rangers.
Well, it didn't quite work out that way. Oakland started very badly, was the best team in baseball for a few months, pulled ahead of the Angels for while, and then faded down the stretch (with ace Rich Harden on the sidelines). Los Angeles, as has become their recent custom, finished well behind the A's statistically but topped them in wins anyway. For a while, Oakland was making us look very good, but in the end, it was everyone else, all those folks who thought the Angels would run away with the division, who were smiling.
Meanwhile, Texas wasn't much of a factor, finishing third-last in the league run prevention and hanging around the .500 mark most of the year. And the Mariners were a disppointment, at least to us, winning only 69 games. Their run margin was good enough to support 75 wins, so they were a little unlucky. But Adrian Beltre created 33 fewer runs than we projected (even after adjusting for his pitcher-friendly home park) and Joel Pineiro was unable to bounce back as we had expected, so they didn't perform well enough to reach our 83-win projection anyway.
Ten predictions were exactly right on the order of finish in this division.
To all intents and purposes, our simulations nailed this one. We had all five teams finishing within 11 games of each other. The actual spread, from top to bottom, was 9 games. We had Philly and Atlanta a little ahead of the pack, with 89 and 86 wins, respectively. Those two teams did indeed pull ahead, though it was Atlanta and Philly, in that order, with 90 and 88 wins. We had the Marlins and the Mets, two teams that were being touted by lots of folks as potential division winners, tied for third with 82 wins. They actually tied for third with 83 wins. And we had Washington in fifth with a very respectable 79 wins. In the real season, they won 81.
The Nationals are perhaps the best example of the point I made earlier about the nature of the scoring system. We picked the Nationals to finish last. So did everyone else. But I'll bet most people thought the Nationals would finish a distant fifth, not be right on the heels of the Marlins and Mets. We can't know what was in the minds of the other predictors, but the Vegas over/under line was set at 70, and the few who did predict win totals had them in the upper 60s or low 70s, so there's some reason to believe that most people viewed Washington as one of the league's worst teams.
In other words, we may have been eight or nine games more "right" about the Nationals, but we don't get any credit for that in the Palmer scoring system. That may sound like sour grapes, but it's really not. Over the years, there have been a few cases where we missed a team by a number of games but got the order of finish right anyway, so it cuts both ways. But this is one of the reasons why we're happy with our 2005 simulation results even though our Palmer score doesn't look very good.
Our simulations presented the division in three tiers. St. Louis was the overwhelming favorite, the Cubs and Astros were in the vicinity of .500, and three bad teams were clustered in the low 70s in wins.
The real-life Cardinals lived up to our expectations, winning 100 games despite the loss of Scott Rolen for much of the year. As was the case with the Nationals, we may have been more "right" than the others who also had the Cardinals winning the division. The Vegas over/under line was at 93 wins, and the other predicted win totals we saw were in the low- to mid-90s. In fact, fifteen of the predictions in our sample had the Cards in second place.
All fifteen of those Cards-in-second predictions had the Cubs winning the division, but we didn't see how they could possibly contend with St. Louis. In our simulations, the Cubs offense was so-so and the pitching was good but not good enough to carry an average attack. Despite an awesome season from Derrek Lee, the offense was a little worse than we expected and the pitching was near expectations, so the Cubs did indeed finish near .500.
Houston was a big surprise. We projected them to have a weak offense after a winter of important free agency losses, and the pitching didn't appear to be anything special. Oddly enough, as of mid-May it appeared as if we were too optimistic, as they floundered at the bottom of the division and looked very much like a team in big trouble. Then they just stopped allowing runs. The offense was near our expectations, meaning that it was below average, but the staff led the league in pitching thanks to unexpectedly great seasons from Roger Clemens, Andy Pettitte, and several middle relievers.
Milwaukee was the other big surprise, and we completely missed it. We projected them to finish last in the league in scoring, and they went on to finish near the league average in runs. We projected them to be a below-average pitching team, and they finished 6th in fewest runs allowed. Bill Hall was the biggest offense surprise on a team where almost every position player was a little better than expected. Defensively, it was another team effort, with many solid contributions to go along with a surprisingly good year from closer Derrick Turnbow.
As expected, the Reds and Pirates were bad teams that finished near 70 wins.
Because this is the game's only 6-team division, it tends to produce the highest Palmer scores. In a four-team division like the AL West, you can't miss by more than three places (nine points), but you can miss by five places in the NL Central (25 points). So it's remarkable that Alan Schwarz and Jackie McMullin accumulated only two points in this division. Both had Chicago third and Milwaukee fourth with the other teams in their correct places.
We missed this division in a big way, but because of the way it happened, we don't feel all that bad about it. Our simulations put the teams in three tiers -- the Dodgers and Giants within two games of each other at the top, San Diego at .500, and the Rockies and Diamondbacks within two games of each other at the bottom.
The Dodgers were ravaged by injuries and missed our projected win total by a whopping 19 games. J. D. Drew, Milton Bradley, Jose Valentin, and Eric Gagne each missed at least half the season. We've never claimed to be able to predict injuries, and it's a good bet that every season is going to produce one disappointment like this one.
When we set up our simulations in March, we didn't know when Barry Bonds would be back. So we made an educated guess, based on published reports, and had him sit out the first two months of the season. As it turned out, of course, he missed all but the last two weeks. The pitching didn't live up to our expectations, either, but most of the decline came on offense.
The Padres finished with one more win in the real season than they did in our simulated seasons, but that 82-80 record was good enough to win the division.
The Diamondbacks were a big surprise, and we paid dearly in our Palmer score as a result, but we weren't wrong about them. Arizona finished with the worst run differential in the division, allowing 160 more runs than they scored. In fact, that was the worst run differential in the National League and the third-worst in the majors. But Arizona finished with a whopping 13 more wins than their run margin would normally support. That's the biggest upside difference in at least the last 32 seasons. That's not skill, folks, that's luck. Differences like this don't persist from season to season, so this is not a group of players that just knows how to win.
It's been a while since the Rockies have had any success, so I was nervous about their ability to finish ahead of the Diamondbacks this year, as our simulations projected they would. A two-game margin is almost meaningless in the first place, and my concern was only heightened when I saw that just about everyone else had picked Colorado to finish last.
After the first two months of the season, the Rockies were in last place with a dismal 15-35 record, including 4-23 on the road, and fourteen games back of the Diamondbacks. After that, it was a different story. From June 1st through the end of the season, the Rockies tied with the Giants for the best record in the division, albeit with a less-than-impressive 52-60 mark. And their road record, at 23-31, was respectable for a change.
We missed big on the White Sox, Indians, Astros, and Brewers, all of whom pitched and played defense at a much higher level than we expected. No excuses there.
We missed big on the Dodgers because we didn't foresee a glut of injuries. It's hard to feel too bad about that.
We missed big on the Giants, partly because Barry was out much longer than we expected, and partly because a few pitchers struggled. We have to take some of the blame for being wrong about the pitching.
And we missed big on the Diamondbacks, but we were not wrong about them. They won far more games than they deserved to win given their overall performance, and that was a fluke of gigantic proportions.
To a lesser extent, we missed on the Angels, who allowed many fewer runs than expected, and the Mariners, who were a moderate disappointment on both side of the ball.
But we were right on target in many other respects. We had the Red Sox and Yankees within a game of each other. We exposed the Devil Rays as a team still a long way from contending. We identified Oakland as a serious contender and saw that Texas wasn't really on the verge of greatness. We pretty much nailed the NL East, including the strong showing of the Nationals. We saw that the Cardinals would dominate their league and that the Cubs were pretenders, not contenders.
We would have liked to do even better, of course, but we're pretty happy with these results. I've pointed out a few areas where I believe our Palmer score makes us appear to have had a very bad year. We didn't get full credit for predicting that the Cardinals would run away with their division and the Nationals were a very respectable team. And we got burned when the Diamondbacks got very lucky.
There's another metric that we use internally to measure our performance. It's a statistic called standard error that compares projected wins to actual wins. Obviously, we can't use it to rank predictions that don't include win totals, but we can and do use it internally to measure our own projections.
We were pleased to see that our standard error for 2005 was the third best in the eight years we've been doing this. And it was better than it was in 2004, when our Palmer score was one of the best out there. That year, the Palmer method made us look a little better than we really were, and this year, it made us look worse. That can happen.
Partly because of luck, and partly because of quirks in the scoring system, one season isn't enough to tell you who has the best approach to predicting the coming season, so we'll devote the rest of this article to ranking the predictions on a multi-year basis.
Here are the rankings for those who were included in our sample every year.
Forecaster 2005 2004 2003 2002 2001 2000 1999 1998 Avg Las Vegas over-under 45.5 32.5 30.0 46.0 65.5 51.5 48.0 52.0 46.4
In 1999, we added some writers from the Boston Globe.
Forecaster 2005 2004 2003 2002 2001 2000 1999 Avg Gordon Edes, Boston Globe 45.0 52.0 32.0 54.0 56.5 26.0 28.0 44.8 Las Vegas over-under line 45.5 32.5 30.0 46.0 65.5 51.5 48.0 45.6
The Diamond Mind simulations missed the mark by quite a bit in 2000. We added a new concept to our projection system that year, but we were unhappy with the results, and we took that out of the model before generating our projections in 2001. The results have been much better since. As you can see, the Las Vegas over-under line has been getting much better in recent years.
Forecaster 2005 2004 2003 2002 2001 2000 Avg Las Vegas over-under line 45.5 32.5 30.0 46.0 65.5 51.5 45.2 Sports Illustrated 45.0 60.0 30.0 48.0 56.5 40.0 46.6 Sporting News 33.0 58.0 44.0 54.0 52.5 38.0 46.6 Gordon Edes, Boston Globe 45.0 72.0 32.0 54.0 56.5 26.0 47.6 Diamond Mind simulations 57.0 42.0 28.0 40.0 54.5 68.0 48.3 Athlon 61.0 48.0 36.0 38.0 67.5 42.0 48.8 USA Today Sports Weekly 44.0 66.0 38.0 42.0 46.5 58.0 49.1 Baseball America 51.0 60.0 28.0 48.0 54.5 54.0 49.3 Previous season standings 49.0 48.0 42.0 48.0 64.5 56.0 51.3 Pete Palmer 42.0 68.0 56.0 50.0 70.5 54.0 56.8 Street & Smith 47.0 62.0 36.0 70.0 68.5 58.0 56.9 Dan Shaughnessy, Globe 69.0 52.0 56.0 70.0 44.5 54.0 57.6 Bob Ryan, Boston Globe 33.0 76.0 40.0 58.0 84.5 58.0 58.3 Payroll ranking 57.0 46.0 64.0 102.0 60.0 88.0 69.5
MSNBC, Lindy's, and the LA Times were added in 2001.
Forecaster 2005 2004 2003 2002 2001 Avg Tony DeMarco, MSNBC.com 37.0 40.0 34.0 34.0 67.5 42.5 Lindy's 47.0 52.0 40.0 42.0 36.5 43.5 Las Vegas over-under line 45.5 32.5 30.0 46.0 65.5 43.9 Diamond Mind simulations 57.0 42.0 28.0 40.0 54.5 44.3 USA Today Sports Weekly 44.0 66.0 38.0 42.0 46.5 47.3 Sports Illustrated 45.0 60.0 30.0 48.0 56.5 47.9 Baseball America 51.0 60.0 28.0 48.0 54.5 48.3 Sporting News 33.0 58.0 44.0 54.0 52.5 48.3 Los Angeles Times 40.0 74.0 18.0 44.0 73.5 49.9 Athlon 61.0 48.0 36.0 38.0 67.5 50.1 Previous season standings 49.0 48.0 42.0 48.0 64.5 50.3 Gordon Edes, Boston Globe 45.0 72.0 32.0 54.0 56.5 51.9 Street & Smith 47.0 62.0 36.0 70.0 68.5 56.7 Pete Palmer 42.0 68.0 56.0 50.0 70.5 57.3 Bob Ryan, Boston Globe 33.0 76.0 40.0 58.0 84.5 58.3 Dan Shaughnessy, Globe 69.0 52.0 56.0 70.0 44.5 58.3 Payroll ranking 57.0 46.0 64.0 102.0 60.0 65.8 Spring training results 120.5 134.0 70.0 86.0 113.5 104.8
Here's how things looked from 2002 to 2005.
Forecaster 2005 2004 2003 2002 Avg Tony DeMarco, MSNBC.com 37.0 40.0 34.0 34.0 36.3 Las Vegas over-under line 45.5 32.5 30.0 46.0 38.5
Here's how things have looked from 2003 to 2005.
Forecaster 2005 2004 2003 Avg Las Vegas over-under line 45.5 32.5 30.0 36.0
Finally, here's how things have looked over the past two years.
Forecaster 2005 2004 Avg New York Times 45.0 30.0 37.5 Tony DeMarco, MSNBC.com 37.0 40.0 38.5 Las Vegas over-under line 45.5 32.5 39.0 APB consensus 50.0 30.0 40.0 Baseball Prospectus 35.0 52.0 43.5 ESPN.com 32.0 56.0 44.0 SABR poll 42.0 48.0 45.0 Sporting News 33.0 58.0 45.5 Previous season standings 49.0 48.0 48.5 Diamond Mind simulations 57.0 42.0 49.5 Lindy's 47.0 52.0 49.5 Eric Mack, Sportsline 54.0 48.0 51.0 Payroll ranking 57.0 46.0 51.5 Dallas Morning News 36.0 68.0 52.0 David Lipman, ESPN.com 60.0 44.0 52.0 Sports Illustrated 45.0 60.0 52.5 ESPN the magazine 32.0 74.0 53.0 Athlon 61.0 48.0 54.5 Street & Smith 47.0 62.0 54.5 Nate Silver, BP 57.0 52.0 54.5 Bob Ryan, Boston Globe 33.0 76.0 54.5 Pete Palmer 42.0 68.0 55.0 Baseball America 51.0 60.0 55.5 USA Today 51.0 61.5 56.3 Peter Gammons, ESPN 39.0 74.0 56.5 USA Today Sports Weekly 44.0 66.0 55.0 Rany Jazeyerli, BP 52.0 58.0 55.0 Jonah Keri, BP 46.0 66.0 56.0 Los Angeles Times 40.0 74.0 57.0 Joe Sheehan, BP 72.0 42.0 57.0