AggPro Early Results

I will be finishing the AggPro Projection Research this week. Most of the work has been completed and the results look good. The weights for all constituent projection systems (Bill James, CHONE, Marcel, PECOTA and ZiPS) are as follows: 0.56 = Bill James, 0.00 = CHONE, 0.21 = Marcel, 0.23 = PECOTA and 0.00 = ZiPS. Recall, these are the weights that when applied to the respective constituent projection systems for the years 2007 and 2008 mimize the standard error of the resulting aggregate projection from the actual 2007 and 2008 Major League Baseball data.

How well did we do?

The previous post described how AggPro would be measured. In short, AggPro will be considered successful if 1) the resulting aggregate projections are more accurate than any of the constituent projections for 2007, 2008, and 2009 and 2) if AggPro projections for each statistical category, for each year are more accurate than any constituent projection for the given category in the given year. The first evaluation criterion is almost trivial. Obviously, AggPro can be as accurate as the most accurate constituent projection system simply by giving a weight of 1.00 to the most accurate projeciton system. I was slightly skeptical if AggPro would be able to meet the second measure of success. It seemed possible a constituent projection system might be very accurate for predicting a statistic in a single statistical year but might mispredict almost every other category in that year or other years.

It seems as if AggPro has met this measures of success. The AggPro projections are more accurate in every statistical category for 2007 and 2008 MLB seasons than any of the constituent projection systems. In 2007 the AggPro projections are a 4.25% improvement over the leading constituent projection system (Bill James). In 2008 the AggPro projections are a 5.9% improvement over the leading constituent projection system (again Bill James). Furthermore, if one was to form a theoretical projection system by taking the most accurate constituent projection for a statistical category each year AggPro would still be more accurate in 2007 and 2008. This system is described in the AggPro baselines table in the previous post. In 2007 AggPro would be a 2.65% improvement over the theoretical system. In 2008 AggPro would be a 4.6% improvement over the theoretical system.

I have several theories as to why the Bill James predictions were so accurate (given AggPro's measures of accuracy) for these years that I may go into in a seperate post. I'll be doing a pro-rated evaluation for the 2009 season later this week. This will be a true test of AggPro's accuracy as the optimization which chose the weights did not consider any 2009 data.

Projected Categories, Players and Measures of Success

Cameron and I have spent the last several weeks acquiring and formatting the projections from Bill James, CHONE, Marcel, PECOTA and ZiPS. We are going to be using the 2007 and 2008 projections from these systems. I will refer to these projections as the constituent projections. Originally, we had planned on using the 2006 projections from these systems too, but as Chone Smith nicely explained in this comment the 2006 projections from CHONE were more experimental than anything else and probably would not be particurally useful. Since we need projections for all systems for all the years that we include in the AggPro analysis we had to eliminate the 2006 projections.

Projected Categories

We will be using the following player performance categories:

Hitters: At Bats, Hits, Runs, Doubles, Triples, Home Runs, RBIs, Strikeouts, Walks and Stolen Bases.

Pitchers: Innings Pitched, Earned Runs, Strikeouts, Walks and Hits.

This is the set of hitter and pitcher performance categories that is common to all the constituent projection systems.

Projected Players

The following lists contain the players AggPro included in the AggPro analysis: 2007, 2008.

The player list for a given year is the list of players that are common to all the constituent projection systems for the year.

AggPro Measures of Success

I also have determined the standard error of the 2007 and 2008 constituent projections from the actual Major League Baseball player performance data in each of the categories. The standard error, as a percentage of the actual population of each category, for each system, is listed below. Double click on the table to enlarge it.

AggPro will be successful if it identifies a single weight to apply to each projection system such that the resulting AggPro projections have less standard error for a given statistical category in a given year than the best constituent system projection for that category, for that year. For example, each category in the AggPro projections for 2007 must have less standard error than corresponding value for the category identified in the righthand most column in the top half of the table. Based on the early results from the simulated annealing optimization it looks like an AggPro cocktail consisting of 2 parts Bill James, 1 part Marcel and 1 part PECOTA comes very close to meeting this goal.