Thursday, July 30, 2009

Official Writeup of AggPro

Tomorrow Cameron and I will be presenting AggPro at SABR 39. The official writeup of AggPro complete with evaluation of the AggPro projections for the 2006, 2007, 2008 and 2009 MLB seasons is availalbe here. The slides we are going to present are available here. AggPro has been a fun project but like most of my other research once I hit the final revision stage of the writeup I really start to HATE the project.

Wednesday, July 1, 2009

AggPro Final Results

The final AggPro results are listed in the table below. They are significantly better than I expected. In each year AggPro is more accurate than any of the constituent systems. Furthermore, each year it more accurate than a theoretical projection system that would be composed of the best constituent projection for each statistical category. Finally, AggPro's projection in every category for every year is more accurate than the any constituent system's projection for a given category in a given year except for one category in one year. (As of June 29th PECOTA's projections for this year (2009) for triples are less than 1% more accurate than AggPro's 2009 projections for triples.) These observations seem to show that these different constituent projection systems work in a complementary nature (i.e. the overall aggregate projection is more than just the sum of its best parts.)

Recall, the AggPro projections were formed by determining the weights for the constituent systems that minimized the standard error from the actual Major League Baseball data for 2007 and 2008. We were able to evaluate the AggPro projections for 2006 because AggPro does not weight the CHONE projections at all. The CHONE projections were the only constituent projections missing for 2006.


YearAggPro Improvement over BCPSAggPro Improvement over TPS
20065.25%5.00%
20074.25%2.65%
20085.90%4.60%
2009 (through 06/29/09)2.4%2.0%


Table Legend: BCPS = Best Constituent Projection System; TPS = Theoretical Projection System

Monday, June 29, 2009

AggPro Early Results

I will be finishing the AggPro Projection Research this week. Most of the work has been completed and the results look good. The weights for all constituent projection systems (Bill James, CHONE, Marcel, PECOTA and ZiPS) are as follows: 0.56 = Bill James, 0.00 = CHONE, 0.21 = Marcel, 0.23 = PECOTA and 0.00 = ZiPS. Recall, these are the weights that when applied to the respective constituent projection systems for the years 2007 and 2008 mimize the standard error of the resulting aggregate projection from the actual 2007 and 2008 Major League Baseball data.

How well did we do?

The previous post described how AggPro would be measured. In short, AggPro will be considered successful if 1) the resulting aggregate projections are more accurate than any of the constituent projections for 2007, 2008, and 2009 and 2) if AggPro projections for each statistical category, for each year are more accurate than any constituent projection for the given category in the given year. The first evaluation criterion is almost trivial. Obviously, AggPro can be as accurate as the most accurate constituent projection system simply by giving a weight of 1.00 to the most accurate projeciton system. I was slightly skeptical if AggPro would be able to meet the second measure of success. It seemed possible a constituent projection system might be very accurate for predicting a statistic in a single statistical year but might mispredict almost every other category in that year or other years.

It seems as if AggPro has met this measures of success. The AggPro projections are more accurate in every statistical category for 2007 and 2008 MLB seasons than any of the constituent projection systems. In 2007 the AggPro projections are a 4.25% improvement over the leading constituent projection system (Bill James). In 2008 the AggPro projections are a 5.9% improvement over the leading constituent projection system (again Bill James). Furthermore, if one was to form a theoretical projection system by taking the most accurate constituent projection for a statistical category each year AggPro would still be more accurate in 2007 and 2008. This system is described in the AggPro baselines table in the previous post. In 2007 AggPro would be a 2.65% improvement over the theoretical system. In 2008 AggPro would be a 4.6% improvement over the theoretical system.

I have several theories as to why the Bill James predictions were so accurate (given AggPro's measures of accuracy) for these years that I may go into in a seperate post. I'll be doing a pro-rated evaluation for the 2009 season later this week. This will be a true test of AggPro's accuracy as the optimization which chose the weights did not consider any 2009 data.

Monday, June 15, 2009

Projected Categories, Players and Measures of Success

Cameron and I have spent the last several weeks acquiring and formatting the projections from Bill James, CHONE, Marcel, PECOTA and ZiPS. We are going to be using the 2007 and 2008 projections from these systems. I will refer to these projections as the constituent projections. Originally, we had planned on using the 2006 projections from these systems too, but as Chone Smith nicely explained in this comment the 2006 projections from CHONE were more experimental than anything else and probably would not be particurally useful. Since we need projections for all systems for all the years that we include in the AggPro analysis we had to eliminate the 2006 projections.

Projected Categories

We will be using the following player performance categories:
Hitters: At Bats, Hits, Runs, Doubles, Triples, Home Runs, RBIs, Strikeouts, Walks and Stolen Bases.
Pitchers: Innings Pitched, Earned Runs, Strikeouts, Walks and Hits.

This is the set of hitter and pitcher performance categories that is common to all the constituent projection systems.

Projected Players

The following lists contain the players AggPro included in the AggPro analysis: 2007, 2008.

The player list for a given year is the list of players that are common to all the constituent projection systems for the year.

AggPro Measures of Success

I also have determined the standard error of the 2007 and 2008 constituent projections from the actual Major League Baseball player performance data in each of the categories. The standard error, as a percentage of the actual population of each category, for each system, is listed below. Double click on the table to enlarge it.

AggPro will be successful if it identifies a single weight to apply to each projection system such that the resulting AggPro projections have less standard error for a given statistical category in a given year than the best constituent system projection for that category, for that year. For example, each category in the AggPro projections for 2007 must have less standard error than corresponding value for the category identified in the righthand most column in the top half of the table. Based on the early results from the simulated annealing optimization it looks like an AggPro cocktail consisting of 2 parts Bill James, 1 part Marcel and 1 part PECOTA comes very close to meeting this goal.

Friday, May 22, 2009

Welcome to AggPro

Welcome to the home of AggPro: The Aggregate Projection System. I have tried to anticipate commonly asked questions about AggPro and provide answers to them below. As AggPro is developed, status updates and results will be posted on this blog. If you have any questions regarding AggPro feel free to contact me. Cameron Snapp will be assisting me with the development of AggPro.

What is AggPro?

AggPro is a proposed Major League Baseball (MLB) player projection system. Currently there exist many different projection systems that predict the performance of MLB players in a variety of different statistical categories. The goal of AggPro is to aggregate the MLB player predictions from the existing systems into a single more accurate prediction. AggPro will use the following projection systems, Bill James Handbook, CHONE, Marcel, PECOTA and ZiPS, for the years 2006, 2007, and 2008 to form the aggregated projections.

Why do we need yet another projection system?

It is important to note that AggPro is not just another projection system. Instead it is a methodology for aggregating effective projections from different systems into a single more accurate projection. That said, we probably don't need another projection system. It appears we are reaching the limit of the accuracy that can be expected from projection systems. While AggPro may improve the accuracy of existing systems I do not expect it to improve the state of the art significantly. However, Greg Rybarczyk believes paradigm shifts that will improve the accuracy of projection systems are on the horizon. If paradigm shifting projection systems are developed, the AggPro methodology will be applicable to improve these systems as well.

My interest in aggregating predictions from effective projection systems to form a better prediction stems from BellKor, the leading solution to the Netflix Prize. In October, 2006 Netflix released a dataset of anonymous movie ratings and challenged researchers to develop systems that could beat the accuracy of its recommendation system, Cinematch. A grand prize, known as the Netflix Prize, of $1,000,000 will be awarded to the first system to beat Cinematch by 10%. The BellKor prediction system, with 8.26% improvement over Cinematch, is the leading solution. BellKor employs 107 different models of varying approaches and uses mathematical optimization methods to weight each prediction. From the weighted predictions of the 107 models BellKor forms its aggregate recommendation. I was curious how well this strategy would work when applied to MLB player projection systems, thus AggPro was born.

How will AggPro work and how will it be evaluated?

AggPro will employ a combination of mathematical optimization methods including Hill climbing, Genetic algorithms, and Simulated annealing to determine the weight for each projected statistic in each system that yields the AggPro projections with the least root mean square error (RMSE) from the MLB players’ actual performance in the 2006, 2007, and 2008 seasons. Using the same weights AggPro will provide projections for the 2009 MLB season.

All the projection systems (AggPro, Bill James Handbook, CHONE, Marcel, PECOTA and ZiPS) will be evaluated against the root mean square error from the actual player performance for the 2006, 2007, 2008 and 2009 MLB seasons.

When will AggPro be presented?

AggPro has been selected to be presented at the 2009 Society for American Baseball Research Convention (SABR 39) in Washington, D.C. on July 29th - August 2nd. The abstract proposing AggPro is available here.