Yes, it is 6.75% better than blind guessing. The 56.75% number is a bit misleading in it's measure of how successful my simulator is because a pure win/loss percentage is not the best measure of what it is doing. It is attempting to output a win probability for each team/game. If Team A and Team B were playing each other 2000 times, with the exact same starting lineups for each game, this is how many times you'd expect each team to win. Well, in real life they aren't playing each other 2000 times. A better measure might be to compare the win probability to something along the lines of the Vegas odds and see which does better. Alot of it depends on which games are simulated. If only close games are simulated then my number is going to approach 50%, if the games that I simulate are not evenly matched teams, then my number is going to be quite higher.
What goes into my simulator is a bit complex when it comes to the actual algorithms I use, but I can mention what some of the algorithms look into. I may forget a few things, as I am doing this off the top of my head. :)
1. 2000 games are played, with the exact same lineup, pitcher and home team.
2. Real 2007 statistics are used for each player, or where sample sizes are deemed to small, ZIPS or CHONE projections are used.
3. All teams follow the same managerial algorithms for pinch hitting, stealing, sac bunting, changing pitchers. In most cases the managerial algorithms are quite conservative as to not have a large effect on the outcome.
4. Pitchers tire based on pitch counts. Pitch counts are determined by the league average pitches thrown for each at bat/outcome. When pitchers tire, they give up more hits. There are sabermetrical algorithms for all of these.
5. Left handed hitters do better against right handed pitchers and right handed hitters do better against left handed pitchers. There is a sabermetrical algorithm based on league average splits.
6. Runners with more speed, are more successful at stealing bases and advancing an extra base on a hit.
7. Against a league average pitcher (LAP), someone with a .400 OBP should get on base at around a .400 clip. Against a LAP someone who gets a home run for every 10th hit they get, will perform at or near this level. Pitchers who give up less home runs, will make it harder for hitters to hit homeruns etc... Defense independent stats follow league averages.
8. Basically, the simulator takes player stats as input, uses sabermetric algorithms to determine the probability of each at-bat event, uses algorithms for managerial moves and plays 2000 games to come up with a win probability. The 2000 game level was picked as to lower the margin of error. This is basically, a Monte Carlo method for predicting the likely outcomes of baseball games. A baseball game is very complex and nearly impossible to model mathematically. But there are mathematical formulas/algorithms developed by some very smart people that can be programmed and used with the help of a random number generator.
The simulator is still far from perfect, as it doesn't take defense into account. Perhaps I will add this at some point, but want to make sure it is done correctly and not subjectively. Some factors that may mitigate against predicting baseball scores or likely outcomes could be small sample sizes, injuries, incorrect modelling (too much or not enough emphasis on something). No science is perfect for doing this in my opinion, you just try to do the best you can and test, test, test and make improvements.
2 comments:
So that's 6.75% better than blind guessing. Is that significantly better?
Can you discuss sometime what goes into the predictions, and what factors militate against predicting baseball scores being a science?
Yes, it is 6.75% better than blind guessing. The 56.75% number is a bit misleading in it's measure of how successful my simulator is because a pure win/loss percentage is not the best measure of what it is doing. It is attempting to output a win probability for each team/game. If Team A and Team B were playing each other 2000 times, with the exact same starting lineups for each game, this is how many times you'd expect each team to win. Well, in real life they aren't playing each other 2000 times. A better measure might be to compare the win probability to something along the lines of the Vegas odds and see which does better. Alot of it depends on which games are simulated. If only close games are simulated then my number is going to approach 50%, if the games that I simulate are not evenly matched teams, then my number is going to be quite higher.
What goes into my simulator is a bit complex when it comes to the actual algorithms I use, but I can mention what some of the algorithms look into. I may forget a few things, as I am doing this off the top of my head. :)
1. 2000 games are played, with the exact same lineup, pitcher and home team.
2. Real 2007 statistics are used for each player, or where sample sizes are deemed to small, ZIPS or CHONE projections are used.
3. All teams follow the same managerial algorithms for pinch hitting, stealing, sac bunting, changing pitchers. In most cases the managerial algorithms are quite conservative as to not have a large effect on the outcome.
4. Pitchers tire based on pitch counts. Pitch counts are determined by the league average pitches thrown for each at bat/outcome. When pitchers tire, they give up more hits. There are sabermetrical algorithms for all of these.
5. Left handed hitters do better against right handed pitchers and right handed hitters do better against left handed pitchers. There is a sabermetrical algorithm based on league average splits.
6. Runners with more speed, are more successful at stealing bases and advancing an extra base on a hit.
7. Against a league average pitcher (LAP), someone with a .400 OBP should get on base at around a .400 clip. Against a LAP someone who gets a home run for every 10th hit they get, will perform at or near this level. Pitchers who give up less home runs, will make it harder for hitters to hit homeruns etc... Defense independent stats follow league averages.
8. Basically, the simulator takes player stats as input, uses sabermetric algorithms to determine the probability of each at-bat event, uses algorithms for managerial moves and plays 2000 games to come up with a win probability. The 2000 game level was picked as to lower the margin of error. This is basically, a Monte Carlo method for predicting the likely outcomes of baseball games. A baseball game is very complex and nearly impossible to model mathematically. But there are mathematical formulas/algorithms developed by some very smart people that can be programmed and used with the help of a random number generator.
The simulator is still far from perfect, as it doesn't take defense into account. Perhaps I will add this at some point, but want to make sure it is done correctly and not subjectively. Some factors that may mitigate against predicting baseball scores or likely outcomes could be small sample sizes, injuries, incorrect modelling (too much or not enough emphasis on something). No science is perfect for doing this in my opinion, you just try to do the best you can and test, test, test and make improvements.
Thanks for the comment/questions.
vr, Xei
Post a Comment