By E.J. Fagan
August 22, 2023
***
NOTE: The following comes from EJ Fagan's substack page and is shared with permission.
Please check out EJ's substack page for more great articles.
***
Recently, former Yankee minor league farm hand wrote about the deficiencies of the Yankee farm system in the comments on Foul Territory. Erik Kratz pretty much confirmed everything he said:
Ruta describes a lot of weird drills that Dillon Lawson and a lot his proteges had hitters going through, like measuring the velocity that a player can throw a medicine ball from his hitting stance. But the important theme to his and Kratz’s comments is that the Yankees were obsessed with maximizing exit velocity. Players would get points, presumably leading to promotions or demotions, based on how often they hit the ball hard.
I’ve been writing about how hitting the ball hard isn’t always a good thing. But let’s ignore that for now. Hitting the ball hard is one way to succeed in baseball. I want to ignore it because I think the Yankees are running into one of the classic mistakes in the use of metrics in an organization: Goodhart’s Law.
Metrics are Dangerous
Goodhart’s Law, which we usually apply to government or private organizations that use metrics to make decisions is that, “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” It means that if you rely on a narrow set of metrics to make important decisions, your employees will adjust their behavior to increase the metric at the cost of other things that might be desirable but don’t increase the metric.
The classic example of Goodhart’s Law is standardized testing in schools. It’s hard to tell if a teacher or school is doing a good job, but we want our schools to be better. In the 2000s, largely due to George W. Bush signing the No Child Left Behind Act into law, we started to use standardized testing to determine if students were being taught successfully. If too many students failed the standardized tests, we punished the school. In some states, individual teachers were rewarded if their students did well on the standardized test.
What happened was entirely predictable: teachers started to teach to the test. They spent more and more of their time making sure that their students learned exactly what they needed to do well on the standardized test. Things that weren’t on the test that teachers used to teach because they thought it was valuable were cut. Students suffered because passing a standardized test is not a very useful skill. The federal standardized testing regime quietly collapsed when both parties decided it wasn’t working in 2015.
When you tell employee that the most important thing is that they maximize a metric, they will maximize the metric. In the case of exit velocity, they will swing harder. All the time.
Regardless of the situation. Swinging hard all the time may be an okay solution in the low minors, but I don’t think it’s a coincidence that so many top Yankee hitting prospects have stalled out at Triple-A or the majors, where batters have to deal with way more good breaking stuff. When you tell prospects that only exit velocity matters, they will sell out for exit velocity.
The Better Way to Make Decisions with Data
Okay, Dr. Fagan, so the Yankees shouldn’t use statistics to evaluate prospects? No! Of course not. Goodhart’s Law doesn’t mean stop using data. It means that you need to do two things: take a more holistic view of data, and use metrics that are better aligned with the incentives that you want to create.
A more holistic view of data just means not to obsess over one metric. Exit velocity can be really good! But you might also care about strikeout rates, situational hitting rates, on base percentage, barrel rate, etc. You want to take a step back and get a complete picture of a player’s performance. You might also use non-metrics, old school evaluative methods to supplement a statistical analysis.
Using metrics that better align with incentives means focusing on outcomes not outputs. An outcome is the end result of something that occurs on a baseball field, such as an out, hit, home run, walk or strikeout. An output is something that is supposed to translate into that outcome, such as velocity, swing speed, spin rate or launch angle.
All outcomes are the product of outputs, but they can come in different ways. A 110 mph ground ball can produce a single, but so can a 70 mph bloop fly ball. Both are worth the same. In the long run*, it doesn’t matter how a batter produced their singles. They all count the same on the scoreboard.
In practice, focusing on outcomes means using real basic statistics like wOBA or slugging percentage to make decisions. Some prospects might increase their wOBA by hitting a bunch of bloop singles while others hit the ball really hard to produce more home runs and hard ground balls, or whatever combination of skills you can imagine.
Of course, the holistic rule still applies. A hitter might adopt a strategy that works in the low minors but won’t in the majors, like taking a ton of walks. For instance, Anthony Siegler hit .236/.405/.369 in 2022 between A-ball levels with more walks than strikeouts. It would be naïve to observe those data and predict that Siegler was about to breakout as a super high-OBP catcher. Higher level pitchers can throw strikes when they think a batter is just sitting there not swinging. And of course, they did. Siegler hit .171/.335/.253 at Double-A this year.
The Yankees need to get smarter about how they make decisions with data. There was a line in Oppenheimer that was something along the lines of, “I see the music in the math.” A good data-literature analyst sees the music, not the notes. They can synthesize a lot of information together and decide if a player is successful. A bad data analyst has a checklist that says, “this thing good, this thing bad.” When those analysts are given power, everyone will conform to their process.
It’s Not Working
Outcomes, not outputs. What is the ultimately outcome for a farm system? Developing major league players. The Yankees have shown success in developing major league pitchers, but utter failure in developing hitters. The Yankees first brought in Dillon Lawson in 2018. The farm system has graduated zero average or better major league hitters since then. Maybe Volpe and Peraza will correct that trend soon, but it’s hard to see their struggles as major league hitters and not attribute some of it to “max exit velocity all the time.”
Ultimately, Brian Cashman is the person responsible for the Yankees playing poorly. I think the Yankees would be better off with someone else, but at the very least he should consider a complete revamp of their minor league hitting coaching system. I wonder if we’re going to hear a lot more stories like Ben Ruta’s soon.
