Part 1 of MLB Org Training Case Study
17 Hitters
One Affiliate
April 15, 2024
Part 1 of MLB Org Training Case Study is a look at how we delivered learning in uHIT to make results on-field in OPS
What you need to know:
Hitters increased OPS by +103 points on average while learning pitch and zone skills with uHIT training
Part 1 and Part 2 Overview
Part 1 of MLB org training case study tells you what effects uHIT Plus training had on hitters of one org’s affiliate. The assessments in zone and pitch recognition were given to all hitters before they began the 2016 AZL season. Re-assessments were given at the end of the season. For Part 1, we will examine the changes from the start to the end of the season, as measured in uHIT. In Part 2, we will do a similar analysis looking at the changes on-field for these hitters. Specifically, we will focus there on year-over-year OPS changes.
Pitch Recognition Changes (Part 1)
In Part 1 of MLB org training case study, we will focus on the three most important metrics of uHIT training. These metrics are important for a hitter using uHIT if using uHIT Plus, uHIT Custom (see here) or uHIT Gameday (see more here). Specifically, we will examine the hitter’s Accuracy, Reaction Time (for ‘yes’ decisions) and XP / Pitch (or Points Per Pitch). These metrics tell on a high-level how well and quickly a hitter is recognizing pitches (see more from other case studies here).
We will examine what changes in pitch recognition skills we saw in this org affiliate’s hitters. Before uHIT Custom was available, these hitters used uHIT Plus, a more self-guided mode to uHIT training. We evaluated whether consistent use of uHIT Plus would 1) increase Accuracy, 2) decrease Reaction Time (for ‘yes’ decisions), and 3) raise XP / Pitch. The last metric most of all was important to this analysis because it is a combination of accuracy and decision speed isolated to each pitch. Below, we discuss what we found.
Zone Recognition Changes (Part 1)
Also in Part 1 of MLB org training case study, we do a similar analysis for zone recognition skills. Here too, all hitters used uHIT Plus as a self-guided practice to improve their in-zone / out-zone recognition skills. Without the automatic customization of uHIT Custom, we evaluated if there was also 1) an increase in Accuracy, 2) decrease in Reaction Time and 3) an increase in XP / Pitch. But zone recognition training was much tougher in 2016 on uHIT (see more here). Since then, we have tempered it to more appropriately match a hitter’s ability in uHIT Custom. We will see impacts from this heightened difficulty when we evaluate the performance changes seen for these hitters from the beginning to end of the 2016 season.
OPS Changes (Part 2)
Part 2 of MLB org training case study will look at what effects we saw on-field. Specifically, we will examine the OPS changes these hitters demonstrated from 2016 ahead to 2017. By looking at the following season, we further tested an observation we’ve made with other pro hitters (see here). In particular, we tested whether the lessons learned for pitch and zone recognition skills would transfer from one year to the next. The data shows us they did. In Part 2, we will dive into those details.
Some Notes About the Data (Part 2)
Finally, we will sum up the last part with some notes on how we found on-field data. Sometimes players are released or play at different levels from year to year. Here, we will talk about our decisions to most judiciously select on-field OPS metrics to test our hypothesis of contributing to year-over-year OPS changes.
Pitch Recognition Changes: Better and Faster
17 Hitters, 12-minute Sessions
Between June 18 and Aug 17, the 17 hitters of the org’s AZL affiliate did pitch recognition skill work in uHIT Plus in 12-minute intervals. The hitters got the standard feedback on each pitch’s outcome deeming their responses correct or incorrect (see here for more). To gauge progress over the season, we took their first and last sessions using uHIT Plus. We recorded their Accuracy, Reaction Time (to ‘yes’ decisions) and XP / Pitch for each of these sessions.
Results Summary
Across June 18-19, the hitters did their first sessions of Pitch Recognition skill training in uHIT Plus. We found average scores across the hitters of:
- Accuracy: 71%
- Reaction Time: 0.480secs
- XP / Pitch: 67
There was not a lot of variability in these scores. We evaluated the spread in these values using the standard error (see here):
- Accuracy: +/-3%
- Reaction Time: +/-0.020secs
- XP / Pitch: +/-7
After 2 months of training, we took followup measurements in uHIT of these 17 hitters. As a group, they did uHIT Plus for 24 sessions on average (+/-2). As a result, we could evaluate changes from the start to the end of the AZL short season. Across last sessions Aug 17-18, we found average scores across the hitters of:
- Accuracy: 80%
- Reaction Time: 0.440secs
- XP / Pitch: 89
There was not a lot of variability in these scores. We evaluated the spread in these values using the standard error (see here):
- Accuracy: +/-3%
- Reaction Time: +/-0.010secs
- XP / Pitch: +/-6
We can visually see the differences between the start to the end of the season below:
Statistically Relevant Changes: Across The Board
Visually, the plots tell us that Accuracy looks largely unchanged. But both Reaction Time and XP / Pitch show changes from beginning to end of season. Our aim with any uHIT training period (Plus, Custom or Gameday) is for Accuracy to increase, Reaction Time (for ‘yes’ decisions) to go down and XP / Pitch to go up. We can test that aim (or hypothesis) statistically by using a one-sided t-test (see here for more). Specifically, we hypothesize that:
- Accuracy June 17-18 is less than Accuracy Aug 16-17
- Reaction Time June 17-18 is greater than Reaction Time Aug 16-17
- XP / Pitch June 17-18 is less than XP / Pitch Aug 16-17
With a one-sided t-test of each of these statements, we can provide the probability chance that the statement is correct. We get for each metric the following:
- Accuracy: 98.9% the statement is correct
- Reaction Time: 96.9% the statement is correct
- XP / Pitch: 99.2% the statement is correct
In other words, we can say with high certainty that all our aims were met for hitters using uHIT Plus for training in this AZL affiliate. By most standards of statistical tests (see here), we would claim that the intervening months of 24 sessions (+/-2) on average helped these hitters perform better at pitch recognition.
Zone Recognition Changes: Better and Faster
17 Hitters, 12-minute Sessions
Between June 18 and Aug 17, the 17 hitters of the org’s AZL affiliate did zone recognition skill work in uHIT Plus in 12-minute intervals. The hitters got the standard feedback on each pitch’s outcome deeming their responses correct or incorrect (see here for more). To gauge progress over the season, we took their first and last sessions using uHIT Plus. We recorded their Accuracy, Reaction Time (to ‘yes’ decisions) and XP / Pitch for each of these sessions.
Results Summary
Across June 18-19, the hitters did their first sessions of Zone Recognition skill training in uHIT Plus. We found average scores across the hitters of:
- Accuracy: 57%
- Reaction Time: 0.350secs
- XP / Pitch: 46
There was not a lot of variability in these scores. We evaluated the spread in these values using the standard error (see here):
- Accuracy: +/-1%
- Reaction Time: +/-0.020secs
- XP / Pitch: +/-3
After 2 months of training, we took followup measurements in uHIT of these 17 hitters. As a group, they did uHIT Plus for 24 sessions on average (+/-2). As a result, we could evaluate changes from the start to the end of the AZL short season. Across last sessions Aug 17-18, we found average scores across the hitters of:
- Accuracy: 58%
- Reaction Time: 0.240secs
- XP / Pitch: 56
There was not a lot of variability in these scores. We evaluated the spread in these values using the standard error (see here):
- Accuracy: +/-2%
- Reaction Time: +/-0.010secs
- XP / Pitch: +/-4
We can visually see the differences between the start to the end of the season below:
Statistically Relevant Changes: XP / Pitch and Reaction Time
Visually, the plots tell us that Accuracy looks largely unchanged. But both Reaction Time and XP / Pitch show changes from beginning to end of season. Our aim with any uHIT training period (Plus, Custom or Gameday) is for Accuracy to increase, Reaction Time (for ‘yes’ decisions) to go down and XP / Pitch to go up. We can test that aim (or hypothesis) statistically by using a one-sided t-test (see here for more). Specifically, we hypothesize that:
- Accuracy June 17-18 is less than Accuracy Aug 16-17
- Reaction Time June 17-18 is greater than Reaction Time Aug 16-17
- XP / Pitch June 17-18 is less than XP / Pitch Aug 16-17
With a one-sided t-test of each of these statements, we can provide the probability chance that the statement is correct. We get for each metric the following:
- Accuracy: 71% the statement is correct
- Reaction Time: 99.9996% the statement is correct
- XP / Pitch: 98% the statement is correct
In other words, we are almost certain that Reaction Time decreased and XP / Pitch increased using uHIT Plus for training in this AZL affiliate. Since XP / Pitch is a combination of Accuracy and decision speed each pitch, we can say that the affiliate hitters got better at recognizing pitches in and out of the zone. As Reaction Time shows clearly, they also got faster at their ‘yes’ decisions.
Why Accuracy Lags
Accuracy though is less certain. By most standards of statistical tests (see here), we would not be able to claim that Accuracy went up for this group, even though there is a sign that it trended that direction. One particular reason for this is that the zone recognition training before 2019/2020 was more difficult than it is now. As a result, it was less adaptive to the strengths / weaknesses of a hitter and less accommodating for a range of skill sets than is the current uHIT Assessment (see here for more on this topic). As a result, skill changes took longer and the 2-month period of this training was not enough time to see that change happen. Current methods employed in uHIT Custom are much faster, allowing changes in this time frame (see here, for instance).
Interim Summary
Across 17 hitters, an AZL org affiliate saw marked improvements in pitch and zone recognition skills in uHIT. Deployed before uHIT Custom, all hitters used uHIT Plus (see here) and did 6-minute sessions for each scale over a 2-month period. The hitters of the affiliate did 23 (+/-2) sessions over that short season period.
This wraps part 1 of MLB org training case study. In Part 2, we will examine what long-term effects these skill improvements had for these hitters on-field.
Leave A Comment