• Catch Framing Predictability

    There has been more and more talk lately about catcher framing, which is the ability of catchers to impact the runs scored in a game by garnering strikes from pitches outside the strike zone. My initial skepticism around it has largely been due to the huge impact it can have. For instance, last year the difference between the best pitch framer (Jonathan LuCroy) and the worst (John Buck) is estimated at 50 runs. That's a five win difference. That's hard to believe.

    But today I wanted to look at different aspect: its predictability. That is, if a catcher is good at framing in one year, can we reasonably assume that he'll be good at it the next year? One way to look at this is to look at all catchers and how they did from year to year. If they did well one year, did they do well the next and vice versa?

    (I wondered about this because I was looking up Kurt Suzuki's framing numbers. They're usually been negative, but there have been some positive ones sprinkled in. I wondered how common that is.)

    There is a neat little statistical gizmo to do this called a correlation coefficient. A correlation coefficient examines two sets of numbers and gives back a number between -1 and 1.
    • 1 means there is a perfect correlation, like between the temperature in Celsius and the temperature in Fahrenheit.
    • -1 means there is a perfectly negative correlation, like the amount you spend in a month and your checking balance.
    • And if it's 0, that means the numbers have no correlation, like Joe Mauer's batting average and the migratory penguin population.


    You can find the results of my study here.

    The bottom line: there is a lot of predictability. The runs per season had a correlation of .76, which is high. But the correlation on pitches per game is even higher .82.

    So catchers who have had a large positive effect end up continuing to do so. Unfortunately, most of the Twins who will play catcher this year didn't have a large positive effect last year. In fact, none of them did:

    Kurt Suzuki: -9.1
    Josmil Pinto: -4.3
    Chris Herrmann:-4.6
    Eric Fryer: -0.8

    I didn't choose that order to emphasize the negative. I chose to list them in my predicted order of innings caught. It's almost as if the worse they are at pitch framing, the more likely they are to play catcher. And this is where John starts rubbing his temples.

    And yet, that still might be better than last year. Because last year Joe Mauer was average (0.4) and Ryan Doumit was horrendous (-15.9). Still, it appears the new catching corps may not do the Twins revamped pitching staff any favors this year.
    ~~~

    Since I'm sure you might want to do something like this yourself (and really, why wouldn't you - YEAY MATH), I thought I'd spell out the steps.

    1. I pulled all the data I could from this great site and pasted it into a spreadsheet. It has all the catcher framing data from 2013 through 2007.
    1b. I forgot to mention - I also limited the study to catchers with at least 3000 "samples" in a season which I assume are pitches.
    2. I added one column to that data: "Prev Yr." You'll see why in a minute.
    3. I imported that spreadsheet into an Access DB twice, once as a table called "Following" and another as "Previous".
    4. I created a query joining those two tables, joining fields First Name, Last Name and "Prev Yr" from following to "Year" from the Previous field. I pulled the Names, Years, Per Game and RAA fields from each table.
    5. Copy and paste the results from the query back into an Excel spreadsheet.
    6. Use the "Correl" function to compare the values in the two "Per Game" and two RAA" fields.

    Ta Da!
    This article was originally published in blog: Catch Framing Predictability started by John Bonnes
    Comments 23 Comments
    1. Seth Stohs's Avatar
      Seth Stohs -
      Very interesting, and yet not too surprising. If you're getting more strikes called, a pitcher will throw less pitches...

      My concerns with pitch framing...

      1.) I think it would be interesting to see how the numbers would look if you removed 3-0 and 0-2 counts. 3-0, if it's pretty close, it's a strike. If it's 0-2, that same pitch in the same location is often called a ball.
      2.) As much as we want to think that all players have the same strike zone, does it not make sense that someone catching Sam Deduno, Kyle Gibson, Ryan Pressly, other rookies and unproven pitchers will have different strike zones than proven pitchers? It's human nature. I bet Greg Maddux's catchers in his prime had pretty good pitch framing numbers if they had had such things then.
      3.) Game situations shouldn't, but likely do, play into it.
    1. Thrylos's Avatar
      Thrylos -
      I have lots of concerns about pitch framing as a metric as well:

      - it has to be normalized per umpire (it is not.) And we know that a 5'7" 300lbs umpire's strike zone is different than a 6'7" 200 lbs umpire's.

      - it has to be normalized per pitcher (it is not). I bet if Greg Maddux was throwing exclusively to Ryan Doumit, Doumit would have been the best pitch framing catcher in the game...

      Until they find a way to normalize their data in at least those 2 dimensions, I am not buying it...
    1. DocBauer's Avatar
      DocBauer -
      I can't add much to what Seth and Thrylos already stated.

      This stat is attempting to quantify human nature, discrepancy and error and provide a measurable and quantifiable statistic.

      You have two leagues, 4 man umpiring crews who rotate who is behind the plate, and they almost ALL call a game differently. Factor in different pitchers, not only power vs control, but vet vs rookie, LH vs RH, and I just think you have way too many mitigating factors to come up with a manageable statistic.
    1. h2oface's Avatar
      h2oface -
      I think it is sad that when we have the technology for instant correct calls, instead, we have to embrace a talent the cheats the rule of the game, and continue to have the game impacted by getting calls that are not correct.
    1. jorgenswest's Avatar
      jorgenswest -
      Good work.

      The correlation suggests it is a skill. Once teams are willing to acknowledge it is a skill and trust the metric, the skill can be improved. Analysis can point out zones and pitch types of strength and weakness in a catcher. Video study can show technique that can be improved with those zones and pitches. Work can be done throughout the minors.

      There is no reason an athletic catcher can't improve in this skill. Particularly those young and in the minors. The first step is for the Twins to acknowledge and trust the metric. I don't think they are there yet.
    1. John Bonnes's Avatar
      John Bonnes -
      Re: the concerns

      With umpires - you would think if the umpires were that important in this study, the results of the correlation would be more random, right? It's not like a catcher's umpires that he happens to catch in front of are consistent from year to year.

      Re: pitching staffs - I just reran the same study, only this time, I only chose catchers who also changed teams from one year to the next. That would seem to negate the impact of pitchers. It came back with only 46 examples (as opposed to 189 from the previous study) so there is a higher deviation, but the results were still ridiculously strong: .68 for the season and .75 for pitches per game.

      This is a skill, folks.
    1. nicksaviking's Avatar
      nicksaviking -
      Good write up John, but I'm still skeptical. As has been mentioned, I think it has more to do with the pitchers. Umpires already have preconceived ideas and biases about the pitchers. If the umpire already has it in his mind that the pitcher is wild or young, I doubt catcher manipulation will work well. On the flip side, well I'm guessing Greg Maddox's personal catchers through the years would be ranked among the elite at framing if the data was available.
    1. biggentleben's Avatar
      biggentleben -
      I'll echo something that has been posted already in a slightly different way. When I was pitching growing up, I threw against a garage that had a board about the size of an old catcher's pud. Every time I was working on a new grip on a pitch, if I couldn't consistently hit that board with it, I wasn't going to use it in a game. Once I started pitching in games, coaches knew I had a strong arm from watching me at third base, but they were astounded at my command and ability to hit the catching mitt every time. I wasn't anywhere near the skill of a major league pitcher, but we've all seen plenty who don't hit their catchers in the mitt at all. I'd say the ratio is very, very high that if your catcher's glove doesn't move, you get a strike called for you, even if it's a bit outside or inside. That's one thing Eddie Perez did masterfully for the Braves for years. He would set up just a tick off the black outside or inside, and Maddux would nail his glove without moving it. Even if you're an inch inside with the pitch, if you hit that spot perfectly, you'll get the call.

      I'd be curious to go through the data and find the difference in fastball vs. changeup vs. breaking pitches and framing of them. From the eye test, one thing that always astounded me with Suzuki was how many breaking pitches he caught from guys in Oakland and how well he made sure they never got by him. A young guy throwing breaking balls will make his catcher's framing look poor, but Suzuki spent years catching entire staffs of young guys living on breaking stuff in Oakland, so it's understandable his numbers were off some. One interesting correlation that Baseball Prospectus mentioned in a discussion on the Suzuki signing is that catchers who have high framing ratings tend to also have more passed balls, and the opposite has also been true, to the coefficient of roughly 0.5, so not amazing, but certainly notable, so perhaps there's something in that holding your position to the very end gets you more calls, but you end up having more balls get by you or the opposite, you never let a ball get by you, but you take your body out of position to frame the pitch best for an umpire by doing that.

      Just some thoughts...I like the idea of rating framing, but I feel like it's where things like UZR and Range Factor were 10-15 years ago. As more and more factors are figured out to get the best measurement, the numbers will balance with the eye test, and right now, there are some catchers who get rated low on catch framing that seem to do a very good job protecting their pitchers, so perhaps it's not always the best indication of success behind the plate.
    1. Wookiee of the Year's Avatar
      Wookiee of the Year -
      Quote Originally Posted by John Bonnes View Post
      Re: the concerns

      With umpires - you would think if the umpires were that important in this study, the results of the correlation would be more random, right? It's not like a catcher's umpires that he happens to catch in front of are consistent from year to year.

      Re: pitching staffs - I just reran the same study, only this time, I only chose catchers who also changed teams from one year to the next. That would seem to negate the impact of pitchers. It came back with only 46 examples (as opposed to 189 from the previous study) so there is a higher deviation, but the results were still ridiculously strong: .68 for the season and .75 for pitches per game.

      This is a skill, folks.
      Yeah, that's the rub--it's easy to come up with reasons why catch framing might not be all it's cracked up to be, but when you're staring at a year-to-year correlation of over .65, you're arguing against reality. The data makes a strong case this is a skill.

      The hard part is translating the impact of that skill into runs and wins.

      I also wonder: We've heard so much about how pitchers love working with Suzuki. I wonder how big a factor pitch selection is vs. pitch framing? We really can't evaluate the latter, but just because you can't (yet?) quantify it doesn't mean it can't have a very large impact.
    1. cmathewson's Avatar
      cmathewson -
      Yes, it is a skill. But I don't think the metric as currently used is a good one for development. Why? Because there are too many uncontrolled variables. If there was a way of breaking down the metric to that which a catcher can control, it would be worth tracking those factors over time.

      Perhaps an anology is needed. Say a player wants to improve his batting average, but fails to do so mostly because of luck. Perhaps his LD percentage goes up but his BABIP goes way down. That is, he hits a lot of liners right at people. Because he used average as his KPI, he is a failure. If he had used LD percentage instead, he might have a better gauge of his improvement.

      I just don't know how to break down the metric into similar smaller bits. And therein lies the problem. It is OK as a gross measure of one aspect of catching, given sufficient sample sizes. But it is a poor measure of ability or improvement. Also, as big an impact as it might have, pitch selection is bigger. It just goes to show how underrated the catching position is. We can only focus on what we can measure, which isn't the half of it and which is subject to the perils of small samples.
    1. jorgenswest's Avatar
      jorgenswest -
      Thought on umpires...

      If you don't believe in the data because of umpires, than you certainly can't believe in pitcher strike outs and walks where fewer umpires would have a greater influence on the smaller sample.

      Thought on developing catchers...

      Ben Lindbergh did several interviews last year with catchers and coaches from teams that seemed to be using the data. Jose Molina credits his development and growth to Tony Pena and Joe Girardi. Others speak about how much they work on the skill. The data set is large enough so that catchers can look at it by zone or by pitch. They can be aware of where they are losing strikes or which pitches they lose strikes on. They can use video study to see how their technique differs from other catchers who are getting those strikes. This skill can absolutely be developed.

      If the Twins don't believe in the data, they are wise not to invest the time. They are wise not to pay for the skill when acquiring catchers. Let other teams waste the time and pay for the skill they don't trust. I hope the Twins are on the right side of this argument.
    1. Nate Haseman's Avatar
      Nate Haseman -
      Nice work! I completely agree that it is a skill but I'm with a lot of the other readers in that I think it's a tough metric to calculate. If Mauer was catching for the Dodgers or Cardinals last year I bet he'd wind up being well above average. Keep up the great work in our quest for the truth about pitch framing!
    1. Seth Stohs's Avatar
      Seth Stohs -
      No question it's a skill. I don't think anyone will question that. It was something I practiced when I did do some catching. And there's no question some are better at it than others, like any skill.
    1. Oxtung's Avatar
      Oxtung -
      I don't think we need to worry about how any given umpire affects a catchers pitch calling stats. While every umpire is going to call the game slightly differently those differences won't matter just because of the sheer number of games the pitch framing metric covers. In the end they'll have umps that call it close and umps that call it loose and they should balance out.

      Here's another way of looking at it. Since 08, which is what John's data covers, Joe Mauer has caught 561 games. How many games would an umpire have to call before he has any discernible impact on Joe's pitch framing skill? An umpire calls ~30 games a year behind the plate.

      Does anyone know how a crew is assigned to a series? Random or by region or ...?
    1. thetank's Avatar
      thetank -
      Quote Originally Posted by h2oface View Post
      I think it is sad that when we have the technology for instant correct calls, instead, we have to embrace a talent the cheats the rule of the game, and continue to have the game impacted by getting calls that are not correct.
      It's suppose to rain on both sides. I assume they would use sensors on the both sides of the plate, but let the umpire determine the top to bottom strike zone?
    1. cmathewson's Avatar
      cmathewson -
      Quote Originally Posted by thetank View Post
      It's suppose to rain on both sides. I assume they would use sensors on the both sides of the plate, but let the umpire determine the top to bottom strike zone?
      Pitch F/X is accurate--more accurate than umpires. What we have seen, especially with the taller Twins catchers, is umpires reluctant to call the low strike. The other side of the coin is that both Mauer and Doumit got more high strikes called. Considering that the staff has consisted of mostly sinkerball guys, that is not a good thing. Hopefully both Suzuki and Pinto can bring the strike zone down a bit. I'm more interested in that trend than the overall percentage of balls brought in or strikes lost.
    1. Jim H's Avatar
      Jim H -
      Quote Originally Posted by cmathewson View Post
      Pitch F/X is accurate--more accurate than umpires. What we have seen, especially with the taller Twins catchers, is umpires reluctant to call the low strike. The other side of the coin is that both Mauer and Doumit got more high strikes called. Considering that the staff has consisted of mostly sinkerball guys, that is not a good thing. Hopefully both Suzuki and Pinto can bring the strike zone down a bit. I'm more interested in that trend than the overall percentage of balls brought in or strikes lost.
      I don't know that the first statement is accurate. There are 3 things about pitchF/X that make that statement doubtful. First, PitchF/X is 2 dimensional while the strike zone is 3 dimentional. The 16 inches of depth of the plate allows moving pitches to drop in or move in from the side. Since PitchF/X is set up at the front of the plate many pitches "called" balls by PitchF/X are often strikes according to the rule book.

      Second, Pitch F/X has a "standard" one size fits all strike zone. Major league ball players range in size from about 5' 5" to about 6' 9". According to the rule book there is no standard strike zone. Pitch F/X has to miss pitches both high and low depending on the height of the player.

      Finally, I think that all of us realize that as wonderful as modern technology is, it is far from perfect. Something as complex as Pitch F/X is going to need some very expert calibrating to actually have that window set up perfectly over the plate and being actually the exact same size in every ballpark.

      While I have no illusions about the perfection of the average big league umpire, there is no way given the limitations of Pitch F/X, that there will not be many, many cases where the umpire is right and Pitch F/X is wrong.
    1. Oxtung's Avatar
      Oxtung -
      Quote Originally Posted by Jim H View Post
      I don't know that the first statement is accurate. There are 3 things about pitchF/X that make that statement doubtful. First, PitchF/X is 2 dimensional while the strike zone is 3 dimentional. The 16 inches of depth of the plate allows moving pitches to drop in or move in from the side. Since PitchF/X is set up at the front of the plate many pitches "called" balls by PitchF/X are often strikes according to the rule book.

      Second, Pitch F/X has a "standard" one size fits all strike zone. Major league ball players range in size from about 5' 5" to about 6' 9". According to the rule book there is no standard strike zone. Pitch F/X has to miss pitches both high and low depending on the height of the player.

      Finally, I think that all of us realize that as wonderful as modern technology is, it is far from perfect. Something as complex as Pitch F/X is going to need some very expert calibrating to actually have that window set up perfectly over the plate and being actually the exact same size in every ballpark.

      While I have no illusions about the perfection of the average big league umpire, there is no way given the limitations of Pitch F/X, that there will not be many, many cases where the umpire is right and Pitch F/X is wrong.
      No way to know but I would place money on the machine being correct more often than the human. The eye-brain unit is absolutely terrible at precision and even worse when it comes to high speed precision.
    1. old nurse's Avatar
      old nurse -
      Quote Originally Posted by Oxtung View Post
      No way to know but I would place money on the machine being correct more often than the human. The eye-brain unit is absolutely terrible at precision and even worse when it comes to high speed precision.
      Are you outright dismissing the limitations of pitch f/x ? Correct more often is hardly a solution
    1. Oxtung's Avatar
      Oxtung -
      Quote Originally Posted by old nurse View Post
      Are you outright dismissing the limitations of pitch f/x ? Correct more often is hardly a solution
      Where did I say I was dismissing the limitations? It's not a perfect solution. I was saying that I believe, imperfections and all, it is still better than the human eye/brain combo.

      If "correct more often is hardly a solution" what does that say about our system of umpires today?
©2014 TwinsCentric, LLC. All Rights Reserved.
Interested in advertising with Twins Daily? Click here.