If you’ve ever been to a baseball game, then you’ve probably noticed someone standing off to the side and waving their hands around in seemingly random ways. What they’re actually doing is signaling the players. If, for instance, the team is on offense, then those signals might indicate that a player should steal a base. The signals are in plain view, but for obvious reasons they don’t want the opposing team to know what they mean. So teams come up with elaborate signals that are hidden among arbitrary, meaningless motions. That’s extremely difficult to decipher mentally, but Mark Rober was able to do it with machine learning.
As Rober describes in the video, he actually came up with two ways to decipher baseball signals. The first technique is done with a simple smartphone app that you can use to input each of the motions in a signal, as well as what the outcome was. After just one inning, the app’s algorithm is able to identify which part of the sequence is the true signal. From then on, it can predict whether a player will steal a base or not. But that only works because the signals are fairly basic — usually just a pair of consecutive hand signals hidden among meaningless motions. If the signals and decoys were truly complex, this algorithm couldn’t decipher them.
To be able to do that, Rober turned to YouTuber Jabrils to create a machine learning model that can figure out incredibly complex signal patterns. The basic idea is the same: the series of hand motions are input into the machine learning model along with the outcomes, in order to train it. But the machine learning model can find far more complex relationships than the app algorithm. To test it, Rober came up with his own elaborate signal where the first “indicator” and then the actual command are separated by an arbitrary motion. A third cancellation signal anywhere in the sequence negates that command, just to make it harder to decipher. But the machine learning model was still able to figure it out in just minutes, and even works in live baseball games.