Score and Text Recognition with Machine Learning, an eSports case

SOLVVE team continues to share its expertise on the trending technologies and solutions. Previously we have rather extensively covered a topic of data forecasting in machine learning based on data science, including the possibility of algorithms to predict the winning numbers in a lottery and understand human emotions. 

However, there is another vast area of applications that deals with the recognition of different objects. And in this article, our machine learning department explains how it has built and trained a score and text recognition system to help automatically extract and register game results in eSports.

Score and Text Recognition Process

Семантична сегментація

We are so used to all the amazing things that technology is capable to do that it is hard to underestimate the amount of work it takes to train a machine to do what a human brain can do in a matter of split seconds. A person’s eye can perform a quick search to locate the score section thanks to color schemes, highlights, some conventions of how sports results are displayed for different disciplines, and so on. 

However, for a machine learning algorithm, the starting point is just a vast field of pixels. To teach an algorithm to look in the right direction and locate sections with the score or other necessary information, engineers use semantic segmentation. This is a process of labeling each pixel in the picture and then figuring out which of them belong to the area of interest. 

There are several methods to perform this manipulation. However, SOLVVE team relies on U-net which is a leveled-up version of a convolutional neural network: a trained network that is very handy in identifying faces and other types of objects in pictures. You have probably seen it in action on social networks when a system suggests you to tag friends on your uploaded pictures.

Unlike the example above, pictures tend to be quite distorted due to horizon tilt, rotation, visual noise, poor or extensive lightning, etc. That is why it is necessary to modify and “normalize” the chosen segment before the actual score and text recognition procedure. SOLVVE team uses OpenCV to eliminate the above-mentioned distortions and get an image that could be used for the extraction of the necessary information.

Training in recognition

The most popular tool for score and text recognition is Tesseract. However, it performs well only on black objects against a white backdrop, like document scans, and cannot be directly used on colorful screenshots of photos of game results.

At this point, if you have familiarized yourself with our previous publication on machine learning and artificial intelligence, you already know that one needs to feed enough data to the algorithm before it is capable of producing reliable results. Thus, SOLVVE team generated a set of scores similar to those shown in the picture below and trained the algorithm until it was able to deliver legit results for the whole scoreline.

I Would Like to Have This Feature in my Product

When speaking in broad-terms, score and text recognition might seem a simple 2-step process where automated calculations work their magic. Nevertheless, it takes a highly-skilled machine learning engineer to train and calibrate the algorithm before it can be seamlessly integrated into service adding value to the user experience and spearing human resources from manual entry of the results. 

While this particular solution is meant to facilitate fir judgment in eSports competitions, it is also possible to apply these techniques to extract and recognize many types of data from pictures, videos, or even live streams. If you have any questions or ideas about using machine learning for score and text recognition in your business, do not hesitate to contact us. Let us make it happen!