CS 4670 Final Project - Blackjack Advisor

Matt Mukerjee (mkm234), Greg Sadowski (gjs33)

Overview

The goal of our final project is to create a program that takes as input an image of a blackjack hand in progress and outputs the optimal strategy for all players at that stage in the hand. There are many points in a blackjack hand where a player must make a decision (most often this decision is whether or not to take another card). Our blackjack advisor automates this decision making process. To do so, the program must find the current state of the game by recognizing which cards are in play and which cards belong to which players. Then, it can look up the best play for each player in a large strategy table. Unfortunately, casinos do not allow computerized devices to assist players playing blackjack, so our blackjack advisor will not able to be used in real-world situations.

We make several assumptions to simplify the problem:

• Table is black - This assumption is not entirely unrealistic, as most standard blackjack tables are almost entirely dark green, and converting dark green pixels to black pixels is relatively simple. We did not have access to a blackjack table, so we used a black cloth instead.
• Image shows only the table - We wanted to focus on card segmentation and recognition, so we made this assumption that assures us that we won't have any noise outside of the table (the region of interest).
• Image is taken from the perspective of the player - The program is designed for use by blackjack players, so this assumption seems reasonable. We need this assumption so that we can tell which hand is the dealer's hand and which hands are the players' hands based on their locations in the image.
• Image is not taken from an extreme angle - We found that exact card rectification was unnecessary for card recognition, so we compute an approximate homography that is sufficient as long as the image is not extremely warped. This is reasonable, as we assumed the picture was taken from the player's perspective

Note: This is an original project and we did not reference any previous work.

Algorithms

The computer vision portions of our algorithms are divided into two main parts. First, the segmentation of individual hands and cards out of the image, and second the determination of the values of the individual cards.

Segmentation begins with a few preprocessing steps. The image is scaled down, color thresholded (we only care about the white pixels), and then converted to grayscale. Next, edges are detected using Canny edge detection, and dilated to increase their size. The edges become the boundaries of the player hands, so we can then detect the dilated edges as contours in the image. Each contour corresponds to one player's hand, and we can then indicate where the hands are in the image:

Next, we segment out individual cards from the hands. To do this, we compute the edges in the hand image with Canny edge detection, and then subtract those edges from the original image. This creates gaps between the individual cards so that when we find the contours in the hand image, each contour corresponds to an individual card. Here is a hand image with individual cards detected:

Now that we have individual card images, we can move on to actually recognizing the value of the card. To do this, we first rectify that card so that it is approximately flat and vertical. We find the four approximate corner points of the card using the visible corner points, known card sizes, and vector arithmetic. We then compute a homography between the four approximate corner points of the card and the four corner points of a flat, vertical card holder.

Once the card has been rectified, we know that the rank character of the card is in the bottom-right corner. We search this space for the first occurrences of non-white pixels to get a bounding box on the character.

Next, we crop the rank character out of the corner of the card, and then run OCR (optical character recognition) methods to determine which character it is. We have a precomputed set of binary masks, one for each possible character. We scale the character that we cropped out of the image to the exact size of the masks, convert it to black and white, and then compare it with each mask. Whichever mask it is most similar to (by a simple sum of squared distances calculation) is the card rank of the card.

Results

Once we have recognized the cards in an image, we do a lookup into a strategy table for blackjack to find the optimal action for each player. We then write this information along with the card information to the image to get our result:

Our blackjack advisor works most of the time in basic cases. However, under certain lighting conditions, edges between cards become extremely hard to recognize and the program will fail. This happens because when two white card edges are overlapping, they are occasionally recognized as the same component (the same card).

Many of the issues that we had came down to parameter tuning. We have many different parameters that affect our results, and it is impossible to get every parameter perfectly tuned. Additionally, some calls in openCV seem to handle differently on the phone, than in linux, leading to further difficulties.

OpenCV reliance

for this project we made use of various CV algorithms. To segment out cards, we need to be able to threshold images, detect edges, and find contours.

• Thresholding is rather trivial to code, simply loop through the image, setting pixel values to high (255) if they are greater than the threshold, and low (0) if they are less than the threshold.
• For edge detection, we used the standard canny algorithm. First we blur the image w/a gaussian, then get the component gradient orientation by computing sobel x and y. We get the overall gradient orientation by taking the inverse of the tangent of sobel_y / sobel_x at each point. We then quantize the gradient into one of four bins, as the edge described by the gradient must pass between 2 of the 8 neighboring pixels, leaving only four possible directions. We then suppress additional noise by using non-maximum suppression. Finally we threshold (as explained above) to clean up the image.
• For contour detection, we simply iterate through a binary image twice. The first round, we go through row by row, scanning left to right. We want to assign each pixel to a group. These groups are just integer values. All black pixels are assigned to group -1. For each white pixel, to if their left neighbor, or any neighbor above them (including diagonals) is not -1, the white pixel is assigned to that group (if these neighbors are in different groups, then just picking one is fine). After iterating through all pixels, we run through a second time, merging different numbered groups together if they have neighbors which are in different groups. This final set of groups are returned as contours.

Even though these algorithms are fairly compact, we found that openCV does a much better job handling their data structures than we do. Since we want the code to be able to run on the phone, we opted to use their versions (cvThreshold, cvCanny, and cvFindContours), greatly speeding up our execution.

For the recognition section, we implement the rectification code ourselves, along with the "rank finding" (looking for the value of the card within the rectified image) code, and the OCR-like character recognition code

Division of Work

Matt worked on the card segmentation code and hacking the code into old phone code to get it running on the phone. Greg worked on the rank recognition (including card rectification) and the actual blackjack simulation code.

Future Work

Here are a few ways we could improve our blackjack advisor if we had more time:

• Handle blackjack hands on a real blackjack table surface rather than a black background
• Track a blackjack hand in real time, doing card recognition and strategy recommendation as the cards are dealt