Let’s see if this works…

It made it through Round 1 and got a score of 56 points.  Time to take a step back and analyze what I’ve observed.

• The biggest problem is losing track of the bomb.  It will be on track to catch the lowest bomb, when suddenly the detection will skip a beat and it’ll take off across the screen for something else and can’t get back fast enough to catch the bomb.  I have to fix this first.  I’m pretty sure I have outlier detection from the Pong game that I can reuse here.
• The segmented calibration that I described back on the first day seems to be working fairly well.  That’s where the robot moves the paddle knob a known number of degrees, then the program sees how far that moved the buckets on screen, and calculates how many pixels a degree is.  This lets the program know that to catch the bomb that’s 70 pixels away, it will have to rotate the paddle 30 degrees.  I suspect that I will have to tweak the algorithm a bit, though.  Mainly, I think it will need to do several calibration turns to refine the numbers.
• The response time is going to be a killer.  I have little doubt that the motor itself can move the paddle into position fast enough.  When it moves, it zips across the screen.  The problem is that it doesn’t get moving as fast as it needs to and there’s an unacceptable lag between commands.  I think I might still be seeing an echo of the waggle.  It’s not as visible, due to the weight on the spinner assembly now, but I think that delay is still there.  If I can fix some of the other issues, I suspect that the response time is going to prevent the robot from getting past around level 3.
• The NXT has an inactivity timer.  The robot turned itself off in the middle of a game.  Gotta take care of that…
• I don’t want to keep hitting the button to start a round.  I’m going to have to implement the button press.  Unfortunately, implementing the button press means I’ll have to change around how the PaddleController class works.  It’s a dirty hack right now, I gotta clean that up.
• I like that Kaboom! is a much faster game cycle than Pong.  With Pong, I couldn’t always tell what was wrong right away.  Sometimes it took a minute or two to get the ball in a situation where something went wrong.  I’d implement a fix and try again, and again, it would take a minute or two.  Kaboom! doesn’t last that long.  The games at this point are less than a minute long, sometimes much less.  If the robot is going to lose, it’s going to lose fast.  It’s not going to end up in a cycle where the ball keeps bouncing around between the same two points over and over and over for five minutes.  If a game in Kaboom! lasts five minutes, then it’s an amazing success.

I think I’m going to have to put together a finite state machine for the robot.  The robot for Pong had essentially two states:  “Playing” and “Totally Flipping Out”.  I need to do a better job than that for this one.

Here’s the basic states for Kaboom!:

• Bombs on screen:  Track the lowest bomb.
• No Bombs on Screen: Press button.
• No Buckets on Screen:  Hit Game Reset switch.

At this time, it won’t be able to hit the Game Reset switch itself, so it should probably put a notice on the screen for human intervention. ¹  For the “No Bombs on Screen” state, it’s going to have to hit the button, then wait before trying to hit the button again, otherwise it’ll hit the button a second time, before the bombs start to fall, and it’ll lose precious reaction time.  When the bombs are on the screen, it’s going to have to be careful and make sure that it doesn’t get fooled by blips where the lowest bomb disappears for a frame.  However, it can’t be too careful, since it’s going to have to move to the next spot as soon as it catches a bomb.

For now, I’ll get the “Bombs on Screen” state going, since that’s the one that’s important.  I can hit the button myself for now, as long as I’m careful to keep my fingers clear of the spinning bricks.

1. And I don’t want to give it the ability to hit the reset switch itself.  You teach it how to do that and you never know what it’s capable of. [↩]

That’s the game.  Those are the elements I need to recognize and react to.  I’m going to need to find the bombs, find the buckets, and find the bomber.

I could try to do this in pretty much the same way that I did for Pong, but that was a bit hacky and prone to failure.  I’d like to learn a bit more about some of the object detection and recognition features in OpenCV, and see if there’s some way for it to identify the components directly, instead of just assuming that different boxes are different parts of the playing field.  I played with some of that on the Wesley Crusher thing, but that was using ready-made classifiers.  This time I’ll have to do it from scratch and see what happens.

At any rate, the game playing logic should be easier this time around.  I don’t have to deal with bouncing balls and linear regression and all of that.  The bomb tracking should be failry straightforward.

Anyway, time to break out the OpenCV.

Last time, the paddle was a bunch of errors and trials, approached from the point of view of the Bluetooth interface of the Mindstorms NXT.  It didn’t work out so well.  Many of the problems I had were the result of telling the NXT to do conflicting things, often queueing up many actions faster than it could process them.  It ended up having about as much control as a fully loaded freight train.  I’d tell it to rotate left and it would rotate left.  While it was rotating, every frame processed would tell it to rotate left again.  And again.

And again.

Then, the paddle would reach the point where I wanted it to stop, so I’d send a stop request.  Of course, I’d already sent a ton of “RotateLeft”s that it hadn’t processed yet, so it would run through them until it had used them all up.  Then it would stop.

By then, though, since the paddle had gone too far, I’d tell it to rotate right to get back where I wanted it.  In the next frame, it was still too far, so I’d tell it to rotate right again.  And again.

And again.

Eventually it would catch up and start rotating right, back toward the target point.  Now the queue was full of “RotateRight”s, so it would spin wildly back across the target.  Then again in the other direction.  Then back.

The ball, of course, wouldn’t wait around for the paddle to make up it’s mind and would sail steadily right past the confused rectangle.

Obviously, that’s bad.

I hacked around the problem by adding limited protection against command queuing.  I also lowered the power of the rotation motor so that the paddle would be less likely to swing past the target point, so it would be less likely to need to swing back.  Unfortunately, the lowered power also meant that it was no longer fast enough to reach from one side of the play field to the other.  If the paddle was at the bottom and the ball went for the top, it couldn’t make it.

In the end, this lack of control limited how well it could play.  It ended up playing just slightly better than the computer paddle.  Despite all of the technology and computing power involved, it was still no match for a reasonably skilled human player.

What I really needed was to be able to tell the paddle to go to a specific point as fast as it could. ROTATE 10 DEGREES LEFT.  And off it would go, just like that, resulting in a point that’s 10 degrees left of where it was.

Actually, I could tell it to do that.  Sort of.  I could tell it to rotate a certain number of degrees, but there were two problems with that.  First, the motor seemed to think that rotating somewhere in the same magnitude as 10 degrees was the same as rotating exactly 10 degrees.  I’d tell it to move 10 degrees and it would go full speed for 10 degrees, then coast for another 15, stopping at 25.  Then after that, if you told it to rotate another 10 degrees in the same direction, it wouldn’t go anywhere.  Since it went a total of 25 degrees with the first movement, it wasn’t going to go anywhere until the cummulative total of future movement requests was greater than 15 degrees.  Great, so now it was moving totally randomly and sometimes refusing to respond to movement requests.  That’s really useful right there.

The second problem is that it didn’t really matter if I could tell it how to rotate exactly 10 degrees, because I had no concept of what in the hell 10 degrees meant in the game world.  All of the calculations and projections were in pixels.  Degrees would be meaningless without some sort of conversion, and a conversion would require crazy calibration code.

I think I eventually may have solved the first problem, outside of the Crazy Weekend Project timeframe.  I think there was a mode for the Mindstorm motor that would force it to rotate the desired amount exactly, but that it was a hidden mode or was not accessible the way I was trying to use it or something like that.  I haven’t tried it out on a game, but I have a reasonable expectation of success there.

I think I’ve also solved the second problem, as well.  All of my initial thoughts on paddle calibration involved some crazy scheme where I’d rotate the paddle to the clockwise extreme of the screen, then take a reading, rotate it to the counter-clockwise extreme, take another reading, and use the numbers to come up with a conversion between pixels and degrees.  A little something like this:

It’s all fine on paper, but in practice, it all falls apart.  First, you have to find the extremes.  You can’t just rotate the paddle all the way to the left, then all the way to the right, because Atari paddles have a useful range somewhere in the middle.  They have a roughly 270 degree arc, but for most games, there’s only about 100 degrees of usefulness.  Also, even if that did work, how will you detect when you’ve hit the end point of the paddle?  The motor will just keep going and going until something breaks.  You might be able to turn left or right until the on screen paddle quits moving, but in some games, like Pong, the paddle will happily sail off the screen.  If the paddle behaved and stayed on the screen, you still have to correlate the rotation angle of the paddle to the movement on screen, and stop moving and make your calculation at the instant the extreme is reached.  There’s a good chance that you’d have to move so cautiously and deliberately that you will have lost the game before you’ve calibrated your paddle.

However, the idea is sound.  The math is a simple transformation that works. So what if you take away the walls and try again?  Why bother with two variables, when you can do it with only one?  Like this:

The extremes don’t matter at all.  What matters is that you have a distance and an angle.  So why not start with specifying the angle?  Rotate 10 degrees clockwise or 25 degrees counter-clockwise.  Then, when the motion is done, see how far the paddle on the screen has moved.  This will be much easier to do, not to mention, much faster.  One movement and you’re done.

This is what I’m going to try to do.  I already have the Pong recognition set up and running from last time, so I’m going to use that as a base to implement and tune the new movement engine.

The general Paddle class interface will be something like this, although it will likely be modified and tuned as I go.

Some notes about the refactor.

Pretty basic stuff.  I really just wanted to upload this to show off the neat graph paper journal I found in Fred Meyer.

Trying to use a real TV screen will pose a couple of problems.  First, I have to find the TV screen in the image.   Then, once I have the TV screen, I’ll have to do something about the distortion.  There will be two types of distortion.  First is perspective distortion, since I can’t assume that the camera is facing the TV screen head on.  Then, of course, since this is a typical CRT TV, there’s the curve of the screen to take into account.

I’m going to rely heavily on user input to calibrate the display.  My idea is that the user will have to click several points on the video image that correspond to points on the playfield.  Then, hopefully, I’ll be able to fix the distortion in the image and have a playfield image that’s nice and square to use.

It will look something like this:

There.  Wasn’t that simple?

Of course, it would be totally awesome if I could simply feed it an image of what I expect to see, then have it find that image in the video and undistort everything for me.  However, I haven’t quite read that chapter yet…

I just realized that there was something missing in the my original design spec.  Here’s the update: