Post 9 - New Videos

Quick update, today I got some footage of 3 more people. Hopefully this will have eliminated the problem of the shadow on the floor, but it might have the problem of a tree blowing in the wind in the background. Also the first video consists of over 20,000 frames! In comparison, the largest video I previously used was about 3,500 frames. Time to cull some of the useless frames I think!

Post 8 - Adding Metrics

TL:DR - Pictures towards the bottom

For the analysis of the animation, we're going to require a lot more information than just the bounding boxes we currently use. To this end, a few new metrics to analyse have now been added to the project.

First thing to mention though is the increased efficiency. The bounding-box search for each frame was improved by using information from the previous frame, which brought the time down for analysing each frame from ~550ms to the 350-400ms range. However after adding the new metrics, the average time now hovers around the 500ms mark. Never mind, did I mention we have more metrics to measure?!


X-Centroid
The first measure is the x-centroid. After the background has been subtracted, we take each scanline through the image and find the average x-coordinate of the remaining pixels. The average of these averages then provides our x-centroid, which roughly corresponds to the center of the person.


X-Eccentricity
The second measure is the x-eccentricity. This is simply the x-distance between the actual centre of the bounding-box and the x-centroid. This gives a nice measure of how far off to one side the person is. For example, if the person punches/kicks, this will change.


X-Velocity
The third and (currently) final measure is the x-velocity. This is the distance from the x-centroid in the previous frame to the x-centroid in the current frame. So if the person moves from left to right between two frames, we will see a positive x-velocity of some value.

Pictures!
Enough words! Pictures explain it better anyway. (Click to enlarge!)
Red Box: Bounding Box
Vertical Red Line: Half Bounding Box
Vertical Green Line: X-Centroid
Space between vertical red/green: X-Eccentricity
Horizontal Green Line: X-Velociy

Standing Still

Moving Left -> Right

Moving Right-> Left

Crouching

Crouching Punch

Why do we not calculate these for y-coordinates? Well, simply put we're taking a short cut in the fact that vertical movement doesn't happen very much, we're much more interested in x-movement for the time being. They may be added at a later time though.

Next up:

• Centralising images for the game.
• Complicated data mining.
• SOUND! If anyone knows a good java library for sound, let me know! I'm currently investigating jOrbis and paulscode, but other possibilities would be greatly appreciated!

Edit:
Nearly forgot, a system for playing animations in the game has now been added too! Bonus!

Post 7 - Menu Screen

Just a quick post, today we have a new menu screen for the game!
Click to enlarge as always.

Post 6 - Analysis and Game Progress

After a brief hiatus for various coursework assignments and group projects, work has resumed on the project.

Animation Creator
A large part of the animation creator code has been completed. This part looks at the frames generated by the background creator, and searches for repeating patterns to use as animations. The code here seem to work, except for two problems:

1. An implementation for choosing letters from an alphabet using some sort of even distribution hasn't been written yet. This is crucial to finding appropriate matches, and without it typically everything is found to match (almost) everything else!
2. The original videos captured for this project probably aren't working particuarly well with the current algorithm. The algorithm will work best with some sort of structured, repeating pattern of various different moves, while the current videos are a bit haphazard and random. A re-shoot is in order.

As a result, when the current implementation runs it finds lots of matches, but none of them are very useful!

Game
Whilst the animation creator is still being completed, I thought it would be a good idea to start work on the actual game itself. Plus there aren't many pictures to show you for the animation creator, and we all love pictures! So here we are (click to embiggen):

Currently you can move left/right, crouch, and punch. There are no animations, hit boxes, or a lot of other things, but it's a start.

As an aside, if I can't get the game up to scratch, I've started looking at an alternative to place animations in. jmugen looks promising, if a little involved.

Post 5 - Populating arrays (again)...

Quick post of a discovery. Yet again I want to create a (comparatively) small array, and initialize it to contain 'false'. In a previous post I discovered the way to do this is:

Boolean[] mask = new Boolean[maskSize];

for (int j = 0; j < maskSize; j++) {
     mask[j] = false;
}

In my travels across the interwebs I have also discovered another way to fill it, which looks a bit neater:

Boolean[] mask = new Boolean[maskSize];

Arrays.fill(mask, false);

I do enjoy cutting down for loops to single lines. However, digging a bit deeper, Arrays.fill() simply uses a for loop inside it, so usually runs in the same time. Worse though, (at least in eclipse) when Arrays is first called, presumably it must load the class, which puts another overhead, increasing the runtime by quite a lot! I suspect if you were to compile it to a .jar file and run it, either way would be equivalent, but seeing as I'll mostly be using this in eclipse I might as well leave it as a for loop!

As an aside, some of my posts here are actually quite embarassing, but I figure if I don't post embarassing stuff, I'll never learn! Not to mention this blog would be a lot shorter too!

Improving Efficiency

Whilst coding up the masks, I took some shortcuts which had a pretty big hit on performance, so I've spent a little time on speeding things up. It doesn't really matter how long the run-time of this part of the project is, but it can get boring watching a screen of updates go by slowly...

Before:

Ouch! That's over 1 second per frame!

After:

Ah, much better!

This was done by simplifying a search through the image for certain pixels. Hopefully after a few more tweaks, it'll be even faster.

Post 3 - Mask

Previously the background was subtracted using a mask, but at no point in the code could you stop it and say 'here is the mask', as it was all calculated in one rather large if statement. The code has now been extracted out to actually create a mask, which is then applied to the image during processing. This runs a bit slower because it's not particuarly efficient, but it allows other adjustments to be made to the mask easily. The first of these is... I forget the name, but essentially it removes strands of individual pixels, e.g. where a shadow fell along an edge, a subtle difference, but it helps.

Post 2 - Bounding Box

Draw a bounding box around the figure. This is useful because it encapsulates data about the x and y movement.

As a side note, I also fixed the background subtraction. Until now I had to run an old version of the project to get that part to work properly. Why? It turns out you can't create an array, then populate the array with an enhanced for-loop. Taken me 2 weeks to find this problem!

Do this:
        int cells = size * size;
        float[] matrix = new float[cells];

        for (int i = 0; i < cells; i++) {
            matrix[i] = 1.0f / (float) cells;
        }
Not this:
        int cells = size * size;
        float[] matrix = new float[cells];

        for (float i : matrix) {
            i = 1.0f / (float) cells;
        }
Not quite sure why the latter doesn't work, possibly something to do with it not being initialised? Any ideas?

Edit (24/1/2011):  As pointed out to me by my friend:

It doesn't work because floats are primitive, so assigning them is not changing what is stored in the array.

Post 1

Dump a video into .png images using ffmpeg command line tool. 1min video = ~1500 images.

Generate a background image by taking a temporal median. Can be created either from frames of the background, or from a sample of the frames with the person in them.

Extract the person using background subtraction and thresholding.

Next time: Create a bounding box around the figure.