Image Processing with Pygame

Example image compressed using uneven block size

In this series of tutorials I will show how various image processing algorithms work. Many of these processes can be done a lot more simply using the Python Image Library (PIL). However, the point of these tutorials is to show how the algorithms work and I think Pygame and Numpy offer a simple low-level way to implement them.

I aim to cover simple processes, such as cropping, sharpening and blurring, as well as some more complex ones, such as edge-detection and depth-stacking. I will also include some experimental ideas of my own, such as the algorithm that created the image above.

Input and output

The first step for any image processing we want to do will be to load the image into a form we can manipulate. To load the image, we use the pygame.image.load() function, which creates a pygame.Surface object. You can manipulate the pixels of these surfaces (using Surface.get_at() and Surface.set_at()), but it's very slow. If we want to do a lot of processing, it faster to first convert the Surface to an array using the pygame.surfarray module. The function below opens a given file and returns an array of pixel intensities.

import pygame

def getPixelArray(filename):
        image = pygame.image.load(filename)
    except pygame.error, message:
        print "Cannot load image:", filename
        raise SystemExit, message
    return pygame.surfarray.array3d(image)

Pixel arrays

The pixel array returned by this functions is three-dimensional, which may seem surprising given that the image is 2D. You can find the dimensions of your array like this:

pixels = getPixelArray('my_image.jpg')
print pixels.shape

If your image is 640x400 image, then you will see the tuple, (640, 400, 3). The extra dimension of size three is because the colour of each pixel is stored as a tuple of three values. These value refer to the intensity of red, blue and green (RGB) respectively, on a scale of 0-255. For example, (255, 0, 0) is pure red, (0, 255, 0) is pure green, (0, 0, 255) is pure blue, (0, 0, 0) is black and (255, 255, 255) is white.

Saving images

Once we've loaded our image and processed it, we will want to save it. Below is a function that converts a pixel array to saved image using The type of file created by this function depends on the extension of the filename. It can save images as jpg, bmp, png or tga.

def saveSurface(pixels, filename):
        surf = pygame.surfarray.make_surface(pixels)
    except IndexError:
        (width, height, colours) = pixels.shape
        surf = pygame.display.set_mode((width, height))
        pygame.surfarray.blit_array(surf, pixels), filename)

Normally the first line of the try statement and the save() call are sufficient, but recently I've been having some issues. The code in the except block gets around the problem by first creating a pygame screen and blitting the image to it before saving the screen.

Now we have some functions to convert images to and from arrays. In the next tutorials we will look at how to process arrays in interesting ways. 

imageProcessing1.txt845 bytes

Array slicing

In the previous tutorial we converted an image to and from and surfarray object. This object is a 3D array, so all our processing will involve working with 3D arrays. The Pygame surfarray is does not work in quite the same way as normal Python arrays, but rather like numpy arrays. in this tutorial I will go over how to work with these arrays, so if you're familiar with numpy, this should all be very easy.

For this tutorial I will be using this example image (of a passion flower):

Passion flower

We can use the function defined in the previous tutorial to convert it into a surfarray object:

pixels = getPixelArray('Passion_flower.jpg')

Array indexing

In Python we normally index an array with pixels[n]. If we try this with our 3D array, it will return a 2D slice with dimensions (200 x 3) which represents the three RGB colour values for every pixel in the nth column in the image. If you set all the values in the 100th slice to 0 and save the image like so:

pixels[100] = 0
saveSurface(pixels, 'Passion_flower_black_column.jpg')

You get an image with all the pixels in the 100th column as black (0, 0, 0).

Passion flower with a black line

If we want a single value, we use three indices, separated by commas. For example, for my image pixels[50, 100, 0] returns 80. This tells us that the pixel 50 across, 100 down from the top left pixel has a red value of 80.

Array slicing

As with normal Python array slicing, we use colons to separate the start and stop value of a slice. For example:

pixels = pixels[50:-50, 200:, :]
saveSurface(pixels, 'Passion_flower_crop.jpg')

This creates a cropped image starting 50 pixels from the left to 50 pixels from the right, and starting 200 pixels from the top to the bottom; it includes all the colour information.

Passion flower crop

Array slicing with steps

Again, like normal Python arrays we can define a step for the slice with a second colon. For example:

pixels = pixels[::2,::2,:]
saveSurface(pixels, 'Passion_flower_scale.jpg')

This takes every second row of pixels and every second column of pixels and thus creates a scaled image:

Scaled passion flower

Compression with uneven block size

Picture of Victoria compressed with uneven block size

I came across this article which shows images that have been compressed using blocks of different sizes. It looked interesting, so I made my own version using Python.

The program works by reading in an image (as described here) and calculates the variance of pixel intensities in a 256 x 256 pixel square. If that variance is above a threshold then it splits the image into a four 128 x 128 pixel squares and repeats the process for each of them until it reaches a single pixel. If the variance is below the threshold at any point, then it creates a square of that size with the mean colour. By changing the threshold I can change how blocky the image is.

I later added an option to output the processing.js code so I can create an image as a Khan Academy computer science program. The only real addition was to make the program first order the squares by colour so all the squares of the same colour are next to each other so fill() needs only to be called once per colour. I used it to create a version of the Mona Lisa which is now my most upvoted program.

[Update: I've put a version online at]

Below, I've attached a version of my Python program which is relatively user-friendly. You need to change the file extension from .txt to .py and you need to have Python and the numpy library. To run, call:

>>> python path/to/file.jpg

A png image will be created with the same filename, but ending in _constrast_blocks. The program uses a default block size of 1200, which seems to work quite well on the images I've tried. To change the value use the -t option, e.g.

>>> python path/to/file.jpg -t2000

To output the code for a Khan Academy program add a -k option.

>>> python path/to/file.jpg -k



findContrastBlocks.txt3.69 KB