Written by Joshua Noble, with images by Robert Hodgin
In this little tutorial we're going to follow a path that starts with a file on the filesystem, say a PNG file, and ends with the image being drawn to the screen using OpenGL.
I'll explain a little of how this process works in general and a lot of how this process works in Cinder because, of course, that's what we're most interested in. The simplest way to load and display an image would be something similar to the following:
// probably in your App's setup() method
gl::Texture texture = loadImage( "image.jpg" );
// and in your App's draw()
gl::draw( texture );
Even though that code simply loads and draws an image, there's actually quite a lot going on behind the scenes. The first line creates a new OpenGL texture from the result of
These two lines highlight one of the core characteristics of Cinder's design: data, its I/O (Input/Output) and manipulation are carefully separated. Objects and functions are designed for distinct logical points in the process of loading, manipulating, and transferring data. For example,
Continuing this theme of the separation of responsibilities, we introduce the
Surface mySurface; // initialized elsewhere
gl::Texture texture = gl::Texture( mySurface );
The rest of this article is going to run through a few different topics in greater depth:
The uint8_t
or a 32 bit float
. These two types are Surface8u and Surface32f, respectively. The majority of the time a Surface8u is just what you need. However if you're doing some advanced image processing, or you want to make use of high dynamic range images, Surface32f is your best bet. Also it's worth noting Surface is just a convenient synonym for Surface8u - you can use either name for the class in your code.
The most important thing to recognize about the
Surface8u regularSurface; // an empty 8 bit surface
Surface32f hdrSurface; // an empty 32 bit high dynamic range surface
When you declare a
Surface mySurface( 640, 480, true ); // width, height, alpha?
So what's that doing? Simply setting aside memory.
Surface mySurface( 640, 480, true, SurfaceChannelOrder::RGBA ); // width, height, alpha?, channel order
Here we've asked that the color channels of a pixel be ordered in memory as red-green-blue-alpha
. This last parameter is optional (notice that we left it out earlier) and it defaults to something reasonable based on whether you have an alpha channel or not. If we had passed something different like blue-green-red-alpha
order in memory instead. Why would we ever want that? Well many graphics APIs - such as Apple's Quartz, Microsoft's GDI+, Cairo, or OpenCV prefer or even require that pixels be ordered in a manner other than RGBA. This feature is part of what allows Cinder
In the image below, the small area outlined by the white box is blown up on the right hand side to show the red, green and blue values, as well as the alpha, which is depicted as a block of white to indicate that the pixel is 100% opaque (a value of 255).
The application deals with the bitmap data sequentially...
... and stores it in memory as an array of numbers (in this case, bytes).
The idea to take away is that the
For advanced users, if you're familiar with C++ templates then you'll recognize what you see when you open up the Surface.h file - the core type is a class uint8_t
or float
. However understanding the implementation is definitely not necessary to understand how to use it. Also, as a side note, Cinder supports a 3rd less common
There are a few different ways to create a new image from the data in another image. First up, using the
newSurface = oldSurface.clone();
This function creates a new
newSurface = oldSurface;
This gives us a very different result. While vector<Surface>
, for example. In the example above, oldSurface and newSurface are able to safely sort out memory managment between themselves automatically. So if oldSurface goes away, it won't take the image data that newSurface is now pointing to with it. However if both oldSurface and newSurface go away, the memory they were sharing is freed for you automatically. Last one out turns off the lights. You'll find this design technique used throughout Cinder. If you're into the Design Pattern literature, you might know this techique as the handle-body idiom. Or if you are familiar with shared_ptr
or other reference-counted pointers, you can think of shared_ptr
's.
We can also copy just a section of a
Surface newSurface( oldSurface.getWidth() / 2, oldSurface.getHeight(), false );
newSurface.copyFrom( oldSurface, newSurface.getBounds() );
That second parameter is an
You can also load
Surface myPicture = loadImage( "myPicture.png" );
gl::Texture myTexture; // initialized elsewhere
Surface fromTex( myTexture );
both of which are discussed in greater detail in the I/O section. These methods are appropriate when you're copying big blocks of data. But what if you want to manipulate a single pixel, or even more surgically, a single channel value within a pixel?
Something that you might want to do with a
Surface bitmap( loadImage( "image.jpg" ) );
Area area( 0, 0, 500, 500 );
Surface::Iter iter = surface->getIter( area );
while( iter.line() ) {
while( iter.pixel() ) {
iter.r() = 255 - iter.r();
iter.g() = 255 - iter.g();
iter.b() = 255 - iter.b();
}
}
Here we construct an instance of a helpful class,
For instance, in the image below, if the Iter is currently pointing at the pixel labeled A in the image, then r(0,-1)
will access the red value of the pixel labeled B, and r(1,1)
, will access the red value of the pixel labeled C in the image.
This saves you the trouble of either using multiple iterators (messy) or maintaining multiple pointers (awful). The
void TwirlSampleApp::twirl( Surface *surface, Area area, float maxAngle )
{
// make a clone of the surface
Surface inputSurface = surface->clone();
// we'll need to iterate the inputSurface as well as the output surface
Surface::ConstIter inputIter( inputSurface.getIter() );
Surface::Iter outputIter( surface->getIter( area ) );
float maxDistance = area.getSize().length() / 2;
Vec2f mid = ( area.getUL() + area.getLR() ) / 2;
while( inputIter.line() && outputIter.line() ) {
while( inputIter.pixel() && outputIter.pixel() ) {
Vec2f current = inputIter.getPos() - mid;
float r = current.length();
float twirlAngle = r / maxDistance * maxAngle;
float angle = atan2( current.y, current.x );
Vec2f outSample( r * cos( angle + twirlAngle ), r * sin( angle + twirlAngle ) );
Vec2i out = outSample - current;
outputIter.r() = inputIter.rClamped( out.x, out.y );
outputIter.g() = inputIter.gClamped( out.x, out.y );
outputIter.b() = inputIter.bClamped( out.x, out.y );
}
}
}
Without getting into too much detail, the algorithm converts the coordinates of each pixel into polar coordinates, adds a value to the angle based on how far it is from the center, and then converts this back to rectangular coordinates. One thing to note in that code is the use of
You may have noticed that channels in a bitmap get special attention, so much so that they have their own class to help you work with them more easily. Let's look at the
You can think of a
A
Surface surface( channel );
You'll get a grayscale image automatically because the red, green and blue channels will all be set to the same values as the
Channel channel( surface );
In this case a high quality grayscale interpretation is automatically made of your RGB data. We say "high quality" because the red, green and blue are weighted to mimick the way your eye perceives luminance (derived from the Rec. 709 high definition video spec, for those interested in such things). Using a
You can also mix the 8u and 32f variants of the
Channel8u myChannel( ... );
Surface32f myHdrSurface( myChannel );
A potential use for the
// only uses the red pixels of the mask Surface
void ChannelDemoApp::surfaceMaskImage( const Surface &mask, Surface *target )
{
Surface::ConstIter maskIter( mask.getIter() ); // using const because we're not modifying it
Surface::Iter targetIter( target->getIter() ); // not using const because we are modifying it
while( maskIter.line() && targetIter.line() ) { // line by line
while( maskIter.pixel() && targetIter.pixel() ) { // pixel by pixel
float maskValue = maskIter.r() / 255.0f;
targetIter.r() *= maskValue;
targetIter.g() *= maskValue;
targetIter.b() *= maskValue;
}
}
}
That could be made more general and potentially accelerated by passing a
void ChannelDemoApp::channelMaskImage( const Channel &mask, Surface *target )
{
Channel::ConstIter maskIter = mask.getIter(); // using const because we're not modifying it
Surface::Iter targetIter( target->getIter() ); // not using const because we are modifying it
while( maskIter.line() && targetIter.line() ) { // line by line
while( maskIter.pixel() && targetIter.pixel() ) { // pixel by pixel
float maskValue = maskIter.v() / 255.0f;
targetIter.r() *= maskValue;
targetIter.g() *= maskValue;
targetIter.b() *= maskValue;
}
}
}
As a side note, this could be optimized substantially, so don't take the above code to be an example of fast image manipulation.
The value of a Channel at any position can be retrieved using the
gl::Texture createEdgeTexture( const Channel &src )
{
Channel temp( src.getWidth(), src.getHeight() );
cinder::ip::edgeDetectSobel( src, &temp );
return gl::Texture( temp );
}
This would be called like so:
myTexture = createEdgeTexture( simpleSurface.getChannelBlue() );
In summary, the Channel is a lightweight tool for working with a particular color channel from a
Very often you'll want to read or write an image file, which is a topic we'll explore in this section. In the beginning of this article you saw loading images using the
loadImage( "data/image.png" );
or, with a slight variation, from a URL:
loadImage( loadUrl( "http://site.com/image.png" ) );
We can also use this method to load images stored in resources (described in a separate guide):
loadImage( loadResource( RES_LOGO_IMAGE ) );
If you look up
gl::Texture myTexture = loadImage( "grayscale.png" );
OpenGL can store textures in a special grayscale-only mode which saves valuable GPU memory. The
And what about the other way around - writing an image out as a file? As you might have guessed, it's done with a function called
writeImage( "/path/to/image.png", surfaceToBeWritten );
Cinder automatically infers the sort of image you want from the file extension. You can also force the image to be written with a certain format by supplying the extension as a string for the 3rd parameter:
writeImage( "/path/to/image_without_extension", surfaceToBeWritten, "jpg" );
It's not just Surfaces that can be saved. For example, you can pass a
writeImage( "output.png", loadImage( "input.jpg" ) );
As we've seen, the
There are a few conceptual things to keep in mind when working with a Texture. Fundamentally it's just a bitmap object, but your program has handed it off to another piece of hardware, the GPU, which has its own memory and processors. Copying from CPU to GPU (by for example, constructing a
The actual data of a
You've seen how
gl::draw( mProcessedImageTex, getWindowBounds() );
The first parameter is of course the
texture = gl::Texture( surfaceInstance );
or by initiailizing the Texture with some dummy data:
gl::Texture texture(500, 500);
Of course if you just draw this, your program won't crash, but you'll just see junk data - whatever random memory happens to be on the graphics card at the moment. Cool if you're making glitch art, not if you're not.
You can also initialize the Texture with a gl::Texture::Format object that contains specs for how to build the Texture. For instance, by default a
gl::Texture::Format fmt;
fmt.setWrap( GL_REPEAT );
texture = gl::Texture( someSurface, fmt );
The differences between clamping and repeating can be seen in the image below:
There are other features that you can set with the Format object too - mipmapping for instance. What's that you may ask? Take a look at the following images:
Notice the edges on the left. As the English say: that's not cricket. We want smoothed edges as the
A mipmap is a miniaturized version of the larger bitmap that helps you reduce artifacts when scaling. The smaller the texture is drawn, the smaller the mipmap that the GPU will select. As an example: a texture has a basic size of 256 by 256 pixels, so a mipmap set will be generated with a series of 8 images, each one-fourth the total area of the previous one: 128x128 pixels, 64x64, 32x32, 16x16, 8x8, 4x4, 2x2, 1x1 (a single pixel).
Artifacts are reduced since the mipmap images are effectively already anti-aliased, taking some of the burden off the real-time renderer and potentially improving performance.
gl::Texture::Format fmt;
fmt.enableMipmapping( true );
fmt.setMinFilter( GL_LINEAR_MIPMAP_LINEAR );
You'll notice we added a call to setMinFilter() to tell it use our mipmap. In general Cinder is designed for you to use the OpenGL constants directly (like GL_LINEAR_MIPMAP_LINEAR
) so if these are unfamiliar to you, we'd recommend you pick up a book like The OpenGL Programming Guide.
And that's that. As promised: image data from the filesystem, to the CPU, to the graphics card, and finally to your screen. Now go make something and have fun. And thanks to Flickr user Trey Ratcliff, whose beautiful photograph is used throughout this tutorial.