Brian’s Brain

I Need a Tagline

Resize Images on Your iPad

In my previous article, I described how I first tried to solve the “large image” problem on the iPad using the tiling technique. In this technique, you scale the image to different sizes, then break down each image into square tiles. Using Cocoa’s CATiledLayer class, you can then draw only the tiles you need at different zoom layers.

However, when running on an iPad 1, I would still periodically run out of memory when trying to compute tiles for large images. Thus, in Version 2.1 of Pholio, I opted for a much simpler approach. When a customer provides a large image, I shrink it down to a single manageable size. I picked images of no more than 1500 pixels on a side. That caps the memory used for displaying the image at 9 MB, and it still lets the user zoom a little to see more detail.

Remember that the tiling technique requires you to resize the image multiple times, and at each size you need to compute and save the tiles. That takes time and memory. The new technique is not only simpler, it’s faster… I just resize the image once and save it. Showing a single resized image in a UIImageViewController instead of a tiled image with a CATiledLayer also gets rid of the annoying flicker each time the user scrolls to a new image. The only thing that keeps this from being a complete win? Users can no longer zoom in to see the actual pixels in their images.

That said, there was still a fair bit of work to efficiently resize images. Let me tell you the tips I use Pholio.

Tip 1: Have a single, simple routine for resizing

In Pholio, that routine is -[IPPhoto optimize]. I’ll talk about it in more detail soon, but at a high level it has these key attributes:

  • It is synchronous. This means it is easy to unit test. It also means it will be slow… more on multithreading concerns in a bit.
  • It fences the excessive memory usage — the whole problem is resizing a large image consumes a lot of memory, and I work to ensure that all possible memory is released by the time the routine exits. Nothing hangs around in an autorelease pool, as little as possible sits in the IPPhoto object. Resize the image, release memory… because the system will be hurting.
  • It does all of the necessary prep work to make the image efficient to show on the iPad. This means resizing large images and computing thumbnails for all images. Call -[IPPhoto optimize] and be done.
  • Critical: I organize the rest of the code to make sure IPPhoto objects do not make it into the data model until after I have called -[IPPhoto optimize] on the object. This means that everything that is in the data model is fair game for showing on the screen.

Having all of the hard work around photo resizing in a single synchronous routine makes it very easy to reason about the correct behavior of the program. However, because this routine takes such a long time, it must be done on a background thread; otherwise, the UI would be locked up for seconds. This brings me to:

Tip 2: Use background threads — but not too many!

One of my first mistakes was naively making my program responsive by pushing all calls to -[IPPhoto optimize] in the default background queue using Grand Central Dispatch. Then, once the photo was done optimizing in the background, I would call a completion routine on the main thread that would insert the photo into the model.

The problem? Too much background optimization! Because each call to -[IPPhoto optimize] can consume so much memory, it would kill the iPad 1 to have more than one optimization happening at the same time. And that’s exactly what would happen when I put different optimizations in the background queue. Not knowing better, Grand Central Dispatch would schedule the work concurrently.

To address this, I introduced a new class, IPPhotoOptimizationManager (.h, .m). The optimization manager does the following tasks:

  • It creates a single NSOperationQueue to use for all memory intensive operations, like resizing. Then, using -[NSOperationQueue setMaxConcurrentOperationCount:], I ensure that at most one option will take place in the background at a time.
  • It defines simple methods to queue an optimization for a photo (or an array of photos) and call a completion routine on the main thread when the work is complete.
  • It maintains a counter of all pending and active optimizations.
  • It notifies a delegate each time the number of active optimizations changes. I use this to display a notice to the user that optimization is happening.

Using the IPPhotoOptimizationManager class, Pholio can control the level of background optimizations.

Tip 3: ImageIO is your friend

My original image resizing code came from the incredibly helpful Trevor’s Bike Shed. It works great, and it’s an incredibly helpful way to get started.

However, with my program crashing at regular intervals because it runs out of memory, I decided to go straight to the source: Apple’s versatile ImageIO library. ImageIO is a C rather than an Objective-C library, so it’s a little more difficult to use, but it’s the most efficient way to read and resize images.

My use of ImageIO comes primarily inside -[IPPhoto optimize]. Here’s how it works. First, to use ImageIO, you start with an image source (CGImageSourceRef). Conveniently, you can create an image source from any file by generating a file URL. From the Pholio code:

1
2
  NSURL *imageUrl = [NSURL fileURLWithPath:self.filename];
  CGImageSourceRef imageSource = CGImageSourceCreateWithURL((CFURLRef)imageUrl, NULL);

Remember that a CGImageSourceRef is a Core Foundation object, so you need to call CFRelease() on it.

Armed with imageSource, Pholio next determines if it needs to resize the image. I want to resize any image where the maximum pixel size is greater than 1500 pixels (in the code, the constant kIPPhotoMaxEdgeSize). Note that ImageIO lets you get the image metadata without reading the entire image into memory; this is the way to determine the image size. So:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
  CFDictionaryRef imageProperties = CGImageSourceCopyPropertiesAtIndex(imageSource, 0, NULL);
  CFNumberRef pixelWidthRef  = CFDictionaryGetValue(imageProperties, kCGImagePropertyPixelWidth);
  CFNumberRef pixelHeightRef = CFDictionaryGetValue(imageProperties, kCGImagePropertyPixelHeight);
  CGFloat pixelWidth = [(NSNumber *)pixelWidthRef floatValue];
  CGFloat pixelHeight = [(NSNumber *)pixelHeightRef floatValue];
  CGFloat maxEdge = MAX(pixelWidth, pixelHeight);

  if (maxEdge > kIPPhotoMaxEdgeSize) {

    //
    //  Need to resize
    //
  }
  CFRelease(imageProperties);

OK, so if I determine I need to resize the image, how does it get done? The answer is ImageIO’s CGImageSourceCreateThumbnailAtIndex() function. This is a particularly awkward function to use because you need to pass most of the meaningful parameters as items in a dictionary. (This is the way all of ImageIO works, but I’ve been able to ignore it in the other ImageIO calls until now.) I’m a bit of a lazy programmer, so I take advantage of the toll-free bridging between CFDictionaryRef and NSDictionary and use the easy convenience constructor for NSDictionary:

1
2
3
4
5
  NSDictionary *thumbnailOptions = [NSDictionary dictionaryWithObjectsAndKeys:(id)kCFBooleanTrue, kCGImageSourceCreateThumbnailWithTransform,
                                    kCFBooleanTrue, kCGImageSourceCreateThumbnailFromImageAlways,
                                    [NSNumber numberWithFloat:kIPPhotoMaxEdgeSize], kCGImageSourceThumbnailMaxPixelSize,
                                    nil];
  CGImageRef thumbnail = CGImageSourceCreateThumbnailAtIndex(imageSource, 0, (CFDictionaryRef)thumbnailOptions);

Taking this apart: this code says:

  • The resized image should have whatever rotation transformations exist on the original image (kCGImageSourceCreateThumbnailWithTransform).
  • The resized image should have at most kIPPhotoMaxEdgeSize pixels on a side (kCGImageSourceThumbnailMaxPixelSize).
  • The function should resize the image exactly to my specifications, rather than reuse a smaller one that may exist in the file (kCGImageSourceCreateThumbnailFromImageAlways).

At this point, I have a valid CGImageRef object. I compress it as a JPEG and save it:

1
2
3
4
  UIImage *resizedImage = [UIImage imageWithCGImage:thumbnail];
  NSData *jpegData = UIImageJPEGRepresentation(resizedImage, 0.8);
  [jpegData writeToFile:self.filename atomically:YES];
  CFRelease(thumbnail);

Tip 4: Don’t rely on UIImage unloading its memory data

The previous three tips made a huge difference in the reliability of Pholio on the iPad 1. However, it was still too easy to crash the program when importing lots of large images.

I spent some time in the Instruments to see if I could figure out what was going on. When your application gets a memory warning, the important thing to look at is what’s creating lots of dirty pages in memory. The VM Tracker instrument can show you information about dirty pages. What I found in instruments: In the simulator, after going through some galleries and importing images, I had over 140 MB of dirty pages — even after simulating a memory warning! The VM Tracker told me that most of the dirty pages were Memory tag 70. If you can believe the Internet, this memory comes from images loaded in memory. Makes sense… that’s what my program does.

Why was I using so much image data after receiving a memory warning? True, I loaded UIImage objects on demand whenever the -[IPPhoto image] method was called, and I never explicitly unloaded the UIImage objects. However, according to the UIImage documentation, “In low-memory situations, image data may be purged from a UIImage object to free up memory on the system.” I thus expected the large images to get purged from memory, even though my UIImage objects were hanging around.

I concluded that UIImage must not be purging images as aggressively as I needed. I decided to step in and manage things manually. I wrote the following simple method on IPPhoto:

1
2
3
  - (void)unloadImage {
    [image_ release], image_ = nil;
  }

In Pholio, only one class displays images at full resolution: IPPhotoScrollView. I made two small changes:

  1. Whenever the IPPhotoScrollView is told to display a new IPPhoto object, it sends an unloadImage message to the old IPPhoto object.
  2. In -[IPPhotoScrollView dealloc], I send an unloadImage message to the current IPPhoto object.

These two changes mean that I explicitly unload images when they are no longer being shown to the user.

The result? In the simulator, the dirty page size reduced to less than 54MB – these simple changes reduced the dirty page footprint by 60%!

After making all of these changes (using Image IO directly, ensuring no more than one photo optimization happens at a time, unloading images when they are no longer shown to the user), I was robustly handling large images on both the iPad 1 and the iPad 2.