In my previous article, I described how I first tried
to solve the “large image” problem on the iPad using the tiling technique.
In this technique, you scale the image to different sizes, then break down each
image into square tiles. Using Cocoa’s
CATiledLayer class, you can then draw
only the tiles you need at different zoom layers.
However, when running on an iPad 1, I would still periodically run out of memory when trying to compute tiles for large images. Thus, in Version 2.1 of Pholio, I opted for a much simpler approach. When a customer provides a large image, I shrink it down to a single manageable size. I picked images of no more than 1500 pixels on a side. That caps the memory used for displaying the image at 9 MB, and it still lets the user zoom a little to see more detail.
Remember that the tiling technique requires you to resize the image multiple
times, and at each size you need to compute and save the tiles. That takes
time and memory. The new technique is not only simpler, it’s faster… I
just resize the image once and save it. Showing a single resized image in
UIImageViewController instead of a tiled image with a
gets rid of the annoying flicker each time the user scrolls to a new image.
The only thing that keeps this from being a complete win? Users can no longer
zoom in to see the actual pixels in their images.
That said, there was still a fair bit of work to efficiently resize images. Let me tell you the tips I use Pholio.
Tip 1: Have a single, simple routine for resizing
In Pholio, that routine is
-[IPPhoto optimize]. I’ll talk about it in
more detail soon, but at a high level it has these key attributes:
- It is synchronous. This means it is easy to unit test. It also means it will be slow… more on multithreading concerns in a bit.
- It fences the excessive memory usage — the whole problem is resizing
a large image consumes a lot of memory, and I work to ensure that all
possible memory is released by the time the routine exits. Nothing hangs
around in an
autoreleasepool, as little as possible sits in the
IPPhotoobject. Resize the image, release memory… because the system will be hurting.
- It does all of the necessary prep work to make the image efficient to
show on the iPad. This means resizing large images and computing thumbnails
for all images. Call
-[IPPhoto optimize]and be done.
- Critical: I organize the rest of the code to make sure
IPPhotoobjects do not make it into the data model until after I have called
-[IPPhoto optimize]on the object. This means that everything that is in the data model is fair game for showing on the screen.
Having all of the hard work around photo resizing in a single synchronous routine makes it very easy to reason about the correct behavior of the program. However, because this routine takes such a long time, it must be done on a background thread; otherwise, the UI would be locked up for seconds. This brings me to:
Tip 2: Use background threads — but not too many!
One of my first mistakes was naively making my program responsive by
pushing all calls to
-[IPPhoto optimize] in the default background
queue using Grand Central Dispatch. Then, once the photo was done optimizing
in the background, I would call a completion routine on the main thread that
would insert the photo into the model.
The problem? Too much background optimization! Because each call to
can consume so much memory, it would kill the iPad 1 to have more than one
optimization happening at the same time. And that’s exactly what would happen
when I put different optimizations in the background queue. Not knowing better,
Grand Central Dispatch would schedule the work concurrently.
- It creates a single
NSOperationQueueto use for all memory intensive operations, like resizing. Then, using
-[NSOperationQueue setMaxConcurrentOperationCount:], I ensure that at most one option will take place in the background at a time.
- It defines simple methods to queue an optimization for a photo (or an array of photos) and call a completion routine on the main thread when the work is complete.
- It maintains a counter of all pending and active optimizations.
- It notifies a delegate each time the number of active optimizations changes. I use this to display a notice to the user that optimization is happening.
IPPhotoOptimizationManager class, Pholio can control the
level of background optimizations.
Tip 3: ImageIO is your friend
My original image resizing code came from the incredibly helpful Trevor’s Bike Shed. It works great, and it’s an incredibly helpful way to get started.
However, with my program crashing at regular intervals because it runs out of memory, I decided to go straight to the source: Apple’s versatile ImageIO library. ImageIO is a C rather than an Objective-C library, so it’s a little more difficult to use, but it’s the most efficient way to read and resize images.
My use of ImageIO comes primarily inside
-[IPPhoto optimize]. Here’s how
it works. First, to use ImageIO, you start with an image source (
Conveniently, you can create an image source from any file by generating a
file URL. From the Pholio code:
Remember that a
CGImageSourceRef is a Core Foundation object, so you need to call
CFRelease() on it.
imageSource, Pholio next determines if it needs to resize the
image. I want to resize any image where the maximum pixel size is greater
than 1500 pixels (in the code, the constant
kIPPhotoMaxEdgeSize). Note that
ImageIO lets you get the image metadata without reading the entire image into
memory; this is the way to determine the image size. So:
1 2 3 4 5 6 7 8 9 10 11 12 13 14
OK, so if I determine I need to resize the image, how does it get done?
The answer is ImageIO’s
This is a particularly awkward function to use because you need to pass
most of the meaningful parameters as items in a dictionary. (This is the
way all of ImageIO works, but I’ve been able to ignore it in the other
ImageIO calls until now.) I’m a bit of a lazy programmer, so I take advantage
of the toll-free bridging between
and use the easy convenience constructor for
1 2 3 4 5
Taking this apart: this code says:
- The resized image should have whatever rotation transformations exist
on the original image (
- The resized image should have at most
kIPPhotoMaxEdgeSizepixels on a side (
- The function should resize the image exactly to my specifications, rather
than reuse a smaller one that may exist in the file (
At this point, I have a valid
CGImageRef object. I compress it as a JPEG
and save it:
1 2 3 4
Tip 4: Don’t rely on UIImage unloading its memory data
The previous three tips made a huge difference in the reliability of Pholio on the iPad 1. However, it was still too easy to crash the program when importing lots of large images.
I spent some time in the Instruments to see if I could figure out what was going on. When your application gets a memory warning, the important thing to look at is what’s creating lots of dirty pages in memory. The VM Tracker instrument can show you information about dirty pages. What I found in instruments: In the simulator, after going through some galleries and importing images, I had over 140 MB of dirty pages — even after simulating a memory warning! The VM Tracker told me that most of the dirty pages were Memory tag 70. If you can believe the Internet, this memory comes from images loaded in memory. Makes sense… that’s what my program does.
Why was I using so much image data after receiving a memory warning? True,
UIImage objects on demand whenever the
-[IPPhoto image] method
was called, and I never explicitly unloaded the
UIImage objects. However,
according to the
“In low-memory situations, image data may be purged from a UIImage object to free up memory on the system.”
I thus expected the large images to get purged from memory, even though my
UIImage objects were hanging around.
I concluded that
UIImage must not be purging images as aggressively as
I needed. I decided to step in and manage things manually. I wrote the following
simple method on
1 2 3
In Pholio, only one class displays images at full resolution:
I made two small changes:
- Whenever the
IPPhotoScrollViewis told to display a new
IPPhotoobject, it sends an
unloadImagemessage to the old
-[IPPhotoScrollView dealloc], I send an
unloadImagemessage to the current
These two changes mean that I explicitly unload images when they are no longer being shown to the user.
The result? In the simulator, the dirty page size reduced to less than 54MB – these simple changes reduced the dirty page footprint by 60%!
After making all of these changes (using Image IO directly, ensuring no more than one photo optimization happens at a time, unloading images when they are no longer shown to the user), I was robustly handling large images on both the iPad 1 and the iPad 2.