Brian’s Brain

I Need a Tagline

Pholio Coder's Guide: Tiling Large Images

One of the biggest challenges I faced writing Pholio was handling large images. Using the Camera Connection Kit, photographers will import full-resolution images onto their iPads. Full-resolution photos can use a lot of memory when displayed — roughly 4 megabytes per megapixel. So each image from the 16 megapixel D7000, for example, will need 64 MB of RAM to show the picture at full resolution. If the photographer is using an iPad 1, with 256 MB of RAM, the device will be straining to show even one full resolution photo. Smooth scrolling requires you load more than one photo into memory, though. So what do you do?

The simplistic answer: Never show full resolution photos! The iPad screen has a resolution of a little under one megapixel. There’s no need to load a full 16 MP image on a display that can show only 1 MP. Your challenge as a programmer: Efficiently turn the 16 MP image that the user gives you to the 1MP image you need for display. This article shares some tips I learned writing Pholio on managing large images.

Attempt #1: Tiling

My first attempt at handling large images relied upon tiling. Tiling holds a lot of promise. You get the efficient memory use of small pictures, but customers can still zoom in and see the detail of the large image. This magic works because when you zoom in to look at high-resolution detail, you aren’t looking at the whole picture. If you break your high-resolution image into tiles and load only the tiles you need, you can avoid the memory hit of loading the entire high resolution image.

The following diagram gives you an idea of Pholio used tiling for large photos. Starting with the full resolution image, I computed how many total levels of detail I wanted to provide. Each level of detail has one quarter the resolution of the prior level (it’s half the size in each dimension). The highest level of detail was the full resolution image; the lowest level of detail is the image that would just fill the screen. Then, I’d rescale the image to the appropriate size and finally divide it into fixed-size square tiles. Each tile was saved as an individual PNG. (You can find the code at IPPhoto.m — start at -[IPPhoto saveTilesForAllScales]. At its core, I use code from the great tutorial at Cocoa is my Girlfriend).

Example of tiling

Once I had the photo broken down into tiles, I relied on a simple custom UIView subclass to show the photo. This subclass, IPPhotoTilingView, uses CATiledLayer to efficiently draw itself to the screen using the tiles I saved previously. The only drawback I could see to the tiling approach was the photo would “flickr” when it first appeared; you could see the boundary between the tiles as they were drawn.

I shipped Version 2.0.2 of Pholio using this strategy for dealing with large images. On demand, I would spawn a background thread to generate tiles of photos. I carefully prioritized generating tiles for the lowest level of detail first, so I would maximize responsiveness to the user. I wrote code to make sure that the user couldn’t zoom to a level of detail that I hadn’t finished tiling yet. Everything worked great in my tests and on my device. I was proud of what I wrote.

Interlude: The iPad 1

After Version 2.0 shipped, I had the moment every developer dreads: A customer contacted me telling me that the program was crashing. After a few emails back and forth, I figured out why: He was working with full-resolution 10 MP images on an iPad 1. I did all my device testing on an iPad 2, which has twice the memory of the iPad 1.

I started testing with full resolution photos on an iPad 1, and I found what the customer had discovered. The process of tiling the photo made it too easy to exhaust memory on the device. Let me tell you now, if you haven’t discovered it yet: There’s nothing more frustrating than debugging memory termination errors when you know you need lots of memory for a short time.

I needed a new approach. In my next article, I will describe the more conservative approach to large images that I use in version 2.1.