There are three ways to store and access large numbers of images in Python.
A Python guide compares PNG files, LMDB, and HDF5 for storing CIFAR-10 images, covering batch loading, read/write speed, and disk usage.
Key Takeaways
- For large Python image datasets, the tutorial compares three storage approaches: PNG files, Lightning Memory-Mapped Databases (LMDB), and Hierarchical Data Format version 5 (HDF5).
- Storage choice depends on scale: simple PNG files work for a few images, while LMDB and HDF5 are better for large training datasets that require batch loading.
- The article evaluates the methods using CIFAR-10, focusing on single-image and many-image read/write performance as well as disk usage.
- Common mistakes include using overly complex formats for small datasets, ignoring memory and disk limits, and treating serialized pickle files as viewable images.
- Developers should test both single-image access and batch access before choosing a storage format for computer vision workloads.
Meta description: Compare Python image storage with Portable Network Graphics files, lightning memory-mapped databases, and hierarchical data format for datasets like CIFAR-10.
Tags: Python image storage, Portable Network Graphics, Lightning Memory-Mapped Databases, Hierarchical Data Format, CIFAR-10
Featured image: Abstract friendly illustration of three Python image storage methods: file folders, memory-mapped database blocks, and hierarchical data trees, with no people, no logos, and no text.
A tutorial at realpython.com compares three Python image storage methods for developers who move from editing a few photos to training on large visual datasets. It contrasts saving images as Portable Network Graphics files, storing images in lightning memory-mapped databases, and grouping images inside hierarchical data format version 5. The example dataset is the Canadian Institute for Advanced Research image dataset, better known as CIFAR-10.
Why the storage method matters
If you are segmenting a handful of images by color or detecting faces one by one using OpenCV, disk files are enough. If you use Python Imaging Library to draw on a few hundred photos, .png and .jpg files remain suitable. The pressure changes when a training loop must load images in batches, perhaps hundreds or thousands of times.
ImageNet is a well-known public image database put together for training models on tasks like object classification, detection, and segmentation, and it consists of over 14 million images. Loading all of those files from disk could take quite a while, long enough to leave your computer and do many other things while you wish you worked at Google or NVIDIA.
What the tutorial covers and what you need
The tutorial covers storing images on disk as .png files, storing images in lightning memory-mapped databases, and storing images in hierarchical data format version 5. It also explores why alternate storage methods are worth considering, what the performance differences are when reading and writing single images, what the performance differences are when reading and writing many images, and how the three methods compare in terms of disk usage.
Readers need a reasonably solid Python foundation and a basic understanding of images, including the fact that they are really composed of multi-dimensional arrays of numbers. A practical sense of relative memory, such as the difference between 10MB and 10GB, also helps.
Dataset facts and setup
The dataset is CIFAR-10, which consists of 60,000 32x32 pixel color images belonging to different object classes, such as dogs, cats, and airplanes. Relatively, CIFAR-10 is not a very large dataset. The credits for the dataset, as described in chapter 3 of the tech report, go to Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The image credit is A. Krizhevsky.
Downloading the Python version sacrifices 163MB of disk space. If you were to use the full TinyImages dataset, you would need about 400GB of free disk space, which would probably be a limiting factor.
After download and unzip, the CIFAR-10 files are not human-readable image files. They have been serialized and saved in batches using cPickle. The tutorial uses pickle only to extract the CIFAR-10 dataset. Python pickle has the key advantage of being able to serialize any Python object without any extra code or transformation on your part. It also has a potentially serious disadvantage: it poses a security risk and does not cope well when dealing with very large quantities of data.
Load CIFAR-10 and compare storage routes
- Download the Python version of CIFAR-10 and unzip it.
- Point the code to the data directory, such as data/cifar-10-batches-py/.
- Use NumPy, pickle, and pathlib to unpickle each of the five batch files.
- For each flattened image, split the red, green, and blue channel blocks of 1024 values and reshape each block into a 32x32 array.
- Stack the three channel arrays to reconstruct the original image and add it to the images list.
- Append the matching label to the labels list.
- Print the shapes to confirm that the loaded training images are in RAM in the images variable, with their corresponding metadata in labels.
- Install the Python packages needed for the three storage methods.
The three routes then serve different needs. Portable Network Graphics files are simple: one image, one file, easy to inspect and share. Lightning memory-mapped databases and hierarchical data format version 5 are presented as alternatives for large collections. The tutorial compares read and write speed for single images and many images, as well as disk usage.
The full article continues at https://realpython.com/storing-images-in-python/. It also offers a bonus guide, Python Face Detection & OpenCV Examples Mini-Guide, which shows practical code examples of real-world Python computer vision techniques, and an Improve Your Python With Python Tricks email delivered every couple of days.
Common mistakes
- Choosing a complex storage format for a few photos. If you only run OpenCV on single images or draw on a few hundred files with Python Imaging Library, .png or .jpg on disk is fine.
- Forgetting the scale. A 10MB working set and a 10GB working set create different pressure on RAM, disk, and training loops.
- Treating cPickle files as viewable images. They are serialized batches, not ordinary image files.
- Loading all data into RAM without checking memory or disk usage first.
- Ignoring pickle security. Untrusted serialized files can be dangerous.
- Skipping the single-image versus many-image test. A method that reads one image quickly may not read thousands efficiently.

Comments (0)