Repository navigation
Add support for high bit depth multichannel images #1888
Description
Pillow (and PIL) is currently able to open 8 bit per channel multi-channel images (such as RGB) but is able to open higher bit depth images (e.g. I16, I32, or Float32 images) if they are single channel (e.g., grayscale).
Previous References
This has been requested many times: #1828, #1885, #1839, #1602, and farther back.
Requirements
- We should be able to support common GIS formats as well as high bit depth RGB(A) images.
- At least 4 channels, but potentially more (see Add tests for opening 2-5 layer uint16 greyscale TIFFs #1839)
- Different pixel formats, including I16, I32, and Float.
- There should be definitions for the array interface to exchange images with numpy/scipy
- There should be enough support to read and write TIFFs and raw image data.
- Support for resize, crop, and convert operations at the very least.
Background Reference Info
The rough sequence for image loading is:
-
Image file is opened
-
Each of the ImagePlugin _accept functions have a chance to look at the first few bytes to determine if they should attempt to open the file
-
The
*ImagePlugin._openmethod is called giving the image plugin a chance to read more of the image and determine if it still wants to consider it a valid image of it's particular type. If it does, it passes back a tile definition which includes a decoder and an image size. -
If there is a successful _open call, at some point later
*ImagePlugin._loadmay be called on the image, which runs the decoder producing a set of bytes in a raw mode. This is where things like compression are handled, but the output of the decoder is not necessarily what we're storing in our internal structures. -
The image is unpacked (
Unpack.c) from the raw mode (e.g.I16;BS) into a storage (Storage.c) mode (I). -
It's now possible to operate on the image (e.g. crop, pixel access, etc)
There are 3 (or 4) image data pointers, as defined in Imaging.h:
struct ImagingMemoryInstance {
/* Format */
char mode[IMAGING_MODE_LENGTH]; /* Band names ("1", "L", "P", "RGB", "RGBA", "CMYK", "YCbCr", "BGR;xy") */
int type; /* Data type (IMAGING_TYPE_*) */
int depth; /* Depth (ignored in this version) */
int bands; /* Number of bands (1, 2, 3, or 4) */
int xsize; /* Image dimension. */
int ysize;
/* Colour palette (for "P" images only) */
ImagingPalette palette;
/* Data pointers */
UINT8 **image8; /* Set for 8-bit images (pixelsize=1). */
INT32 **image32; /* Set for 32-bit images (pixelsize=4). */
/* Internals */
char **image; /* Actual raster data. */
char *block; /* Set if data is allocated in a single block. */
int pixelsize; /* Size of a pixel, in bytes (1, 2 or 4) */
int linesize; /* Size of a line, in bytes (xsize * pixelsize) */
/* Virtual methods */
void (*destroy)(Imaging im);
};
The only one that is guaranteed to be set is **image, which is an array of pointers to row data.
Changes Required
- Definitions for all of the modes that we're planning, and potentially a [format];MB[#bands] style generic mode.
Core Imaging Structure
- The imaging structure has the fields required to add the additional channels. (type, bands, pixelsize, linesize)
- The
**imagepointer can be used for any width of pixel. - We may or may not want to set the
**image32pointer. - Currently type of
IMAGING_TYPE_INT32andIMAGING_TYPE_FLOAT32imply 1 band. This will change. - Consider promoting int16 to
IMAGING_TYPE_INT16
Storage
- Updates to
Storage.c,Unpack.c,Pack.c,Access.c,PyAccess.py, andConvert.c
Ways to Help
We need a better definition of the format requirements. What are the various types of images that are used in GIS, Medical, or other fields that we'd want to interpret? We need small, redistributable versions of images that we can test against.
[in progress]
Activity
terramars commentedon May 23, 2016
I'm having the same problem with 16 bit single-channel paletted TIFFs, created by GDAL. It would be "really" nice if Pillow could play nicely with GIS and scientific image formats, as GDAL is a pain in the ass and I'd rather not use it.
tiffinfo as follows:
TIFFReadDirectory: Warning, Unknown field with tag 33550 (0x830e) encountered.
TIFFReadDirectory: Warning, Unknown field with tag 33922 (0x8482) encountered.
TIFFReadDirectory: Warning, Unknown field with tag 34735 (0x87af) encountered.
TIFFReadDirectory: Warning, Unknown field with tag 34737 (0x87b1) encountered.
TIFFReadDirectory: Warning, Unknown field with tag 42113 (0xa481) encountered.
TIFF Directory at offset 0x34293c6 (54694854)
Image Width: 10774 Image Length: 12577
Bits/Sample: 16
Sample Format: unsigned integer
Compression Scheme: LZW
Photometric Interpretation: palette color (RGB from colormap)
Samples/Pixel: 1
Rows/Strip: 1
Planar Configuration: single image plane
Color Map: (present)
Tag 33550: 4.999617,4.999789,0.000000
Tag 33922: 0.000000,0.000000,0.000000,679006.067110,9955209.915048,0.000000
Tag 34735: 1,1,0,7,1024,0,1,1,1025,0,1,1,1026,34737,22,0,2049,34737,7,22,2054,0,1,9102,3072,0,1,32736,3076,0,1,9001
Tag 34737: WGS 84 / UTM zone 36S|WGS 84|
Tag 42113: 0
Predictor: horizontal differencing 2 (0x2)
bodokaiser commentedon Mar 14, 2017
Any updates on this?
wiredfool commentedon Mar 14, 2017
Unfortunately, no.
vfdev-5 commentedon Feb 20, 2018
@wiredfool what do you think about to add the support of multichannel images as sequence of Image ? For example, 4 channels image with uint16 is represented (more less equivalently) by
['<PIL.Image.Image image mode=I;16 size=... >', '<PIL.Image.Image image mode=I;16 size=...>', ..., '<PIL.Image.Image image mode=I;16 size=...>']. I mean by that, maybe, to provide a class inheriting fromImageandtupleand override all method to work on a tuple of images... Sure that it looks like a hack, however it could unlock more features (and create issues :) ) at least while working withImage.fromarray.wiredfool commentedon Feb 21, 2018
To do anything useful with it, we'd have to have support in the C layer, so it would have to be at the core imaging layer, and especially Unpack/Pack.
vfdev-5 commentedon Feb 21, 2018
@wiredfool following your "Ways to help",
For GIS, as there is a huge amount of different formats (for example, gdal format list), this can be left for GIS libraries as gdal, rasterio etc.
However, a support of
Image.fromarrayon input multi-channel (3,4,5,...) arrays of dtypenp.uint16, np.float32would be, imho, essential.For GIS imagery, this can be easily created manually with gdal, rasterio.
I would like to give a hand on this, so, feel free to ask me.
edowson commentedon Jun 7, 2018
PIL cannot handle processing multi-channel images. They get truncated to 3-ch images if you perform any transformation using PIL. #3160
akinuri commentedon Jun 8, 2018
Related: How to open a TIF (CMYK, 16-bit) image file?
bthorsted commentedon Feb 15, 2019
What is the status of this issue? It has been almost three years since the first proposal. I am unfortunately unable to provide any help since I have zero experience with coding in C, but I am among the people that is awaiting support for e.g. multi-channel floating-point images (with possibilities for negative pixel values). This especially useful in deep learning, where it is preferable to have all values normalized with zero mean. PIL has some really awesome ImageOps, which is one of the reasons for wanting this support.
hugovk commentedon Feb 17, 2019
@bjtho08 No updates.
#2485 links to a multipage RGB TIFF containing
float64values.omaghsoudi commentedon Jul 5, 2019
Please fix the issue with multi-channel 16 bit images.
Thank you!
24 remaining items
aclark4life commentedon May 29, 2024
Can anyone suggest some test data we can use to develop this feature? This event is happening tomorrow and would be nice to have a success target in mind e.g. "If we can read/write this type of data …" https://www.meetup.com/dcpython/events/301086016/
rbavery commentedon May 30, 2024
In case it isn't too much pain to work with more than 4 bands, we host this example subset of Eurosat, here is an example image
s3://wherobots-examples/data/eurosat_small/Highway/Highway_1.tif.Each image is 13 bands, uint16, planar
aclark4life commentedon May 31, 2024
@wiredfool If we use Arrow that implies adding a dependency on pyarrow, ideally optionally via extras like
pip install pillow[arrow], correct?wiredfool commentedon Jun 1, 2024
@aclark4life Maybe. There's definitely a C-only implementation (nanoarrow) that might be what we want, since all of our image allocations are in the C layer now. PyArrow might be easier for integration/interop at the high level, but my sense here is that it wouldn't necessarily be giving us a whole lot that we'd not already have with a C arrow implementation + our usual set of accessors.
aclark4life commentedon Jun 20, 2024
Folks interested in this issue, please test #8224 and give feedback, thanks all
MyleneSimon commentedon Dec 2, 2024
Hi, any updates on a potential merge of #8224? If I understand properly it would add support for reading pyramidal tiffs?
Our use case is that we deal with scientific images that are stored as pyramidal tiled tiff, and a lot of ML libraries and some image annotation platforms such as CVAT use Pillow as a reader or expect Pillow images as an input, and so we find ourselves having to "hack" some of the pipelines/code bases to support our images. Pyramidal tiff support would be really helpful.
aclark4life commentedon Dec 2, 2024
@MyleneSimon No, but I haven't given up! May need to move ETA for more progress to Q1 2025. What we really need is some code reviews and opinions about which direction to go in… thanks for the interest.
bigcat88 commentedon Dec 13, 2024
Can the community somehow help with the work on this feature?
It is possible to divide what is possible into minimal parts and start implementing and testing them, at least those that can be divided, if any.
wiredfool commentedon Jan 24, 2025
@MyleneSimon I don't think that the #8224 PR would add support for pyramids, that's an orthogonal issue to the high bit depth problem. It may be possible to read individual images in a pyramid now (assuming that they're a format compatible with our current platform), but I highly doubt that there's a way to save them. It would depend on being able to select the subimage, which may or may not be possible.
wiredfool commentedon Jan 24, 2025
From some further thoughts along the lines of #1888 (comment) and the work that I've been doing with the Arrow support. I'm pretty sure that the best/easiest way forward is to use planar image storage for complicated images.
(There's also tiled images, which are very useful for large tiffs, where each nxn block is stored in a chunk. Ideally what we're doing here would support extensions in that direction, but it's not a primary goal)
Planar image storage would get us:
The complexity would be any place where we have multi-channel ops that can't be decomposed into the same op on all channels independently. Shuffle, Un/pack, Luminance/Color Space Conversion are the ones that jump out at me.
As a general guide to how I think this could be done, I see three phases:
Python level handling for assembling planar images out of arbitrary single channel core images.
__arrow_c_array__for existing planar arrays corresponding to channels.Support for shuffle/unpack for appropriate file load/save
Support for multichannel ops,
I'm breaking this out this way because I think 2 and 3 will be incremental, multi part things, but part 1 can be done in one chunk and be complete enough to be able to test on its own, where part 2 requires the basic infra to already be available.
aclark4life commentedon Oct 12, 2026
OK ~1.5 years later, I've opened a draft design proposal in #10189 based on @wiredfool's planar storage approach above (#1888 (comment)). Each band would be stored as an existing single-band image (L, I;16, I, F) along with metadata describing it. Once we agree on the design, the work can be done in three phases:
Feedback on the open questions is welcome on PR #10189!