nbdkit-indexed-gzip-filter - access .gz contents efficiently
nbdkit --filter=indexed-gzip PLUGIN gzip-index-path=PATH
[gzip-index-span=SIZE]nbdkit-indexed-gzip-filter is a filter for nbdkit that provides fast and efficient random-access to gzip-compressed data.
The standard nbdkit-gzip-filter(1) decompresses the entire archive to a temporary file, which means you need to have enough storage or memory to store the complete uncompressed input.
With this filter, the entire archive still needs to be decompressed once, but only an index file needs to be stored in storage or memory, which (depending on user-specified distance between index points) may be 0.3% to 3% of the size of the uncompressed archive.
Path to the complete gzip index file (created if it doesn't exist, reused if it does).
This parameter is required.
Number of bytes between index points. Eg. 1M, 10M etc. (default: 1MB)
A span of 10M produces an index file of eg, ~0.3% of uncompressed input
A span of 1M produces an index file of eg, ~3% of uncompressed input.
Smaller span improves random-access performance, since on average it means fewer bytes to decompress until reaching the requested byte (with the trade-off of a larger index file).
The filter.
Use nbdkit --dump-config to find the location of $filterdir.
nbdkit-indexed-gzip-filter first appeared in nbdkit 1.46.
nbdkit(1), nbdkit-filter(3), nbdkit-gzip-filter(1)
Shasheen Ediriweera
Mark Adler
Copyright (C) 2025 Shasheen Ediriweera
Copyright (C) 2005, 2012, 2018, 2023, 2024 Mark Adler
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
Neither the name of Red Hat nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.