An uncompressed NDJSON export is a lot of JSON. Bulk exports for a mid-size health plan run in the tens of gigabytes uncompressed. Compression reduces both delivery bandwidth and storage, and the choice of compression algorithm matters more than most teams give it credit for. Between gzip, zstd, and xz there is a real speed-vs-ratio trade-off. The site's NDJSON export peeker transparently handles gzip; the choice for delivery is a server-side decision. For the wider FHIR framing, FHIR provider data exchange guides have more.
Why NDJSON Compresses Well
- Repeating keys — every Patient has
resourceType,name,birthDate - Repeating values — same code system URLs, same profile URLs
- Repeating structure — every line has a similar JSON shape
- Long strings — narrative text, base64 attachments
Every one of those is what dictionary-based compressors exploit. Typical ratios: 5-10× on NDJSON.
Gzip
- Universal support — every language, every tool, every browser
- Reasonable ratio — 5-8× on NDJSON
- Fast decompression, moderate compression speed
- Streaming-friendly
The default for bulk data delivery. If you cannot pick, pick gzip.
Zstd
- Better ratio than gzip — 6-10× on NDJSON
- Much faster decompression than gzip
- Faster compression at equivalent ratios
- Streaming-friendly
- Growing but not universal support
The right choice for high-throughput pipelines. Modern languages and modern proxies support it. Older infrastructure may not.
Xz
- Best ratio — 8-12× on NDJSON
- Slower decompression than gzip
- Much slower compression
- Streaming-friendly with quirks
- Common on Unix, less so elsewhere
The right choice for archival or one-time deliveries where storage cost dominates. Wrong choice for interactive pipelines.
For the streaming ingestion side, streaming an NDJSON file into a database is the entry.
Snappy
- Fast compression and decompression
- Modest ratio — 3-5× on NDJSON
- Streaming-friendly
- Common in big-data ecosystems (Parquet, Hadoop)
Rarely the right choice for FHIR delivery. Included here because you may encounter it in downstream pipelines.
The HTTP Content-Encoding Header
Servers should:
- Set
Content-Encoding: gzip(orzstd) - Accept
Accept-Encodingfrom clients - Fall back to uncompressed if the client cannot handle the chosen encoding
Clients should:
- Send
Accept-Encoding: gzip, zstd - Handle both
- Decompress transparently
Most well-designed clients do all of this automatically.
Compression And Range Requests
Compressed files complicate HTTP range requests — you can request bytes 1000-2000 of the compressed stream but the decompressed content depends on preceding bytes.
If your pipeline needs to resume partial downloads, either compress in chunks (multi-part gzip) or accept the resume-from-start cost. Streaming servers usually just re-serve from the start.
Streaming Decompression
Never decompress the whole file to disk before reading. Chain the decompressor into the read stream:
- File → decompressor → line-splitter → parser → pipeline
Memory stays bounded. The overall throughput matches the decompressor's speed.
For the format basics, the NDJSON format and why bulk data uses it is the entry.
Compression Ratios In Practice
- Patient.ndjson — 6-7× with gzip, 7-9× with zstd
- Observation.ndjson — 8-10× with gzip (repetitive), 10-12× with zstd
- Attachments-heavy files — 2-3× (base64 does not compress well)
Test on your own data. The above are typical, not universal.
Cost Trade-Offs
- Storage-heavy workflow — pick the best ratio (xz or zstd)
- Bandwidth-heavy workflow — pick the best ratio
- Latency-heavy workflow — pick the fastest decompression (zstd)
- Compatibility-heavy workflow — pick gzip
For the monitoring implications, monitoring the ingestion of an NDJSON batch covers throughput measurement.
The Short Version
Gzip is the safe default. Zstd is better on ratio and decompression speed if your infrastructure supports it. Xz is for archival. Streaming-decompress into your ingestion pipeline. Test ratios on your own data.
Sources
- HL7 canonical Bulk Data Access IG covering delivery formats - HL7 canonical Bulk Data Access IG covering delivery formats