A fresh $export lands in a bucket. Before running it through the ingestion pipeline, a five-second sanity check saves the hour of debugging you would otherwise do when the whole pipeline runs on a bad export. That check is a small NDJSON sample and a quick visual scan. The site's NDJSON export peeker is built for exactly this. For the wider FHIR framing, related provider-side ePA guides have more.
What A Sanity Check Verifies
- File is not empty
- First line parses as JSON
- resourceType matches the file name convention
- Structure looks like the expected profile
- Random spot-check confirms consistent shape
Five bullets. Two minutes. Prevents the hour of "why is the pipeline crashing" that comes from processing a corrupt export.
Sampling Strategies
- First N lines — cheap, biased toward file start
- Last N lines — cheap, biased toward file end
- Random N lines — unbiased, requires the file (harder on streams)
- Stratified sample — one from each 10% chunk
For most sanity checks, first + last is enough. Random is worth the extra work for compliance-relevant workflows.
For the read-side pattern, reading an NDJSON export without loading the whole file is the entry.
The Peeker Defaults
The peeker shows:
- Line count (from
wc -l) - Resource type tally (first N lines)
- Malformed-line report
- Pretty-printed first three resources
That is enough to catch most bad exports before the pipeline touches them.
What To Look For In The Sample
- resourceType matches the file (Patient.ndjson has Patient resources, not Observation)
- The resources have the fields you expect for your profile
- Extensions look populated
- References look reasonable (not all pointing at "unknown")
Each is a smoke check. Failure means the export is not what you asked for.
The First Line Is The Best First Look
If the first line parses cleanly and matches the expected type, the rest usually will too. A malformed first line means something upstream is wrong.
Never skip the first-line check. It is the cheapest signal you get.
Count Types Fast
For a multi-type NDJSON (rare but possible), tally resource types in the sample:
`` head -100 file.ndjson | jq -r '.resourceType' | sort | uniq -c ``
That gives you a distribution of types in the first 100 lines. Wildly off from expected means the export is not what you thought.
For the mixed-type case, handling mixed-resource NDJSON in a $export is the entry.
The Ratio Sanity Check
Compare the file's line count against expected volumes:
- Patient file → line count matches your expected patient count
- Observation file → line count matches expected observation count (usually much larger than patient count)
Wildly off — order of magnitude off — means the export failed silently. Line count is the cheapest signal for that.
Sample Corruption Detection
The sample also catches:
- Encoding issues (UTF-8 errors surface in the first non-ASCII line)
- Truncation (last line is a partial JSON)
- Duplicate rows (if the sample has clear duplicates, the whole file probably does)
For the deeper corruption detection, detecting corruption in a large NDJSON stream is the entry.
The Compression Trick
Sampling a compressed NDJSON:
`` zcat file.ndjson.gz | head -100 ``
Works for gzip. xzcat for xz. zstdcat for zstd. Every serious compressor has a -cat companion that streams to stdout.
Automation
Every ingestion pipeline should sanity-check the file before processing:
- Line count within expected range
- First line parses
- resourceType matches expectation
Alert on failure. Do not process. For the monitoring side, monitoring the ingestion of an NDJSON batch is the entry.
The Short Version
Head + tail + line count + resource-type tally + pretty-print. Two minutes at the human level. A few CI checks at the automation level. Prevents most bad exports from reaching the pipeline.
Sources
- HL7 canonical Bulk Data Access IG - HL7 canonical Bulk Data Access IG