Corruption in an NDJSON stream is rarely announced. The file downloads cleanly, the checksum matches, and the pipeline chugs happily until row 4.2 million turns out to be malformed and everything past it silently misinterprets. Building corruption detection into the read path catches these failures early. The site's NDJSON export peeker flags the common corruption signatures on paste-in. For the wider FHIR framing, more on FHIR attribution patterns has more.

The Corruption Signatures

• Line does not parse as JSON — partial write or encoding issue
• Line is empty — extra newline
• resourceType missing or wrong — schema drift
• Duplicate resource id within one file — writer bug or race
• Character encoding not UTF-8 — producer misconfiguration
• Line-ending mixing (LF vs CRLF) — cross-platform generation

Each has a different cause and a different remediation.

Fail-Open Or Fail-Close

• Fail-close — one corrupt line stops the whole ingestion; alert; wait for investigation
• Fail-open — log the corrupt line, count it, continue

Fail-close for regulated workflows where every row matters. Fail-open for volume workflows where the aggregate is what counts.

Neither is universally right. The peeker defaults to fail-open with clear counts.

Per-Line Independence Helps

Because each NDJSON line is independent, corruption is bounded — a bad line does not corrupt neighbors. The pipeline can identify the specific bad line without discarding the whole file.

For the format basics, the NDJSON format and why bulk data uses it is the entry.

Detecting Duplicates

Duplicate resource ids within a single NDJSON file are a real corruption category. Producers sometimes write the same resource twice due to a retry or race condition.

Detection: as you stream, keep a set of seen (resourceType, id) tuples. Add each new resource. Alert on any duplicate.

Memory cost: bounded by unique-id count, which is usually much smaller than line count.

Detecting Truncation

A file may have been cut off mid-write. Signatures:

• Last line is a partial JSON value
• File does not end with a newline (some producers require it)
• HTTP download completed but file size does not match Content-Length

The pipeline should verify the last line parses. If not, the file is truncated.

For the sanity-check side, sampling an NDJSON to sanity-check the export is the entry.

Character Encoding Verification

• Read as UTF-8
• Reject non-ASCII bytes that do not form valid UTF-8 sequences
• Reject byte-order marks (BOMs) at the file start

Producers should emit clean UTF-8. Consumers should verify.

Line-Ending Mismatch

• Standard: \n only
• Occasionally: \r\n (Windows-generated)
• Rare: \r alone (very old Mac)

The pipeline should be tolerant of the first two and reject the third with a clear error.

Structural Corruption

Even a line that parses as JSON may be structurally wrong:

• Missing resourceType
• resourceType does not match the file name
• Fields present that do not belong on this resource type
• Value types that violate the schema

Structural corruption is caught by FHIR validation, not by JSON parsing. Consider running structural validation on a sample before ingesting the whole file.

Reporting

Every ingestion should emit a corruption summary:

• Total lines
• Successful parses
• Malformed lines (with line numbers)
• Structural failures (with sample paths)
• Duplicates detected

Downstream reviewers use the summary. Silent pipelines drop corruption reports; noisy pipelines make them actionable. For the monitoring side, monitoring the ingestion of an NDJSON batch is the entry.

The Short Version

Detect malformed lines, empty lines, wrong resourceType, duplicate ids, encoding issues, truncation. Decide fail-open vs fail-close per workload. Emit a corruption summary. Every serious pipeline does this — silent processing is silently wrong.

Sources

• HL7 canonical Bulk Data Access IG - HL7 canonical Bulk Data Access IG