Streaming an NDJSON file into a database is the ingestion path for every real bulk-data pipeline. The naïve "load the file, insert row by row" approach hits memory ceilings on large exports. The naïve "insert every row in its own transaction" approach hits throughput ceilings. Getting both right is a small pile of ingestion patterns that every serious bulk-data engineer learns once. The site's NDJSON export peeker is the pre-flight for humans; the ingestion is where the automation lives. For the wider FHIR framing, deeper Bulk Data engineering coverage has more.

The Ingestion Pipeline

• Open the NDJSON as a byte stream
• Decompress if gzip
• Read line by line
• Parse each line to a JSON value
• Validate against the expected resource type
• Buffer rows in batches
• Bulk-insert each batch
• Commit per batch

Each stage is bounded in memory. The overall pipeline processes files of arbitrary size without stalling.

Batch Size Matters

• Too small (1 row per transaction) — throughput drops from network round-trips
• Too large (100000 rows) — memory pressure, long-running transactions, retry cost

Reasonable defaults: 500-5000 rows per batch. Tune per database and per resource type.

For the read-side mechanic, reading an NDJSON export without loading the whole file is the entry.

Validation In The Pipeline

Every incoming line is a candidate corrupt row. Validate before insertion:

• Parseable JSON
• Has resourceType field
• resourceType matches the expected type for this file
• Basic structural validity

Fail-open for validation: log the bad row, continue with the batch. Fail-close for critical paths: reject the batch and alert.

For the corruption side specifically, detecting corruption in a large NDJSON stream is the entry.

Idempotency

Bulk pipelines get re-run. The ingestion should be idempotent:

• Use INSERT ... ON CONFLICT DO UPDATE for upsert semantics
• Key on resourceType + id to detect duplicates
• Emit a "already loaded" count rather than failing on duplicates

That way a re-run does not double-insert.

Transactions And Failure Modes

• Batch fails mid-way — the batch rolls back, pipeline continues from the next batch
• Whole file fails — restart from where it stopped (idempotency helps)
• Downstream error after insert — separate retry logic

Design for restartability. Every serious pipeline has a "resume from row N" story.

Throughput vs Correctness

• Bulk insert is fast; validation is slow
• Validation before insert catches bad data early
• Validation after insert requires a cleanup pass

Pick per workload. Regulated workflows validate before. Volume-oriented workflows validate as a background pass.

Downstream Processing

Ingestion is one stage. Downstream stages consume the ingested rows:

• Index building
• Reference resolution
• Terminology validation
• Aggregation

Each is its own pipeline. Do not conflate them with ingestion — separation makes retry and monitoring cleaner. For the ingestion-monitoring side, monitoring the ingestion of an NDJSON batch is the entry.

Memory-Bounded Buffers

Every stage should have a bounded buffer:

• Read buffer — bounded by file chunk size
• Parse buffer — bounded by batch size
• Write buffer — bounded by DB batch size

Unbounded buffers produce OOM crashes on large files. Bounded buffers produce backpressure — the pipeline slows down when downstream is slow.

Language-Specific Tools

• Node — stream-json, readline, custom stream wiring
• Python — ijson, orjson, generator-based line reading
• Go — bufio.Scanner, encoding/json
• Java — Jackson Streaming API, buffered reader

All support bounded-memory streaming. Pick the language your infrastructure uses.

The Short Version

Read line by line. Validate before insert. Batch. Upsert for idempotency. Bounded buffers. Separate downstream processing. Design for restart. Monitor. That is the streaming-ingestion playbook. For the format itself, the NDJSON format and why bulk data uses it is the entry.

Sources

• HL7 canonical Bulk Data Access IG - HL7 canonical Bulk Data Access IG