Handling Mixed-Resource NDJSON in a $export

FHIR $export usually delivers one NDJSON file per resource type — Patient.ndjson, Observation.ndjson, and so on. Sometimes the spec-compliant output mixes resource types in a single file. Handling that case is a specific dispatch pattern most pipelines miss on the first pass. The site's NDJSON export peeker detects mixed-type files and reports the distribution. For the wider FHIR framing, the 5-year history reference has more.

When It Happens

  • $export with resource-type-specific request produces per-type files
  • $export without type filter may produce mixed files depending on server
  • Downstream tools may produce mixed files by concatenating
  • Historical exports from older servers may not separate types

Every one of these produces a single NDJSON where reading requires dispatching per line.

The Dispatch Pattern

Every line has a resourceType field. Read the line, parse the field, dispatch to the type-specific handler:

  • Patient → the Patient ingestion path
  • Observation → the Observation path
  • Condition → the Condition path
  • Unknown type → log and skip or fail

That is the base pattern. Every serious ingester supports it whether the input is per-type or mixed.

For the format side, the NDJSON format and why bulk data uses it is the entry.

Detecting Mixed-Type At Read

The peeker's first-few-lines check:

  • Sample the first N lines
  • Tally resourceType values
  • Report the distribution

If the distribution shows one type, treat as per-type. If it shows several, treat as mixed.

For the sampling approach, sampling an NDJSON to sanity-check the export is the entry.

Ingest Per Type Or Ingest Together

  • Per-type — split the mixed file into per-type files first, then ingest each
  • Together — stream the mixed file and dispatch each line to the type handler

Per-type is simpler to reason about but requires an extra pass. Together is more efficient but requires the dispatcher to be robust.

Most modern pipelines dispatch together. The split-first pattern is easier to write but doubles the I/O.

Reference Resolution With Mixed Files

Mixed files often carry references between resources in the same file — Patient referenced by Observation, both in one file. That is exactly the pattern urn:uuid: in Bundle solves for transactions.

For NDJSON, references usually use real ids or Reference.identifier. The dispatcher builds an id index as it goes and resolves within-file references at end-of-file.

For the streaming-ingest side, streaming an NDJSON file into a database is the entry.

Order Sensitivity

Some mixed exports emit resources in dependency order — Patient first, then Observations referencing that Patient. The dispatcher can assume forward references work.

Other exports emit in arbitrary order. The dispatcher has to handle out-of-order references — process Observations that reference Patients not yet seen, resolve later.

Do not assume order. Design for arbitrary.

Reporting

The pipeline should report:

  • Total lines
  • Distribution by resource type
  • Unknown types (with resourceType values encountered)
  • Per-type success/failure counts

That summary is useful for the operator and for downstream validation. For the monitoring side, monitoring the ingestion of an NDJSON batch is the entry.

Filename Conventions

Some producers name mixed files with a resource-type suffix (misleading if the file is mixed). Others use generic names (bulk.ndjson).

Never trust the filename. Trust the content. Sample first, then process.

The Short Version

Dispatch per line by resourceType. Handle both per-type and mixed files with the same code path. Build an id index for within-file references. Do not assume dependency order. Report distribution. Never trust the filename.

Metal-engraving diagram of a mixed-resource NDJSON stream being dispatched per line to type-specific handlers with an id index growing alongside, drawn as detailed engraved shading with rich-blue accents on ivory

Sources

Share: Facebook Twitter Linkedin

Comments are closed.