Key concepts

DRRDRR Digital Regulatory Reporting. An industry‑developed, machine‑executable interpretation of regulatory rules that produces consistent, transparent and fully traceable reporting outputs from standardised CDM data. supports 31 report types across ten regulators, and holds 211 FpMLFpML Financial Products Markup Language sample trades to exercise them against. Every supported regulation has to run against the right samples, and the expected output of every one of those runs has to stay current as the model changes.

That is the problem the test pack machinery exists to solve. This page explains the pieces and how they fit together. For the practical steps when adding a regime, see Adding a new jurisdiction.

The four building blocks

A sample is one FpMLFpML Financial Products Markup Language record-keeping XMLXML Extensible Markup Language. Text-based format used to store and transport data in a structured way that both humans and machines can read. document — a single trade or lifecycle event. Samples live under rosetta-source/src/main/resources/cdm-sample-files/.

A test pack is a named group of samples, organised by asset class or scenario. There are twelve, and the directory names under cdm-sample-files/ are the test pack IDs: rates, credit, commodity, equity, fx, etd, events, exotic, custom-scenarios, delegated-reporting, cftc-event-scenarios and pre-enrich.

Test packs exist because reporting rules differ by asset class. A fixed-float swap and a commodity option are reported differently under the same regulation, so a regulation is only meaningfully tested against a spread of products.

A pipeline is one transform: a function, the type it consumes, and the type it produces. Pipelines chain together — each names the pipeline that feeds it, so the whole run from raw FpMLFpML Financial Products Markup Language to a submitted report is a chain of them:

{
  "id" : "pipeline-report-...-esma-mifir-rt-s22",
  "name" : "ESMA / MIFIR RTS22",
  "transform" : {
    "type" : "REPORT",
    "function" : "drr.regulation.esma.mifir.rewrite.trade.reports.ESMAMIFIRRTS22ReportFunction",
    "inputType" : "drr.regulation.common.TransactionReportInstruction",
    "outputType" : "drr.regulation.esma.mifir.rewrite.trade.MIFIRTransactionReportRTS22"
  },
  "upstreamPipelineId" : "pipeline-enrich-...-test-pack-default"
}

A transform type says which stage of the chain a pipeline belongs to. There are four:

Transform type	Turns	Into
TRANSLATE	FpMLFpML Financial Products Markup Language XMLXML Extensible Markup Language. Text-based format used to store and transport data in a structured way that both humans and machines can read.	ReportableEvent (CDMCDM Common Domain Model. A standardised, machine-readable and machine-executable blueprint for how financial products are traded and managed across the transaction lifecycle. It is represented as a domain model and distributed in open source.)
ENRICH	ReportableEvent	TransactionReportInstruction
REPORT	TransactionReportInstruction	a regulator's transaction report
PROJECTION	a transaction report	a submission format (ISO 20022 XMLXML Extensible Markup Language. Text-based format used to store and transport data in a structured way that both humans and machines can read., DTCC, CSVCSV Comma-Separated Values. Simple file format used to store tabular data (like spreadsheets or databases) in plain text.)

What links samples to regulations

A pipeline says how to transform. A test pack says which samples to run through it. The join between them is a test pack config — one file naming a pipeline and listing the samples to feed it:

{
  "id" : "test-pack-report-...-esma-mifir-rt-s22-rates",
  "pipelineId" : "pipeline-report-...-esma-mifir-rt-s22",
  "name" : "Rates",
  "samples" : [ {
    "id" : "ir-capfloor-ex01-cap-premium",
    "name" : "IR CapFloor ex01 Cap Premium",
    "inputPath" : "enrich/output/.../rates/IR-CapFloor-ex01-Cap-Premium.json",
    "outputPath" : "regulatory-reporting/output/.../esma-mifir-rt-s22/rates/IR-CapFloor-ex01-Cap-Premium.json",
    "assertions" : {
      "modelValidationFailures" : 54,
      "runtimeError" : false
    }
  } ]
}

The relationship is many-to-many, which is the reason the join is a file of its own. One pipeline runs against several test packs; one test pack feeds many pipelines. ESMAESMA European Securities and Markets Authority – EU‑level regulator responsible for securities markets, including EMIR reporting. MiFIRMiFIR Markets in Financial Instruments Regulation – alongside MiFID II, it forms the EU’s core rulebook for how financial markets operate. RTS 22, for instance, is a single report pipeline joined to ten test packs covering 186 samples, while the rates test pack feeds every regulation that reports rates products.

Hence the file counts on master: 46 report pipelines but 145 report test pack configs, and 44 projection pipelines against 133 projection test pack configs.

Each sample entry also carries its expected output. outputPath points at a committed JSONJSON JavaScript Object Notation. Text-based, language-independent format with key-value pairs (eg Name: Dave). file holding the report this sample produced, and assertions records counts alongside it — how many model validation failures that output had, and whether the run threw. Both the outputs and the counts are committed, so a model change that alters a report shows up as a reviewable diff rather than as a silent behaviour change.

Where DrrTestPackCreator fits

DrrTestPackCreator is a Java utility in the tests module. Despite living among the tests it is not itself a test — it asserts nothing. It is the thing that writes everything described above.

The distinction that matters: the Java is the source of truth, and the JSONJSON JavaScript Object Notation. Text-based, language-independent format with key-value pairs (eg Name: Dave). is generated output. The configs and expected outputs under rosetta-source/src/main/resources/ are committed to the repository, but nobody edits them by hand. To change which samples a regulation runs against, you edit DrrTestPackCreator.generatePipelines() and regenerate.

Running it does three things: deletes the generated directories, rebuilds every pipeline and test pack config from the Java, and re-runs every sample through every pipeline to refresh the expected outputs and assertion counts.

mvn -pl tests -am clean install -Pupdate-expectations -DskipTests

It runs regularly, not just when a regime is added — any model change that alters report output needs the expectations regenerated so the diff is visible in review.

Two copies of the samples

The samples exist twice, and both copies are committed:

• cdm-sample-files/ — input to the synonym-based ingestion.
• ingest/input/ — input to the function-based ingestion, which is the Translate 2.0 path the pipelines use.

The two trees hold the same 211 XMLXML Extensible Markup Language. Text-based format used to store and transport data in a structured way that both humans and machines can read. files and are expected to be byte-identical. SynonymToFunctionIngestSampleTest enforces that: it walks both trees and asserts every sample is present in each and that their contents match.

This catches you out when editing samples, because the tools do not treat the two trees alike. TestPackModifierMain — the utility that stamps jurisdiction data into every sample — is pointed at cdm-sample-files only, so changes it makes have to be mirrored into ingest/input to keep that test passing.

Where things live

Path (under rosetta-source/src/main/resources/)	Contents
cdm-sample-files/<test-pack>/	FpMLFpML Financial Products Markup Language samples, synonym ingest input
ingest/input/<test-pack>/	The same samples, function ingest input
ingest/config/, ingest/output/	Translate pipeline and its outputs
enrich/config/, enrich/output/	Enrich pipelines and their outputs
regulatory-reporting/config/, .../output/	Report pipelines, test packs and expected reports
projection/config/, projection/output/	Projection pipelines, test packs and expected submissions

Everything except the two sample trees is generated by DrrTestPackCreator.