Adding a source

A source adapter is the only part of this library that knows what a particular government publishes. It declares what the list is and where it lives, and it turns that publisher's bytes into ActiveSanction::Entity objects. Everything either side of that — conditional GET, the payload cache, checksumming the result into a Snapshot, storage, and eventually the matcher — is written against the canonical model and never against a list, so adding a jurisdiction is a parsing problem and not a plumbing one.

class UnConsolidated < ActiveSanction::Sources::Base
  key          :un_consolidated
  jurisdiction :un
  authority    "United Nations Security Council"
  format       :xml
  url          :main, "https://scsanctions.un.org/resources/xml/en/consolidated.xml"

  def parse(raw)
    ...   # => [Entity, ...]
  end
end

ActiveSanction::Sources.register(UnConsolidated)

#parse is the whole of what you must write. This document is the rest: how to decide what goes in it, and what the surrounding checklist is.

It applies whether the adapter lands in this gem or in your own application — the registry is open and duck-typed, and section 9 covers both.

Read the closest existing adapter first
Look at what the publisher actually serves
Declare the list
Choose how to read it
Map the publisher's fields onto the canonical model
Give every record a stable id
Report what you could not read
Capture a fixture and wire up the conformance spec
Register it
Typed, linted, and green
A worked example, end to end
Checklist

1. Read the closest existing adapter first

There is no scaffold generator, deliberately. This project will have roughly eight adapters at maturity, and a generator maintained for that many uses has to be kept in step with every change to Sources::Base, the conformance spec and the parser toolkits — and goes stale silently when it is not. An existing adapter to copy does the same job with none of the upkeep.

Pick the one whose shape matches your publisher's, not the one whose jurisdiction is nearest:

The publisher serves	Copy from	Its spec
One delimited file, nothing unusual	`spec/support/contract_example_source.rb` — a whole adapter in twenty lines	`spec/active_sanction/sources/conformance_spec.rb`
Several delimited files that only mean something joined	`lib/active_sanction/sources/ofac.rb`, `ofac_sdn.rb`	`spec/active_sanction/sources/ofac_sdn_spec.rb`
One XML document, one flat record shape	`lib/active_sanction/sources/canada_sema.rb`	`spec/active_sanction/sources/canada_sema_spec.rb`
One XML document, several record shapes	`lib/active_sanction/sources/un_consolidated.rb`	`spec/active_sanction/sources/un_consolidated_spec.rb`
One XML document, tens of megabytes, data in attributes	`lib/active_sanction/sources/eu_fsf.rb`	`spec/active_sanction/sources/eu_fsf_spec.rb`
One XML document, deeply nested containers, data in elements	`lib/active_sanction/sources/uk_sanctions_list.rb`	`spec/active_sanction/sources/uk_sanctions_list_spec.rb`
One spreadsheet, one row per name, records joined on a reference	`lib/active_sanction/sources/australia_dfat.rb`	`spec/active_sanction/sources/australia_dfat_spec.rb`
Two lists from one publisher in one format	`lib/active_sanction/sources/ofac.rb` holds the reading; `ofac_sdn.rb` and `ofac_consolidated.rb` declare only which list they are

Each of those files opens with a class comment describing the list, its quirks, and what the adapter refuses to do about them. That comment is part of the deliverable, not decoration: it is where the next person finds out why Canadian aliases are not split on commas without having to reconstruct the argument from the code.

Note the two-file convention in the XML adapters. canada_sema.rb says what the list is and drives the read; canada_sema/record.rb says what the publisher's elements mean. They are separated because they are two jobs, and all the judgment sits in the second one. A mapping small enough to read at a glance — the contract example, a five-column CSV — does not need the split.

2. Look at what the publisher actually serves

Before writing a line, get the bytes and answer eight questions about them. Every one of them changes the adapter, and every one of them is cheaper to answer now than after a fixture has been committed.

curl -sSL -o /tmp/list.xml "https://example.gov/sanctions/list.xml"
file /tmp/list.xml && wc -c /tmp/list.xml

What encoding is it, really? OFAC serves Windows-1252 and says nothing about it in a header; the UN serves UTF-8 and says nothing either. Read as the wrong one, thousands of accented names arrive as replacement characters. The adapter states the encoding; it does not guess.
What does the publisher write where it means nothing? OFAC writes -0- — with a trailing space — roughly a quarter of a million times. Blank and all-whitespace are already nil; a sentinel has to be declared.
Is there an identifier, and is it stable? See section 6. This is the question most likely to cost you a rewrite.
How many records, and of what kinds? Count them. The counts belong in the class comment, because a comment saying "736 individuals, 275 organizations" is checkable against the file and a comment saying "some people and some organizations" is not.
What separates repeated values? Semicolons are usually unambiguous. Commas usually are not — see the alias discussion in canada_sema.rb for what splitting on an ambiguous separator manufactures.
Which fields have no home in Entity? They go into remarks behind the [source fields] marker, never on the floor.
Does one element or column mean two different things? The UN's QUALITY grades an alias under <INDIVIDUAL_ALIAS> and names the alias kind under <ENTITY_ALIAS>. Canada's date element is a date of birth or a ship's build date depending on the record. These are the bugs that produce a list which parses cleanly and is wrong.
Does the document carry its own version marker? The UN stamps dateGenerated on the root element. If yours does, override #source_version to return it — it is the string an examiner asking "which version was this screened against" will recognise, and it is more precise than the Last-Modified header Base falls back to.

3. Declare the list

Declarations come from Sources::Definition, extended into Sources::Base, so an adapter's class body reads as a description of the list:

key          :ofac_sdn      # required, never inherited
jurisdiction :us            # required
authority    "U.S. Department of the Treasury, Office of Foreign Assets Control"
format       :csv           # optional, informational

url :sdn, "https://sanctionslistservice.ofac.treas.gov/api/download/SDN.CSV"
url :alt, "https://sanctionslistservice.ofac.treas.gov/api/download/ALT.CSV"
url :add, "https://sanctionslistservice.ofac.treas.gov/api/download/ADD.CSV"

key is the public name of the list: what config.sources names, what a stored snapshot is filed under, what the payload cache uses as a directory name, and what a match result cites. Lowercase snake_case, and it is never inherited — a subclass silently taking its parent's key would try to register under a name already taken.
authority is what a compliance report prints beside a hit, so spell it the way the body spells itself: "United Nations Security Council", not "UN".
format is a label a CLI prints. It is not dispatched on and is not checked against a list of known formats, so a source arriving as :fixed_width can say so.
url is declared once per file. Each is fetched, validated and cached independently, because they change independently: a sync in which only ALT.CSV moved should download only ALT.CSV. A source declaring several files has each filed under "#{key}-#{name}" in the cache; one declaring a single file is filed under the key itself.

Everything except key resolves up the superclass chain, so two lists from one publisher share a base that holds the jurisdiction, the authority and the format — that is exactly what Sources::Ofac is.

A source that is not fetched over HTTP at all — a bank's internal watchlist backed by a database table — declares no URL and overrides #retrieve instead. It does not have to subclass Base at all; see section 9.

What `#parse` is handed

Whatever the declaration implies, and never what a caller happened to pass:

one declared URL → #parse receives the bytes as a String;
several → it receives a Hash keyed by the names you declared, as in raw[:sdn].

That rule is applied by Base before #parse is called and again by the conformance spec, so an adapter's signature does not change when a fixture is handed to #snapshot directly:

source = ActiveSanction::Sources[:un_consolidated].new
source.snapshot(File.binread("spec/fixtures/un_consolidated/consolidated.xml"))
source.sync                       # the same path, with the network in front of it

The bytes arrive undecoded. An adapter reading the Windows-1252 OFAC serves is held to doing its own decoding rather than to being handed a String that somebody already fixed up — which is what the format toolkits do for you when you tell them the encoding.

4. Choose how to read it

There is no field-mapping DSL

A declarative field_map DSL was planned (#13) and deliberately not built. The intent was to make a new list a mapping rather than a hand-written parser, but every launch source has at least one quirk that needed a transform hook to absorb — -0- sentinels, bilingual values, one name spread across four elements, an id that has to be synthesized — and when the escape hatches carry most of the weight, the abstraction has not been found. Two real adapters later the shared part turned out to be the format layer, not the mapping layer.

So: #parse is yours to write, and the reuse lives in ActiveSanction::Parsers. Nothing in Parsers knows about Entity — they are file readers, and keeping them ignorant of the canonical model is what lets an adapter for a list nobody here has seen reuse them.

`Parsers::DelimitedTable` — CSV, TSV, anything delimited

LIST = ActiveSanction::Parsers::DelimitedTable.new(
  columns:  %i[ent_num sdn_name sdn_type program title remarks],
  null:     "-0-",
  encoding: Encoding::WINDOWS_1252
)

LIST.read(raw).each { |row| row[:sdn_name] }

A table is a description of the file, built once at class-definition time and reused for every sync; #read is one pass over one payload.

columns: names a headerless file's columns and pins its width. If the publisher inserts a column, every row arrives the wrong width and says so in #warnings — a far better failure than 19,321 entities quietly built from shifted fields. Pass columns: nil (the default) for a file that carries its own header; the names are read from the first row, lowercased and snake_cased, so City/State/Province/ZIP/Postal Code and city_state_province_zip_postal_code are the same column no matter how the publisher capitalized it this quarter.
null: takes one sentinel or several. It is matched after stripping surrounding whitespace, and blank fields are nil regardless.
col_sep: / quote_char: for TSV and the rest.
liberal_parsing: defaults to true, because an unescaped quote inside a company name is common in published data and failing the row is the wrong default. Turn it off where a stray quote should be loud.

A Row is deliberately not a Hash: row[:sdn_nme] raises rather than answering nil and letting a typo look like an empty column all the way into a snapshot. row.fetch(:col) is there for the genuinely optional case, row.null?(:col) asks whether the publisher left it empty, and row.line is the line number a warning has to cite.

`Parsers::XmlRecords` — record-oriented XML

LIST = ActiveSanction::Parsers::XmlRecords.new(records: %w[INDIVIDUAL ENTITY])

reader = LIST.read(raw)
reader.each do |record|
  record.name                          # => "INDIVIDUAL"
  record["FIRST_NAME"]                 # => "ERIC"
  record.values("NATIONALITY/VALUE")   # => ["Chad", "Sudan"]
  record.nodes("INDIVIDUAL_ALIAS")     # => [Record, ...]
  record["@dateGenerated"]             # an attribute, XPath-style
end
reader.root["dateGenerated"]           # the publisher's own version marker

It streams. One record's depth is held on a stack and dropped as soon as you are done with it. Every list here today is small enough to have loaded whole, which is exactly why it does not: the adapter written against this interface is the one that reads OFAC's 126 MB SDN_ADVANCED.XML later, unchanged.
Name only the record elements. Scaffolding — <CONSOLIDATED_LIST>, <INDIVIDUALS> — is skipped rather than built into nodes nobody asked for. Naming several is normal and gives you one pass over both shapes.
Namespace prefixes are stripped before matching, so a publisher adding an xmlns next quarter does not silently stop matching anything.
Missing, self-closing and whitespace-only elements all read as nil. That is the only reading that survives real documents: the UN files 294 placeholder aliases as <INDIVIDUAL_ALIAS><QUALITY/><ALIAS_NAME/></INDIVIDUAL_ALIAS> and means nothing at all by them. Use record.fetch(path) for a field you consider mandatory — it names the record and what it does carry when the path is missing — and record.present to explore an unfamiliar document from bin/console.

Do not pass backend:. Which XML library parses a list is an installation's decision (config.xml_backend), not a list's.

`Parsers::Spreadsheet` — an `.xlsx` workbook

LIST = ActiveSanction::Parsers::Spreadsheet.new(sheet: "Consolidated List")

reader = LIST.read(raw)
reader.each do |row|
  row[:name_of_individual_or_entity]   # => "MOHAMMAD HASSAN AKHUND"
  row[:date_of_birth]                  # => "1962-08-24"
  row.number                           # => 2, the sheet's own row number
end
reader.sheet_names                     # => ["Consolidated List"]
reader.modified                        # the publisher's own save time

Reach for this only when the publisher offers nothing else — Australia does not, which is why it exists. It reads the workbook with zlib and Parsers::XmlRecords, so it adds no dependency; an .xlsx is a ZIP of XML parts and the only thing missing was a ZIP header unpacker.

The first row is always the header and is never yielded as a record. Its cells name the columns by default, lowercased and snake_cased the way DelimitedTable normalizes a CSV header. Passing columns: renames them by position instead; the header is still consumed, because it is still a header.
sheet: takes a name, a zero-based index, or nil for the first sheet. Name it: a publisher adding a tab should be an error rather than a silent change of which sheet is read.
A cell arrives as a String, or nil. Shared strings are resolved, inline strings are read, a boolean reads TRUE/FALSE, and Excel's _x000D_ escapes are undone.
A date cell arrives as ISO 8601 at the precision its format displays — 2026-04-14, 1973-11, 1975 — which PartialDate::Parser reads directly. A number under any non-date format is passed through as the publisher wrote it, which is what keeps 1958 a year rather than becoming a day in 1905. Getting this right requires xl/styles.xml: the two cells are otherwise identical.
A row states only the cells it filled, so cells are placed by their address (r="C7") and never by position. A value past the last named column is dropped and warned about; a row with nothing in it is spacing and is skipped.
Formulas are not evaluated. A formula cell reads as the value cached in it, which is what the file displays and what an export contains. Merged cells, comments, charts and styling are ignored, and the older binary .xls is a different format that is not read.

A Row behaves like DelimitedTable::Row — row[:typo] raises, row.fetch is the optional case, row.null? asks whether the cell was empty — except that it reports row.number, the spreadsheet's own row number, rather than a line.

`Parsers::Join` — several files, one logical record

join = ActiveSanction::Parsers::Join.new(
  on: :ent_num, aliases: ALT.read(raw[:alt]), addresses: ADD.read(raw[:add])
)

join.each(PRIMARY.read(raw[:sdn])) do |row, related|
  related[:aliases]     # => [Row, ...] -- always an Array, never nil
  related[:addresses]   # => [Row, ...]
end

join.orphans            # => { aliases: 0, addresses: 0 }

The primary file streams; the child files are indexed and held in memory for the length of the join. A child row matching no primary row is dropped and counted, and a nonzero orphan count after a sync usually means the files were downloaded at different moments and do not describe the same version of the list — a data problem no amount of careful parsing fixes.

Neither

If the publisher serves JSON, a fixed-width file, or something with no shape worth abstracting, parse it however it needs parsing and build Entities. The toolkits are a convenience, not an obligation, and the conformance spec does not know or care which you used. What it does care about is the rules in sections 5 to 7, which are not optional.

5. Map the publisher's fields onto the canonical model

Entity is the single source-agnostic record every adapter produces. Nothing downstream — storage, the diff between two syncs, the index, the matcher, the report an examiner reads — should ever need to know which government published a record.

ActiveSanction::Entity.new(
  source:         :ofac_sdn,          # required -- your own key
  source_ref:     "2674",             # the publisher's own reference, if it has one
  id:             nil,                # derived from source + source_ref when omitted
  type:           :individual,        # individual | organization | vessel | aircraft
  names:          [Name, ...],
  addresses:      [Address, ...],
  identifiers:    [Identifier, ...],
  dates_of_birth: [PartialDate, ...],
  nationalities:  ["EG"],
  programs:       ["SDGT"],
  listed_on:      PartialDate,
  remarks:        "..."
)

Instances are frozen and compare by value. The four collection members are declared types, so srb tc refuses an adapter that hands over the string a publisher wrote where a PartialDate belongs.

The rules that are not negotiable

Every date is a PartialDate, never the string it was written as. A date left as the publisher's string cannot be compared with one from another list, and PartialDate exists because no two lists agree on how precise a date of birth is. PartialDate.parse reads ISO (1972, 1972-04, 1972-04-29), worded forms (29 Apr 1972, April 29, 1972, Apr 1972), approximations (circa 1962, ~1962) and spans (between 1971 and 1973, 1971-1973), and returns nil on anything it cannot read. A publisher writing 14/03/2019 is outside that vocabulary and the adapter converts before parsing; it never stores the string.

Types are the matcher's coarsest filter. vessel and aircraft are first-class because they are about 10% of the OFAC SDN list and carry name-like strings; without a distinct type, a search for a person can rank a ship. Map the publisher's vocabulary onto the four, warn about anything you did not recognise, and default to :organization rather than dropping the record.

A record with no name is not a record. It cannot be screened against. Return nil for it and record a warning naming the publisher's reference, so the day one appears it is visible rather than absent.

Never drop the publisher's free text. Entity#remarks keeps it verbatim. This is where the fields the canonical model has no home for still live — OFAC's dates of birth and passport numbers are nowhere else — and an adapter that drops it loses data no later issue can get back.

Fields with no home: `Sources::Remarks`

Append them behind one shared marker rather than inventing a convention per source:

Remarks.build(row[:remarks], [["Vessel flag", "Panama"], ["Tonnage", "8000"]])
# => "Registered in Panama [source fields] Vessel flag: Panama; Tonnage: 8000"

Sources::Base.published_remarks(entity.remarks)   # => "Registered in Panama"

Remarks.build drops a label whose value is blank or missing, and a value may be an Array — the UN files three designations under one element. Remarks.published strips everything an adapter appended back off, which is what anything reading the remark for what the publisher actually wrote must use: OFAC's remarks parser must never see a vessel flag and read it as a nationality. The conformance spec checks that at least one record still has publisher text after stripping, so appending is safe and swallowing is not.

The value objects

Class	Notes
`Name`	`value:` plus `kind:` (`:primary`, `:aka`, `:fka`, `:nka` — defaults to `:primary`), `quality:` (`:good`, `:low`, or `nil` for unstated), `script:` (a closed list; map the publisher's vocabulary onto it — OFAC's "Farsi" is `:arabic`). A blank value raises: a blank-valued name is a record that matches everything.
`Address`	`street:`, `city:`, `state_province:`, `postal_code:`, `country:`, `note:`. An address that located nothing raises `InvalidArgument` (an `ArgumentError`); rescue and drop it rather than keeping an empty one.
`Identifier`	`value:` plus `kind:` (`:passport`, `:national_id`, `:tax_id`, `:registration_number`, `:other`), `country:`, `issued_on:`, `expires_on:`, `note:`. `:other` is a real answer — a document we cannot classify still matches on its number. A document element with a type and no number has nothing to match on: drop it.
`PartialDate`	`.parse`, `.range(from, to)`, `.new(year:, month:, day:, approximate:)`. Never collapse a year to January 1st.

The mistakes this section exists to prevent

Each of these is caught by the conformance spec, but knowing why is cheaper than reading the failure:

one field read as another, because a publisher uses one name for two things;
a date left as a string, which quietly cannot be compared with anything;
free text dropped to keep the schema tidy, which loses screening signal permanently;
an empty payload read as a list with nobody on it — see section 7.

6. Give every record a stable id

Entity#id is namespaced and never nil. When you pass source_ref: and no id:, it is derived as "#{source}:#{source_ref}", which is what you want whenever the publisher supplies a reference of its own.

The id has to satisfy two properties, and the conformance spec checks both:

unique within a sync — two records under one id are one record to storage, and the second silently replaces the first;
identical on a second read of the same bytes — the snapshot diff compares records under their ids, and an id that moves because a Hash iterated differently or a counter was involved reports the whole list as removed and re-added. A diff that says everything changed says nothing at all.

So an id may never depend on position in the file, iteration order, a counter, the wall clock, or anything else outside the record's own bytes.

When the publisher supplies no id

Canada is the case. Global Affairs publishes no identifier of any kind; what it publishes is a citation — which regulation, which schedule, which item — and CanadaSema::SourceRef derives a deterministic id from it. The recipe generalizes:

module SourceRef
  NORMALIZE = /[[:space:]]+/
  SEPARATOR = "\u0000"  # cannot occur in XML character data
  LENGTH    = 16        # 64 bits of SHA-256

  module_function

  def for(country:, schedule:, item:, name:)
    parts = [country, schedule, item, name].map { |part| normalize(part) }
    -Digest::SHA256.hexdigest(parts.join(SEPARATOR))[0, LENGTH]
  end

  def normalize(value) = value.to_s.split(NORMALIZE).join(" ").downcase
end

Four decisions in there are worth copying deliberately:

Hash the citation and the name. The citation alone is already unique across all 5,690 published records, so the name looks redundant — until a schedule is amended. Item numbers are positions in a list: delete item 5 and everything after it moves up one, and hashing the citation alone would hand item 6's old id to the person who used to be item 7. The diff would then report that one person quietly changed their name, which is the same shape as a correction and reads as one. With the name in the hash that amendment reports as a removal and an addition — noisier, and true. The cost runs the other way: correcting a typo in a published name re-ids that record. Churn in a diff is a nuisance; one id covering two different people is a screening failure, so the trade goes this way.
Normalize only case and whitespace. Canada publishes Venezuela both with and without a trailing space, and a record must not change id when a space does. Nothing further is folded — not punctuation, not diacritics — because every additional fold is another way for two genuinely different records to collide into one id.
Join with a separator that cannot occur in the data. Otherwise ("a", "bc") and ("ab", "c") hash the same.
Treat the constants as versioned. Changing the normalization, the separator or the length re-ids every record of that source ever stored. That makes each of them a decision with a migration attached, not a cleanup.

Document the choice in the class comment. A reader of a stored snapshot has to be able to find out where an id came from.

7. Report what you could not read

A sanctions list is not a file we control, and the two failure modes are different:

A row that could not be read is a Parsers::Warning, kept rather than raised. OFAC ships 19,321 rows, and a single unbalanced quote in the middle must not cost the other 19,320 — refusing to load a list because one record is malformed fails exactly when the list is most needed. The toolkits collect their own warnings; the convention is that an adapter exposes them together with anything it noticed itself:

attr_reader :warnings

def parse(raw)
  reader = LIST.read(raw)
  @unmapped = []
  entities = reader.filter_map { |record| entity(record) }
  @warnings = reader.warnings + @unmapped
  entities
end

Read #warnings after #parse; sync orchestration reports them. Give every warning a line number where the parser can supply one — a 5.6 MB file is only debuggable if the complaint says where.

A payload that is not the list at all raises ParseError (the toolkits raise it under the name Parsers::ParseError; it is the same class). Both toolkits already refuse an empty payload for you, and the delimited reader also gives up after enough consecutive unparseable rows, on the grounds that the publisher probably served an error page. Do not rescue that into an empty array. No sanctions list has ever been published empty, so an empty payload is a failed download, a moved URL or an outage — never a day on which nobody is sanctioned, and letting a sync succeed at screening against nothing is the most expensive way this library can fail.

A truncated download is the middle case. It may be worth salvaging — the XML reader keeps the records it read before the break and warns — and what the contract requires is only that half a list never comes back looking exactly like the whole one.

Raising one of your own is fine, as long as it is in the hierarchy — a payload this adapter can see is wrong is a ParseError, and give it a locator if you have one:

raise ActiveSanction::ParseError.new("the sheet has no listing column", line: 1)

Nothing an adapter raises may be a bare RuntimeError, an ArgumentError or an exception class belonging to a library it happens to parse with: the whole point of the error hierarchy is that a host application rescues ActiveSanction::Error and gets your adapter's failures along with everything else. #source_id is filled in for you as the error leaves the adapter, so there is no need to name your own list in it.

#parse should not rescue ActiveSanction::Error at all. #sync does not rescue either: one source's failure being isolated from the others is a decision about a run, and it belongs to sync orchestration, which needs an exception here to notice.

What the doctor reads, and the two hooks it offers

ActiveSanction.doctor diagnoses whether a list still parses the way we think it does, and it works entirely from what an adapter already produces: the record count, the fill rates of every canonical field, #warnings, #orphans and #remarks_coverage where those exist. An adapter that follows the convention above is diagnosed with no extra work — the doctor asks what your adapter responds to and measures whatever it finds.

Two things are worth declaring on top of that, and both are optional.

A floor, for the run that has nothing to compare against. The doctor's real baseline is the snapshot the last sync stored, which never goes stale; a floor is a coarse backstop for a first sync, a new source, or a store that was cleared:

floor :remarks_coverage, 0.90
floor :record_count, 400

The name is a doctor check name (:record_count, :remarks_coverage, :fill_addresses, and so on) and the value is the least it may be without the diagnosis saying so. Declare few, and declare them wide. A number committed here goes stale on its own, and the day somebody widens one to make a build pass is the day it stops being read.

A column shape, if — and only if — the publisher ships a positional file. Declaring the column names already pins the width, so a column inserted upstream arrives as a wrong-width row and every row warns. A column reordered upstream keeps the width, parses cleanly, and builds every record out of shifted fields; nothing raises and nothing warns. Assert what the values are, not only how many there are:

def column_shapes(raw)
  rows = LIST.read(raw).to_a
  [Parsers::ColumnShape.new(name: :ent_num, matches: /\A\d+\z/, description: "numeric")]
    .map { |shape| shape.tally(rows.map { |row| row[shape.name] }) }
end

A file that names its own columns needs none of this, which is most of them. Pick a threshold you could defend on a published file rather than on your fixture: these are not validated files, and one row where somebody typed a letter into a numeric column is a curiosity rather than a format change.

8. Capture a fixture and wire up the conformance spec

The fixture

Use real published records. A conformance run against a fixture somebody wrote to pass it proves nothing about the list. Download the real file and cut it down.

Choose records by quirk, not at random. The Canadian fixture is sixteen records covering all three record shapes, a bilingual value split on / and another split on |, a value bilingual in neither, a name padded with U+00A0, a vessel type wrapped across three lines, a semicolon-separated alias field and a comma-separated one, a year-only date, a full date, two candidate dates, a date nothing can read, a record with no schedule, and a padded schedule and country. Every one of those is a line in the spec. Aim for the same: one record per thing the adapter had to decide, plus one ordinary record.

Keep the bytes verbatim. The fixture is read with File.binread and reaches #parse undecoded, which is the point — it holds the adapter to doing its own decoding. The committed OFAC fixture is still Windows-1252, and an editor that helpfully re-saves it as UTF-8 turns that spec into a test of nothing. Line endings and the trailing 0x1A byte DOS-lineage export tooling writes matter less — the delimited reader trims that byte for you — but the safe cut is one made by a tool that does not touch what it was not asked to:

# delimited: the header, if there is one, plus the rows you picked
head -1 /tmp/SDN.CSV > spec/fixtures/my_source/SDN.CSV
grep -a '^36,' /tmp/SDN.CSV >> spec/fixtures/my_source/SDN.CSV

For XML, keep the document element and its attributes — that is where a dateGenerated version marker lives — and any intermediate scaffolding the records sit inside, then delete whole record elements. Verify the result parses before committing it.

For a multi-file source, keep the join keys aligned. If you trim SDN.CSV to eight rows and leave ALT.CSV whole, every unmatched alias becomes an orphan and the fixture stops describing a coherent version of the list. The committed OFAC fixture is 8 primary rows, 7 aliases and 5 addresses, all keyed to the same entities.

Keep it small. The existing fixtures run from 250 bytes to 18 KB. They are read on every suite run and reviewed by hand.

The spec

RSpec.describe ActiveSanction::Sources::CanadaSema do
  it_behaves_like "a sanction source", fixture: "canada_sema/sema.xml"
end

Paths are relative to spec/fixtures. A source whose publisher splits the list names one fixture per declared URL, under the keys the adapter declared:

it_behaves_like "a sanction source",
                fixture: { sdn: "ofac_sdn/SDN.CSV", alt: "ofac_sdn/ALT.CSV", add: "ofac_sdn/ADD.CSV" }

There is one option: remarks: false, for a list that publishes no free text of its own anywhere — Canada is the only launch source that qualifies. It has to be asked for, so that dropping a remark by accident stays a failure.

The group lives in lib/active_sanction/testing/sanction_source.rb, reaches a suite through require "active_sanction/testing", and checks what everything downstream assumes and cannot check for itself: that the adapter declares a key, a jurisdiction, an authority and a URL and registers itself; that #parse returns Entities with unique, deterministic ids, a canonical type and at least one name; that dates are PartialDates; that every record survives the round-trip through #to_h; that the publisher's own text survives in remarks; that an empty payload is refused; and that a truncated one is not reported as the whole list.

It is the floor, not the ceiling. Nothing in it knows that OFAC writes -0- for null, that the UN means two different things by QUALITY, or which of your fixture's records is a vessel. Only a spec that knows what is in the fixture can check that the list was read correctly, so write one — every adapter here does. Look up records the way a person would (by reference, or by name for a source with no reference) and assert the meaning:

def entity(ref) = entities.find { |candidate| candidate.source_ref == ref }

it "types a record carrying an IMO number as a vessel" do
  expect(entity("RT/2023/0330").type).to eq(:vessel)
end

The suite is hermetic: WebMock blocks outbound connections, so an un-stubbed HTTP call raises rather than quietly reaching a government server. A spec that genuinely needs a live endpoint is tagged :live, excluded from the default run, and executed with bundle exec rspec --tag live.

9. Register it

An adapter whose file is required but which never registers is invisible: config.sources cannot name it, sync will not run it, and nothing says so out loud. The conformance spec checks for exactly this.

Inside this gem

One explicit line at the bottom of the adapter's own file, and a require in lib/active_sanction.rb:

# lib/active_sanction/sources/my_source.rb
ActiveSanction::Sources.register(ActiveSanction::Sources::MySource)

Registration is explicit rather than hooked onto inherited because auto-registering every subclass would also enrol the abstract intermediates that adapters sharing a publisher want — Sources::Ofac is one, and there is no such list as "OFAC" — along with every throwaway subclass a test defines.

From outside this gem

The registry is open, and that is deliberate. Nothing there requires Sources::Base: registration is duck-typed on .key and .new, so a source backed by a database table rather than a published file — no URL to declare, no payload to fetch — is a first-class citizen rather than something that has to pretend to be a file download.

# in your own gem or initializer
ActiveSanction::Sources.register(MyCompany::InternalWatchlist)

ActiveSanction.configure { |c| c.sources = %i[ofac_sdn my_internal_watchlist] }

Subclassing Base is the convenient way to write an adapter, not the price of admission. If you do subclass it and your list is not fetched over HTTP, override #retrieve to return a Hash of name => bytes (or nil when nothing has changed) and leave the rest alone.

To replace a built-in adapter with a patched one, unregister first — two lists cannot answer to one name, and registering over a claimed key raises at load time, which is where the collision is cheap to fix:

ActiveSanction::Sources.unregister(:ofac_sdn)
ActiveSanction::Sources.register(MyCompany::PatchedOfacSdn)

There is deliberately no clear!.

And commit a canary baseline

The upstream canary (#69) fetches every registered source on weekdays and holds it against .github/baselines/<key>.json. A newly registered adapter has no such file, and nothing breaks: it is held to the coarse floor declarations you made in section 7 instead, and the run reports compared: false rather than reading as a clean comparison.

Commit one anyway, in the same pull request as the adapter:

$ CANARY_SOURCES=my_source bundle exec rake canary          # what it measures
$ CANARY_SOURCES=my_source bundle exec rake canary:refresh  # write the baseline

That reaches the real publisher, which is the point: the numbers a reviewer sees in .github/baselines/my_source.json are the numbers the file actually yields, and from the next weekday onwards a change in any of them opens an issue. It is also the cheapest review anybody will ever give your parser — a fill rate that reads 4% where you expected 90% is a field you mapped to the wrong element, and it is far easier to see in that diff than in a fixture of forty records.

.github/baselines/README.md documents the format and the per-key tolerances.

10. Typed, linted, and green

Every file in lib/ is # typed: strict, and new files are born that way: a signature written beside the code costs a line, and one retrofitted a milestone later costs an afternoon of reading the code back. In practice that means extend T::Sig, a sig on every method, T.let on every instance variable and on constants, and override. on #parse, which Base declares. The worked example below is written the way it would land in lib/.

An adapter in your own application is under no such obligation — nothing in the public API requires signatures.

bundle exec rake        # rspec, then rubocop, then srb tc; all three must pass

A worked example, end to end

Here is a complete adapter, its fixture and its spec. The list is invented for this document — a real adapter's fixture must be real published bytes, as section 8 says. It is otherwise exactly what lands in the repository, and it passes the conformance spec, RuboCop and Sorbet unchanged.

The Ruritanian Ministry of Finance publishes one CSV with a header row, N/A where it means nothing, a stable reference per record, semicolon-separated aliases, a Position column the canonical model has no home for, and dates in ISO.

The fixture

spec/fixtures/ruritania_fsl/list.csv:

Reference,Name,Aliases,Type,Date of Listing,Regime,Date of Birth,Passport Number,Position,Notes
RT/2019/0014,ACME TRADING LIMITED,ACME TRADE; ACME TRADING LTD,Entity,2019-03-14,Ruritania (Financial Measures) Order 2019,N/A,N/A,N/A,Registered in Nicosia; ships through two subsidiaries
RT/2021/0102,JOHN AGYEMAN OKORO,N/A,Individual,2021-11-02,Ruritania (Financial Measures) Order 2019,1974,RT884213,Deputy Minister of Trade,Travels on a diplomatic passport
RT/2023/0330,MV NORTHERN STAR,NORTHERN STAR; SEVERNAYA ZVEZDA,Ship,2023-07-30,Ruritania (Shipping Measures) Order 2022,N/A,N/A,N/A,Reflagged twice in 2022
RT/2024/0007,MARIA ELENA VASQUEZ DE LEON,N/A,Individual,2024-01-19,Ruritania (Financial Measures) Order 2019,1988-06-04,N/A,N/A,N/A

Four records: an organization with two aliases and no dates, an individual with a year-only date of birth and a passport, a vessel, and an individual with a full date of birth and no free text of any kind.

The adapter

lib/active_sanction/sources/ruritania_fsl.rb:

# typed: strict
# frozen_string_literal: true

require "sorbet-runtime"

require "active_sanction/parsers"
require "active_sanction/sources"
require "active_sanction/sources/base"

module ActiveSanction
  module Sources
    # The Ruritanian consolidated financial sanctions list: everyone named in
    # an order made under the Financial Measures Act, plus the vessels listed
    # under the Shipping Measures Order.
    #
    #   snapshot = ActiveSanction::Sources[:ruritania_fsl].new.sync
    #
    # One file, one flat record shape, a stable reference per record, and
    # `N/A` wherever the Ministry means nothing.
    class RuritaniaFsl < Base
      extend T::Sig

      key :ruritania_fsl
      jurisdiction :rt
      authority "Ruritanian Ministry of Finance"
      format :csv

      url :main, "https://finance.gov.rt/sanctions/consolidated.csv"

      # The Ministry ships a header row, so the column names are read from the
      # file rather than declared; `N/A` is what it writes where it means
      # nothing.
      LIST = T.let(Parsers::DelimitedTable.new(null: "N/A"), Parsers::DelimitedTable)

      # `Type` is free text and the Ministry writes three values.
      TYPES = T.let(
        { "individual" => :individual, "entity" => :organization, "ship" => :vessel }.freeze,
        T::Hash[String, Symbol]
      )

      # The alias column is separated by semicolons and by nothing else.
      ALIASES = T.let(";", String)

      sig { returns(T::Array[Parsers::Warning]) }
      attr_reader :warnings

      sig { params(args: T.untyped, options: T.untyped).void }
      def initialize(*args, **options)
        super
        @warnings = T.let([], T::Array[Parsers::Warning])
        @unmapped = T.let([], T::Array[Parsers::Warning])
      end

      sig { override.params(raw: T.untyped).returns(T::Array[Entity]) }
      def parse(raw)
        reader = LIST.read(raw)
        @unmapped = []
        entities = reader.filter_map { |row| entity(row) }
        @warnings = reader.warnings + @unmapped
        entities
      end

      private

      sig { params(row: Parsers::DelimitedTable::Row).returns(T.nilable(Entity)) }
      def entity(row)
        published = names(row)
        return note_nameless(row) if published.empty?

        Entity.new(source: key, source_ref: row[:reference], type: type(row), names: published,
                   identifiers: identifiers(row), dates_of_birth: dates_of_birth(row),
                   programs: [row[:regime]].compact, listed_on: PartialDate.parse(row[:date_of_listing]),
                   remarks: Remarks.build(row[:notes], [["Position", row[:position]]]))
      end

      sig { params(row: Parsers::DelimitedTable::Row).returns(T::Array[Name]) }
      def names(row)
        primary = row[:name]
        return [] if primary.nil?

        [Name.new(value: primary)] + aliases(row)
      end

      sig { params(row: Parsers::DelimitedTable::Row).returns(T::Array[Name]) }
      def aliases(row)
        row[:aliases].to_s.split(ALIASES).filter_map do |value|
          text = value.strip
          Name.new(value: text, kind: :aka) unless text.empty?
        end
      end

      sig { params(row: Parsers::DelimitedTable::Row).returns(Symbol) }
      def type(row)
        published = row[:type].to_s.downcase
        return T.must(TYPES[published]) if TYPES.key?(published)

        @unmapped << Parsers::Warning.new(
          line: row.line, message: "unknown Type #{row[:type].inspect}; treated as an organization"
        )
        :organization
      end

      sig { params(row: Parsers::DelimitedTable::Row).returns(T::Array[PartialDate]) }
      def dates_of_birth(row) = [PartialDate.parse(row[:date_of_birth])].compact

      sig { params(row: Parsers::DelimitedTable::Row).returns(T::Array[Identifier]) }
      def identifiers(row)
        return [] if row.null?(:passport_number)

        [Identifier.new(kind: :passport, value: row[:passport_number])]
      end

      # A record with no name cannot be screened against. None of the records
      # published today is nameless; the warning exists so that the day one is,
      # it is visible rather than absent.
      sig { params(row: Parsers::DelimitedTable::Row).returns(NilClass) }
      def note_nameless(row)
        @unmapped << Parsers::Warning.new(
          line: row.line, message: "row #{row[:reference].inspect} has no Name and was skipped"
        )
        nil
      end
    end
  end
end

ActiveSanction::Sources.register(ActiveSanction::Sources::RuritaniaFsl)

Then add the require to lib/active_sanction.rb:

require "active_sanction/sources/ruritania_fsl"

The spec

spec/active_sanction/sources/ruritania_fsl_spec.rb:

# frozen_string_literal: true

RSpec.describe ActiveSanction::Sources::RuritaniaFsl do
  def raw = File.binread(File.expand_path("../../fixtures/ruritania_fsl/list.csv", __dir__))

  let(:adapter) { described_class.new }
  let(:entities) { adapter.parse(raw) }

  def entity(ref) = entities.find { |candidate| candidate.source_ref == ref }

  it_behaves_like "a sanction source", fixture: "ruritania_fsl/list.csv"

  it "reads the publisher's reference as the entity id" do
    expect(entity("RT/2021/0102").id).to eq("ruritania_fsl:RT/2021/0102")
  end

  it "splits the alias column on semicolons" do
    expect(entity("RT/2023/0330").names.map(&:value))
      .to eq(["MV NORTHERN STAR", "NORTHERN STAR", "SEVERNAYA ZVEZDA"])
  end

  it "reads a year-only date of birth without inventing a day" do
    expect(entity("RT/2021/0102").dates_of_birth.first.to_h).to include(year: 1974, month: nil, day: nil)
  end

  it "keeps the Position column behind the source-fields marker" do
    expect(entity("RT/2021/0102").remarks)
      .to eq("Travels on a diplomatic passport [source fields] Position: Deputy Minister of Trade")
  end

  it "reads N/A as nothing at all" do
    expect(entity("RT/2019/0014").identifiers).to be_empty
  end
end

That is 19 conformance examples and 5 of its own, and the list is now reachable everywhere:

ActiveSanction::Sources[:ruritania_fsl]              # => the adapter class
ActiveSanction.configure { |c| c.sources = %i[ofac_sdn ruritania_fsl] }
snapshot = ActiveSanction::Sources[:ruritania_fsl].new.sync

What a harder list adds to that

The example above is the easy shape. In roughly the order they bite:

No publisher id → section 6, and copy CanadaSema::SourceRef.
Several files → declare a url per file, read raw[:name], join with Parsers::Join, and watch the orphan counts.
XML → Parsers::XmlRecords, and split the mapping into its own Record class the way un_consolidated/record.rb does.
An encoding the publisher does not declare → pass encoding: to the table.
A field that means two things → map it twice, per record shape, and say so in the class comment.
Free text carrying the identifiers → OFAC's shape. Sources::Ofac::RemarksParser is the precedent, along with its coverage reporting: extraction is additive, the remark is kept verbatim either way, and every sync reports how much of the text was understood.

Checklist

[ ] Read the closest existing adapter, and its spec.
[ ] Downloaded the real file and answered the eight questions in section 2.
[ ] Declared key, jurisdiction, authority, format, and one url per file.
[ ] Overrode #source_version if the document carries its own version marker.
[ ] #parse returns Entity objects and nothing else.
[ ] Every id is unique and identical on a second read of the same bytes.
[ ] Every date is a PartialDate.
[ ] Every type is one of the four canonical types, and an unrecognised one warns rather than dropping the record.
[ ] The publisher's free text is kept verbatim; extra fields are appended with Remarks.build.
[ ] Unreadable rows become warnings; a payload that is not the list raises.
[ ] A floor for anything worth a coarse bound on a first sync, and #column_shapes if the publisher ships a positional file.
[ ] A trimmed fixture of real bytes, one record per quirk, committed under spec/fixtures/<key>/.
[ ] it_behaves_like "a sanction source" passes, plus a spec that knows what is in the fixture.
[ ] Sources.register at the bottom of the file, and a require in lib/active_sanction.rb.
[ ] A canary baseline committed under .github/baselines/<key>.json, written by CANARY_SOURCES=<key> bundle exec rake canary:refresh.
[ ] A class comment describing the list, its record counts, its quirks, and what the adapter refuses to do about them.
[ ] bundle exec rake is green.

Adding a source

Contents

1. Read the closest existing adapter first

2. Look at what the publisher actually serves

3. Declare the list

What #parse is handed

4. Choose how to read it

There is no field-mapping DSL

Parsers::DelimitedTable — CSV, TSV, anything delimited

Parsers::XmlRecords — record-oriented XML

Parsers::Spreadsheet — an .xlsx workbook

Parsers::Join — several files, one logical record

Neither

5. Map the publisher's fields onto the canonical model

The rules that are not negotiable

Fields with no home: Sources::Remarks

The value objects

The mistakes this section exists to prevent

6. Give every record a stable id

When the publisher supplies no id

7. Report what you could not read

What the doctor reads, and the two hooks it offers

8. Capture a fixture and wire up the conformance spec

The fixture

The spec

9. Register it

Inside this gem

From outside this gem

And commit a canary baseline

10. Typed, linted, and green

A worked example, end to end

The fixture

The adapter

The spec

What a harder list adds to that

Checklist

What `#parse` is handed

`Parsers::DelimitedTable` — CSV, TSV, anything delimited

`Parsers::XmlRecords` — record-oriented XML

`Parsers::Spreadsheet` — an `.xlsx` workbook

`Parsers::Join` — several files, one logical record

Fields with no home: `Sources::Remarks`