Module: ActiveSanction::Snapshot::Bundle

Extended by:
T::Helpers, T::Sig
Defined in:
lib/active_sanction/snapshot/bundle.rb,
lib/active_sanction/snapshot/bundle/header.rb,
lib/active_sanction/snapshot/bundle/payload.rb,
lib/active_sanction/snapshot/bundle/signature.rb

Overview

One list, in one file, that another machine can load and trust without ever reaching the publisher.

File.open("ofac_sdn.asb", "wb") do |io|
ActiveSanction::Snapshot::Bundle.write(snapshot, io: io, sign_with: private_key)
end

File.open("ofac_sdn.asb", "rb") do |io|
snapshot = ActiveSanction::Snapshot::Bundle.read(io, verify_with: public_key)
snapshot.trusted?   # => true
end

ActiveSanction.export and .import are the sugar over this that most applications want; this is the format itself, and it is public API. The byte-level specification is docs/bundle_format.md, which is written so that a bundle can be produced and read by something that is not this gem and not Ruby.

Why a file, when there is already a store

Storage::FileSystem writes gzipped JSON too, and its layout is explicitly private and expected to change. This is the opposite thing: a published artifact with a stability contract, which three situations need and a directory cannot serve.

  • Publishers go down. OFAC breaks, changes format, and rate-limits. A bundle produced once and copied is the difference between a bad afternoon at Treasury and a failed deploy for everyone downstream.
  • Air-gapped and privacy-sensitive installations. A compliance team that will not send subject names to a third-party API will happily consume fresh data. Only a file serves them.
  • Audit. A checksum proves a list is internally intact. A signature proves it is the one that was published, which is the claim an examiner is actually asking about.

The shape of a bundle

ACTIVESANCTION-BUNDLE/1\n     magic, and the format version
{"format_version":1,...}\n    one canonical line -- see Header
ecdsa-sha256 MEUCIQ...\n      or "-" -- see Signature
<deflated NDJSON>             to EOF -- see Payload

The first three lines are text on purpose: head -c 512 on a bundle tells an operator what list it holds, how many records, from when, and who says so, without a tool and without decompressing anything.

Determinism

Two writes of one snapshot produce one file. Records are ordered by the fingerprint Snapshot's own checksum is built from rather than by whatever order a publisher's file happened to arrive in, the header's keys have a fixed order, the compression level is named by the specification rather than taken from a build's default, and there is deliberately no written-at timestamp anywhere in the file.

What that buys is comparability: two mirrors that bundled the same snapshot can be held against each other byte for byte. The invariant that survives everything, including a zlib that packs differently, is the header line -- it pins the content, through payload_digest, and the provenance. A signature line is the one part that may differ between two signings, because ECDSA is randomized.

What it refuses to do

Return anything it cannot prove, on the same rule the stores follow. A flipped byte, a truncated download and an edited record all raise Corrupt rather than screening against a list that is quietly missing somebody. A format version from a newer gem raises UnsupportedFormat before the payload is touched, because a newer shape will usually deserialize into plausible, wrong records. And a snapshot only comes back trusted? when a key was supplied and the signature verified under it.

Defined Under Namespace

Classes: Corrupt, Header, Unsigned, UnsupportedFormat, UntrustedSignature

Class Method Summary collapse

Class Method Details

.build(header, entities, trusted) ⇒ Snapshot

The snapshot itself, which re-derives its own checksum over the records that actually arrived and refuses to be built if it does not match the one in the header. The same defence a store relies on, applied to a file that came from somebody else.

Parameters:

  • header (Header)
  • entities (Array<T.untyped>)
  • trusted (Boolean)

Returns:



323
324
325
326
327
328
329
330
 
# File 'lib/active_sanction/snapshot/bundle.rb', line 323

def build(header, entities, trusted)
  Snapshot.new(source: header.source, entities: entities, fetched_at: header.fetched_at,
               checksum: header.snapshot_checksum, record_count: header.record_count,
               schema_version: header.schema_version, source_version: header.source_version,
               trusted: trusted)
rescue Snapshot::ChecksumMismatch, InvalidArgument => e
  raise Corrupt, "this bundle does not hold the list its header describes (#{e.message})"
end

.digest_message(header, digest, bytes) ⇒ String

Parameters:

  • header (Header)
  • digest (String)
  • bytes (Integer)

Returns:

  • (String) 


370
371
372
373
374
 
# File 'lib/active_sanction/snapshot/bundle.rb', line 370

def digest_message(header, digest, bytes)
  "this bundle's records hash to #{digest} over #{bytes} bytes, not the #{header.payload_digest} over " \
    "#{header.payload_bytes} its header declares. The file was damaged in transit or edited after it was " \
    "written -- fetch it again rather than screening against it"
end

.entities(io, header) ⇒ Array<T.untyped>

The records, streamed, held against everything the header promised about them.

Parameters:

  • io (T.untyped)
  • header (Header)

Returns:

  • (Array<T.untyped>) 


301
302
303
304
305
306
307
308
309
 
# File 'lib/active_sanction/snapshot/bundle.rb', line 301

def entities(io, header)
  built = T.let([], T::Array[T.untyped])
  digest, bytes = Payload.unpack(io, limit: header.payload_bytes) { |record| built << entity!(record) }
  unless digest == header.payload_digest && bytes == header.payload_bytes
    raise Corrupt, digest_message(header, digest, bytes)
  end

  built
end

.entity!(record) ⇒ T.untyped

Parameters:

  • record (String)

Returns:

  • (T.untyped) 


312
313
314
315
316
 
# File 'lib/active_sanction/snapshot/bundle.rb', line 312

def entity!(record)
  Entity.from_h(JSON.parse(record))
rescue JSON::ParserError, ArgumentError, TypeError, KeyError => e
  raise Corrupt, "a record in this bundle cannot be read (#{e.message})"
end

.format_message(version) ⇒ String

Parameters:

  • version (Integer)

Returns:

  • (String) 


350
351
352
353
354
 
# File 'lib/active_sanction/snapshot/bundle.rb', line 350

def format_message(version)
  "this bundle is written in bundle format version #{version}; active_sanction #{VERSION} reads " \
    "#{READABLE_FORMAT_VERSIONS.first}-#{READABLE_FORMAT_VERSIONS.last}. Upgrade the gem -- nothing here " \
    "can read it partially, and a format this code guessed at would produce a list nobody could defend"
end

.header(io) ⇒ Header

What a bundle says about itself, without reading its records:

header = File.open(path, "rb") { |io| Bundle.header(io) }
header.record_count       # => 19015
header.snapshot_checksum  # => "sha256:9f86d081884c7d65..."

Storage::Meta's job for a file somebody sent you, and cheap for the same reason: deciding whether a 25 MB bundle holds a list you already have should cost a few hundred bytes.

Parameters:

  • io (T.untyped)

Returns:



227
228
229
230
231
 
# File 'lib/active_sanction/snapshot/bundle.rb', line 227

def header(io)
  io.binmode if io.respond_to?(:binmode)
  version = magic!(io)
  supported!(parse!(line!(io, "header")), version)
end

.line!(io, what) ⇒ String

One line of the container, refused rather than read without limit.

Parameters:

  • io (T.untyped)
  • what (String)

Returns:

  • (String)

Raises:



334
335
336
337
338
339
340
 
# File 'lib/active_sanction/snapshot/bundle.rb', line 334

def line!(io, what)
  line = io.gets("\n", MAX_LINE_BYTES)
  raise Corrupt, "this bundle ends before its #{what} line" if line.nil? || line.empty?
  raise Corrupt, "this bundle's #{what} line is longer than #{MAX_LINE_BYTES} bytes" unless line.end_with?("\n")

  line.chomp.force_encoding(Encoding::UTF_8)
end

.magic!(io) ⇒ Integer

The format version off the first line, checked before anything else in the file is looked at.

Parameters:

  • io (T.untyped)

Returns:

  • (Integer)

Raises:



252
253
254
255
256
257
258
259
260
 
# File 'lib/active_sanction/snapshot/bundle.rb', line 252

def magic!(io)
  match = MAGIC_PATTERN.match(line!(io, "magic"))
  raise Corrupt, "this is not an active_sanction bundle: it does not begin with #{MAGIC}/<version>" if match.nil?

  version = Integer(T.must(match[1]))
  return version if READABLE_FORMAT_VERSIONS.cover?(version)

  raise UnsupportedFormat, format_message(version)
end

.parse!(line) ⇒ Header

Parameters:

  • line (String)

Returns:



263
264
265
266
267
 
# File 'lib/active_sanction/snapshot/bundle.rb', line 263

def parse!(line)
  Header.parse(line)
rescue JSON::ParserError, ArgumentError, TypeError, Sources::DeclarationError => e
  raise Corrupt, "this bundle's header cannot be read (#{e.message})"
end

.payload_message(header) ⇒ String

Parameters:

Returns:

  • (String) 


364
365
366
367
 
# File 'lib/active_sanction/snapshot/bundle.rb', line 364

def payload_message(header)
  "this bundle's payload is #{header.payload_encoding}/#{header.payload_compression}; active_sanction " \
    "#{VERSION} reads #{ENCODING}/#{COMPRESSION}. Upgrade the gem"
end

.read(io, verify_with: nil) ⇒ Snapshot

Reads a bundle, verifying it as it goes, and returns the Snapshot.

Bundle.read(io)                     # => Snapshot, trusted? false
Bundle.read(io, verify_with: key)   # => Snapshot, trusted? true, or an exception

Without verify_with: the signature is not looked at: an unsigned bundle is a first-class bundle, and a signed one read without a key is exactly as useful as an unsigned one -- its records are still proven against the digest in its header and against the snapshot checksum. What it is not is attested, and trusted? says so.

With a key, the signature is checked before a byte of the payload is inflated. Compressed data from a party that has not authenticated is the last thing anybody should expand.

Parameters:

  • io (T.untyped)
  • verify_with (T.untyped) (defaults to: nil)

Returns:



208
209
210
211
212
213
214
215
 
# File 'lib/active_sanction/snapshot/bundle.rb', line 208

def read(io, verify_with: nil)
  io.binmode if io.respond_to?(:binmode)
  version = magic!(io)
  line = line!(io, "header")
  header = supported!(parse!(line), version)
  trusted = verified!(line!(io, "signature"), line, verify_with)
  build(header, entities(io, header), trusted)
end

.readable_payload?(header) ⇒ Boolean

Parameters:

Returns:

  • (Boolean) 


284
285
286
 
# File 'lib/active_sanction/snapshot/bundle.rb', line 284

def readable_payload?(header)
  header.payload_encoding == ENCODING && header.payload_compression == COMPRESSION
end

.records(snapshot) ⇒ Array<String>

The records of a snapshot, in the order a bundle lays them out.

Sorted by the fingerprint Snapshot's checksum is built from, so that two snapshots holding the same entities in the order two publishers happened to emit them produce identical files -- and so that the order of a payload and the meaning of a checksum can never drift apart. Serializing each entity twice, once to fingerprint it and once to write it, is what one definition of "the fingerprint of an entity" costs; an export is not a hot path.

Parameters:

Returns:

  • (Array<String>) 


243
244
245
246
247
 
# File 'lib/active_sanction/snapshot/bundle.rb', line 243

def records(snapshot)
  snapshot.entities
          .sort_by { |entity| Snapshot.fingerprint(entity) }
          .map { |entity| JSON.generate(entity.to_h) }
end

.schema_message(header) ⇒ String

Parameters:

Returns:

  • (String) 


357
358
359
360
361
 
# File 'lib/active_sanction/snapshot/bundle.rb', line 357

def schema_message(header)
  "this bundle holds records written under snapshot schema_version #{header.schema_version} by " \
    "#{header.generator}; active_sanction #{VERSION} reads #{READABLE_SCHEMA_VERSIONS.first}-" \
    "#{READABLE_SCHEMA_VERSIONS.last}. Upgrade the gem, or ask the publisher for a bundle this old"
end

.snapshot!(value) ⇒ Snapshot

Parameters:

  • value (T.untyped)

Returns:

Raises:



343
344
345
346
347
 
# File 'lib/active_sanction/snapshot/bundle.rb', line 343

def snapshot!(value)
  return value if value.is_a?(Snapshot)

  raise InvalidArgument, "Bundle.write takes an ActiveSanction::Snapshot, got #{value.class}"
end

.supported!(header, version) ⇒ Header

Everything about a header that decides whether this code may read the payload at all. All of it happens before the payload is touched.

Parameters:

  • header (Header)
  • version (Integer)

Returns:

Raises:



272
273
274
275
276
277
278
279
280
281
 
# File 'lib/active_sanction/snapshot/bundle.rb', line 272

def supported!(header, version)
  if header.format_version != version
    raise Corrupt, "this bundle is labelled #{MAGIC}/#{version} and its header says format_version " \
                   "#{header.format_version}. The two have to agree -- only one of them is signed"
  end
  raise UnsupportedFormat, schema_message(header) unless READABLE_SCHEMA_VERSIONS.cover?(header.schema_version)
  raise UnsupportedFormat, payload_message(header) unless readable_payload?(header)

  header
end

.verified!(signature, header_line, key) ⇒ Boolean

Whether this bundle was proven to come from the holder of key, or the exception that says why it was not. False -- rather than an exception -- only when no key was given, which is the caller saying they do not care.

Parameters:

  • signature (String)
  • header_line (String)
  • key (T.untyped)

Returns:

  • (Boolean) 


292
293
294
295
296
 
# File 'lib/active_sanction/snapshot/bundle.rb', line 292

def verified!(signature, header_line, key)
  return false if key.nil?

  Signature.verify!(signature, header_line, key)
end

.write(snapshot, io:, sign_with: nil, generator: nil) ⇒ Header

Writes snapshot to io and returns the Header it wrote.

Bundle.write(snapshot, io: io)                        # unsigned, and fully usable
Bundle.write(snapshot, io: io, sign_with: key)        # OpenSSL::PKey, or a PEM
Bundle.write(snapshot, io: io, generator: "acme/2.0") # whose bundle this is

generator is the one field a publisher other than this gem should set. It is who published the file; gem_version, which is not overridable, is what serialized the records.

Parameters:

  • snapshot (T.untyped)
  • io (T.untyped)
  • sign_with (T.untyped) (defaults to: nil)
  • generator (T.untyped) (defaults to: nil)

Returns:



183
184
185
186
187
188
189
190
191
 
# File 'lib/active_sanction/snapshot/bundle.rb', line 183

def write(snapshot, io:, sign_with: nil, generator: nil)
  stored = snapshot!(snapshot)
  payload, digest, bytes = Payload.pack(records(stored))
  header = Header.from_snapshot(stored, payload_digest: digest, payload_bytes: bytes, generator: generator)
  line = header.to_line
  io.binmode if io.respond_to?(:binmode)
  io.write("#{MAGIC}/#{FORMAT_VERSION}\n", "#{line}\n", "#{Signature.sign(line, sign_with)}\n", payload)
  header
end