Class: ActiveSanction::Sources::Base

Inherits:
Object
  • Object
show all
Extended by:
Definition, T::Sig
Defined in:
lib/active_sanction/sources/base.rb

Overview

The contract every sanctions list adapter implements: declare what the list is and where it lives, then turn its bytes into Entities.

class UnConsolidated < ActiveSanction::Sources::Base
key          :un_consolidated
jurisdiction :un
authority    "United Nations Security Council"
format       :xml
url          :main, "https://scsanctions.un.org/resources/xml/en/consolidated.xml"

def parse(raw)
  ...   # => [Entity, ...]
end
end

ActiveSanction::Sources.register(UnConsolidated)

#parse is the whole of what an adapter must write. Everything above it is declaration (Definition), and everything below it -- conditional GET, payload caching, checksumming the result into a Snapshot -- is here, the same for every list, so that adding a jurisdiction is a parsing problem and not a plumbing one.

snapshot = ActiveSanction::Sources[:un_consolidated].new.sync
snapshot                      # => Snapshot, or nil if nothing changed

What #parse is handed

A source declaring one URL gets the bytes. One declaring several gets a Hash keyed by the names it declared, because OFAC's three files only mean anything joined:

def parse(raw)
join(raw[:sdn], raw[:alt], raw[:add])
end

Which of the two it is follows from the declaration, not from what a caller happened to pass, so an adapter's signature does not change under it when a fixture is handed to #snapshot directly.

The bytes arrive as a String. A list too large to hold in memory wants #15's streaming parse rather than this path; the cached Entry, which knows how to hand out a verified file handle, is where that will start.

What sync does not do

It does not store the snapshot, and it does not rescue anything. One source's failure being isolated from the others, and the previous good snapshot being kept when a list fails, are decisions about a run rather than about a list -- they belong to sync orchestration (#34), which needs an exception here to notice.

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods included from Definition

declarations, declared?, declared_floors, declared_urls, floor, key!, licence_notice, licence_url, multi_url?, to_h

Constructor Details

#initialize(fetcher: Fetcher.new, cache: PayloadCache.new, logger: ActiveSanction.config.logger, instrumenter: ActiveSanction.config.instrumenter) ⇒ void

cache: nil turns off payload caching, which costs one thing worth knowing: a multi-file source can no longer answer a sync where some of its files changed and others came back 304, so the unchanged ones are downloaded again in full.

Parameters:

  • fetcher (Fetcher) (defaults to: Fetcher.new)
  • cache (PayloadCache, nil) (defaults to: PayloadCache.new)
  • logger (T.untyped) (defaults to: ActiveSanction.config.logger)
  • instrumenter (T.untyped) (defaults to: ActiveSanction.config.instrumenter) 


99
100
101
102
103
104
105
106
 
# File 'lib/active_sanction/sources/base.rb', line 99

def initialize(fetcher: Fetcher.new, cache: PayloadCache.new, logger: ActiveSanction.config.logger,
               instrumenter: ActiveSanction.config.instrumenter)
  @fetcher = T.let(fetcher, Fetcher)
  @cache = T.let(cache, T.nilable(PayloadCache))
  @logger = T.let(logger, T.untyped)
  @instrumenter = T.let(instrumenter, T.untyped)
  @results = T.let({}, T::Hash[Symbol, Fetcher::Result])
end

Instance Attribute Details

#cache ⇒ PayloadCache? (readonly)

nil turns payload caching off -- see #initialize.

Returns:

  • (PayloadCache, nil) 


80
81
82
 
# File 'lib/active_sanction/sources/base.rb', line 80

def cache
  @cache
end

#fetcher ⇒ Fetcher (readonly)

Returns:

  • (Fetcher) 


76
77
78
 
# File 'lib/active_sanction/sources/base.rb', line 76

def fetcher
  @fetcher
end

#instrumenter ⇒ T.untyped (readonly)

Where this adapter's :parse event goes, or nil for nothing listening. See Instrumentation.

Returns:

  • (T.untyped) 


89
90
91
 
# File 'lib/active_sanction/sources/base.rb', line 89

def instrumenter
  @instrumenter
end

#logger ⇒ T.untyped (readonly)

Anything Logger-shaped, or nil, as Configuration#logger has it.

Returns:

  • (T.untyped) 


84
85
86
 
# File 'lib/active_sanction/sources/base.rb', line 84

def logger
  @logger
end

Class Method Details

.published_remarks(remarks) ⇒ String?

A remark with everything this adapter appended stripped back off -- the publisher's own words and nothing else. Inherited, so it reads the same for every source and a caller does not have to know which list a remark came from before it can strip one. See Sources::Remarks.

Parameters:

  • remarks (T.untyped)

Returns:

  • (String, nil) 


140
 
# File 'lib/active_sanction/sources/base.rb', line 140

def self.published_remarks(remarks) = Remarks.published(remarks)

Instance Method Details

#authority ⇒ String

Returns:

  • (String) 


115
 
# File 'lib/active_sanction/sources/base.rb', line 115

def authority = self.class.authority

#column_shapes(_raw) ⇒ Array<Parsers::ColumnShape::Tally>

The hook #column_tallies dispatches to, handed exactly what #parse is handed. Overridden by an adapter over a positional file.

Parameters:

  • _raw (T.untyped)

Returns:

  • (Array<Parsers::ColumnShape::Tally>) 


233
 
# File 'lib/active_sanction/sources/base.rb', line 233

def column_shapes(_raw) = []

#column_tallies(payloads = nil, **files) ⇒ Array<Parsers::ColumnShape::Tally>

The positional-column assertions this source's raw files satisfy, or do not. Takes what #retrieve returned, or what #snapshot would be given, and answers with one Parsers::ColumnShape::Tally per declared column.

Empty here, because most publishers ship a file that names its own fields and a named field cannot be quietly swapped with the one beside it. An adapter over a headerless file overrides #column_shapes -- see Sources::Ofac, and Parsers::ColumnShape for why a declared width is not enough on its own.

Parameters:

  • payloads (T.untyped) (defaults to: nil)
  • files (T.untyped)

Returns:

  • (Array<Parsers::ColumnShape::Tally>) 


226
227
228
 
# File 'lib/active_sanction/sources/base.rb', line 226

def column_tallies(payloads = nil, **files)
  column_shapes(parse_argument(payloads || files))
end

#file_key(name) ⇒ Symbol

Parameters:

  • name (T.untyped)

Returns:

  • (Symbol) 


127
 
# File 'lib/active_sanction/sources/base.rb', line 127

def file_key(name) = self.class.file_key(name)

#floors ⇒ Hash{Symbol => Numeric}

The lower bounds this list is held to when there is nothing to compare it against. See Definition#floor, and Doctor, which is the only thing that reads them.

Returns:

  • (Hash{Symbol => Numeric}) 


133
 
# File 'lib/active_sanction/sources/base.rb', line 133

def floors = self.class.floors

#format ⇒ Symbol?

Returns:

  • (Symbol, nil) 


118
 
# File 'lib/active_sanction/sources/base.rb', line 118

def format = self.class.format

#fresh? ⇒ Boolean

Returns:

  • (Boolean) 


242
 
# File 'lib/active_sanction/sources/base.rb', line 242

def fresh? = !stale?

#inspect ⇒ String

Returns:

  • (String) 


252
253
254
255
 
# File 'lib/active_sanction/sources/base.rb', line 252

def inspect
  name = self.class.declared?(:key) ? key : "(no key)"
  "#<#{self.class} #{name} #{urls.size} url(s)>"
end

#jurisdiction ⇒ Symbol

Returns:

  • (Symbol) 


112
 
# File 'lib/active_sanction/sources/base.rb', line 112

def jurisdiction = self.class.jurisdiction

#key ⇒ Symbol

Returns:

  • (Symbol) 


109
 
# File 'lib/active_sanction/sources/base.rb', line 109

def key = self.class.key

#parse(_raw) ⇒ Array<Entity>

The one method an adapter must write: bytes in, canonical records out.

Parameters:

  • _raw (T.untyped)

Returns:

Raises:



144
145
146
147
 
# File 'lib/active_sanction/sources/base.rb', line 144

def parse(_raw)
  raise UnsupportedError,
        "#{self.class} must implement #parse(raw) and return an Array of ActiveSanction::Entity"
end

#parse_argument(payloads) ⇒ T.untyped

What #parse is handed, worked out from what #retrieve returned: the bytes for a source declaring one file, the Hash keyed by declaration name for one declaring several. Public because a caller that has already fetched -- Doctor, which parses and then reads the same payloads a second time -- has to be able to produce the same argument without knowing how many files this source declares.

Parameters:

  • payloads (T.untyped)

Returns:

  • (T.untyped) 


264
265
266
267
268
 
# File 'lib/active_sanction/sources/base.rb', line 264

def parse_argument(payloads)
  return payloads if payloads.is_a?(String)

  self.class.multi_url? ? payloads.to_h : payloads.to_h.values.first
end

#retrieve(force: false) ⇒ Hash{Symbol => T.untyped}?

Every declared file, conditionally: a Hash of name => bytes, or nil when the publisher answered 304 for all of them.

A file that came back unchanged is served from the payload cache, so a sync in which one of OFAC's three files moved downloads one file and not three. If the cache has nothing to serve -- a first run against a store that already has validators, a cache directory a user deleted -- that file alone is re-fetched in full.

Parameters:

  • force (Boolean) (defaults to: false)

Returns:

  • (Hash{Symbol => T.untyped}, nil) 


205
206
207
208
209
210
211
212
213
214
 
# File 'lib/active_sanction/sources/base.rb', line 205

def retrieve(force: false)
  raise DeclarationError, "#{self.class} declares no URL to retrieve" if urls.empty?

  @results = urls.to_h { |name, address| [name, fetch_file(name, address, force)] }
  return nil if @results.each_value.all?(&:unchanged?)

  @results.keys.to_h { |name| [name, payload(name)] }
rescue ActiveSanction::Error => e
  raise e.in_source(declared_key)
end

#snapshot(payloads = nil, **files) ⇒ Snapshot

Parses payloads already in hand into a Snapshot. What #sync calls, and what an adapter's own spec calls with a fixture and no network:

source.snapshot(main: File.read("spec/fixtures/un_consolidated.xml"))

The files may be named as keywords, as above, or passed as one Hash -- or, for a source that declares a single file, as the bytes themselves.

Parameters:

  • payloads (T.untyped) (defaults to: nil)
  • files (T.untyped)

Returns:



182
183
184
185
186
187
188
189
190
191
192
193
194
 
# File 'lib/active_sanction/sources/base.rb', line 182

def snapshot(payloads = nil, **files)
  raw = parse_argument(payloads || files)
  entities = Instrumentation.instrument(instrumenter, :parse,
                                        { source: declared_key, bytes: byte_count(raw) }) do |event|
    parsed = parse(raw)
    event[:records] = parsed.size
    event[:warnings] = warnings.size
    parsed
  end
  Snapshot.new(source: key, entities: entities, fetched_at: Time.now.utc, source_version: source_version)
rescue ActiveSanction::Error => e
  raise e.in_source(declared_key)
end

#source_version ⇒ String?

The publisher's own marker for the version just fetched. Last-Modified is the only one every launch source serves; an adapter whose document carries a generation date inside it should override this and say so, because that is the string an examiner will recognise.

Returns:

  • (String, nil) 


249
 
# File 'lib/active_sanction/sources/base.rb', line 249

def source_version = @results.values.first&.last_modified

#stale? ⇒ Boolean

Whether any of this source's files is due a fetch, answered locally and without a request. See Fetcher#stale? for what that does and does not claim.

Returns:

  • (Boolean) 


239
 
# File 'lib/active_sanction/sources/base.rb', line 239

def stale? = urls.any? { |name, address| fetcher.stale?(file_key(name), url: address) }

#sync(force: false) ⇒ Snapshot?

Fetches, parses, and checksums -- or returns nil when the publisher says nothing has changed, which is the outcome to expect on most runs and the reason conditional GET exists.

Parameters:

  • force (Boolean) (defaults to: false)

Returns:



167
168
169
170
171
172
 
# File 'lib/active_sanction/sources/base.rb', line 167

def sync(force: false)
  payloads = retrieve(force: force)
  return nil if payloads.nil?

  snapshot(payloads)
end

#url(name = nil) ⇒ String

Parameters:

  • name (T.untyped) (defaults to: nil)

Returns:

  • (String) 


124
 
# File 'lib/active_sanction/sources/base.rb', line 124

def url(name = nil) = name.nil? ? self.class.url : self.class.url(name)

#urls ⇒ Hash{Symbol => String}

Returns:

  • (Hash{Symbol => String}) 


121
 
# File 'lib/active_sanction/sources/base.rb', line 121

def urls = self.class.urls

#warnings ⇒ Array<Parsers::Warning>

What the last #parse could not read: a Parsers::Warning per row that was skipped or could not be mapped, kept rather than raised. Every shipped adapter overrides this with the parser's own warnings plus whatever it noticed itself, which is what the adapter rules require of a new one.

Empty here rather than abstract, because an adapter that genuinely cannot fail to read a row should not have to say so, and because the :parse event counts these for every source and a count that is sometimes a NoMethodError is not a metric. See Doctor, which reads the warnings themselves rather than the count.

Returns:



161
 
# File 'lib/active_sanction/sources/base.rb', line 161

def warnings = []