Class: ActiveSanction::Doctor

Inherits:
Object
  • Object
show all
Extended by:
T::Sig
Defined in:
lib/active_sanction/doctor.rb,
lib/active_sanction/doctor/report.rb,
lib/active_sanction/doctor/checkup.rb,
lib/active_sanction/doctor/finding.rb,
lib/active_sanction/doctor/profile.rb,
lib/active_sanction/doctor/diagnosis.rb

Overview

Diagnoses whether a source's format has drifted: fetches each list, parses it, measures it, and compares the measurements against the last version that was stored.

report = ActiveSanction.doctor              # every configured source
report = ActiveSanction.doctor(:ofac_sdn)   # one

report.ok?        # => false
report.findings   # => [Finding(source:, severity:, check:, message:, observed:, baseline:)]
exit report.exit_code

The failure this exists to catch

Sanctions lists change format on three clocks. A whole-format migration is announced years ahead and fails loudly. A column added or an element renamed happens quietly, in months. A new document label or a new designation vocabulary happens continuously, weekly.

Only the first of those fails loudly. The dangerous ones are the changes where the file still parses cleanly and means something different: 19,321 entities carrying zero passports looks exactly as healthy as 19,321 carrying 23,429 if all anyone counts is records. Nothing in a sync would notice that for months, and a screening run against it returns a clean result for a customer whose passport is on the list.

So this measures what a sync does not: the share of records carrying each field, the vocabulary the parser recognized, the shape of the values in a positional column, the classes of warning the parse produced. See Profile for what is measured and Checkup for what is made of it.

It never writes anything

Not the snapshot, not the payload cache, not the conditional-GET validators. Each adapter the doctor builds gets a fetcher over an in-memory validator store and no payload cache, which has two consequences worth stating:

  • every run downloads every list in full, because a diagnosis of a list the publisher answered 304 for is a diagnosis of nothing; and
  • a doctor run before a sync cannot make that sync skip a changed list. Sharing the validators would do exactly that -- the doctor's fetch would learn the new ETag, the sync that followed would be answered 304, and the list it decided was unchanged would be the one the doctor had just seen change. Diagnosing a source must not be able to stop it being updated.

Nothing is repaired either. Deciding that a 40% drop in record count is a delisting wave rather than a broken parse is a judgment call, and making it automatically is how a compliance tool ends up quietly screening against nothing.

It is not a cheap run, and it is not meant to be: every list is downloaded, parsed, and parsed again where a positional file's columns are asserted, and the stored snapshot is read in full so its fill rates can be recomputed as the baseline. That is the price of comparing two parses rather than two file sizes, and it is charged once a night rather than once a sync.

One source failing does not stop the others

The same rule sync orchestration runs under, and for the same reason: government endpoints go down, and a UN outage must not stop OFAC being diagnosed. Each source runs inside its own rescue and a failure becomes an error finding on that source alone.

Where this is meant to run

In a nightly job, not in a terminal. A doctor invoked by hand only confirms a regression that was already suspected; the whole value here is noticing one nobody suspected, which means something has to run it when nobody is looking and alert when it says something. exit_code is for the cron job, to_h is for the metrics pipeline, and to_s is for the CLI verb (#36) that will print it.

Defined Under Namespace

Classes: Checkup, Diagnosis, Finding, Profile, Report

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(sources: nil, store: nil, baseline: nil, tolerance: nil, logger: ActiveSanction.config.logger) ⇒ void

sources: takes source keys, adapter classes, adapter instances, or nil for whatever config.sources names.

baseline: is what a previous run measured -- a Doctor::Report, or a Hash of source to Profile -- for the checks a stored snapshot cannot supply. Everything derived from the entities is recomputed from what is in storage and needs nothing passed here; the warning classes and the free-text coverage exist only during a parse, so a host that wants those compared week to week keeps the last report and hands it back:

yesterday = JSON.parse(File.read("doctor.json"))
report = ActiveSanction.doctor(baseline: Doctor::Report.from_h(yesterday))
File.write("doctor.json", JSON.generate(report.to_h))

A supplied profile is used only where it describes the same list version that is in storage; where it does not, storage wins, because a profile from three syncs ago would report drift that has already been reviewed.

Parameters:

  • sources (T.untyped) (defaults to: nil)
  • store (T.untyped) (defaults to: nil)
  • baseline (T.untyped) (defaults to: nil)
  • tolerance (T.untyped) (defaults to: nil)
  • logger (T.untyped) (defaults to: ActiveSanction.config.logger) 


134
135
136
137
138
139
140
141
142
143
 
# File 'lib/active_sanction/doctor.rb', line 134

def initialize(sources: nil, store: nil, baseline: nil, tolerance: nil, logger: ActiveSanction.config.logger)
  @sources = T.let(resolve(sources), T::Array[T.untyped])
  @keys = T.let(@sources.map { |source| Sources::Definition.key!(source.key) }, T::Array[Symbol])
  @store = T.let(store || ActiveSanction.storage, T.untyped)
  @recorded = T.let(baselines!(baseline), T::Hash[Symbol, Profile])
  @tolerance = T.let(
    Configuration.doctor_tolerance!(tolerance || ActiveSanction.config.doctor_tolerance), Float
  )
  @logger = T.let(logger, T.untyped)
end

Instance Attribute Details

#keys ⇒ Array<Symbol> (readonly)

Returns:

  • (Array<Symbol>) 


97
98
99
 
# File 'lib/active_sanction/doctor.rb', line 97

def keys
  @keys
end

#logger ⇒ T.untyped (readonly)

Returns:

  • (T.untyped) 


108
109
110
 
# File 'lib/active_sanction/doctor.rb', line 108

def logger
  @logger
end

#sources ⇒ Array<T.untyped> (readonly)

The adapters this run covers.

Returns:

  • (Array<T.untyped>) 


94
95
96
 
# File 'lib/active_sanction/doctor.rb', line 94

def sources
  @sources
end

#store ⇒ T.untyped (readonly)

Returns:

  • (T.untyped) 


100
101
102
 
# File 'lib/active_sanction/doctor.rb', line 100

def store
  @store
end

#tolerance ⇒ Float (readonly)

How far a measurement may move from its baseline before it is worth a finding, as a share of what it was.

Returns:

  • (Float) 


105
106
107
 
# File 'lib/active_sanction/doctor.rb', line 105

def tolerance
  @tolerance
end

Class Method Details

.call(**options, &block) ⇒ Report

Parameters:

  • options (T.untyped)
  • block (T.untyped)

Returns:



111
 
# File 'lib/active_sanction/doctor.rb', line 111

def self.call(**options, &block) = T.unsafe(self).new(**options).call(&block)

Instance Method Details

#call(&block) ⇒ Report

Runs the diagnosis and returns the Report. Never raises for a source that could not be read -- that is what an error finding is for.

The optional block is the progress hook: it is called with each Diagnosis as that source finishes. Sources are diagnosed one at a time, because this is a job nobody is waiting on and downloading four government lists at once to save four minutes of it is not a trade worth making.

Parameters:

  • block (T.proc.params(diagnosis: Diagnosis).void, nil)

Returns:



153
154
155
156
157
158
159
160
161
162
163
164
165
166
 
# File 'lib/active_sanction/doctor.rb', line 153

def call(&block)
  started_at = Time.now.utc
  began = monotonic
  log(:info, "diagnosing #{keys.size} source(s): #{keys.join(", ")}")
  diagnoses = keys.each_with_index.map do |key, at|
    diagnose(key, sources.fetch(at)).tap do |diagnosis|
      log_diagnosis(diagnosis)
      block&.call(diagnosis)
    end
  end
  report = Report.new(diagnoses: diagnoses, started_at: started_at, duration: elapsed(began))
  log(report.ok? ? :info : :warn, "diagnosed #{report.summary}")
  report
end

#inspect ⇒ String

Returns:

  • (String) 


169
 
# File 'lib/active_sanction/doctor.rb', line 169

def inspect = "#<#{self.class} #{keys.join(", ")} tolerance=#{tolerance}>"