Class: ActiveSanction::Diff

Inherits:
Object
  • Object
show all
Extended by:
T::Sig
Defined in:
lib/active_sanction/diff.rb,
lib/active_sanction/diff/change.rb

Overview

What changed between two snapshots of one source.

diff = ActiveSanction.diff(:ofac_sdn, from: last_months_snapshot, to: todays_snapshot)

diff.added      # => [Entity], newly listed
diff.removed    # => [Entity], delisted
diff.modified   # => [Diff::Change], amended, with the fields that moved
diff.changed    # => [Entity], what a consuming service should re-screen against
puts diff       # => the summary below

Why this exists

Screening is not a one-time event. A customer cleared last month may be listed today, and the obligation is to notice. Re-running an entire book of business against an entire list every night is how most services answer that, and it is why most services answer it weekly instead. A diff turns the nightly job into "screen everyone against the eleven records that moved", which is a job small enough to run every time a list is synced.

Delistings matter as much as listings, and they are the half a re-screen against added records only would miss: a delisting is what lets a customer back through the door, and a service that never notices one goes on blocking somebody the government stopped sanctioning in March.

It rests on ids being stable

The two snapshots are joined by entity id, so an amendment reports as one modification rather than as a delisting and a new listing. That only holds while a record keeps its id between syncs, which is why the adapter conformance group asserts id stability (#16) and why the Canada adapter (#22) hashes a citation and a name into a synthetic one. Ids that move would make every sync look like a full replacement, and a diff full of delistings that did not happen is worse than no diff at all.

A first sync is a baseline, not 19,015 new listings

With no previous snapshot there is nothing to compare, and reporting the whole list as added would be false: those records were not listed today, they were listed over twenty years and we are only now looking. So a diff with no from is a baseline -- added, removed and modified are all empty, baseline? is true, and changed is empty because the right response to a first sync is a deliberate full screening run rather than one driven by a diff that is really a list.

Computed here rather than taken from a publisher

OFAC serves a delta feed of its own at /changes/latest. This does not read it, and the reason is that a diff has to describe the two list versions we hold: a publisher's delta describes the change between two versions it chose, and a run that skipped a day, or held a stale list because a fetch failed (#34), is not on either end of it. Cross-checking a computed diff against that feed is worth doing -- it is how a parser regression that quietly drops records gets caught -- but it belongs in the OFAC adapter, as one publisher's answer to a question every source has to answer, rather than in the general shape of a diff.

The output

ofac_sdn: 19015 -> 19023 records, 12 added, 4 removed, 5 modified (0.1% of the previous list)
+ ofac_sdn:41234  IVANOV, Ivan Ivanovich  [SDGT]
- ofac_sdn:2674   ABBAS, Abu  [SDGT]
~ ofac_sdn:36     names +1, programs +1

That is diff.to_s, and #summary is its first line on its own -- the one a sync wrapper logs. There is no CLI to print either from, #36 being closed as not planned, so the human-readable form is a method on the object and a rake task or a scheduled job prints it. #to_h is the machine-readable form of the same thing, JSON-ready and carrying the two snapshots' checksums so a diff says which pair of list versions produced it.

There is no .from_h: a diff is derived rather than stored, and those two checksums are what makes it reproducible -- keep them and the diff can always be computed again, keep the diff and you have a copy of an answer nobody can check.

Instances are frozen on construction and compare by value.

Defined Under Namespace

Classes: Change

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(to:, from: nil, source: nil) ⇒ void

from: is nil for a first sync, which is a baseline rather than a list of additions -- see the class comment. source: is optional and is checked against the snapshots rather than trusted, since a diff of the wrong pair of lists reports every record on both as having moved.

Parameters:

  • to (T.untyped)
  • from (T.untyped) (defaults to: nil)
  • source (T.untyped) (defaults to: nil) 


156
157
158
159
160
161
162
163
164
165
166
167
 
# File 'lib/active_sanction/diff.rb', line 156

def initialize(to:, from: nil, source: nil)
  current = snapshot!(:to, to)
  previous = from.nil? ? nil : snapshot!(:from, from)
  @source = T.let(source!(source, previous, current), Symbol)
  @from = T.let(previous && Storage::Meta.from_snapshot(previous), T.nilable(Storage::Meta))
  @to = T.let(Storage::Meta.from_snapshot(current), Storage::Meta)
  added, removed, modified = compare(previous, current)
  @added = T.let(added, T::Array[T.untyped])
  @removed = T.let(removed, T::Array[T.untyped])
  @modified = T.let(modified, T::Array[Change])
  freeze
end

Instance Attribute Details

#added ⇒ Array<T.untyped> (readonly)

Newly listed: on the new snapshot, not on the old, sorted by id.

Returns:

  • (Array<T.untyped>) 


106
107
108
 
# File 'lib/active_sanction/diff.rb', line 106

def added
  @added
end

#from ⇒ Storage::Meta? (readonly)

What the old snapshot was: fetched_at, checksum, record_count. Nil for a baseline. The whole snapshot is deliberately not held -- a diff of eleven records would otherwise pin two lists and tens of megabytes of entities in memory for as long as anything holds it.

Returns:



121
122
123
 
# File 'lib/active_sanction/diff.rb', line 121

def from
  @from
end

#modified ⇒ Array<Change> (readonly)

Amended, with the fields that moved. See Diff::Change.

Returns:



114
115
116
 
# File 'lib/active_sanction/diff.rb', line 114

def modified
  @modified
end

#removed ⇒ Array<T.untyped> (readonly)

Delisted: on the old snapshot, not on the new.

Returns:

  • (Array<T.untyped>) 


110
111
112
 
# File 'lib/active_sanction/diff.rb', line 110

def removed
  @removed
end

#source ⇒ Symbol (readonly)

Returns:

  • (Symbol) 


102
103
104
 
# File 'lib/active_sanction/diff.rb', line 102

def source
  @source
end

#to ⇒ Storage::Meta (readonly)

What the new snapshot is, which is what screening runs against now.

Returns:



125
126
127
 
# File 'lib/active_sanction/diff.rb', line 125

def to
  @to
end

Class Method Details

.call(source = nil, from: nil, to: nil, store: nil) ⇒ T.attached_class

Sugar, and what ActiveSanction.diff calls:

ActiveSanction.diff(:ofac_sdn, from: old, to: new)
ActiveSanction.diff(:ofac_sdn, from: old)   # `to:` is what is stored now
ActiveSanction.diff(from: old, to: new)     # the source comes from the snapshots

to: defaults to the stored snapshot because that is what a re-screen is about to run against, and it raises rather than defaulting to nothing: diffing against a list that is not there would report every record on it as delisted, which is a clean report for every customer on it.

Parameters:

  • source (T.untyped) (defaults to: nil)
  • from (T.untyped) (defaults to: nil)
  • to (T.untyped) (defaults to: nil)
  • store (T.untyped) (defaults to: nil)

Returns:

  • (T.attached_class) 


138
139
140
141
 
# File 'lib/active_sanction/diff.rb', line 138

def self.call(source = nil, from: nil, to: nil, store: nil)
  key = source.nil? ? nil : Sources::Definition.key!(source)
  new(source: key, from: from, to: to || stored!(key, store))
end

Instance Method Details

#==(other) ⇒ Boolean Also known as: eql?

Parameters:

  • other (T.untyped)

Returns:

  • (Boolean) 


252
253
254
255
256
 
# File 'lib/active_sanction/diff.rb', line 252

def ==(other)
  return false unless other.instance_of?(self.class)

  to_h == other.to_h
end

#any? ⇒ Boolean

Returns:

  • (Boolean) 


178
 
# File 'lib/active_sanction/diff.rb', line 178

def any? = !empty?

#baseline? ⇒ Boolean

No previous snapshot: the source was synced for the first time, and this says what it holds rather than claiming every record on it is new.

Returns:

  • (Boolean) 


172
 
# File 'lib/active_sanction/diff.rb', line 172

def baseline? = from.nil?

#changed ⇒ Array<T.untyped>

What a consuming service should re-screen its book against: the records that are on the list now and were not, or were not the same.

Delistings are deliberately not in here -- they are not something to screen against, they are records to clear existing alerts on, which is a different job done from removed. And nothing here judges a change too small to matter: a corrected passport number and a reworded remark reach the scorer through different paths, and a library that decided on a host's behalf which amendments were worth re-screening would be deciding which sanctions hits it is willing to miss.

Returns:

  • (Array<T.untyped>) 


195
 
# File 'lib/active_sanction/diff.rb', line 195

def changed = added + modified.map(&:entity)

#churn ⇒ Float?

How much of the previous list moved, as a fraction. The number to alert on: two consecutive syncs of a live sanctions list move a fraction of a percent, so a diff that says a third of the list changed is a parse regression, an id scheme that shifted, or a publisher who reissued the file -- and all three are things to look at before re-screening anybody against the result. Nil for a baseline, and for a previous list that was empty.

Returns:

  • (Float, nil) 


205
206
207
208
209
210
 
# File 'lib/active_sanction/diff.rb', line 205

def churn
  count = from&.record_count
  return nil if count.nil? || count.zero?

  (size.to_f / count).round(6).to_f
end

#details(limit: nil) ⇒ Array<String>

One line per record that moved, marked +, - or ~. limit: caps how many are returned and adds a line saying how many were not; nil returns every one, which is what a consumer writing a report wants.

Parameters:

  • limit (Integer, nil) (defaults to: nil)

Returns:

  • (Array<String>) 


239
240
241
242
243
244
245
246
 
# File 'lib/active_sanction/diff.rb', line 239

def details(limit: nil)
  lines = added.map { |entity| "  + #{label(entity)}" } +
          removed.map { |entity| "  - #{label(entity)}" } +
          modified.map { |change| "  ~ #{change}" }
  return lines if limit.nil? || lines.size <= limit

  lines.first(limit) + ["  ... and #{lines.size - limit} more"]
end

#empty? ⇒ Boolean

Returns:

  • (Boolean) 


175
 
# File 'lib/active_sanction/diff.rb', line 175

def empty? = added.empty? && removed.empty? && modified.empty?

#hash ⇒ Integer

Returns:

  • (Integer) 


260
 
# File 'lib/active_sanction/diff.rb', line 260

def hash = [self.class, to_h].hash

#inspect ⇒ String

Returns:

  • (String) 


263
 
# File 'lib/active_sanction/diff.rb', line 263

def inspect = "#<#{self.class} #{source} +#{added.size} -#{removed.size} ~#{modified.size}>"

#size ⇒ Integer

Records that moved, in either direction.

Returns:

  • (Integer) 


182
 
# File 'lib/active_sanction/diff.rb', line 182

def size = added.size + removed.size + modified.size

#summary ⇒ String

The one line at the top of #to_s, and the line worth logging on its own after a sync.

Returns:

  • (String) 


227
228
229
230
231
232
233
 
# File 'lib/active_sanction/diff.rb', line 227

def summary
  return "#{source}: first snapshot, #{to.record_count} records (baseline, nothing to re-screen)" if baseline?
  return "#{source}: #{to.record_count} records, unchanged" if empty?

  "#{source}: #{T.must(from).record_count} -> #{to.record_count} records, #{added.size} added, " \
    "#{removed.size} removed, #{modified.size} modified#{churn_note}"
end

#to_h ⇒ Hash{Symbol => T.untyped}

Returns:

  • (Hash{Symbol => T.untyped}) 


213
214
215
216
217
218
219
220
221
222
 
# File 'lib/active_sanction/diff.rb', line 213

def to_h
  {
    source: source,
    from: from&.to_h,
    to: to.to_h,
    added: added.map(&:to_h),
    removed: removed.map(&:to_h),
    modified: modified.map(&:to_h)
  }
end

#to_s ⇒ String

Returns:

  • (String) 


249
 
# File 'lib/active_sanction/diff.rb', line 249

def to_s = ([summary] + details(limit: DETAIL_LIMIT)).join("\n")