Class: ActiveSanction::Doctor::Profile

Inherits:
Object
  • Object
show all
Extended by:
T::Sig
Defined in:
lib/active_sanction/doctor/profile.rb

Overview

What one parse of one list measured, in numbers small enough to keep.

profile = Doctor::Profile.measure(snapshot, adapter: source)

profile.record_count          # => 19015
profile.cohorts[:individual]  # => 11704
profile.fill[:identifiers]    # => 0.341
profile.remarks_coverage      # => 0.973
profile.warnings              # => { "unknown SDN_Type \"syndicate\"" => 41 }

This is the thing a diagnosis compares. A snapshot is tens of megabytes and a profile of it is a few hundred bytes, so a host that wants to watch a fill rate over ninety days keeps ninety of these and no lists at all.

Fill rates are the check that catches a clean parse of a changed file

Record counts do not move when a publisher renames an element. Fill rates do: 19,321 entities carrying zero passports looks exactly like 19,321 carrying 23,429 if the only thing anyone counts is records, and the first one screens a passport number against nothing. Measuring the share of records that carry each field is what turns that from invisible into a number that halved.

Each field is measured over the records that could have one

A date of birth is measured over individuals, because an organization never has one and including them would make the rate a function of how many companies a designation round happened to name. Everything else is measured over every record: an address, an identifier or a program is something any kind of listed party can carry, and a list where the organizations lost their registration numbers is the same failure as one where the people lost their passports.

Half of a profile survives a stored snapshot, and half does not

Everything derived from the entities -- counts, fill rates -- can be recomputed from a snapshot that was stored months ago, which is what makes the last sync usable as a baseline for free. Everything that falls out of the parse itself -- warnings, orphaned child rows, how much of OFAC's free text was understood, the column tallies -- exists only while the parse is running and is nil in a profile rebuilt from storage. A caller that wants those compared too keeps the profile: see Doctor#baseline.

Instances are frozen on construction and compare by value.

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(source:, record_count:, checksum: nil, cohorts: {}, fill: {}, warnings: nil, orphans: nil, remarks_coverage: nil, unrecognized: nil, columns: []) ⇒ void

Parameters:

  • source (T.untyped)
  • record_count (T.untyped)
  • checksum (T.untyped) (defaults to: nil)
  • cohorts (T.untyped) (defaults to: {})
  • fill (T.untyped) (defaults to: {})
  • warnings (T.untyped) (defaults to: nil)
  • orphans (T.untyped) (defaults to: nil)
  • remarks_coverage (T.untyped) (defaults to: nil)
  • unrecognized (T.untyped) (defaults to: nil)
  • columns (T.untyped) (defaults to: []) 


281
282
283
284
285
286
287
288
289
290
291
292
293
294
 
# File 'lib/active_sanction/doctor/profile.rb', line 281

def initialize(source:, record_count:, checksum: nil, cohorts: {}, fill: {}, warnings: nil,
               orphans: nil, remarks_coverage: nil, unrecognized: nil, columns: [])
  @source = T.let(symbol!(:source, source), Symbol)
  @record_count = T.let(Integer(record_count), Integer)
  @checksum = T.let(string_or_nil(checksum), T.nilable(String))
  @cohorts = T.let(counts!(cohorts), T::Hash[Symbol, Integer])
  @fill = T.let(ratios!(fill), T::Hash[Symbol, Float])
  @warnings = T.let(warnings.nil? ? nil : tally!(warnings), T.nilable(T::Hash[String, Integer]))
  @orphans = T.let(orphans.nil? ? nil : counts!(orphans), T.nilable(T::Hash[Symbol, Integer]))
  @remarks_coverage = T.let(remarks_coverage.nil? ? nil : Float(remarks_coverage), T.nilable(Float))
  @unrecognized = T.let(unrecognized.nil? ? nil : tally!(unrecognized), T.nilable(T::Hash[String, Integer]))
  @columns = T.let(columns!(columns), T::Array[T::Hash[Symbol, T.untyped]])
  freeze
end

Instance Attribute Details

#checksum ⇒ String? (readonly)

The checksum of the snapshot this was measured over, so a stored profile can say which list version it describes.

Returns:

  • (String, nil) 


110
111
112
 
# File 'lib/active_sanction/doctor/profile.rb', line 110

def checksum
  @checksum
end

#cohorts ⇒ Hash{Symbol => Integer} (readonly)

How many records of each type, plus :all.

Returns:

  • (Hash{Symbol => Integer}) 


114
115
116
 
# File 'lib/active_sanction/doctor/profile.rb', line 114

def cohorts
  @cohorts
end

#columns ⇒ Array<Hash{Symbol => T.untyped}> (readonly)

Positional column assertions, as Parsers::ColumnShape::Tally#to_h wrote them. Empty for a source whose file names its own columns.

Returns:

  • (Array<Hash{Symbol => T.untyped}>) 


145
146
147
 
# File 'lib/active_sanction/doctor/profile.rb', line 145

def columns
  @columns
end

#fill ⇒ Hash{Symbol => Float} (readonly)

Field name to the share of its cohort carrying at least one, 0.0 to 1.0.

Returns:

  • (Hash{Symbol => Float}) 


118
119
120
 
# File 'lib/active_sanction/doctor/profile.rb', line 118

def fill
  @fill
end

#orphans ⇒ Hash{Symbol => Integer}? (readonly)

Child rows that matched no entity, by file. A nonzero count means the publisher's files were downloaded at different moments, or that the key they join on has moved.

Returns:

  • (Hash{Symbol => Integer}, nil) 


129
130
131
 
# File 'lib/active_sanction/doctor/profile.rb', line 129

def orphans
  @orphans
end

#record_count ⇒ Integer (readonly)

Returns:

  • (Integer) 


105
106
107
 
# File 'lib/active_sanction/doctor/profile.rb', line 105

def record_count
  @record_count
end

#remarks_coverage ⇒ Float? (readonly)

The share of the publisher's free text the adapter understood, for a source that reads any. nil for one that does not, and for a profile rebuilt from storage.

Returns:

  • (Float, nil) 


135
136
137
 
# File 'lib/active_sanction/doctor/profile.rb', line 135

def remarks_coverage
  @remarks_coverage
end

#source ⇒ Symbol (readonly)

Returns:

  • (Symbol) 


102
103
104
 
# File 'lib/active_sanction/doctor/profile.rb', line 102

def source
  @source
end

#unrecognized ⇒ Hash{String => Integer}? (readonly)

The free-text shapes the parser did not recognize, to how many segments each cost -- what turns a coverage drop into the label that caused it.

Returns:

  • (Hash{String => Integer}, nil) 


140
141
142
 
# File 'lib/active_sanction/doctor/profile.rb', line 140

def unrecognized
  @unrecognized
end

#warnings ⇒ Hash{String => Integer}? (readonly)

Shaped parser warning to how many rows carried it. nil in a profile rebuilt from a stored snapshot -- see the class comment.

Returns:

  • (Hash{String => Integer}, nil) 


123
124
125
 
# File 'lib/active_sanction/doctor/profile.rb', line 123

def warnings
  @warnings
end

Class Method Details

.from_h(hash) ⇒ T.attached_class

Parameters:

  • hash (T.untyped)

Returns:

  • (T.attached_class)

Raises:



162
163
164
165
166
167
168
 
# File 'lib/active_sanction/doctor/profile.rb', line 162

def self.from_h(hash)
  attributes = hash.to_h.transform_keys(&:to_sym)
  unknown = attributes.keys - MEMBERS
  raise InvalidArgument, "unknown Doctor::Profile attribute(s): #{unknown.join(", ")}" if unknown.any?

  T.unsafe(self).new(**attributes)
end

.measure(snapshot, adapter: nil, columns: nil) ⇒ T.attached_class

Measures a parsed snapshot, and everything the adapter that parsed it is willing to say about the parse. adapter is optional: without one this is the entity-derived half, which is exactly what a stored snapshot can supply.

Parameters:

  • snapshot (T.untyped)
  • adapter (T.untyped) (defaults to: nil)
  • columns (T.untyped) (defaults to: nil)

Returns:

  • (T.attached_class) 


152
153
154
155
156
157
158
159
 
# File 'lib/active_sanction/doctor/profile.rb', line 152

def self.measure(snapshot, adapter: nil, columns: nil)
  entities = snapshot.entities
  new(source: snapshot.source, record_count: snapshot.record_count, checksum: snapshot.checksum,
      cohorts: count_cohorts(entities), fill: measure_fill(entities),
      warnings: shape_warnings(adapter), orphans: count_orphans(adapter),
      remarks_coverage: coverage_of(adapter), unrecognized: unrecognized_of(adapter),
      columns: (columns || []).map(&:to_h))
end

.shape(message) ⇒ String

A warning's class, rather than the warning: the message with its digits masked, so "row 4711 has no SDN_Name" and "row 4712 has no SDN_Name" are one complaint counted twice rather than two complaints.

Masking rather than truncating, because what distinguishes one class of warning from another on these lists is usually the value the publisher put in a field -- unknown SDN_Type "syndicate" is a different thing to notice from unknown SDN_Type "trust" -- while what makes two warnings the same complaint is that only their row numbers and identifiers differ.

Parameters:

  • message (String)

Returns:

  • (String) 


271
272
273
274
 
# File 'lib/active_sanction/doctor/profile.rb', line 271

def self.shape(message)
  masked = message.gsub(/\d/, "#").gsub(/\s+/, " ").strip
  -(masked.length > SHAPE_LENGTH ? "#{masked[0, SHAPE_LENGTH]}..." : masked)
end

Instance Method Details

#==(other) ⇒ Boolean Also known as: eql?

Parameters:

  • other (T.untyped)

Returns:

  • (Boolean) 


339
340
341
342
343
 
# File 'lib/active_sanction/doctor/profile.rb', line 339

def ==(other)
  return false unless other.instance_of?(self.class)

  to_h == other.to_h
end

#cohort_name(field) ⇒ String

What to call the records a field was measured over: "individuals", "records".

Parameters:

  • field (Symbol)

Returns:

  • (String) 


304
305
306
307
 
# File 'lib/active_sanction/doctor/profile.rb', line 304

def cohort_name(field)
  cohort = FIELDS.fetch(field, :all)
  COHORT_NAMES.fetch(cohort, cohort.to_s)
end

#cohort_size(field) ⇒ Integer

How many records the field's rate was measured over, for a message that says "12% of 11,704 individuals" rather than "12%".

Parameters:

  • field (Symbol)

Returns:

  • (Integer) 


299
 
# File 'lib/active_sanction/doctor/profile.rb', line 299

def cohort_size(field) = cohorts.fetch(FIELDS.fetch(field, :all), 0)

#empty? ⇒ Boolean

Returns:

  • (Boolean) 


329
 
# File 'lib/active_sanction/doctor/profile.rb', line 329

def empty? = record_count.zero?

#hash ⇒ Integer

Returns:

  • (Integer) 


347
 
# File 'lib/active_sanction/doctor/profile.rb', line 347

def hash = [self.class, to_h].hash

#inspect ⇒ String

Returns:

  • (String) 


350
 
# File 'lib/active_sanction/doctor/profile.rb', line 350

def inspect = "#<#{self.class} #{source} #{record_count} records #{fill.size} fill rate(s)>"

#orphan_count ⇒ Integer

Returns:

  • (Integer) 


326
 
# File 'lib/active_sanction/doctor/profile.rb', line 326

def orphan_count = (orphans || {}).values.sum

#to_h ⇒ Hash{Symbol => T.untyped}

Returns:

  • (Hash{Symbol => T.untyped}) 


332
333
334
335
336
 
# File 'lib/active_sanction/doctor/profile.rb', line 332

def to_h
  { source: source, record_count: record_count, checksum: checksum, cohorts: cohorts, fill: fill,
    warnings: warnings, orphans: orphans, remarks_coverage: remarks_coverage,
    unrecognized: unrecognized, columns: columns }
end

#top_warnings(count = 5) ⇒ Array<T.untyped>

Warning classes this parse produced, most rows first.

Parameters:

  • count (Integer) (defaults to: 5)

Returns:

  • (Array<T.untyped>) 


311
312
313
 
# File 'lib/active_sanction/doctor/profile.rb', line 311

def top_warnings(count = 5)
  (warnings || {}).sort_by { |shape, rows| [-rows, shape] }.first(count)
end

#warning_count ⇒ Integer

Returns:

  • (Integer) 


323
 
# File 'lib/active_sanction/doctor/profile.rb', line 323

def warning_count = (warnings || {}).values.sum

#worst_unrecognized ⇒ Array<T.untyped>?

The unrecognized free-text shape that cost the most, as [shape, count] -- what a coverage finding names as the likely cause.

Returns:

  • (Array<T.untyped>, nil) 


318
319
320
 
# File 'lib/active_sanction/doctor/profile.rb', line 318

def worst_unrecognized
  (unrecognized || {}).max_by { |shape, count| [count, shape] }
end