Class: ActiveSanction::Parsers::ColumnShape

Inherits:
Object
  • Object
show all
Extended by:
T::Sig
Defined in:
lib/active_sanction/parsers/column_shape.rb

Overview

An assertion about what one column of a positional file contains, measured over the whole file.

ENT_NUM = ColumnShape.new(name: :ent_num, matches: /\A\d+\z/, at_least: 0.99)

tally = ENT_NUM.tally(rows.map { |row| row[:ent_num] })
tally.ok?       # => false
tally.ratio     # => 0.0
tally.sample    # => ["AEROCARIBBEAN AIRLINES", "AEROTAXI EJECUTIVO"]
tally.to_s      # => "ent_num numeric on 0.0% of 19321 rows (expected 99%)"

Why a declared width is not enough

A file that names its own columns cannot have them quietly swapped: the header moves with the data and an adapter reading sdn_type still gets the type. A headerless file has no such protection, and OFAC ships three of them. Declaring the column names pins the width, so a column inserted upstream arrives as a wrong-width row and every row says so -- but a column reordered upstream keeps the width, parses cleanly, and produces 19,321 entities built from shifted fields. Nothing raises, nothing warns, and the list means something different.

So the shape of the values is asserted separately from the shape of the row. ent_num is a number on essentially every row of OFAC's file, and a version of that file where it is a company name is not a version this library should screen against.

at_least, rather than "every row"

These are published files, not validated ones. A single row where a publisher typed a letter into a numeric column is a curiosity; a file where a third of them are is a format change. The threshold is what separates the two, and it defaults to 99% -- high enough that a real swap cannot hide under it, loose enough that one bad row does not stop a sync being diagnosed as healthy.

Blank values are not counted at all. A column the publisher leaves empty is saying nothing about its shape, and OFAC leaves most of its columns empty most of the time -- -0- roughly a quarter of a million times across the six files. Counting those as failures would make every assertion about an optional column fail on the day it was written.

Instances are frozen on construction.

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(name:, matches: nil, allowing: nil, satisfying: nil, at_least: DEFAULT_AT_LEAST, description: nil) ⇒ void

One of matches: (a Regexp), allowing: (the values the column may hold, compared case-insensitively after stripping), or satisfying: (a callable taking the value and returning truthy). description: names the expectation in the failure message; the first two derive a readable one when it is not given.

Parameters:

  • name (T.untyped)
  • matches (T.untyped) (defaults to: nil)
  • allowing (T.untyped) (defaults to: nil)
  • satisfying (T.untyped) (defaults to: nil)
  • at_least (T.untyped) (defaults to: DEFAULT_AT_LEAST)
  • description (T.untyped) (defaults to: nil) 


87
88
89
90
91
92
93
94
95
 
# File 'lib/active_sanction/parsers/column_shape.rb', line 87

def initialize(name:, matches: nil, allowing: nil, satisfying: nil, at_least: DEFAULT_AT_LEAST,
               description: nil)
  @name = T.let(name!(name), Symbol)
  @allowed = T.let(allowing.nil? ? nil : allowed!(allowing), T.nilable(T::Array[String]))
  @rule = T.let(rule!(matches, satisfying), T.nilable(T.proc.params(value: String).returns(T.untyped)))
  @at_least = T.let(at_least!(at_least), Float)
  @description = T.let(description!(description, matches), String)
  freeze
end

Instance Attribute Details

#at_least ⇒ Float (readonly)

The share of non-blank values that must satisfy the rule.

Returns:

  • (Float) 


71
72
73
 
# File 'lib/active_sanction/parsers/column_shape.rb', line 71

def at_least
  @at_least
end

#description ⇒ String (readonly)

What the column is supposed to hold, in words, for the message a failure prints: "numeric", "a known SDN_Type".

Returns:

  • (String) 


76
77
78
 
# File 'lib/active_sanction/parsers/column_shape.rb', line 76

def description
  @description
end

#name ⇒ Symbol (readonly)

Returns:

  • (Symbol) 


67
68
69
 
# File 'lib/active_sanction/parsers/column_shape.rb', line 67

def name
  @name
end

Class Method Details

.percentage(ratio) ⇒ String

A percentage as a report prints one: no decimal where there is nothing after the point, since "99%" is what was declared and "99.0%" is not.

Parameters:

  • ratio (Float)

Returns:

  • (String) 


137
138
139
140
 
# File 'lib/active_sanction/parsers/column_shape.rb', line 137

def self.percentage(ratio)
  value = (ratio * 100).round(1)
  value == value.to_i ? "#{value.to_i}%" : "#{value}%"
end

Instance Method Details

#inspect ⇒ String

Returns:

  • (String) 


132
 
# File 'lib/active_sanction/parsers/column_shape.rb', line 132

def inspect = "#<#{self.class} #{self}>"

#percentage(ratio) ⇒ String

Parameters:

  • ratio (Float)

Returns:

  • (String) 


143
 
# File 'lib/active_sanction/parsers/column_shape.rb', line 143

def percentage(ratio) = ColumnShape.percentage(ratio)

#satisfied_by?(value) ⇒ Boolean

Whether one value satisfies the assertion. Blanks never reach here -- see #tally.

Parameters:

  • value (String)

Returns:

  • (Boolean) 


100
101
102
103
104
105
 
# File 'lib/active_sanction/parsers/column_shape.rb', line 100

def satisfied_by?(value)
  allowed = @allowed
  return allowed.include?(value.strip.downcase) unless allowed.nil?

  !!T.must(@rule).call(value)
end

#tally(values) ⇒ Tally

Measures the assertion over one file's worth of values, in one pass. Takes anything enumerable, so a caller can hand it a lazy reader rather than materializing 19,321 rows.

Parameters:

  • values (T.untyped)

Returns:

  • (Tally) 


111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
 
# File 'lib/active_sanction/parsers/column_shape.rb', line 111

def tally(values)
  checked = 0
  matched = 0
  blank = 0
  sample = T.let([], T::Array[String])
  values.each do |value|
    string = value.nil? ? "" : value.to_s.strip
    next blank += 1 if string.empty?

    checked += 1
    next matched += 1 if satisfied_by?(string)

    sample << string if sample.size < SAMPLE_SIZE
  end
  Tally.new(shape: self, checked: checked, matched: matched, blank: blank, sample: sample)
end

#to_s ⇒ String

Returns:

  • (String) 


129
 
# File 'lib/active_sanction/parsers/column_shape.rb', line 129

def to_s = "#{name} #{description} on at least #{percentage(at_least)} of rows"