Class: ActiveSanction::Parsers::DelimitedTable

Inherits:
Object
  • Object
show all
Extended by:
T::Sig
Defined in:
lib/active_sanction/parsers/delimited_table.rb,
lib/active_sanction/parsers/delimited_table/row.rb,
lib/active_sanction/parsers/delimited_table/reader.rb

Overview

Reads a record-oriented delimited list -- CSV, TSV, anything stdlib CSV can be told about -- into rows an adapter can map onto Entities.

A table is a description of the file, built once and reused for every sync; a Reader is one pass over one payload.

SDN = ActiveSanction::Parsers::DelimitedTable.new(
columns:  %i[ent_num sdn_name sdn_type program title remarks],
null:     "-0-",
encoding: Encoding::WINDOWS_1252
)

SDN.read(bytes).each { |row| row[:sdn_name] }

Headerless files, and why columns are declared

OFAC ships all three of its files with no header row, so the names have to come from somewhere. Declaring them in the adapter also pins the file's shape: if OFAC inserts a column, rows arrive the wrong width and every one of them says so in #warnings, which is a far better failure than 19,321 entities quietly built from shifted fields.

A file that does carry a header is read with columns: nil, and the names come from its first row.

The null sentinel

OFAC does not leave a field empty; it writes -0- , with a trailing space, and it does this roughly a quarter of a million times:

36,"AEROCARIBBEAN AIRLINES",-0- ,"CUBA",-0- ,-0- ,...

Any declared sentinel is matched after stripping surrounding whitespace, and a field that is empty or all whitespace is nil as well -- a list that uses both conventions in one file (they all do) should not make an adapter check for both.

Defined Under Namespace

Classes: Row

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(columns: nil, null: nil, col_sep: ",", quote_char: '"', encoding: DEFAULT_ENCODING, liberal_parsing: true) ⇒ void

liberal_parsing is on by default because these files are published, not validated: an unescaped quote inside a company name is common enough in OFAC and UK OFSI data that failing the row is the wrong default. Turn it off for a source where a stray quote should be loud.

Parameters:

  • columns (T.untyped) (defaults to: nil)
  • null (T.untyped) (defaults to: nil)
  • col_sep (T.untyped) (defaults to: ",")
  • quote_char (T.untyped) (defaults to: '"')
  • encoding (T.untyped) (defaults to: DEFAULT_ENCODING)
  • liberal_parsing (Boolean) (defaults to: true) 


84
85
86
87
88
89
90
91
92
93
 
# File 'lib/active_sanction/parsers/delimited_table.rb', line 84

def initialize(columns: nil, null: nil, col_sep: ",", quote_char: '"',
               encoding: DEFAULT_ENCODING, liberal_parsing: true)
  @columns = T.let(columns!(columns), T.nilable(T::Array[Symbol]))
  @nulls = T.let(nulls!(null), T::Array[String])
  @col_sep = T.let(col_sep.to_s, String)
  @quote_char = T.let(quote_char.to_s, String)
  @encoding = T.let(encoding!(encoding), Encoding)
  @liberal_parsing = T.let(liberal_parsing, T::Boolean)
  freeze
end

Instance Attribute Details

#col_sep ⇒ String (readonly)

Returns:

  • (String) 


65
66
67
 
# File 'lib/active_sanction/parsers/delimited_table.rb', line 65

def col_sep
  @col_sep
end

#columns ⇒ Array<Symbol>? (readonly)

nil where the file names its own columns -- see #headers?.

Returns:

  • (Array<Symbol>, nil) 


62
63
64
 
# File 'lib/active_sanction/parsers/delimited_table.rb', line 62

def columns
  @columns
end

#encoding ⇒ Encoding (readonly)

Returns:

  • (Encoding) 


74
75
76
 
# File 'lib/active_sanction/parsers/delimited_table.rb', line 74

def encoding
  @encoding
end

#nulls ⇒ Array<String> (readonly)

Returns:

  • (Array<String>) 


71
72
73
 
# File 'lib/active_sanction/parsers/delimited_table.rb', line 71

def nulls
  @nulls
end

#quote_char ⇒ String (readonly)

Returns:

  • (String) 


68
69
70
 
# File 'lib/active_sanction/parsers/delimited_table.rb', line 68

def quote_char
  @quote_char
end

Instance Method Details

#coerce(names, values) ⇒ Hash{Symbol => String, nil}

Zips a row's values against the column names. Extra values are dropped and missing ones are nil; the Reader has already warned about both.

Parameters:

  • names (Array<Symbol>)
  • values (Array<T.untyped>)

Returns:

  • (Hash{Symbol => String, nil}) 


107
108
109
 
# File 'lib/active_sanction/parsers/delimited_table.rb', line 107

def coerce(names, values)
  names.each_with_index.to_h { |name, index| [name, value(values[index])] }.freeze
end

#col_sep_name ⇒ String

What to call this format in an error message, so a complaint about OFAC's file says "CSV" rather than "delimited text".

Returns:

  • (String) 


120
 
# File 'lib/active_sanction/parsers/delimited_table.rb', line 120

def col_sep_name = SEPARATOR_NAMES.fetch(col_sep, "delimited text")

#csv_options ⇒ Hash{Symbol => T.untyped}

Returns:

  • (Hash{Symbol => T.untyped}) 


112
113
114
115
 
# File 'lib/active_sanction/parsers/delimited_table.rb', line 112

def csv_options
  { col_sep: col_sep, quote_char: quote_char, headers: false,
    liberal_parsing: @liberal_parsing, skip_blanks: true }
end

#headers? ⇒ Boolean

Whether the file names its own columns.

Returns:

  • (Boolean) 


102
 
# File 'lib/active_sanction/parsers/delimited_table.rb', line 102

def headers? = columns.nil?

#inspect ⇒ String

Returns:

  • (String) 


123
124
125
126
127
 
# File 'lib/active_sanction/parsers/delimited_table.rb', line 123

def inspect
  declared = columns
  shape = declared.nil? ? "headers from file" : "#{declared.size} columns"
  "#<#{self.class} #{col_sep_name} #{shape}#{" null=#{nulls.first.inspect}" if nulls.any?}>"
end

#read(payload) ⇒ Reader

A pass over one payload. Takes the bytes as a String, which is what Sources::Base hands #parse.

Parameters:

  • payload (T.untyped)

Returns:

  • (Reader) 


98
 
# File 'lib/active_sanction/parsers/delimited_table.rb', line 98

def read(payload) = Reader.new(table: self, payload: payload)