Class: ActiveSanction::Normalizer::Dictionary

Inherits:
Object
  • Object
show all
Extended by:
T::Sig
Defined in:
lib/active_sanction/normalizer/dictionary.rb,
lib/active_sanction/normalizer/dictionary/stoplist.rb

Overview

The token lists the fold applies per entity type, and the one list that overrides them.

dictionary = ActiveSanction::Normalizer::Dictionary.default
dictionary.stoplist(:organization).reject(%w[rosneft oil company])  # => ["rosneft", "oil"]
dictionary.stoplist(:individual).reject(%w[hajji abdallah])         # => ["abdallah"]

Stage 1b of the matching pipeline. Form settles what a name looks like; this settles which of its tokens carry no identifying information -- LTD on a company, SHAYKH on a person -- so that "Rosneft Oil Company" and "Rosneft" can score as the near-identical pair they are.

Contextual, because the same token means different things

Legal forms are stripped from organizations and honorifics from individuals, and neither is stripped from the other or from a vessel or an aircraft. This is not tidiness. CO is a legal form in "Bank of Kunlun Co Ltd" and the first syllable of a great many personal names; AS is a Norwegian company and an English word. Applying a list to the type it was written for is what keeps it from being a source of false matches everywhere else. A caller that does not know the type says so by passing none, and gets the fold and nothing else.

The preserve list wins

Whatever the strip lists say, no entry containing a particle from particles.txt is applied. bin, abu, al and abd look like noise to a stopword filter and are structural parts of the names they appear in; dropping them turns "Osama bin Laden" into a different name rather than a shorter one. The collision is real and shipped: AL is in organization_stopwords.txt and never strips anything, which is what the rule is for and what the suite holds it to.

Data files, not constants

The lists live in lib/active_sanction/normalizer/dictionaries/*.txt, one entry per line with # comments, because what belongs on them is settled by reading government lists rather than by reading this code -- and a contributor adding OYJ should be sending a one-line diff, not editing a Ruby array.

Entries are written as a publisher writes them (L.L.C., not l l c) and folded by the same Form the names are folded by, so a file never spells the casing, the accents or the marks: one LTD covers Ltd, ltd. and LTD. An entry that folds to several tokens is matched as a contiguous phrase, which is what lets L.L.C. reach a name a publisher wrote as L L C -- see Stoplist.

What folding does not do is join tokens, so LLC and L.L.C. are one token and three and the file carries both. That is the one thing a contributor has to know when adding an abbreviation.

Extending it

A host adds to the shipped lists with a Hash, or replaces them wholesale by building a Dictionary of its own:

ActiveSanction.configure do |c|
c.normalizer_dictionary = { legal_forms: %w[OYJ TBK], particles: %w[ben] }
end

Instances are frozen on construction and compare by value.

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(legal_forms:, honorifics:, organization_stopwords:, particles:) ⇒ void

Every keyword is required, because a Dictionary built by hand is a replacement for the shipped one and a replacement that forgot to carry the particles over would strip al out of several hundred SDN names without saying anything. Dictionary.default.merge(...) is the way to add to the lists rather than replace them.

Parameters:

  • legal_forms (T.untyped)
  • honorifics (T.untyped)
  • organization_stopwords (T.untyped)
  • particles (T.untyped) 


119
120
121
122
123
124
125
126
 
# File 'lib/active_sanction/normalizer/dictionary.rb', line 119

def initialize(legal_forms:, honorifics:, organization_stopwords:, particles:)
  @legal_forms = T.let(entries(legal_forms), T::Array[String])
  @honorifics = T.let(entries(honorifics), T::Array[String])
  @organization_stopwords = T.let(entries(organization_stopwords), T::Array[String])
  @particles = T.let(entries(particles), T::Array[String])
  @stoplists = T.let(build_stoplists, T::Hash[Symbol, Stoplist])
  freeze
end

Instance Attribute Details

#honorifics ⇒ Array<String> (readonly)

Returns:

  • (Array<String>) 


101
102
103
 
# File 'lib/active_sanction/normalizer/dictionary.rb', line 101

def honorifics
  @honorifics
end

#legal_forms ⇒ Array<String> (readonly)

Returns:

  • (Array<String>) 


98
99
100
 
# File 'lib/active_sanction/normalizer/dictionary.rb', line 98

def legal_forms
  @legal_forms
end

#organization_stopwords ⇒ Array<String> (readonly)

Returns:

  • (Array<String>) 


104
105
106
 
# File 'lib/active_sanction/normalizer/dictionary.rb', line 104

def organization_stopwords
  @organization_stopwords
end

#particles ⇒ Array<String> (readonly)

The entries no strip list may touch. See the class comment.

Returns:

  • (Array<String>) 


108
109
110
 
# File 'lib/active_sanction/normalizer/dictionary.rb', line 108

def particles
  @particles
end

Class Method Details

.default ⇒ Dictionary

The shipped lists. Built at load rather than memoized on first use, so nothing has to synchronize reading four files; the constant behind it is private because this is the way to reach it.

Returns:



184
 
# File 'lib/active_sanction/normalizer/dictionary.rb', line 184

def default = DEFAULT

.from_files(directory = DIRECTORY) ⇒ Dictionary

Reads <directory>/<list>.txt for each of LISTS. Public because it is how a host ships its own set of files rather than a Ruby literal, and how the suite builds a dictionary it can vary.

Parameters:

  • directory (String) (defaults to: DIRECTORY)

Returns:



190
191
192
193
194
 
# File 'lib/active_sanction/normalizer/dictionary.rb', line 190

def from_files(directory = DIRECTORY)
  # `new(**hash)` past required keyword parameters is one of the few
  # things Sorbet cannot check statically. The keys are LISTS itself.
  T.unsafe(self).new(**LISTS.to_h { |list| [list, read(File.join(directory, "#{list}.txt"))] })
end

.read(path) ⇒ Array<String>

One entry per line; blank lines and # comments ignored. Comments are whole-line only -- no entry contains a #, and a rule that stripped from the middle would be a rule to remember when one does.

Parameters:

  • path (String)

Returns:

  • (Array<String>) 


200
201
202
 
# File 'lib/active_sanction/normalizer/dictionary.rb', line 200

def read(path)
  File.readlines(path, chomp: true).map(&:strip).reject { |line| line.empty? || line.start_with?("#") }
end

Instance Method Details

#==(other) ⇒ Boolean Also known as: eql?

Parameters:

  • other (T.untyped)

Returns:

  • (Boolean) 


164
165
166
167
168
 
# File 'lib/active_sanction/normalizer/dictionary.rb', line 164

def ==(other)
  return false unless other.instance_of?(self.class)

  to_h == other.to_h
end

#hash ⇒ Integer

Returns:

  • (Integer) 


172
 
# File 'lib/active_sanction/normalizer/dictionary.rb', line 172

def hash = [self.class, to_h].hash

#inspect ⇒ String

Returns:

  • (String) 


175
 
# File 'lib/active_sanction/normalizer/dictionary.rb', line 175

def inspect = "#<#{self.class} #{to_h.map { |list, values| "#{list}=#{values.size}" }.join(" ")}>"

#merge(legal_forms: nil, honorifics: nil, organization_stopwords: nil, particles: nil) ⇒ Dictionary

This dictionary's lists with more entries added. Duplicates are dropped, so merging a list a file already carries is a no-op rather than an error.

Parameters:

  • legal_forms (T.untyped) (defaults to: nil)
  • honorifics (T.untyped) (defaults to: nil)
  • organization_stopwords (T.untyped) (defaults to: nil)
  • particles (T.untyped) (defaults to: nil)

Returns:



151
152
153
154
155
156
157
158
 
# File 'lib/active_sanction/normalizer/dictionary.rb', line 151

def merge(legal_forms: nil, honorifics: nil, organization_stopwords: nil, particles: nil)
  self.class.new(
    legal_forms: @legal_forms + entries(legal_forms),
    honorifics: @honorifics + entries(honorifics),
    organization_stopwords: @organization_stopwords + entries(organization_stopwords),
    particles: @particles + entries(particles)
  )
end

#stoplist(type) ⇒ Stoplist?

What is stripped from a name of this type, or nil when nothing is -- an unknown type, a type no list applies to, or a host that emptied the lists that did. Nil is the fold's fast path, not a degraded one.

Parameters:

  • type (Symbol, nil)

Returns:

  • (Stoplist, nil) 


132
133
134
135
136
137
138
139
140
141
142
 
# File 'lib/active_sanction/normalizer/dictionary.rb', line 132

def stoplist(type)
  return nil if type.nil?

  unless Entity::TYPES.include?(type)
    raise InvalidArgument,
          "unknown entity type #{type.inspect}; expected one of #{Entity::TYPES.join(", ")} or nil"
  end

  stoplist = @stoplists[type]
  stoplist unless stoplist.nil? || stoplist.empty?
end

#to_h ⇒ Hash{Symbol => Array<String>}

Returns:

  • (Hash{Symbol => Array<String>}) 


161
 
# File 'lib/active_sanction/normalizer/dictionary.rb', line 161

def to_h = LISTS.to_h { |list| [list, T.unsafe(public_send(list))] }