Class: ActiveSanction::Storage::ActiveRecord

Inherits:
Base
  • Object
show all
Extended by:
T::Sig
Defined in:
lib/active_sanction/storage/active_record.rb,
lib/active_sanction/storage/active_record/row.rb,
lib/active_sanction/storage/active_record/reader.rb,
lib/active_sanction/storage/active_record/writer.rb

Overview

Snapshots in the host application's database.

$ rails generate active_sanction:install && rails db:migrate

store = ActiveSanction::Storage::ActiveRecord.new
store.write_snapshot(ActiveSanction::Sources[:ofac_sdn].new.sync)
store.snapshot_meta(:ofac_sdn).age   # one row read

Optional means optional

ActiveRecord is not a dependency of this gem and must not become one. Nothing requires this file unless a host has already loaded ActiveRecord itself -- see the guard at the bottom of storage.rb -- and the gem is fully usable, including Storage::FileSystem, with ActiveRecord absent. The default adapter costs a directory; this one costs a migration, and it is for an installation that already has a database and wants to query its lists, not a prerequisite for using this library.

What the database buys, and what it does not

It buys the prefilter. Scoring 19,015 OFAC records against one name in Ruby is the cost the matcher (#33) wants to avoid paying, and an equality probe on an indexed column narrows that to a few hundred candidates before any of them are loaded:

Row::Name.matching("Aiman al-Zawahiri").pluck(:entity_id)
Row::Identifier.matching("AB-123 456").pluck(:entity_id)

It does not buy a different reading model. each_entity is inherited from Storage::Base rather than reimplemented as a cursor, and that is a decision rather than an omission: a snapshot's checksum is computed over the whole list, so a store that streamed rows straight to the matcher would be handing it records it cannot prove are all of them. Reading a list here materializes it and verifies it, exactly as the other adapters do. Tens of megabytes is a fine price for that; screening against a list that is quietly missing people is not.

Writing

One transaction per list, and insert_all in batches inside it. A sync that dies partway through 19,015 entities rolls back to the list that was there before it -- there is no half-updated state to inspect, and none to screen against. Row-at-a-time saves would be the obvious alternative and are not: 19,015 entities plus some 65,000 rows hanging off them is not work to do one INSERT at a time.

Reading

Nothing partial is ever returned. The Snapshot is rebuilt with the checksum stored beside it, so construction re-derives the digest over the records that actually came back and raises CorruptSnapshot when they do not agree -- a row deleted by hand, a write that half landed, a column edited in a console. A schema_version this gem does not know raises UnsupportedSchema before a record is read.

Concurrency

The database's problem, which is the point. A write is one transaction, so a reader sees the list as it was before it or as it is after it, and readers on other processes and other machines get that for free rather than from a rename that only holds within one filesystem.

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods inherited from Base

#clear, #each_entity, #empty?, #fetch_snapshot, #inspect, #size, #stored?

Constructor Details

#initialize(batch_size: DEFAULT_BATCH_SIZE) ⇒ void

Parameters:

  • batch_size (T.untyped) (defaults to: DEFAULT_BATCH_SIZE) 


148
149
150
151
 
# File 'lib/active_sanction/storage/active_record.rb', line 148

def initialize(batch_size: DEFAULT_BATCH_SIZE)
  @batch_size = T.let(batch_size!(batch_size), Integer)
  super()
end

Instance Attribute Details

#batch_size ⇒ Integer (readonly)

Rows per insert_all -- see DEFAULT_BATCH_SIZE.

Returns:

  • (Integer) 


145
146
147
 
# File 'lib/active_sanction/storage/active_record.rb', line 145

def batch_size
  @batch_size
end

Class Method Details

.installed? ⇒ Boolean

Whether the migration has been run. Not checked on construction: an adapter built in a Rails initializer must not open a connection to say hello, and a host running rails db:migrate would then be unable to boot the application that migrates it.

Returns:

  • (Boolean) 


137
138
139
140
141
 
# File 'lib/active_sanction/storage/active_record.rb', line 137

def self.installed?
  Row::ALL.all?(&:table_exists?)
rescue ::ActiveRecord::ActiveRecordError
  false
end

.prefilter_key(value) ⇒ String

The key a name is filed under in active_sanction_names.normalized_value and the key a query has to build to find it:

prefilter_key("Aiman  al-ZAWAHIRI!")   # => "aiman al zawahiri"
prefilter_key("Ayman al-Ẓawāhirī")     # => "ayman al zawahiri"

Deliberately crude, and deliberately not the matcher's normalizer (#26). Its only job is candidate generation, where the cost of the two kinds of error is wildly asymmetric: a key that collides too eagerly costs a few extra records to score in Ruby, and a key that misses costs a sanctioned person who never reaches the scorer at all. So it folds width and diacritics, cases down, and reduces everything that is not alphanumeric to a single space -- and it stops there. It does not transliterate, drop legal forms (#27), or reorder tokens; those change what a name means and belong where a human can see the decision.

Because it is stored, changing this fold makes the stored keys stale. A release that changes it will say so, and the fix is a re-sync.

Parameters:

  • value (T.untyped)

Returns:

  • (String) 


127
128
129
130
 
# File 'lib/active_sanction/storage/active_record.rb', line 127

def self.prefilter_key(value)
  folded = value.to_s.unicode_normalize(:nfkd).gsub(COMBINING_MARKS, "").downcase
  folded.gsub(NON_ALPHANUMERIC, " ").strip.squeeze(" ").slice(0, PREFILTER_KEY_LIMIT).to_s
end

Instance Method Details

#delete_snapshot(source) ⇒ Boolean

Parameters:

  • source (T.untyped)

Returns:

  • (Boolean) 


190
191
192
193
194
195
196
197
198
199
 
# File 'lib/active_sanction/storage/active_record.rb', line 190

def delete_snapshot(source)
  key = source_key!(source)
  connected do
    Row::Base.transaction do
      row = Row::Snapshot.find_by(source: key.to_s)
      row&.discard!
      !row.nil?
    end
  end
end

#read_snapshot(source) ⇒ Snapshot?

Parameters:

  • source (T.untyped)

Returns:



169
170
171
172
173
174
175
 
# File 'lib/active_sanction/storage/active_record.rb', line 169

def read_snapshot(source)
  row = snapshot_row(source_key!(source))
  return nil if row.nil?

  schema_version!(row)
  build(row)
end

#snapshot_meta(source) ⇒ Meta?

One row read, and no entities. What makes printing how old six lists are six primary-key lookups rather than six full deserializations.

Parameters:

  • source (T.untyped)

Returns:



180
181
182
183
184
185
186
187
 
# File 'lib/active_sanction/storage/active_record.rb', line 180

def snapshot_meta(source)
  row = snapshot_row(source_key!(source))
  return nil if row.nil?

  Meta.new(source: row.source, fetched_at: row.fetched_at.to_time, checksum: row.checksum,
           record_count: row.record_count, schema_version: row.schema_version,
           source_version: row.source_version)
end

#sources ⇒ Array<Symbol>

Sorted in Ruby rather than by the database, so a summary does not reshuffle itself when the same lists are read through a connection with a different collation.

Returns:

  • (Array<Symbol>) 


205
 
# File 'lib/active_sanction/storage/active_record.rb', line 205

def sources = connected { Row::Snapshot.pluck(:source) }.map(&:to_sym).sort

#write_snapshot(snapshot) ⇒ Snapshot

Replaces the source's list inside one transaction: the previous generation is dropped and the new one written, or neither happens.

Parameters:

  • snapshot (T.untyped)

Returns:



156
157
158
159
160
161
162
163
164
165
166
 
# File 'lib/active_sanction/storage/active_record.rb', line 156

def write_snapshot(snapshot)
  stored = snapshot!(snapshot)
  key = source_key!(stored.source)
  connected do
    Row::Base.transaction do
      Row::Snapshot.find_by(source: key.to_s)&.discard!
      Writer.new(create_row(key, stored), stored, batch_size: batch_size).call
    end
  end
  stored
end