Class: ActiveSanction::Sources::EuFsf

Inherits:
Base
  • Object
show all
Extended by:
T::Sig
Defined in:
lib/active_sanction/sources/eu_fsf.rb,
lib/active_sanction/sources/eu_fsf/record.rb

Overview

The EU Consolidated Financial Sanctions List: every person and entity subject to an EU financial sanction, in one document, as the Financial Sanctions Files (FSF) export.

snapshot = ActiveSanction::Sources[:eu_fsf].new.sync

One file, one record shape, ten times the size

https://webgate.ec.europa.eu/fsd/fsf/public/files/xmlFullSanctionsList_1_1/content

<export><sanctionEntity>   6,234 records, 25.7 MB

An order of magnitude larger than the UN's 2.2 MB or Canada's 2.9 MB, and the first list to actually exercise the streaming interface the XML toolkit was built around. It reads in one pass at roughly 90 MB of resident memory, most of which is the payload itself; nothing here ever holds two records at once.

The token in the URL

The public endpoint takes a token query parameter. Omitting it answers 403 and a wrong one answers 500, so it is not optional -- but it is also not a credential: dG9rZW4tMjAxNw is base64 for token-2017, it is the value the Commission's own public download page has carried since that year, and it is the same string for every caller. It is declared inline rather than configured, the way any other part of a published URL is.

If the Commission ever rotates it, no release of this gem is needed:

ActiveSanction::Sources::EuFsf.token = "..."           # or
ActiveSanction::Sources::EuFsf.url :main, "https://..."

Conditional GET does not work here, and that is the publisher's doing

The endpoint serves Last-Modified but answers If-Modified-Since with 200 and the whole file, and sends no ETag and Cache-Control: no-store. So every sync of this list downloads 25.7 MB where the other four usually download nothing. The conditional request is still sent -- it costs nothing and the day the Commission honours it, it works -- and the snapshot checksum is unchanged when the content is, so a re-download of identical bytes still diffs (#35) to nothing.

The two traps in this list

No name is the name. The EU marks no <nameAlias> as the official one: all 31,053 carry strong="true", and every other candidate signal is wrong somewhere in the file. Document order files Qusay Hussein's French transliteration ahead of his English name; nameLanguage files a Cyrillic spelling of Anatoliy Sidorov's name under EN; ordering by logicalId picks a non-Latin name for 3,203 of the 5,502 multi-name records. So Record picks one by a stated rule -- see #primary_name -- and what "primary" means for this source is the first name the EU published that the EU did not itself annotate as an alias, which is weaker than what it means for OFAC. It costs nothing in score: the scorer takes the best of an entity's names and the kind only reaches the reason line.

Four birth dates are not in the Gregorian calendar. calendarType is ISLAMIC on four records, where year, monthOfYear and dayOfMonth hold a Hijri date -- year="1343" for a man born in 1964. Read as published those become dates in the fourteenth century, and a fourteenth-century date does not merely fail to match a real one, it conflicts with it, and the scorer penalizes the record. Three of the four carry no Gregorian equivalent at all, so they produce no date of birth and the published Hijri date goes to remarks instead.

What a clean EU result is worth

More than a Canadian one and less than an OFAC one. 3,013 identification documents across 6,234 records is real corroborating signal, and 2,764 citizenships arrive as ISO codes rather than as prose. What is thin is the alias grading: the EU publishes no quality column, and the 499 gradings this adapter does read are prose the Commission happened to put in a <remark> -- so most EU aliases arrive ungraded, which the scorer correctly treats as unstated rather than as good.

What this adapter does not do

It does not infer a vessel. subjectType publishes only person and enterprise, and the 41 records carrying an imo document are shipping companies holding IMO company numbers -- Chongchongang Shipping, Korea Ansan Shipping -- not the ships themselves. Typing those as :vessel on the strength of the document kind would be a guess that hides them from every organization search.

Instance Attribute Summary collapse

Attributes inherited from Base

#cache, #fetcher, #instrumenter, #logger

Class Method Summary collapse

Instance Method Summary collapse

Methods inherited from Base

#authority, #column_shapes, #column_tallies, #file_key, #floors, #format, #fresh?, #inspect, #jurisdiction, #key, #parse_argument, published_remarks, #retrieve, #snapshot, #stale?, #sync, #url, #urls

Methods included from Definition

#authority, #declarations, #declared?, #declared_floors, #declared_urls, #file_key, #floor, #floors, #format, #jurisdiction, #key, key!, #licence_notice, #licence_url, #multi_url?, #to_h, #url, #urls

Constructor Details

#initialize(*args, **options) ⇒ void

Parameters:

  • args (T.untyped)
  • options (T.untyped) 


154
155
156
157
158
159
 
# File 'lib/active_sanction/sources/eu_fsf.rb', line 154

def initialize(*args, **options)
  super
  @warnings = T.let([], T::Array[Parsers::Warning])
  @unmapped = T.let([], T::Array[Parsers::Warning])
  @generated_at = T.let(nil, T.nilable(String))
end

Instance Attribute Details

#warnings ⇒ Array<Parsers::Warning> (readonly)

Records that could not be used, and fields that could not be read. Read after #parse; sync orchestration (#34) reports them.

Returns:



151
152
153
 
# File 'lib/active_sanction/sources/eu_fsf.rb', line 151

def warnings
  @warnings
end

Class Method Details

.token=(value) ⇒ String

Re-points the list at the same endpoint with a different token, for the day the Commission rotates the one baked in above.

ActiveSanction::Sources::EuFsf.token = "..."

Parameters:

  • value (T.untyped)

Returns:

  • (String) 


144
145
146
 
# File 'lib/active_sanction/sources/eu_fsf.rb', line 144

def self.token=(value)
  url :main, "#{ENDPOINT}?token=#{value}"
end

Instance Method Details

#parse(raw) ⇒ Array<Entity>

Parameters:

  • raw (T.untyped)

Returns:



162
163
164
165
166
167
168
 
# File 'lib/active_sanction/sources/eu_fsf.rb', line 162

def parse(raw)
  reader = LIST.read(raw)
  entities = build(reader)
  @generated_at = reader.root[GENERATED_AT]
  @warnings = reader.warnings + @unmapped
  entities
end

#source_version ⇒ String?

Returns:

  • (String, nil) 


171
 
# File 'lib/active_sanction/sources/eu_fsf.rb', line 171

def source_version = @generated_at || super