Class: ActiveSanction::Sources::AustraliaDfat

Inherits:
Base
  • Object
show all
Extended by:
T::Sig
Defined in:
lib/active_sanction/sources/australia_dfat.rb,
lib/active_sanction/sources/australia_dfat/record.rb,
lib/active_sanction/sources/australia_dfat/published_date.rb

Overview

Australia's Consolidated List: everybody the Foreign Minister has designated or declared under the Autonomous Sanctions Regulations 2011, together with every UN Security Council listing Australia gives effect to.

snapshot = ActiveSanction::Sources[:australia_dfat].new.sync

It is published as a spreadsheet, and as nothing else

https://www.dfat.gov.au/sites/default/files/Australian_Sanctions_Consolidated_List.xlsx

"Consolidated List"   11,163 rows, 19 columns, 1.3 MB

One .xlsx file, one sheet. There is no CSV, no XML and no JSON: DFAT's Consolidated List page offers exactly one download, and this is it. So reading a spreadsheet is not a convenience here, it is the price of screening against Australian sanctions at all.

That was the open question on this list, and the answer was not to take a dependency. Parsers::Spreadsheet reads the workbook with zlib and the XML toolkit the other five adapters already use -- an .xlsx being a ZIP of XML parts -- so this adapter costs a host nothing to install. See that class for what it does and does not read.

The URL moved, and the old one still answers

The path this gem was scoped against, regulation8_consolidated.xlsx, is gone. What is left in its place is a redirect to regulation8_consolidated_2.xls -- a real file, served 200, in the old binary Excel format, last modified in March 2022. An adapter pointed at it would download a list four years stale on every sync and never once look unhealthy. The URL above is the one DFAT's own page links today.

DFAT's edge rejects this gem's User-Agent

Verified against the live endpoint: active_sanction/x.y.z (+https://...) gets no response at all -- not a 403, a dropped connection -- while curl/8.7.1, Wget/1.21 and python-requests/2.31.0 are all served. The filter is on the leading product token, and an unrecognised one is dropped, so identifying ourselves honestly is what gets us blocked.

What is sent instead is the configured agent inside the form written for exactly this: Mozilla/5.0 (compatible; <agent>), which is how a well-behaved crawler has identified itself since Googlebot. It is not a disguise -- the agent, its version and the contact URL an operator configured are all still in the string, and DFAT can still see who we are and block us on purpose. It is the same string in a shape the edge parses. See #fetch_file, which is the only place this source departs from Base.

A record is several rows

DFAT publishes one row per name. Reference 1000 is a primary name and 1000a, 1000b are its aliases, with every other column repeated on each -- so 11,163 rows are 3,906 records: 2,543 people, 1,041 organizations and 322 vessels. Record says how they are joined, and what the Control Date is not.

What a clean Australian result is worth

About what a UN one is worth, which is unsurprising: 1,172 of the 3,906 records are listings by a UN committee that Australia has given effect to, and they arrive with the UN's own text intact. Alias grading is published as a field on all 6,802 aliases. Birth dates are published on 6,823 rows in nine spellings, which PublishedDate reads to within four fragments of the whole list.

Against that, DFAT publishes no document numbers of any kind -- no passport, no national identity number, no company registration -- for any of the 3,906 records. The only identifier on the list is an IMO number, on 344 vessel rows. So an Australian name match has nothing behind it to make it decisive, in the way an OFAC passport number usually settles one, and a screening policy should know that before it sets a threshold.

What this adapter does not do

It does not read the Listing Information prose for anything but the listing date. The column holds relisting histories, UN committee references and the reasons for a designation, in a dozen shapes, and reading it properly is the job OFAC's RemarksParser does and wants the same treatment -- a measured coverage figure -- rather than a regex added here in passing. It is kept verbatim in remarks.

It does not decompose an address. DFAT publishes one free-text column and no parts, so the whole published string is the street; see Record.

Instance Attribute Summary collapse

Attributes inherited from Base

#cache, #fetcher, #instrumenter, #logger

Class Method Summary collapse

Instance Method Summary collapse

Methods inherited from Base

#authority, #column_shapes, #column_tallies, #file_key, #floors, #format, #fresh?, #inspect, #jurisdiction, #key, #parse_argument, published_remarks, #retrieve, #snapshot, #stale?, #sync, #url, #urls

Methods included from Definition

#authority, #declarations, #declared?, #declared_floors, #declared_urls, #file_key, #floor, #floors, #format, #jurisdiction, #key, key!, #licence_notice, #licence_url, #multi_url?, #to_h, #url, #urls

Constructor Details

#initialize(*args, **options) ⇒ void

Parameters:

  • args (T.untyped)
  • options (T.untyped) 


162
163
164
165
166
167
 
# File 'lib/active_sanction/sources/australia_dfat.rb', line 162

def initialize(*args, **options)
  super
  @warnings = T.let([], T::Array[Parsers::Warning])
  @unmapped = T.let([], T::Array[Parsers::Warning])
  @modified = T.let(nil, T.nilable(String))
end

Instance Attribute Details

#warnings ⇒ Array<Parsers::Warning> (readonly)

Records that could not be used, and fields that could not be read. Read after #parse; sync orchestration (#34) reports them.

Returns:



159
160
161
 
# File 'lib/active_sanction/sources/australia_dfat.rb', line 159

def warnings
  @warnings
end

Class Method Details

.enumerated(cell) ⇒ Array<String>

One cell split on the a) ... b) ... enumeration DFAT writes several values with, or the whole cell where it wrote only one.

Parameters:

  • cell (T.untyped)

Returns:

  • (Array<String>) 


197
198
199
 
# File 'lib/active_sanction/sources/australia_dfat.rb', line 197

def self.enumerated(cell)
  cell.to_s.split(ENUMERATOR).map { |part| part.gsub(SPACE, " ").strip }.reject(&:empty?)
end

.group(reference) ⇒ String?

A reference with its alias suffix removed: 1000a and 1000b are both part of 1000. Public because it is what Record identifies a group by and what a caller reconciling a hit against DFAT's own site needs.

Parameters:

  • reference (T.untyped)

Returns:

  • (String, nil) 


189
190
191
192
 
# File 'lib/active_sanction/sources/australia_dfat.rb', line 189

def self.group(reference)
  base = reference.to_s.strip.sub(ALIAS_SUFFIX, "")
  base.empty? ? nil : base
end

Instance Method Details

#parse(raw) ⇒ Array<Entity>

Parameters:

  • raw (T.untyped)

Returns:



170
171
172
173
174
175
176
 
# File 'lib/active_sanction/sources/australia_dfat.rb', line 170

def parse(raw)
  reader = LIST.read(raw)
  entities = build(reader)
  @modified = reader.modified
  @warnings = reader.warnings + @unmapped
  entities
end

#source_version ⇒ String?

2026-09-04T05:37:12Z -- the moment DFAT saved the workbook, which it stamps inside the file and reports on its own download page as the date the list was last updated. Falls back to Last-Modified for a payload handed straight to #snapshot.

Returns:

  • (String, nil) 


183
 
# File 'lib/active_sanction/sources/australia_dfat.rb', line 183

def source_version = @modified || super