Class: ActiveSanction::Storage::FileSystem

Inherits:
Base
  • Object
show all
Extended by:
T::Sig
Defined in:
lib/active_sanction/storage/file_system.rb

Overview

Snapshots as gzipped JSON in a directory. The default adapter, and the reason this gem screens a name without an application having provisioned anything first.

store = ActiveSanction::Storage::FileSystem.new           # ~/.active_sanction
store = ActiveSanction::Storage::FileSystem.new(root: "/srv/lists")

store.write_snapshot(ActiveSanction::Sources[:ofac_sdn].new.sync)
store.snapshot_meta(:ofac_sdn).age   # without opening the list
store.read_snapshot(:ofac_sdn)       # => Snapshot, checksum verified

Nothing here is required. zlib and json are stdlib, so the cost of persisting 19,015 OFAC records is a directory -- which is what makes this usable from a cron job, a CLI (#36), a CI run, and an air-gapped host that only ever gets a copied directory. Storage::ActiveRecord (#25) is for an installation that already has a database and wants to query the lists; it is not a prerequisite for using this library.

The layout is private

Under #62 the contents of root are @api private. What is on disk is optimized for reading and rewriting locally, and it is expected to change -- the portable, cross-machine, signature-verified representation is the bundle format (#57), which has its own stability contract and is specified in docs/bundle_format.md. An application that reads these files itself makes every future storage optimization a breaking change for it.

As it stands:

root/ofac_sdn/meta.json
root/ofac_sdn/snapshot-sha256-9f86d081884c7d65....json.gz

Why the snapshot file is named after its checksum

Because "a sync interrupted mid-write leaves the previous good snapshot intact" cannot be honoured by two files at fixed names. Replacing a list means replacing both the list and the sidecar describing it, and whatever order those two renames happen in, a process killed between them leaves a snapshot and a meta that do not describe each other -- the new list under the old checksum, or a sidecar advertising records that are not there. Either way the previous list is gone and the source is unreadable until the next successful sync.

Naming the list file after the content it holds removes the conflict. A new list is written to a name nothing else occupies, so it cannot destroy the list already there, and meta.json -- one small file, replaced by one atomic rename -- is the single point at which the new generation becomes the live one. Interrupt anywhere before that rename and the store is exactly as it was, plus a stray file the next write sweeps. Interrupt after it and the new list is live and complete. There is no third state.

This is the pattern PayloadCache uses for raw payloads, for the same reason and with the same tradeoff: a brief second copy on disk.

What it refuses to do

Return anything it cannot prove. Snapshot.from_h re-derives the checksum from the records that came back and construction fails if it does not match what was stored, so a truncated file, an edited file and a half-written file all raise CorruptSnapshot instead of screening against a list that is missing records. A schema version this code does not know raises UnsupportedSchema before the list is parsed, since a snapshot from a newer gem will usually deserialize into a valid-looking, quietly wrong record set.

Concurrency

Many readers and one writer, across processes, which is the arrangement it exists for: a scheduled sync replacing a list while web workers screen against it. Committing is a rename, so a reader sees the whole previous generation or the whole new one; a reader that had already read the old meta.json when the new one landed re-reads it if the file it was sent to has since been swept.

Two processes writing the same source at once is not supported and is not made safe by anything here -- run one sync.

Instance Attribute Summary collapse

Instance Method Summary collapse

Methods inherited from Base

#clear, #each_entity, #empty?, #fetch_snapshot, #size, #stored?

Constructor Details

#initialize(root: nil) ⇒ void

Parameters:

  • root (T.untyped) (defaults to: nil) 


153
154
155
156
 
# File 'lib/active_sanction/storage/file_system.rb', line 153

def initialize(root: nil)
  @root = T.let(-::File.expand_path((root || ActiveSanction.config.storage_dir).to_s), String)
  super()
end

Instance Attribute Details

#root ⇒ String (readonly)

The directory every source is filed under. Its layout is private -- see the class comment.

Returns:

  • (String) 


150
151
152
 
# File 'lib/active_sanction/storage/file_system.rb', line 150

def root
  @root
end

Instance Method Details

#delete_snapshot(source) ⇒ Boolean

Parameters:

  • source (T.untyped)

Returns:

  • (Boolean) 


193
194
195
196
197
198
 
# File 'lib/active_sanction/storage/file_system.rb', line 193

def delete_snapshot(source)
  directory = directory_for(source_key!(source))
  stored = ::File.file?(::File.join(directory, META_FILENAME))
  FileUtils.rm_rf(directory)
  stored
end

#inspect ⇒ String

Returns:

  • (String) 


214
 
# File 'lib/active_sanction/storage/file_system.rb', line 214

def inspect = "#<#{self.class} #{root} #{list}>"

#read_snapshot(source) ⇒ Snapshot?

Parameters:

  • source (T.untyped)

Returns:



175
176
177
178
179
180
181
182
183
 
# File 'lib/active_sanction/storage/file_system.rb', line 175

def read_snapshot(source)
  key = source_key!(source)
  directory = directory_for(key)
  meta = read_meta(directory)
  meta, json = read_list(directory, meta) if meta
  return nil if json.nil? || meta.nil?

  build(key, meta, json, snapshot_path(directory, meta.checksum))
end

#snapshot_meta(source) ⇒ Meta?

Off the sidecar, without opening the list. What makes sources in the CLI (#36) and the per-source summary in sync (#34) cheap: printing how old six lists are reads six small JSON files rather than inflating and deserializing tens of megabytes.

Parameters:

  • source (T.untyped)

Returns:



190
 
# File 'lib/active_sanction/storage/file_system.rb', line 190

def snapshot_meta(source) = read_meta(directory_for(source_key!(source)))

#sources ⇒ Array<Symbol>

Every directory under root holding a committed sidecar. Deliberately does not parse them: this is what stored?, empty? and clear are built on, and one unreadable list must not make the store impossible to inspect or to repair.

Returns:

  • (Array<Symbol>) 


205
206
207
208
209
210
211
 
# File 'lib/active_sanction/storage/file_system.rb', line 205

def sources
  return [] unless ::File.directory?(root)

  Dir.children(root)
     .select { |name| name.match?(SOURCE_PATTERN) && ::File.file?(::File.join(root, name, META_FILENAME)) }
     .map(&:to_sym).sort
end

#write_snapshot(snapshot) ⇒ Snapshot

Writes the list, then publishes it by replacing meta.json. The previous generation stays readable until that rename lands and is swept immediately after it.

Parameters:

  • snapshot (T.untyped)

Returns:



162
163
164
165
166
167
168
169
170
171
172
 
# File 'lib/active_sanction/storage/file_system.rb', line 162

def write_snapshot(snapshot)
  stored = snapshot!(snapshot)
  directory = directory_for(source_key!(stored.source))
  FileUtils.mkdir_p(directory)

  path = snapshot_path(directory, stored.checksum)
  write_atomically(path) { |file| compress(JSON.generate(stored.to_h), file) }
  commit(directory, Meta.from_snapshot(stored))
  prune(directory, path)
  stored
end