Ammitto

Purpose

Ammitto is a Ruby gem for retrieving sanctioned entities (persons, organizations, vessels, and aircraft) from various international sources. It provides a unified JSON-LD based data model for querying sanctions data from multiple jurisdictions.

Features

Unified Data Model - Consistent JSON-LD schema across all sources
Multiple Entity Types - Supports persons, organizations, vessels, and aircraft
15 Data Sources - EU, UN, US (OFAC), World Bank, UK, Australia, Canada, Switzerland, China, Russia, Japan, New Zealand, Turkey, EU Vessels, UN Vessels
Knowledge Graph Export - Individual JSON-LD node files with @id references
Lightweight Search Index - search-index/ directory: a manifest.json plus one shard per authority (rows with no usable authority go to unknown.json), for client-side search
Ontology Browser Data - Classes, properties, hierarchy for schema exploration
Rich Metadata - Legal bases, effects, reasons, status history, and official announcements
Local Caching - Downloads and caches data locally for fast queries
Search API - Simple and powerful search interface

Installation

Add this line to your application’s Gemfile:

gem 'ammitto'

And then execute:

bundle install

Or install it yourself as:

gem install ammitto

Usage

Configuration

require 'ammitto'

Ammitto.configure do |config|
  config.api_base_url = "https://www.ammitto.org/api/v1"   # default
  config.cache_dir = File.expand_path("~/.ammitto")       # default
  config.cache_ttl = 3600                                  # 1 hour, default
  config.verbose = true                                    # default: false
end

Refreshing Cache

# Refresh specific sources
Ammitto.refresh_cache(sources: [:eu, :un, :us])

# Refresh all sources
Ammitto.refresh_cache(all: true)

# Force refresh even if cache is fresh
Ammitto.refresh_cache(all: true, force: true)

Searching

# Search across all sources
results = Ammitto.search("Kim Jong Un")

# Search specific sources
results = Ammitto.search("Kim Jong Un", sources: [:un, :us])

# Search with pagination
results = Ammitto.search("Putin", limit: 50, offset: 0)

# A search that could not read a source, or was given a source code this gem
# does not know, still returns what it could read, so check before treating
# an empty result as a clean negative
unless results.complete?
  warn "Incomplete: could not read #{results.skipped_sources.join(', ')}"
end

# Iterate results (entities)
results.each do |entity|
  puts entity.display_name
  puts "  Type: #{entity.entity_type}"
end

Checking Cache Status

status = Ammitto.cache_status
status.each do |source, info|
  if info[:cached]
    puts "#{source}: cached, updated #{info[:updated_at]}"
  else
    puts "#{source}: not cached"
  end
end

Getting JSON-LD Output

results = Ammitto.search("test")
json_ld = results.to_json_ld
puts JSON.pretty_generate(json_ld)

Available Sources

Ammitto.sources
# => [:au, :ca, :ch, :cn, :eu, :eu_vessels, :jp, :nz, :ru, :tr, :uk, :un, :un_vessels, :us, :wb]

Data Model

Entity Types

PersonEntity - Individual persons with birth info, nationalities, identifications
OrganizationEntity - Companies and organizations with registration info
VesselEntity - Ships with IMO numbers, flag states, and ownership
AircraftEntity - Aircraft with serial numbers, manufacturers, and registrations

Sanction Entry

A SanctionEntry represents a single sanction against an entity, including:

Authority (EU, UN, US, etc.)
Regime (DPRK, Russia/Ukraine, etc.)
Legal bases (resolutions, regulations, executive orders)
Effects (asset freeze, travel ban, etc.)
Reasons (terrorism, proliferation, etc.)
Status (active, suspended, delisted, etc.)
Status history (tracking changes over time)
Official announcement (link to source document)
Raw source data (original XML/JSON preserved)

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    Ammitto Ruby Gem                              │
├─────────────────────────────────────────────────────────────────┤
│  Ammitto.search() ──► QueryBuilder ──► Sources ──► ResultSet    │
│                                                                  │
│  Ammitto.refresh_cache() ──► CacheManager ──► ApiClient         │
│                                   │                              │
│                                   ▼                              │
│                              Cache (~/.ammitto/)                 │
└─────────────────────────────────────────────────────────────────┘
                              │
                              │ Downloads from
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                    www.ammitto.org/api/v1/                      │
│                                                                  │
│  /sources/eu.jsonld    /sources/un.jsonld    /sources/us.jsonld │
│  /sources/gb.jsonld    /sources/cn.jsonld    /index.jsonld      │
└─────────────────────────────────────────────────────────────────┘

The API download catalogue at index.jsonld points bulk JSON-LD and Turtle consumers to a pinned GitHub Release. Source JSON-LD and individual records remain on the API host.

Data Sources

Code	Authority	Source
EU	European Union	webgate.ec.europa.eu
UN	United Nations	scsanctions.un.org
US	United States (OFAC)	treasury.gov
WB	World Bank	worldbank.org
UK	United Kingdom (OFSI)	gov.uk
AU	Australia (DFAT)	dfat.gov.au
CA	Canada (SEFO)	international.gc.ca
CH	Switzerland (SECO)	seco.admin.ch
CN	China (MOFCOM/MFA)	mofcom.gov.cn, mfa.gov.cn
RU	Russia (MID)	mid.ru
JP	Japan (MOFA)	mofa.go.jp
NZ	New Zealand (MFAT)	mfat.govt.nz
TR	Turkey (Ministry of Interior)	icisleri.gov.tr
EU_VESSELS	EU Vessels List	webgate.ec.europa.eu
UN_VESSELS	UN Vessels List	scsanctions.un.org

For Ammitto Operators

This section documents the complete data pipeline for operators maintaining the Ammitto data infrastructure.

Architecture Overview

The Ammitto data pipeline consists of four phases:

┌─────────────────────────────────────────────────────────────────────────────────────┐
│                          AMMITTO DATA PIPELINE ARCHITECTURE                           │
├─────────────────────────────────────────────────────────────────────────────────────┤
│                                                                                      │
│  PHASE 1: SOURCE DATA REPOS (GitHub: ammitto/data-{source})                         │
│  ══════════════════════════════════════════════════════════════════════             │
│                                                                                      │
│   GHA cron (daily 6:00-8:00 UTC)                                                     │
│         │                                                                            │
│         ▼                                                                            │
│   ammitto fetch {source} --format yaml --output-dir processed                       │
│         │                                                                            │
│         │  1. Download raw XML/JSON/HTML from source API                             │
│         │  2. Parse using Lutaml::Model (Ammitto::Sources::{Source})                │
│         │  3. Save as individual YAML files per entity                               │
│         │  4. Create _index.yaml with metadata                                       │
│         ▼                                                                            │
│   ┌───────────────────────────────────────────────────────────────────────────┐     │
│   │  data-uk/  data-eu/  data-un/  data-us/  data-wb/                          │     │
│   │  data-au/  data-ca/  data-ch/  data-cn/  data-ru/                          │     │
│   │  └── processed/*.yaml  (one file per sanctioned entity)                    │     │
│   └───────────────────────────────────────────────────────────────────────────┘     │
│                                                                                      │
│  ══════════════════════════════════════════════════════════════════════             │
│                                                                                      │
│  PHASE 2: CENTRAL DATA REPO (GitHub: ammitto/data)                                   │
│  ══════════════════════════════════════════════════════════════════════             │
│                                                                                      │
│   GHA schedule (daily 9:00 UTC)                                                      │
│         │                                                                            │
│         ▼                                                                            │
│   1. Checkout all source repos                                                       │
│         │                                                                            │
│         ▼                                                                            │
│   2. ammitto harmonize --all --sources-dir ./sources --output-dir ./api/v1          │
│         │                                                                            │
│         │  For each source:                                                          │
│         │    - Read YAML files from sources/data-{source}/processed/                │
│         │    - Parse using Lutaml::Model source models                               │
│         │    - Transform using Transformers (UkTransformer, etc.)                    │
│         │    - Convert to harmonized ontology models                                 │
│         │    - Export as JSON-LD with @context and @graph                            │
│         │                                                                            │
│         ▼                                                                            │
│   3. Create combined all.jsonld + stats.json                                         │
│         │                                                                            │
│         ▼                                                                            │
│   ┌───────────────────────────────────────────────────────────────────────────┐     │
│   │  api/v1/                                                                   │     │
│   │  ├── sources/uk.jsonld  sources/eu.jsonld  ...                            │     │
│   │  ├── all.jsonld            (combined data)                                │     │
│   │  └── stats.json            (entity counts, last updated)                  │     │
│   └───────────────────────────────────────────────────────────────────────────┘     │
│                                                                                      │
│  ══════════════════════════════════════════════════════════════════════             │
│                                                                                      │
│  PHASE 3: WEBSITE (GitHub: ammitto/ammitto.github.io)                                │
│  ══════════════════════════════════════════════════════════════════════             │
│                                                                                      │
│   Triggered by repository_dispatch from data repo                                    │
│         │                                                                            │
│         ▼                                                                            │
│   https://www.ammitto.org/api/v1/*.jsonld                                            │
│                                                                                      │
│  ══════════════════════════════════════════════════════════════════════             │
│                                                                                      │
│  PHASE 4: GEM USERS                                                                  │
│  ══════════════════════════════════════════════════════════════════════             │
│                                                                                      │
│   Ammitto.refresh_cache → Downloads from ammitto.org/api/                       │
│   ~/.ammitto/cache/ → Local JSON-LD cache                                            │
│   Ammitto.search("query") → Queries against cached data                              │
│                                                                                      │
└─────────────────────────────────────────────────────────────────────────────────────┘

Repository Structure

Repository	Purpose
`ammitto/ammitto`	Core Ruby gem with CLI, source models, and transformers
`ammitto/data-uk`	UK (OFSI) sanctions data in YAML format
`ammitto/data-eu`	EU sanctions data in YAML format
`ammitto/data-eu-vessels`	EU vessels sanctions data in YAML format
`ammitto/data-un`	UN sanctions data in YAML format
`ammitto/data-un-vessels`	UN vessels sanctions data in YAML format
`ammitto/data-us`	US (OFAC) sanctions data in YAML format
`ammitto/data-wb`	World Bank debarments data in YAML format
`ammitto/data-au`	Australia (DFAT) sanctions data in YAML format
`ammitto/data-ca`	Canada (SEFO) sanctions data in YAML format
`ammitto/data-ch`	Switzerland (SECO) sanctions data in YAML format
`ammitto/data-cn`	China (MOFCOM/MFA) sanctions data in YAML format
`ammitto/data-ru`	Russia (MID) sanctions data in YAML format
`ammitto/data-jp`	Japan (MOFA) sanctions data in YAML format
`ammitto/data-nz`	New Zealand (MFAT) sanctions data in YAML format
`ammitto/data-tr`	Turkey sanctions data in YAML format
`ammitto/data`	Central repo with harmonized JSON-LD output
`ammitto/ammitto.github.io`	Website serving the API with search and ontology browser

Source Information

Code	Authority	Source URL	Format
UK	United Kingdom (OFSI)	sanctionslist.fcdo.gov.uk	XML
EU	European Union	webgate.ec.europa.eu	XML
UN	United Nations	scsanctions.un.org	XML
US	United States (OFAC)	treasury.gov	XML
WB	World Bank	worldbank.org	JSON
AU	Australia (DFAT)	dfat.gov.au	CSV
CA	Canada (SEFO)	international.gc.ca	XML
CH	Switzerland (SECO)	seco.admin.ch	XML
CN	China (MOFCOM/MFA)	mofcom.gov.cn, mfa.gov.cn	HTML
RU	Russia (MID)	mid.ru	HTML

CLI Commands for Operators

Fetch Data from Source

# Fetch UK data as YAML
ammitto fetch uk --format yaml --output-dir ./processed --verbose

# Fetch multiple sources
ammitto fetch uk eu un --format yaml --output-dir ./processed

# Fetch every source with a working fetch path (skips cn, ru and jp)
ammitto fetch --all --format yaml --output-dir ./processed

# Dry run (show what would be fetched)
ammitto fetch uk --dry-run

Harmonize Data (YAML → JSON-LD)

# Auto-detect and harmonize all data-* repositories
ammitto harmonize --scan --sources-dir /path/to/repos --output-dir ./api/v1 --verbose

# Harmonize specific source
ammitto harmonize uk --input-dir ../data-uk/processed --output-dir ./api/v1

# Create combined all.jsonld and all.ttl
ammitto harmonize --scan --combine --output-dir ./api/v1

The harmonization process generates:

Output	Description
`api/v1/node/entity/{source}/{id}.jsonld`	Individual entity nodes (list-agnostic)
`api/v1/node/entry/{source}/{list_type}/{id}.jsonld`	Individual entry nodes (list-specific)
`api/v1/node/list/{source}/{list_type}.jsonld`	List definition nodes
`api/v1/node/authority/{code}.jsonld`	Authority nodes (deduplicated)
`api/v1/node/regime/{code}.jsonld`	Regime nodes (deduplicated)
`api/v1/search-index/manifest.json`	Search index manifest: totals and the list of shard files
`api/v1/search-index/{authority}.json`	Lightweight search index shard, one per authority (plus `unknown.json` for rows without a usable authority)
`api/v1/facets/*.json`	Facet counts for filters (includes list_types.json)
`api/v1/by-list/{source}/{list_type}.jsonld`	Entries grouped by list type
`api/v1/ontology/classes.jsonld`	Ontology class definitions
`api/v1/ontology/properties.jsonld`	Ontology property definitions
`api/v1/ontology/hierarchy.json`	Class hierarchy tree
`api/v1/all.jsonld`	Combined JSON-LD graph
`api/v1/all.ttl`	Combined Turtle/RDF graph
`api/v1/stats.json`	Statistics and metadata

IRI Structure

The knowledge graph uses normalized IRIs:

# Entities - LIST-AGNOSTIC (can appear on multiple lists)
https://www.ammitto.org/entity/{source}/{local_id}

# Entries - LIST-SPECIFIC (junction records linking entity to list)
https://www.ammitto.org/entry/{source}/{list_type}/{local_id}

# Lists
https://www.ammitto.org/list/{source}/{list_type}

# Announcements - LIST-AGNOSTIC
https://www.ammitto.org/announcement/{source}/{local_id}

# Legal Instruments - LIST-AGNOSTIC
https://www.ammitto.org/legal_instrument/{source}/{local_id}

Example IRIs:

https://www.ammitto.org/entity/cn/mitsubishi-heavy-industries
https://www.ammitto.org/entry/cn/import-export-control-list/mitsubishi-heavy-industries
https://www.ammitto.org/list/cn/import-export-control-list

Check Status

# Show cache and data status
ammitto status

# List available sources
ammitto sources

# Export to different formats
ammitto export jsonld --output-dir ./data
ammitto export ttl --output-dir ./data

GitHub Actions Workflows

Source Repo Workflows (`data-{source}/.github/workflows/fetch.yml`)

Each source repository has a workflow that:

Runs daily at a scheduled time (6:00-8:00 AM UTC, staggered by source)
Installs the ammitto gem from GitHub
Downloads raw data from the source API
Parses using Lutaml::Model source models
Saves individual YAML files to processed/
Commits and pushes changes

Table 1. Schedule Times (UTC)

Source	Cron Time	Description
UK	6:00	UK OFSI sanctions
EU	6:15	EU sanctions
UN	6:30	UN consolidated list
US	6:45	US OFAC SDN list
WB	7:00	World Bank debarments
AU	7:15	Australia DFAT
CA	7:30	Canada SEFO
CH	7:45	Switzerland SECO
CN	8:00	China (HTML parsing)
RU	8:15	Russia (HTML parsing)

Central Repo Workflow (`data/.github/workflows/harmonize.yml`)

Runs daily at 9:00 AM UTC (after all fetches complete)
Checks out all source repositories
Runs ammitto harmonize --all --combine
Saves JSON-LD output to api/v1/
Triggers website deployment

Adding a New Source

To add a new sanctions source:

Create source models in lib/ammitto/sources/{source}/ using Lutaml::Model

# lib/ammitto/sources/xx/designation.rb
module Ammitto
  module Sources
    module Xx
      class Designation < Lutaml::Model::Serializable
        attribute :id, :string
        attribute :name, :string
        # ... other attributes

        xml do
          root "Designation"
          map_element "ID", to: :id
          map_element "Name", to: :name
        end
      end
    end
  end
end

Create transformer in lib/ammitto/transformers/xx_transformer.rb

module Ammitto
  module Transformers
    class XxTransformer < BaseTransformer
      def initialize
        super(:xx)
      end

      def transform(source)
        # Transform source model to ontology models
      end
    end
  end
end

Register the source in lib/ammitto/config/defaults.rb:

ALL_SOURCES = %i[eu un us wb uk au ca ch cn ru xx].freeze

Create data repo on GitHub:

gh repo create ammitto/data-xx --public --description "Ammitto Data repository for XX Source"

Add fetch workflow to .github/workflows/fetch.yml in the new repo
Update harmonize workflow in ammitto/data to checkout the new source

Environment Variables

Variable	Description	Default
`AMMITTO_CACHE_DIR`	Local cache directory	`~/.ammitto`
`AMMITTO_API_BASE_URL`	API base URL	`https://www.ammitto.org/api/v1`
`AMMITTO_LOG_LEVEL`	Logging level (debug, info, warn, error)	`info`
`AMMITTO_SOURCES`	Comma-separated sources to process	all sources

Troubleshooting

Manual Fetch and Harmonize

# Fetch a specific source manually
cd /path/to/data-uk
ammitto fetch uk --format yaml --output-dir processed --verbose

# Check the output
ls -la processed/ | head -20
cat processed/_index.yaml

# Harmonize manually
cd /path/to/data
ammitto harmonize uk --sources-dir ./sources --output-dir ./api/v1 --verbose

# Check JSON-LD output
jq '.["@graph"] | length' ./api/v1/sources/uk.jsonld

Common Issues

Empty YAML files: Check that source models correctly parse the XML/JSON structure
Missing entities in JSON-LD: Verify transformers are registered in the Registry
Workflow failures: Check the GitHub Actions logs for error messages

Cross-source identity links

Harmonization preserves each source entity and its sanction entries. It adds symmetric owl:sameAs links only when different sources supply a matching, country-qualified identifier (passport, national ID, tax ID), a checksum-valid vessel IMO number, or a complete birth date with a conservative name match. Names use Unicode normalization, token ordering, and at most one character insertion, deletion, or substitution. Short names, approximate or partial dates, contradictory dates/identifiers, different entity types, and ambiguous groups do not establish automatic identity. Every linked group must have compatible evidence between every pair and at most one entity per source. These safeguards favor precision over recall; absence of a link is not evidence that two records describe different people.

Name similarities remain review leads when automatic identity is unsupported or ambiguous, including matching alternative birth dates. Review leads never become identity assertions. With --report PATH, entity_matching records published pair counts, a bounded sample of matching evidence, name-similarity review leads, and skipped oversized candidate blocks. Both samples are limited to 1,000 pairs, with explicit limit indicators. Name-only review sampling stops after 20,000 pair attempts; its truncation flag and attempt count are reported independently of the complete strong identifier/date candidate pass. Candidate generation uses identifier, full-date, and name-token indexes; blocks exceeding 50 records are skipped rather than compared without bounds. This is a conservative matching policy, not a measured guarantee of real-world identity or exhaustive cross-source deduplication.

Development

After checking out the repo, run bin/setup to install dependencies. Then, run bundle exec rake spec to run the tests.

# Run tests
bundle exec rake spec

# Run linter
bundle exec rubocop

# Interactive console
bin/console

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/ammitto/ammitto.

License

The gem is available as open source under the terms of the BSD-2-Clause License.

ammitto

Runtime