DAP: The Data Analysis Pipeline
DAP was created to transform text-based data on the command-line, specializing in transforms that are annoying or difficult to do with existing tools.
DAP reads data using an input plugin, transforms it through a series of filters, and prints it out again using an output plugin. Every record is treated as a document (aka: hash/dict) and filters are used to reduce, expand, and transform these documents as they pass through. Think of DAP as a mashup between sed, awk, grep, csvtool, and jq, with map/reduce capabilities.
DAP was written to process terabyte-sized public scan datasets, such as those provided by https://scans.io/. Although DAP isn't particularly fast, it can be used across multiple cores (and machines) by splitting the input source and wrapping the execution with GNU Parallel.
Installation
Prerequisites
DAP requires Ruby and is best suited for systems with a relatively current version with 2.6.x being the minimum requirement.
Ideally, this will be managed with either
rbenv
or rvm
with the bundler gem also installed and up to date.
Using system managed/installed Rubies is possible but fraught with peril.
Maxmind IP Location Databases
If you intend on using any of the geo_ip*
or geo_ip2*
filters, you must
install the databases that provide the data for these filters. If you do not
intend on using these filters, you can skip this step.
dap
versions 1.4.x and later depend on Maxmind's geoip2/geolite2
databases to be able to append
geographic and related metadata to analyzed datasets. In order to use this
functionality you must put your copy of the relevant Maxmind databases in the
correct location in /var/lib/geoip2
or the data
directory of your dap
installation or override with an environment variable that specifies the full
path to the database in question:
- ASN:
GeoLite2-ASN.mmdb
(environment override:GEOIP2_ASN_DATABASE_PATH
) - City:
GeoLite2-City.mmdb
(environment override:GEOIP2_CITY_DATABASE_PATH
) - ISP:
GeoIP2-ISP.mmdb
(environment override:GEOIP2_ISP_DATABASE_PATH
)
NOTE: Prior to dap
version 1.4.x there was a dependency on Maxmind's geoip
database
to be able to append geographic metadata to analyzed datasets. However, since
that time Maxmind has dropped support for these legacy databases. If you
intend to continue using this deprecated functionality, you must put your copy
of the relevant Maxmind legacy databases in the correct location in
/var/lib/geoip
or the data
directory of your dap
installation or override
with an environment variable that specifies the full path to the database in question:
- ASN:
GeoIPASNum.dat
(environment override in 1.4.x+:GEOIP_ASN_DATABASE_PATH
) - City:
geoip_city.dat
(environment override in 1.4.x+:GEOIP_CITY_DATABASE_PATH
) - Org:
geoip_org.dat
(environment override in 1.4.x+:GEOIP_ORG_DATABASE_PATH
)
Ubuntu 16.04+
sudo apt-get install zlib1g-dev ruby ruby-dev gcc make ruby-bundler
gem install dap
OS X
# Install the GeoIP C library required by DAP
brew update
brew install geoip
gem install dap
Usage
In its simplest form, DAP takes input, applies zero or more filters which modify the input, and then outputs the result. The input, filters and output are separated by plus signs (+
). As seen from dap -h
:
Usage: dap [input] + [filter] + [output]
--inputs
--outputs
--filters
To see which input/output formats are supported and what filters are available, run dap --inputs
,dap --outputs
or dap --filters
, respectively.
This example reads as input a single IP address from STDIN
in line form, applies geo-ip transformations as a filter on that line, and then returns the output as JSON:
$ echo 8.8.8.8 | bin/dap + lines + geo_ip2_city line + json | jq .
{
"line": "8.8.8.8",
"line.geoip2.city.city.geoname_id": "0",
"line.geoip2.city.continent.code": "NA",
"line.geoip2.city.continent.geoname_id": "6255149",
"line.geoip2.city.country.geoname_id": "6252001",
"line.geoip2.city.country.iso_code": "US",
"line.geoip2.city.country.is_in_european_union": "false",
"line.geoip2.city.location.accuracy_radius": "1000",
"line.geoip2.city.location.latitude": "37.751",
"line.geoip2.city.location.longitude": "-97.822",
"line.geoip2.city.location.metro_code": "0",
"line.geoip2.city.location.time_zone": "America/Chicago",
"line.geoip2.city.postal.code": "",
"line.geoip2.city.registered_country.geoname_id": "6252001",
"line.geoip2.city.registered_country.iso_code": "US",
"line.geoip2.city.registered_country.is_in_european_union": "false",
"line.geoip2.city.represented_country.geoname_id": "0",
"line.geoip2.city.represented_country.iso_code": "",
"line.geoip2.city.represented_country.is_in_european_union": "false",
"line.geoip2.city.represented_country.type": "",
"line.geoip2.city.traits.is_anonymous_proxy": "false",
"line.geoip2.city.traits.is_satellite_provider": "false",
"line.geoip2.city.continent.name": "North America",
"line.geoip2.city.country.name": "United States",
"line.geoip2.city.registered_country.name": "United States"
}
There are also several examples of how to use DAP along with sample datasets here.