TLAW - The Last API Wrapper

TLAW (pronounce it like "tea+love"... or whatever) is the last (and only) API wrapper framework you'll ever need for accessing GET-only APIs^* in a consistent way (think weather, search, economical indicators, geonames and so on).

• Features
• Why TLAW?
• Usage
  • URLs and params description
  • Response processing
    • Flat hashes
    • DataTable
    • Post-processing
    • All at once
  • Documentability
• Some demos
• Installation & compatibility
• Upcoming features
• Get-only API
• Current status
• Links
• Author
• License

• Pragmatic: thorougly designed with tens of real world examples in mind, not just your textbook's "perfect orthogonal REST API";
• Opinionated: goal is clean and logical Ruby libary, not just mechanical 1-by-1 endpoint-per-method wrapper for every fancy hivemind invention;
• Easy and readable definitions;
• Discoverable: once API defined in TLAW terms, you can easily investigate it in runtime, obtain meaningful errors like "param foo is missing while trying to access endpoint bar" and so on;
• Sane metaprogramming: allows to define entire branchy API wrapper with tons of pathes and endpoints in really concise manner, while creating all corresponding classes/methods at definition time: so, at runtime you have no 20-level dynamic dispatching, just your usual method calls with clearly defined arguments and compact backtraces.

Take a look at our "model" OpenWeatherMap wrapper and demo of its usage, showing how all those things work in reality.

There are ton of small (and not-so-small) useful APIs about world around: weather, movies, geographical features, dictionaries, world countries statistics... Typically, when trying to use one of them from Ruby (or, to be honest, from any programming language), you are stuck with two options:

1. Study and use (or invent and build) some custom hand-made Wrapper Library™ with ton of very custom design decisions (should responses be just hashes, or Hashie, or real classes for each kind of response? What are the inputs? Where should api key go, to global param?); or
2. Just "go commando" (sorry for the bad pun): construct URLs yourself, parse responses yourself, control params (or ignore the control) yourself.

TLAW tries to close this gap: provide a base for breath-easy API description which produces solid, fast and reliable wrappers.

See also a showcase blog post with on-the-fly API wrapper building for GIPHY.

class Example < TLAW::API
  define do
    base 'http://api.example.com'

    param :api_key, required: true # this would be necessary for API instance creation
    # So, the API instance would be e = Example.new(api_key: '123')
    # ...and parameter ?api_key=123 would be added to any request

    endpoint :foo # The simplest endpoint, will query "http://api.example.com/foo"
    # And then you just do e.foo and obtain result

    endpoint :bar, '/baz.json' # Path to query rewritten, will query "http://api.example.com/baz.json"
    # Method is still e.bar, though.

    # Now, for params definition:
    endpoint :movie do
      param :id
    end
    # Method call would be movie(id: '123')
    # Generated URL would be "http://api.example.com/movie?id=123"

    # When param is part of the path, you can use RFC 6570
    # URL template standard:
    endpoint :movie, '/movies/{id}'
    # That would generate method which is called like movie('123')
    # ...and call to "http://api.example.com/movies/123"

    # Now, we can stack endpoints in namespaces
    namespace :foo do # adds /foo to path
      namespace :bar, '/baz' do # optional path parameter works
        endpoint :blah # URL for call would be "http://api.example.com/foo/baz/blah"
        # And method call would be like e.foo.bar.blah(parameters)
      end

      # URL normalization works, so you can stack in namespaces even
      # things not related to them in source API, "redesigning" API on
      # the fly.
      endpoint :books, '/../books.json' # Real URL would be "http://api.example.com/books"
      # Yet method call is still namespaced like e.foo.books
    end

    # Namespaces can have their own input parameters
    namespace :foo, '/foo/{id}' do
      endpoint :bar # URL would be "http://api.example.com/foo/123/bar
      # method call would be e.foo(123).bar
    end
  end

  # ...and everything works in all possible and useful ways, just check
  # docs and demos.
end

See DSL module docs for full description of all features (there are few, yet very powerful).

TLAW is really opinionated about response processing. Main things:

1. Hashes are "flattened";
2. Arrays of hashes are converted to DataTables;
3. Post-processors for fields are easily defined

The main (and usually top-level) answer of (JSON) API is a Hash/dictionary. TLAW takes all multilevel hashes and make them flat.

Here is an example.

Source API responds like:

{
  "meta": {
    "code": "OK",
  },
  "weather": {
    "temp": 10,
    "precipitation": 138
  },
  "location": {
    "lat": 123,
    "lon": 456
  }
  ...
}

But TLAW response to api.endpoint(params) would return you a Hash looking this way:

{
  "meta.code": "OK",
  "weather.temp": 10,
  "weather.precipitation": 138,
  "location.lat": 123,
  "location.lon": 456
  ...
}

Reason? If you think of it and experiment with several examples, typically with new & unexplored API you'll came up with code like:

p response
# => 3 screens of VERY IMPORTANT RESPONSE
p response.class
# => Hash, ah, ok
p response.keys
# => ["meta", "weather", "location"], hmmm...
p response['weather']
# => stil 2.5 screens of unintelligible details
p response['weather'].class
# => Hash, ah!
p response['weather'].keys
# => and ad infinitum, real APIs are easily go 6-8 levels down

Now, with "opinionated" TLAW's flattening, for any API you just do the one and final response.keys and that's it: you see every available data key, deep to the deepest depth.

NB: probably, in the next versions TLAW will return some Hash descendant, which would also still allow you to do response['weather'] and receive that "slice". Or it would not :) We are experimenting!

The second main type of a (JSON) API answer, or of a part of an answer is an array of homogenous hashes, like:

• list of data points (date - weather at that date);
• list of data objects (city id - city name - latitude - longitude);
• list of views to the data (climate model - projected temperature);
• and so on.

TLAW wraps this kind of data (array of homogenous hashes, or tables with named columns) into DataTable structure, which you can think of as an Excel spreadsheet (2d array with named columns), or loose DataFrame pattern implementation (just like daru or pandas, but seriously simpler—and much more suited to the case).

Imagine you have an API responding something like:

{
  "meta": {"count": 20},
  "data": [
    {"date": "2016-09-01", "temp": 20, "humidity": 40},
    {"date": "2016-09-02", "temp": 21, "humidity": 40},
    {"date": "2016-09-03", "temp": 16, "humidity": 36},
    ...
  ]
}

With TLAW, you'll see this response this way:

pp response
{"meta.count"=>20,
 "data"=>#<TLAW::DataTable[date, temp, humidity] x 20>}
# ^ That's all. Small and easy to grasp what is what. 3 named columns,
#   20 similar rows.

d = response['data']
# => #<TLAW::DataTable[date, temp, humidity] x 20>

d.count # Array-alike
# => 20
d.first
# => {"date" => "2016-09-01", "temp" => 20, "humidity" => 40}

d.keys # Hash-alike
# => ["date", "temp", "humidity"]
d["date"]
# => ["2016-09-01", "2016-09-02", "2016-09-03" ...

# And stuff:
d.to_h
# => {"date" => [...], "temp" => [...] ....
d.to_a
# => [{"date" => ..., "temp" => ..., "humidity" => ...}, {"date" => ...

d.columns('date', 'temp') # column-wise slice
# => #<TLAW::DataTable[date, temp] x 20>
d.columns('date', 'temp').first # and so on
# => {"date" => "2016-09-01", "temp" => 20}

Take a look at DataTable docs and join designing it!

When you are not happy with result representation, you can post-process them in several ways:

# input is entire response, block can mutate it
post_process { |hash| hash['foo'] = 'bar' }

# input is entire response, and response is fully replaced with block's
# return value
post_process { |hash| hash['foo'] } # Now only "foo"s value will be response

# input is value of response's key "some_key", return value of a block
# becames new value of "some_key".
post_process('some_key') { |val| other_val }

# Post-processing each item, if response['foo'] is array:
post_process_items('foo') {
  # mutate entire item
  post_process { |item| item.delete('bar') }

  # if item is a Hash, replace its "bar" value
  post_process('bar') { |val| val.to_s }
}

# More realistic examples:
post_process('meta.count', &:to_i)
post_process_items('daily') {
  post_process('date', &Date.method(:parse))
}
post_process('auxiliary_value') { nil } # Nil's will be thrown away completely

See full post-processing features descriptions in DSL module docs.

All described response processing steps are performed in this order:

• parsing and initial flattening of JSON (or XML) hash;
• applying post-processors (and flatten the response after each of them);
• make DataTables from arrays of hashes.

You do it this way:

class MyAPI < TLAW::API
  desc %Q{
    This is API, it works.
  }

  docs 'http://docs.example.com'

  namespace :ns do
    desc %Q{
      It is some interesting thing.
    }

    docs 'http://docs.example.com/ns'

    endpoint :baz do
      desc %Q{
        Should be useful.
      }

      docs 'http://docs.example.com/ns#baz'

      param :param1,
        desc: %Q{
          You don't need it, really.
        }
    end
  end
end

All of above is optional, but when provided, allows to investigate things at runtime (in IRB/pry or test scripts). Again, look at OpenWeatherMap demo, it shows how docs could be useful at runtime.

• Full-featured API wrappers:
  • OpenWeatherMap: source API docs, wrapper, extensively commented & explained demo code;
  • ForecastIO: API docs, wrapper, demo code;
• Demos of "fire-and-forget" wrappers:
  • Urbandictionary's small and unofficial API wrapper;
  • Partial wrapper only for some features of large TMDB API. It also shows on-the-fly updating of already existing API wrapper to add some features.

Just gem install tlaw or add it to your Gemfile, nothing fancy.

Required Ruby version is 2.1+, JRuby works, Rubinius seems like not.

(in no particular order)

• Expose Faraday options (backends, request headers);
• Request-headers based auth;
• Responses caching;
• Response headers processing DSL;
• Paging support;
• Frequency-limited API support (requests counting);
• YARD docs generation for resulting wrappers;
• More solid wrapper demos (weather sites, geonames, worldbank);
• Approaches to testing generated wrappers (just good ol' VCR should work, probably);
• Splat parameters.

What is those "Get-only APIs" TLAW is suited for?

• It is only for getting data, not changing them (though, API may use HTTP POST requests in reality—for example, to transfer large request objects);
  • It would be cool if our weather APIs could allow things like POST /weather/kharkiv {sky: 'sunny', temp: '+21°C'} in the middle of December, huh? But we are not leaving in the world like this. For now.
• It has utterly simple authentication protocol like "give us api_key param in query" (though, TLAW plans to support more complex authentication);
• It typically returns JSON answers (though, TLAW supports XML via awesome crack).

Alongside already mentioned examples (weather and so on), you can build TLAW-backed "get-only" wrappers for bigger APIs (like Twitter), when "gathering twits" is all you need. (Though, to be honest, TLAW's current authorization abilities is far simpler than Twitter requirements).

It is version 0.0.1. It is tested and documented, but not "tested in battle", just invented. DSL is subject to be refined and validated, everything could change (or broke suddenly). Tests are lot, though.

We plan to heavily utilize it for reality, that would be serious evaluation of approaches and weeknesses.

• API Docs

Victor Shepelev

MIT.