url_expander¶ ↑

The idea is simple, One expander to expand them all. I’ve been working on an analytical tool for twitter.com and i wanted a way to expand all the shortened urls. Url_expander is a good start, it covers (13,231 bit.ly domains + 28 different services). I tried to make it as simple as possible to extend & add new services.

gem install url_expander

Download the config used to authenticate with some of the shortening services

curl -s https://raw.github.com/gist/1086114/c5c401e811effc3e99ed37ae9503f267877d8e68/url_expander_credentials.yml -o ~/url_expander_credentials.yml

Require and Use

require 'rubygems' # if ruby < 1.9
require 'url_expander'
UrlExpander::Client.expand('http://tinyurl.com/66sekq5')

• :nested_shortening – Nested shortening is turned on by default (:nested_shortening => true)

  UrlExpander::Client.expand("http://t.co/ZGEGdas")  => "http://www.blink182.com/upallnight/"
  UrlExpander::Client.expand("http://t.co/ZGEGdas", :nested_shortening => false)  => "http://bit.ly/oi029o"
  

• :limit – If the nested_shortening is turned on, then :limit controls the max number of redirection allowed. Default value is 10

  UrlExpander::Client.expand('http://tinyurl.com/66sekq5', :limit => 4)
  

• :config_file – Some services Require credentials to access the api (such as bitly). You need to provide YAML file with the access credentials. Default is “~/url_expander_credentials.yml”.

  UrlExpander::Client.expand('http://bit.ly/qpshuI', :config_file => '/Users/moski/url_expander_credentials.yml')
  

• bit.ly and bitly domains(13,231 domains as of July 19-2011)

  UrlExpander::Client.expand("http://bit.ly/qpshuI")
  UrlExpander::Client.expand("http://4sq.com/pQkuZk")
  UrlExpander::Client.expand("http://fxn.ws/pBewvL")
  UrlExpander::Client.expand("http://tcrn.ch/oe50JN")
  UrlExpander::Client.expand("http://nyti.ms/dzy2b7")
  ....

• j.mp

  UrlExpander::Client.expand("http://j.mp/qpshuI")
  

• budurl.com

  UrlExpander::Client.expand("http://budurl.com/EYOS2")
  

• cli.gs

  UrlExpander::Client.expand("http://cli.gs/2BAzKa")
  

• decenturl.com

  UrlExpander::Client.expand("http://decenturl.com/youtube/medieval")
  UrlExpander::Client.expand("http://youtube.decenturl.com/medieval")
  

• goo.gl

  UrlExpander::Client.expand("http://goo.gl/DRppM")
  

• is.gd

  UrlExpander::Client.expand("http://is.gd/wsJRhR")
  

• xrl.us

  UrlExpander::Client.expand("http://xrl.us/bkz5iy")
  

• adjix.com

  UrlExpander::Client.expand("http://adjix.com/cm4m")
  

• digbig.com

  UrlExpander::Client.expand("http://digbig.com/3bbd")
  

• doiop.com

  UrlExpander::Client.expand("http://doiop.com/dz8896")
  

• easyurl.jp

  UrlExpander::Client.expand("http://easyurl.jp/1qdv")
  

• just.as

  UrlExpander::Client.expand("http://just.as/amMF3i")
  

• moourl.com

  UrlExpander::Client.expand("http://moourl.com/flsho")
  

• notlong.com

  UrlExpander::Client.expand("http://moski.notlong.com")
  

• nutshellurl.com

  UrlExpander::Client.expand("http://nutshellurl.com/1v38")
  

• ow.ly

  UrlExpander::Client.expand("http://ow.ly/5EVkL")
  

• shrt.st

  UrlExpander::Client.expand("http://shrt.st/148u")
  

• snipurl.com, sn.im, cl.lk, snipr.com, snurl.com

  UrlExpander::Client.expand("http://snipurl.com/209hem")
  UrlExpander::Client.expand("http://sn.im/209hem")
  UrlExpander::Client.expand("http://cl.lk/209hem")
  UrlExpander::Client.expand("http://snipr.com/209hem")
  UrlExpander::Client.expand("http://snurl.com/209hem")
  

• t.co

  UrlExpander::Client.expand("http://t.co/ZGEGdas")
  

• tighturl.com

  UrlExpander::Client.expand("http://tighturl.com/3eoz")
  

• tiny.cc

  UrlExpander::Client.expand("http://tiny.cc/pabx5")
  

• tinyurl.com

  UrlExpander::Client.expand("http://tinyurl.com/66sekq5")
  

• twurl.nl

  UrlExpander::Client.expand("http://twurl.nl/e6mglc")
  

• url.ie

  UrlExpander::Client.expand("http://url.ie/cert")
  

• youtu.be

  UrlExpander::Client.expand("http://youtu.be/bINUfbLV_0M")
  

• y2u.be

  UrlExpander::Client.expand("http://y2u.be/bINUfbLV_0M")
  

• qsr.li

  UrlExpander::Client.expand("http://qsr.li/5Zg9")
  

• shorl.com

  UrlExpander::Client.expand("http://shorl.com/nigekohalenu")
  

• simurl.com

  UrlExpander::Client.expand("http://simurl.com/fendaz")
  

• fb.me

  UrlExpander::Client.expand("http://fb.me/KLfffqy3")
  

• itun.es

  UrlExpander::Client.expand("http://itun.es/igG8XL")
  

• ur1.ca

  UrlExpander::Client.expand("http://ur1.ca/4qcly")
  

• su.pr

  UrlExpander::Client.expand("http://su.pr/Ad5dk1")
  

• t11.me

  UrlExpander::Client.expand("http://t11.me/YZI-Y6")
  

• dld.bz

  UrlExpander::Client.expand("http://dld.bz/ahsJ2")
  

• tnw.to

  UrlExpander::Client.expand("http://tnw.to/1A3qT")
  

• dlvr.it

  UrlExpander::Client.expand("http://dlvr.it/ccG3g")
  

• plizy.com

  UrlExpander::Client.expand("http://plizy.com/jzW29w")
  

url_expander has 3 main classes

• UrlExpander::Expanders::Basic: For services that provides 301 redirect.

• UrlExpander::Expanders::API: For services that provide expand api.

• UrlExpander::Expanders::Scrape: No api, no 301 redirect, manual scraping.

Each service must subclass one of these classes.

class Tinyurl < UrlExpander::Expanders::Basic
  # A pattern is used by the basic class to match the url and extract the
  # the shortning key.
  PATTERN = %r'(http://tinyurl\.com(/[\w/]+))'

  # Reference to the class used by the base class to access Request class within
  attr_reader :parent_klass

  # initialize function to do custom initialization
  # make sure to set the @parent_kass and call super
  # Calling super with will extract the key and calls fetch_url
  def initialize(short_url="", options={})
    @parent_klass = self
    super(short_url, options)
  end

  # Request class will include httpartty and set the base_uri for that
  # service
  class Request
    include HTTParty
    base_uri 'http://tinyurl.com'
  end
end

All you have to do now is register you class. Simple enough :)

module UrlExpander
  module Expanders
    autoload :Tinyurl, 'basic/tinyurl' 
  end
end

If the service provides an api why not use it. UrlExpander::Expanders::API gives you a little more flexibly that the basic one but requires more code.

class Googl < UrlExpander::Expanders::API
  # A pattern is used by the basic class to match the url and extract the the shortening key.
  # NOTICE: We ignored the / before the key
  # http://goo.gl/DRppM => 'DRppM' without /
  PATTERN = %r'(http://goo\.gl/([\w/]+))'

  # Reference to the class used by the base class to access Request class within
  attr_reader :parent_klass

  # initialize function to do custom initialization
  # make sure to set the @parent_kass and call super
  # calling super will only exract the shortening for the url using the pattern defined above.
  # You need to manually call fetch_url.
  def initialize(short_url, options={})
    @parent_klass = self
    super(short_url,options)
    fetch_url
  end

  # Request class will include httpartty and set the base_uri for that service.
  class Request
    include HTTParty
    base_uri 'https://www.googleapis.com'
    headers 'Content-Type' => 'application/json'
    headers "Content-length" => "0"
  end

  private

  # A custom fetcher function expand the url using the api.
  def fetch_url
    response = Request.get("/urlshortener/v1/url?shortUrl=http://goo.gl/#{@shortner_key}")
    if response.code == 200
      @long_url  = response['longUrl']
    else
      error = (JSON.parse response.body)['error']
      raise UrlExpander::Error.new(error['message'],response.code)
    end
  end
end

Thats about it, don’t forget to register your class.

• Check out the latest master to make sure the feature hasn’t been implemented or the bug hasn’t been fixed yet

• Check out the issue tracker to make sure someone already hasn’t requested it and/or contributed it

• Fork the project

• Start a feature/bugfix branch

• Commit and push until you are happy with your contribution

• Make sure to add tests for it. This is important so I don’t break it in a future version unintentionally.

• Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.

Copyright © 2011 Moski. See LICENSE.txt for further details.