AdsenseCrawlerForPrivate

Easy way to enable AdSense crawler to login and see private or custom pages in your rails application. Basically one custom login filter. Gem enables you to easily slightly increase revenues from Google AdSense/AdWords. It makes it easy to enable crawling on private pages and so get better targeted ads even in pages behind login screen.

Usage

Install: In RAILS_ROOT/Gemfile.rb

gem 'adsense_crawler_for_private'

Initialize: In RAILS_ROOT/config/initializers/adsense_crawler_for_private.rb

# Configure adsense_crawler_for_private

AdsenseCrawlerForPrivate.cookie_name = "adsense_crawler"
AdsenseCrawlerForPrivate.cookie_domain = :all
AdsenseCrawlerForPrivate.crawler_name = "test_name"
AdsenseCrawlerForPrivate.crawler_password = "test_password"

# If you are paranoid you can specify ip addresses that are ok to the crawlers to access
# AdsenseCrawlerForPrivate.ip_ranges = [IPAddr.new("127.0.0.1"), IPAddr.new("192.168.0.1/20")]

Routes:

  # Give url "http://domain.dom/adsense_crawler_for_private/login" for crawlers to log in
  # or directly "http://domain.dom/adsense_crawler_for_private/login?name=test_name&password=test_password"
  mount AdsenseCrawlerForPrivate::Engine => "/adsense_crawler_for_private"

Rendering etc usage: in controller define what to render for crawlers


def SomeController << ApplicationController

  before_action :adsense_crawler_private_specific_page # normal authentication filters after this one

  def adsense_crawler_private_specific_page
    if AdsenseCrawlerForPrivate.login_check(cookies, request)
      # here info how to render page for crawler
      # e.g render 'crawler_ad_page'
      # or creating a dummy login info
      return false # so that normal authentication filters etc are not effective
    end
  end

  # Here would be rest of the controller

end

Or if you want only to check alongside other authentication you can call in your own authentication filters AdsenseCrawlerForPrivate.login_check(request)-method directly.

Testing your own stuff

You can test your filters by setting cookie in the same way as in AdsenseCrawlerLoginController#login .

If you have enabled ip_ranges option you have to make sure that the ip that tests use is enabled for crawlers.

test "here would be a test for logged crawler in functional tests"
  # Dummy login for crawler, These should be configured in initializers
  crawler_name="adsense_crawler"; crawler_password = "adsense_pass";
  @request.remote_addr = ip = "127.0.0.1"

  # In some frameworks cookies.signed would be enough. Some will need you to sign the cookie by yourself.
  # If you have a better way, let me know
  # same as cookies.signed[AdsenseCrawlerForPrivate.cookie_name] = AdsenseCrawlerForPrivate.cookie_hash(crawler_name, ip)
  @request.cookies[AdsenseCrawlerForPrivate.cookie_name] =
        ActiveSupport::MessageVerifier.new(Dummy::Application.config.secret_token).generate(
            AdsenseCrawlerForPrivate.cookie_str(crawler_name, crawler_password, 2.days.from_now, ip))


  #Normal test in here
  get :index
  assert_response :success
  assert response.body.include?("Hi normal crawler")
end

test "here would be a test for non-logged crawler"
  get :index
  assert_response :success
  assert response.body.include?("Hi normal user")
end

robots.txt

Remember to update robots.txt if you have previously forbidden adsense to crawl certain pages.

Requirements

See https://github.com/holli/adsense_crawler_for_private/blob/master/.travis.yml for more info about tests coverage. We try to test with few of the newest Ruby & Rails combination.

http://travis-ci.org/#!/holli/adsense_crawler_for_private

Support

Submit suggestions or feature requests as a GitHub Issue or Pull Request. Remember to update tests. Tests are quite extensive.

License

Released under the MIT license (http://www.opensource.org/licenses/mit-license.php)