AdsPower Client
Ruby gem for stealthly web-scraping and data-extraction using AdsPower.com and proxies.
Outline:
- Installation
- Scraping
- Advanced
- Headless
- Logging
- New
driver2
method
1. Installation
First thing first, you need to install the environment. Then, you have to install the gem. Finally, you can write code for AdsPower automation.
Installing Environment:
wget https://raw.githubusercontent.com/leandrosardi/environment/main/sh/install.ubuntu.20_04.sh -O - | bash
Installing Gem:
gem install adspower-client
Writing Code:
client = AdsPowerClient.new(
key: '************************',
)
Remember to open and login to the AdsPower app, as is shown in the picture below. In the chatper 4 (Headless) you will see how to run AdsPower in headless mode.
2. Scraping
The html
method perform the following operations in order to scrape any webpage stealthly:
- create a new profile
- start the browser
- visit the page
- grab the html
- quit the browser from webdriver
- stop the broser from adspower
- delete the profile
- return the html
ret = client.html('http://foo.com')
p ret[:profile_id]
# => "jc8y0yt"
p ret[:status]
# => "success"
p ret[:html]
# => ...
3. Advanced
Internal methods that you should handle to develop advanced bots.
Checking AdsPower Status
The online?
method returns true
if AdsPower API is available.
p client.online?
# => true
Creating Profile
p client.create
# => "jc8y0yt"
Deleting Profile
client.delete('jc8y0yt')
Starting Profile
p client.start('jc8y5g3')
# => {"code"=>0, "msg"=>"success", "data"=>{"ws"=>{"puppeteer"=>"ws://127.0.0.1:43703/devtools/browser/60e1d880-e4dc-4ae0-a2d3-56d123648299", "selenium"=>"127.0.0.1:43703"}, "debug_port"=>"43703", "webdriver"=>"/home/leandro/.config/adspower_global/cwd_global/chrome_116/chromedriver"}}
Stopping Profile
client.stop('jc8y5g3')
Checking Profile
The check
method returns true
if the profile has been started.
client.check('jc8y5g3')
# => true
Operating Browser
id = 'jc8y5g3'
url = 'https://google.com'
driver = client.driver(id)
driver.get(url)
4. Headless
This chapter explains the operations for working with the AdsPower server and browser in headless mode.
Starting the AdsPower server
To start the AdsPower server, use the server_start
method:
client = AdsPowerClient.new(key: YOUR_API_KEY)
client.server_start
The server will listen the port 50325
by default.
You can set a custom port:
client = AdsPowerClient.new(
key: YOUR_API_KEY,
port: 8082,
)
Stopping the AdsPower server
To stop the AdsPower server, use the server_stop
method:
client.server_stop
Checking if the server is running
You can verify whether the server is running with the online?
method:
puts client.online? ? "Server is running" : "Server is stopped"
Starting a browser in headless mode
Pass true
as the second parameter to the driver
method to start a browser in headless mode:
client = AdsPowerClient.new(key: YOUR_API_KEY)
driver = client.driver(PROFILE_ID, true)
5. Logging
The server_start
method seen in chatper 4 (Headless) runs a bash line to start the AdsPower server.
Such a bash line redirects both stdout
and stderr
to ~/adspower-client.log
.
Check such a logfile if you face any problem to start the AdsPower server.
Feel free to change the location and name for the log:
client = AdsPowerClient.new(
key: '************************',
server_log: '~/foo.log'
)
6. New driver2
method
From version 1.0.14
, I added a new method driver2
that is an improvement of legacy driver
method.
# open the browser
driver = client.driver2(PROFILE_ID,
headless: HEADLESS,
read_timeout: 200
)