Project

bubbles

0.0
No commit activity in last 3 years
No release in over 3 years
bubblesequivalent/bubblesHomepageDocumentationSource CodeBug TrackerWiki
Daemonized file uploader that watch a folder and uploads any files files to AWS S3. Designed for Raspberry pi zero
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Popularity
Downloads
12,680
Stars
0
Forks
0
Watchers
3
 Releases
Current version
0.0.5
Total releases
6
First release
2017-04-18
Latest release
2017-06-07
 Development
Primary Language
Ruby
Licenses
MIT
Average date of last 50 commits
2017-04-28
Reverse Dependencies
0

 Dependencies

Development

bundler
~> 1.14
rake
~> 10.0
rspec
~> 3.0

Runtime

aws-sdk
~> 2
 Project Readme

Bubbles

Build Status

Daemonized file uploader that watch a folder and uploads any files files to AWS S3 or Local directory (e.g. mounted NAS volume). Designed for Raspberry pi zero

Notes:

  • for AWS S3 upload we use AWS-SDK s3 put_object the single file upload should not exceed 5GB. I'm not planing to introduce S3 multipart uploads but Pull Requests are welcome.

Installation

Add this line to your application's Gemfile:

gem 'bubbles'

And then execute:

$ bundle

Or install it yourself as:

$ gem install bubbles

Usage

The easy way

Gem by default looks into to two locations for configuration file. So either create ~/.bubbles/config or /var/lib/bubbles/config.yml

note: make sure you have correct read access permission so that ruby can read that file

...with content similar to this:

s3_access_key_id:  xxxxxxxxxxxxxxxxxxxx
s3_secret_access_key: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
s3_bucket: bucket_name
s3_region: eu-west-1

source_dir: /home/myuser/source_folder
processing_dir: /home/myuser/processing_folder

Now all is left is to lunch bubbles with bubbles or bundle exec bubbles

note: files in source_folder will get removed after successful upload

More advanced config
source_dir: /var/myuser/source_folder              # source from where to pick up files
processing_dir: /home/myuser/processing_folder

log_level: 0                                       # debug log level
log_path: /var/log/bubbles.log                     # default is STDOOT

sleep_interval: 1                                  # sleep between next command
num_of_files_to_schedule: 1                        # how many files schedule for processing at the same time

uploaders:
  - 'Bubbles::Uploaders::S3EnsureConnection'       # this will check if s3 connection can be established
  - 'Bubbles::Uploaders::S3'                       # by default only S3 uploader is used
  - 'Bubbles::Uploaders::LocalDir'

s3_access_key_id:  xxxxxxxxxxxxxxxxxxxx
s3_secret_access_key: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
s3_bucket: bucket_name
s3_region: eu-west-1
s3_path: foo_folder/bar_folder/car_folder          # will upload to s3://bucket/foo_folder/bar_folder/car_folder
s3_acl: private                                    # # accepts private, public-read, public-read-write, authenticated-read, aws-exec-read, bucket-owner-read, bucket-owner-full-control

local_dir_uploader_path: /mnt/network_smb_storage/
local_dir_metadata_file_path: /var/log/uploads_metadata.yaml

Look into lib/bubbles/config.rb for more details.

Full Ruby way

You can create custom ruby file and you initialize custom Bubbles::Config.new instance and set all the options you want there:

# ~/my_bubbles_runner.rb
require 'bubbles'

c = Bubbles::Config.new

c.log_level = 0
c.source_dir = /var/myuser/source_folder
c.processing_dir = /var/myuser/processing_folder
c.uploader_classes = [Bubbles::Uploaders::S3, Bubbles::Uploaders::LocalDir]
c.local_dir_uploader_path = /mnt/network_smb_storage
c.s3_region = 'eu-west-1'
c.s3_bucket = 'mybckt'
c.s3_access_key_id     = 'xxxxxxxxxxx'
c.s3_secret_access_key = 'yyyyyyyyyyy'

# ...

command_queue = Bubbles::CommandQueue.new(config: c)
command_queue << Bubbles::DirWatcher.new(config: c, command_queue: command_queue)

loop do
  command_queue.call_next
  sleep c.sleep_interval
end

..and execute:

ruby my_bubbles_runner.rb
# or
bundle exec ruby my_bubbles_runner.rb

Look into lib/bubbles/config.rb for more details.

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment. To install this gem onto your local machine, run bundle exec rake install.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/equivalent/bubbles.