YumS3Sync
Synchronize a Yum repository over HTTP to S3. This can be useful when you want to mirror Internet Yum repositories 'locally' or want to run the synchronization from within EC2 rather than pushing it from outsede.
There are two command line tools supplied:
-
yums3sync- Synchronizes a single Yum repository to a bucket:/prefix -
yums3syncdir- Scan a directory of links for repositories then synchronize all of them to bucket:/prefix
YumS3sync is smart enough to only copy files changes between runs, and does almost nothing if nothing changed on the source end.
Syncdir is effectively a wrapper around sync to make it a little easier to synchronize a large amount of repositories without having to know exactly which ones you're synchronizing. I use it to synchronize internal Yum repositories made by other developers to AWS without having to make configuration changes.
After the syncronization you can use the repository using Yum with an S3 plugin or configure a webserver to act as a proxy between S3 and Yum. One of these two options is required if you do not want to make your S3 bucket internet-readable.
Installation
Gem
gem install yum_s3_sync
Manual
Clone this git repository and build a Gem. The program should work without additional dependencies on RHEL/Centos 6 with EPEL enabled. It is tested on Ruby 1.8.7 and 2.1
Usage
Yums3sync
Usage: yums3sync [options]
-s, --source SOURCE HTTP source URL
-b, --bucket BUCKET Target bucket name
-p, --prefix PREFIX Target bucket prefix
-k, --keep Never overwrite exitant files
-n, --dry-run Don't make any changes
Example usage:
# yums3sync --source http://mrepo.corporate.com/mrepo/rhel6/x86_64/epel6/ --bucket my-org-repository --prefix rhel6/x86_64/epel6
This will make the contents of the 'epel6' repository available under my-org-repository::/rhel6/x86_64/epel6. Running the command again will copy only changes, if any.
Yums3syncdir
Usage: yums3syncdir [options]
-s, --source SOURCEDIR HTTP source base URL
-b, --bucket BUCKET Target bucket name
-p, --prefix PREFIX Target bucket prefix
-x, --exclude prefix1,prefix2 Exclude prefixes
-k, --keep Never overwrite exitant files
-n, --dry-run Don't make any changes
Example usage:
# yums3syncdir --source http://mrepo.corporate.com/mrepo/rhel6/x86_64/ --bucket my-org-repository --prefix rhel6/x86_64 --exclude disc1,iso
Assuming http://mrepo.corporate.com/mrepo/rhel6/x86_64/ has an index.html listing other repositories this will synchronize all of them to my-org-repository:/rhel6/x86_64
Serving the repository
Serving the repository is easy with an Apache (or other) reverse proxy. We first configure the S3 bucket to only allow access from certain referrers. We will use this as a secret to authenticate our proxies to S3.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "PublicReadGetObject",
"Effect": "Allow",
"Principal": "*",
"Action": [
"s3:List*",
"s3:Get*"
],
"Resource": "arn:aws:s3:::my-org-repository/*",
"Condition": {
"StringEquals": {
"aws:Referer": "MySecretString"
}
}
}
]
}
Next we configure Apache to serve up the repository through mod_proxy.
<IfModule proxy_module>
<IfModule proxy_http_module>
<IfModule headers_module>
ProxyRequests off
ProxyErrorOverride On
# We use '/mrepo/' to match our internal repositories.
<LocationMatch "/mrepo/">
ProxyPass http://my-org-repository.s3-eu-west-1.amazonaws.com/
Header set Cache-Control "max-age=300, public"
RequestHeader set Referer MySecretString
</LocationMatch>
ErrorLog logs/error_log
LogLevel warn
# Don't log our secret in our logfiles
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{User-Agent}i\"" proxylogger
CustomLog logs/access_log proxylogger
</IfModule>
</IfModule>
</IfModule>
Optionally add an ELB in front of a couple of these and serve!