Blog

Introduction

Amazon S3 Backup is the most reliable, fast and simple to use solution for keeping the data backed up online. S3 achieves high availability by replicating data across multiple servers within Amazon’s data centers, and offers 99.999999999% durability. S3 typical use cases involve media sharing, media distribution, Server/PC backup, online storage, and application storage. This blog shows how we backup our code base to S3 twice a day using different interesting tools like S3cmd, and S3 browser.

Use Case

This use case shows how we backup our bare git repository to S3 twice a day using different tools like S3cmd, and S3 browser. This use case is kept intentionally simple to demonstrate S3.

What we want to do:

• Ensure Python is set up
• Install & Configure S3cmd tools
• Download & Configure S3 browser
• Create a sample bare git repository
• Create a script to upload the repository to S3

Solution

Ensure Python is set up:

• There are so many blogs and articles explaining about how to install Python, and the links are referenced below. For this use case, we use Ubuntu and Python 2.7.5

http://askubuntu.com/questions/101591/how-do-i-install-python-2-7-2-on-ubuntu

http://heliumhq.com/docs/installing_python_2.7.5_on_ubuntu

Install & Configure S3cmd tools:

• Install S3cmd:
  
  $ sudo apt-get install s3cmd

  1	$ sudo apt-get install s3cmd

• Configure S3Cmd:
  
  $ s3cmd --configure Enter new values or accept defaults in brackets with Enter. Refer to user manual for detailed description of all options. Access key and Secret key are your identifiers for Amazon S3 Access Key: [Enter Amazon Access Key] Secret Key: [Enter Amazon Secret Key] Encryption password is used to protect your files from reading by unauthorized persons while in transfer to S3 Encryption password: treselle (Choose your own password) Path to GPG program [/usr/bin/gpg]: When using secure HTTPS protocol all communication with Amazon S3 servers is protected from 3rd party eavesdropping. This method is slower than plain HTTP and can't be used if you're behind a proxy Use HTTPS protocol [No]: Yes New settings: Access Key: [Your Access Key will appear here] Secret Key: [Your Secret Key will appear here] Encryption password: treselle Path to GPG program: /usr/bin/gpg Use HTTPS protocol: True HTTP Proxy server name: HTTP Proxy server port: 0 Test access with supplied credentials? [Y/n] Y Please wait... Success. Your access key and secret key worked fine :-) Now verifying that encryption works... Success. Encryption and decryption worked fine :-) Save settings? [y/N] y Configuration saved to '/home/treselle/.s3cfg'

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37

  $ s3cmd --configure   Enter new values or accept defaults in brackets with Enter. Refer to user manual for detailed description of all options.   Access key and Secret key are your identifiers for Amazon S3 Access Key: [Enter Amazon Access Key] Secret Key: [Enter Amazon Secret Key]   Encryption password is used to protect your files from reading by unauthorized persons while in transfer to S3 Encryption password: treselle (Choose your own password) Path to GPG program [/usr/bin/gpg]:   When using secure HTTPS protocol all communication with Amazon S3 servers is protected from 3rd party eavesdropping. This method is slower than plain HTTP and can't be used if you're behind a proxy Use HTTPS protocol [No]: Yes   New settings:   Access Key: [Your Access Key will appear here]   Secret Key: [Your Secret Key will appear here]   Encryption password: treselle   Path to GPG program: /usr/bin/gpg   Use HTTPS protocol: True   HTTP Proxy server name:   HTTP Proxy server port: 0   Test access with supplied credentials? [Y/n] Y Please wait... Success. Your access key and secret key worked fine :-)   Now verifying that encryption works... Success. Encryption and decryption worked fine :-)   Save settings? [y/N] y Configuration saved to '/home/treselle/.s3cfg'

• Verify S3Cmd:
  
  $ s3cmd ls 2014-03-22 03:46 s3://backupdemo

  1 2 3

  $ s3cmd ls   2014-03-22 03:46  s3://backupdemo

Download & Configure S3 browser:

• Download S3browser:

Download and Install S3browser from http://s3browser.com/download/s3browser-4-1-1.exe

• Configure S3browser:

Specify Amazon access and secret key along with any unique account name as shown in the below screenshot

• Verify S3browser:

The below screenshot shows a sample S3 bucket with name “backupdemo” created for this blog purpose that doesn’t have any objects yet.

Create a sample bare git repository:

• Create a bare blessed repository
  
  $ mkdir blessed_repo.git $ cd blessed_repo.git $ git –bare init

  1 2 3

  $ mkdir blessed_repo.git $ cd blessed_repo.git $ git –bare init

• Create a sample working directory and push the master to blessed repo
  
  $ mkdir sample_project $ cd sample_project $ git – init $ vi sample.txt $ git add . $ git commit –m “Initial Project commit” $ git remote add origin treselle@blessedrepo.com:/home/treselle/blessed_repo.git $ git push origin master

  1 2 3 4 5 6 7 8

  $ mkdir sample_project $ cd sample_project $ git – init $ vi sample.txt $ git add . $ git commit –m “Initial Project commit” $ git remote add origin treselle@blessedrepo.com:/home/treselle/blessed_repo.git $ git push origin master

Create a script to upload the repository to S3 :

• Backup Script:
  
  code_snapshot.sh

  #!/bin/sh rm /home/treselle/code_backup/blessed_repo.tgz filename=blessed_repo.tgz tar czf /home/treselle/code_backup/$filename blessed_repo.git echo 'Done Compressing' s3cmd put $filename s3://backupdemo echo 'Done Upload to S3 Bucket'

  1 2 3 4 5 6 7 8

  #!/bin/sh   rm /home/treselle/code_backup/blessed_repo.tgz filename=blessed_repo.tgz tar czf /home/treselle/code_backup/$filename blessed_repo.git echo 'Done Compressing' s3cmd put $filename s3://backupdemo echo 'Done Upload to S3 Bucket'

• Execute the script to upload the repo to S3:
  
  ./code_snapshot.sh

  1	./code_snapshot.sh

• Verify:

The blessed repo is uploaded in compressed format to the S3 bucket ‘backupdemo”

Conclusion

• Amazon S3 is the popular cloud storage solution that can be used to backup any type of files and comes with various features like Access Control List to buckets, Lifecycle configuration, Server Side Encryption and so on.
• Tools like S3cmd and S3browser come in handy to automate and verify many S3 activities similar to what was done above.

References

• S3cmd: http://s3tools.org/s3cmd-howto
• S3browser: http://s3browser.com/