Comments on: Rsync to S3 http://alexking.org/blog/2006/12/04/rsync-to-s3 Alex King, Denver Web Developer Wed, 08 Feb 2012 13:45:57 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: Jayant http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-143724 Jayant Thu, 28 Oct 2010 04:47:20 +0000 http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-143724 I've had a bad experience trying to use s3rsync. They dont publish details of their how to use th service upfront - you get that only when you pay up. And then because of the way they've setup their command line, you're basically limited to use the service only if you're running a full blown linux box where you are the admin and have root access to setup things to work with their service. That rules out any appliances, nas boxes or other sytems connecting to their service. Unfortunately, to top it off, customer support is not very responsive. They also claim not to get an email if you write to them via the contact form on their own website! Wonder why its there. I’ve had a bad experience trying to use s3rsync.
They dont publish details of their how to use th service upfront – you get that only when you pay up. And then because of the way they’ve setup their command line, you’re basically limited to use the service only if you’re running a full blown linux box where you are the admin and have root access to setup things to work with their service. That rules out any appliances, nas boxes or other sytems connecting to their service. Unfortunately, to top it off, customer support is not very responsive. They also claim not to get an email if you write to them via the contact form on their own website! Wonder why its there.

]]> By: Philippe http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-63852 Philippe Fri, 11 Jul 2008 13:25:10 +0000 http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-63852 s3rsync.com works quite reliably and it is true rsync. See my post for details: http://www.niquille.com/2008/06/15/amazon-s3-tripple-encrypted-true-rsync-backup/ s3rsync.com works quite reliably and it is true rsync. See my post for details: http://www.niquille.[...]sync-backup/

]]> By: seth http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-63219 seth Thu, 29 May 2008 23:20:51 +0000 http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-63219 @addady: --inplace *is* required for S3. it is because the underlying storage does not actually support move. The webdav abstraction (as pointed out - arrogantly :) - before) sort of hides this problem. But omitting --inplace would get you to upload the file *twice* - see the proper doc by JungleDisk @Michaelb: i suppose that you *do* have access to the data you want to up-and-store at e.g. S3? In that case: no root access is required to run the linux commandline version of jungledisk (just single binaries. Obvious dependencies: libz, libm, libgcc, libpthread - nothing out of the ordinary. I say: download and have at it!) @addady: –inplace *is* required for S3. it is because the underlying storage does not actually support move. The webdav abstraction (as pointed out – arrogantly :) – before) sort of hides this problem. But omitting –inplace would get you to upload the file *twice* – see the proper doc by JungleDisk

@Michaelb: i suppose that you *do* have access to the data you want to up-and-store at e.g. S3? In that case: no root access is required to run the linux commandline version of jungledisk (just single binaries. Obvious dependencies: libz, libm, libgcc, libpthread – nothing out of the ordinary. I say: download and have at it!)

]]> By: Dan m http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-62120 Dan m Sun, 23 Mar 2008 06:47:21 +0000 http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-62120 Dose any one try s3rsync.com ? They claim to provide full rsync on top of s3. Is it read work? Dan Dose any one try s3rsync.com ?
They claim to provide full rsync on top of s3.
Is it read work?

Dan

]]> By: addady http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-61744 addady Sun, 02 Mar 2008 13:14:06 +0000 http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-61744 Using "--inplace" is a bad idea. It mean much more bandwidth and slower backup. Form the rsync man page: --inplace meaning that the rsync algorithm can't extract the full amount of network reduction it might otherwise. This option is useful systems that are disk bound, not network bound. Using “–inplace” is a bad idea. It mean much more bandwidth and slower backup.

Form the rsync man page:
–inplace meaning that the rsync algorithm can’t extract the full
amount of network reduction it might otherwise.
This option is useful systems that are disk bound, not network bound.

]]> By: Price Tipping Point: S3 vs. Local | alexking.org http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-55622 Price Tipping Point: S3 vs. Local | alexking.org Wed, 18 Apr 2007 18:18:54 +0000 http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-55622 [...] excited about my recent experience backing up some critical files to Amazon S3, I asked Adam Tow if he’d considered using S3 for his photo backups. Adam and I have talked [...] [...] excited about my recent experience backing up some critical files to Amazon S3, I asked Adam Tow if he’d considered using S3 for his photo backups. Adam and I have talked [...]

]]> By: JungleDisk Tip | alexking.org http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-55621 JungleDisk Tip | alexking.org Wed, 18 Apr 2007 18:18:22 +0000 http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-55621 [...] you’re using JungleDisk on Mac OS X, double check that you’ve un-mounted (ejected) your JungleDisk volume [...] [...] you’re using JungleDisk on Mac OS X, double check that you’ve un-mounted (ejected) your JungleDisk volume [...]

]]> By: Michaelb http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-53853 Michaelb Sat, 06 Jan 2007 01:27:06 +0000 http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-53853 My problem is I have almost 112 GB of files on a linux server in a datacenter that I don't have root access to that I want to get to my s3 bucket. Downloading to home and then re-upping using JungleDisk would take ages and I haven't found any solution Googling yet. Anyone have a script to use rsync to S3 via the CLI yet? My problem is I have almost 112 GB of files on a linux server in a datacenter that I don’t have root access to that I want to get to my s3 bucket. Downloading to home and then re-upping using JungleDisk would take ages and I haven’t found any solution Googling yet. Anyone have a script to use rsync to S3 via the CLI yet?

]]> By: Brett http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-53809 Brett Wed, 03 Jan 2007 04:53:01 +0000 http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-53809 My arrangement: - mirror local clone nightly using SuperDuper - mirror clone over local network weekly with SuperDuper - mirror local drive monthly using SuperDuper and transport off-site Questionable and need to implement, but not certain how: - mySQL dumps to S3 on remote Unix hosted server (websites, etc.) - rsync to S3 How? - rotational backups using S3 with possibly rsync My arrangement:
- mirror local clone nightly using SuperDuper
- mirror clone over local network weekly with SuperDuper
- mirror local drive monthly using SuperDuper and transport off-site

Questionable and need to implement, but not certain how:
- mySQL dumps to S3 on remote Unix hosted server (websites, etc.)
- rsync to S3 How?
- rotational backups using S3 with possibly rsync

]]> By: Michael Visser http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-53678 Michael Visser Wed, 27 Dec 2006 12:56:29 +0000 http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-53678 Cool tips, pity about the comments. I'm throwing anything of value over FTP at the moment which is less than convenient. Cool tips, pity about the comments. I’m throwing anything of value over FTP at the moment which is less than convenient.

]]> By: Dan http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-53589 Dan Wed, 20 Dec 2006 00:19:21 +0000 http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-53589 Geof, tell us: when does Sergey learn to stop masquerading as a successful person by ceasing to be a condescending, petty asshole with an obvious inferiority complex? Geof, tell us: when does Sergey learn to stop masquerading as a successful person by ceasing to be a condescending, petty asshole with an obvious inferiority complex?

]]> By: Geof F. Morris http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-53346 Geof F. Morris Fri, 08 Dec 2006 23:51:00 +0000 http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-53346 Sergey: When you find yourself in a hole, stop digging. I would have chalked up your previous comments to being overly enthusiastic about being better than your competition, but now your non-apology just makes you look like an ass. Like Alex, I'm a Mac user and can't use your product, but you've guaranteed that I won't use anything you do now. I hardly think that I'm the only person thinking the same thing right now. I wish you the best in the future, and that involves learning when to keep your mouth shut---probably the most important lesson to learn in business. [One I'm still learning every day, in some small way.] Sergey: When you find yourself in a hole, stop digging. I would have chalked up your previous comments to being overly enthusiastic about being better than your competition, but now your non-apology just makes you look like an ass.

Like Alex, I’m a Mac user and can’t use your product, but you’ve guaranteed that I won’t use anything you do now. I hardly think that I’m the only person thinking the same thing right now.

I wish you the best in the future, and that involves learning when to keep your mouth shut—probably the most important lesson to learn in business. [One I'm still learning every day, in some small way.]

]]> By: Sergey http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-53340 Sergey Fri, 08 Dec 2006 17:36:21 +0000 http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-53340 Alex, I agree my comments on JungleDisk were kinda arrogant, sorry about that, but the point holds true. Alex, I agree my comments on JungleDisk were kinda arrogant, sorry about that, but the point holds true.

]]> By: Alex http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-53275 Alex Wed, 06 Dec 2006 19:02:42 +0000 http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-53275 I looked at your app - it looks like it could be nice. If your Mac version was available maybe I'd have used it. Since the Mac version isn't available, I found something else and I'm quite happy with JungleDisk. And may I add, disparaging your competition only serves to make you look low-brow and unprofessional. Besides I don't need a versioned backup (that's what SVN is for), I need a current copy and a system (rsync) to only mirror the changed files. I looked at your app – it looks like it could be nice. If your Mac version was available maybe I’d have used it. Since the Mac version isn’t available, I found something else and I’m quite happy with JungleDisk.

And may I add, disparaging your competition only serves to make you look low-brow and unprofessional.

Besides I don’t need a versioned backup (that’s what SVN is for), I need a current copy and a system (rsync) to only mirror the changed files.

]]> By: Sergey http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-53274 Sergey Wed, 06 Dec 2006 18:55:58 +0000 http://alexking.org/blog/2006/12/04/rsync-to-s3#comment-53274 There are better choices than rsync over jungledisk for, like my app: S3 Backup http://s3bk.com/ The point is that it's a backup solution only changed files are uploaded and such. There will also be an option for keeping multiple previous files too. It's really much neater than jungledisk because it doesn't hide the reality of what S3 is under webdav abstraction (which doesn't really fit) There are better choices than rsync over jungledisk for, like my app: S3 Backup http://s3bk.com/

The point is that it’s a backup solution only changed files are uploaded and such. There will also be an option for keeping multiple previous files too. It’s really much neater than jungledisk because it doesn’t hide the reality of what S3 is under webdav abstraction (which doesn’t really fit)

]]>