backing up data off-site to nas

6

60221654

Messages
109
Edit My Images
No
can anybody recommend any efficient sync software (either hosted on windows or osx) for backing up data off-site where the transfer/upload speed is close to that of your isp's rate? ideally what i'm looking for is software recommendations people use to sync/make off-site backups to a NAS box as opposed to storing at cloud level. i'm sure it's not fool-proof, but one big advantage i've read about backing up this way is that the limits on storage are determined only by the amount of space available in the NAS, but obviously there's a hefty investment in boxes, drives and your time.

thanks
 
I worked it out that it would take 10mins to transfer 1GB of data online, so I gave up any idea of online backup.
 
1gb despite sensor sizes of today's cameras is still a fair amount of images and 10mins whilst being able to go off and attend to something else would be perfectly acceptable. have you tried this or just as you inferred, done the maths?
 
Last edited:
Can you ssh into the backup device? If so, I'd use rsync. Designed to do exactly what you want.
 
60221654 said:
1gb despite sensor sizes of today's cameras is still a fair amount of images and 10mins whilst being able to go off and attend to something else would be perfectly acceptable. have you tried this or just as you inferred, done the maths?
Click to expand...

I have about 3 - 3.5 TB of images so I've not tried it.
 
quite possibly but i'll need to check a few things first. i found this on rsync just in case anybody else is interested but looks like it will require a fair bit of reading and commitment to understand things. looks interesting though so thank you
 
I have WHS but I don't know if I can do an online backup in the way you mention, will have a play later.
 
60221654 said:
quite possibly but i'll need to check a few things first. i found this on rsync just in case anybody else is interested but looks like it will require a fair bit of reading and commitment to understand things. looks interesting though so thank you
Click to expand...
It really isn't difficult.

Once you have ssh access (i.e. you can log into the remote box via a secure connection provided by the program ssh - this can be passwordless once you set it up properly) the command line is pretty simple. I run the following command every 20 minutes on my main PC (file paths shortened):

rsync --exclude-from=<excludefiles.txt> -ave ssh --delete <sourcedir> <user>@<machine>:<destdir>

All that says is copy everything from <sourcedir> to <destdir> on the computer <machine> using account <user> skipping the files in <excludefiles.txt>. The -ave ssh says use an ssh connection to transport the data which means the data is encrypted as it is copied and copy it in a special format and log what you are doing, the --delete tells rsync to delete files from the destination if they have been deleted from the source.

The excludefiles.txt is a simple text file telling rsync what to ignore and has the format of a line per file/directory you want to exclude.

I have variations of this command line to backup my backups between machines overnight and to share certain data I want on all machines.

ssh/rsync is a fundamental pair of tools and available on every o/s I can think of.

As I said, if you have command line access, it is by far the best way of moving data around as it is designed for supporting slower links.
 
60221654 said:
i found this on rsync just in case anybody else is interested
Click to expand...
PS. that page talks about setting up an rsync server. The way I do it, you don't need to have an rsync server set up - just an ssh server. If you have command line access to the remote box, it is HIGHLY likely this will already exist.
 
I tried uploading a 3GB folder remotely to WHS but fell at the 1st hurdle, you can only upload 1 file at a time.
 
I'm going to try and install FTP software
 
I've tried this a quite few times ftp-ing large amounts of stuff stuff back and fro, and it's been a pain in the arse nearly every time. I have a good fibreoptic broadband but even with that it's slow and eratic, I usually end up having to upload the file a few times to get it to work, then end up re-downloading to check it's all gone (all too often it hasn't)
As a back up solution unless it's only small amounts I wouldn't even consider it personally, a better bet is put an external drive in the shed or back up to blueray disk.
 
Well I've installed all the software so will do a test later today
 
arad85 said:
It really isn't difficult.

Once you have ssh access (i.e. you can log into the remote box via a secure connection provided by the program ssh - this can be passwordless once you set it up properly) the command line is pretty simple. I run the following command every 20 minutes on my main PC (file paths shortened):

rsync --exclude-from=<excludefiles.txt> -ave ssh --delete <sourcedir> <user>@<machine>:<destdir>

All that says is copy everything from <sourcedir> to <destdir> on the computer <machine> using account <user> skipping the files in <excludefiles.txt>. The -ave ssh says use an ssh connection to transport the data which means the data is encrypted as it is copied and copy it in a special format and log what you are doing, the --delete tells rsync to delete files from the destination if they have been deleted from the source.

The excludefiles.txt is a simple text file telling rsync what to ignore and has the format of a line per file/directory you want to exclude.

I have variations of this command line to backup my backups between machines overnight and to share certain data I want on all machines.

ssh/rsync is a fundamental pair of tools and available on every o/s I can think of.

As I said, if you have command line access, it is by far the best way of moving data around as it is designed for supporting slower links.
Click to expand...


thank you very much for that explanation. you make it sound so easy! perhaps if you've come from a terminal/coding background or invested the time to learn about it then maybe, but the last couple of weeks have been me drilling down through eye-watering chunks of information on the topic and to be honest, i've come :razz: close to throwing the towel in. however learning something new is both a rewarding/frustrating learning curve and there's definitely renewed vigor in trying to get this protocol set up and running efficiently.

the box in answer to your question will do ssh access (with an installed add-on) but it invalidates the warranty on support for software due to root access so that takes me back to the ssh reading drawing board for now. i have though managed to implement a solution for getting around not having a static ip, familiarized myself with the all important admin panel plus discovered ways of accessing the box locally via both ftps and https. what is required now is to test it remotely over these two protocols and see if with all the firmware/software updates carried out, the transfer speeds are any better
 
60221654 said:
perhaps if you've come from a terminal/coding background
Click to expand...
Guilty as charged :geek:

60221654 said:
the box in answer to your question will do ssh access (with an installed add-on) but it invalidates the warranty on support for software due to root access so that takes me back to the ssh reading drawing board for now.
Click to expand...
Can you not create a user and ssh to that user? I can understand people wasshing their hands of support if you do a lot through root access, but surely not shell access where you've created a user specifically with lower privileges for a specific job.

Anyway, well done on getting this far. Whilst most baulk at command lines, once you start out down that road, you get to see how much more powerful command line is over any GUI wrapper :)
 
it's a readynas neil. bought a couple of years ago and couldn't then get it to work efficiently so put it back in its box! have you got an up and running working nas system working?
 
netgear? they do (or at least did when i had mine) rsync by default without having to mod, used to just be a backup option im sure. i used to use that to sync my two nv+ boxes, albeit locally but shouldnt be hard using a dynDNS service.

edit - ah sorry doesnt support SSH though.

double edit - actually seems unclear, some say it is on business class devices only, others say on newer firmware. when you create a new backup do you have the SSH option? http://www.readynas.com/media/images/rsync_ssh/rsync_ssh_1b.jpg
 
Last edited:
neil_g said:
netgear? they do (or at least did when i had mine) rsync by default without having to mod, used to just be a backup option im sure. i used to use that to sync my two nv+ boxes, albeit locally but shouldnt be hard using a dynDNS service.

edit - ah sorry doesnt support SSH though.

double edit - actually seems unclear, some say it is on business class devices only, others say on newer firmware. when you create a new backup do you have the SSH option? http://www.readynas.com/media/images/rsync_ssh/rsync_ssh_1b.jpg
Click to expand...

yes netgear nvx. they do have an ssh enable add-on but activating this invalidates the warranty. you're correct in that rsync is built into the box (or possibly added through newer firmware) but surely you need root in order to access rsync?
 
like yourself backing up locally isn't a problem but it's another can of worms broaching the net hence rsync.

on a side note i had downloaded what i thought was rsync (via samba) but couldn't get the terminal to show up like i could with putty. i was just about to post a question to you about rsync when i decided to have another look for an executable rsync that would run on windows and although i've yet to try it, this download definitely looks a lot more like the interface i had (hoped for and) expected. another piece of the jigsaw...
 
What are you trying to backup? From a windows machine to a remote NAS somewhere? You should just be able to type rsync in the putty terminal. TBH, I'd just install a minimal install of cygwin and add the rsync executable. You can even then add cron support to automatically backup overnight...
 
I use a D-Link 320 NAS combined with Allway Sync software. Both continue to work flawlessly.
 
neil_g said:
indeed i think we need clarification of exactly what is trying to be acheived.
Click to expand...


first and foremost it will be for off-site back-up/retrieval of images from a local computer/laptop running windows over wifi (hence the need for a secure tunnel) with the remote nas attached via a gigabit router. the backup batches range anywhere from 0.5 to a couple of gig each and the plan is to have these backed up (preferably via a schedule) overnight. backing up locally isn't a problem but previous attempts to get this working outside of the local network has bottle-necked at approx 40kb/s which at the time rendered this unusable. obviously upload rate will be something determined by the isp but the then 40kb/s pointed to some major fundamental flaw in the original set up i had which i sadly never managed to solve.

however incremental backup as suggested by arad via rsync sounds in theory to quite a desirable strategy and if with the right settings, interface and protocol, (hell even block level will do!) transfer produced something in the region of 10mb/s uploads, then i think i could categorically state that i'd pretty much wet myself.

so where do i go next?
 
60221654 said:
so where do i go next?
Click to expand...
IMHO...

  • Setup ssh/rsync on the NAS. If it invalidates s/w support, then so be it. You need these to work securely.
  • Install cygwin on Windows machine (http://www.cygwin.com)
  • Add openssh, rsync, mintty to cygwin
  • Generate keys to allow passwordless logins (you need to create one on the PC and ad it to the server)
  • ssh into the NAS to authenticate and prove you have passwordless setup correctly
  • try rsync from the windows machine to the nas

Let us know how you get on ;)
 
You must log in or register to reply here.
Back
Top