Backing up account

[Matt]

Hi,
I would like to do the following:
Telnet into account, create zip backup of entire account that preserves directories and subdirectories, and use FTP to back up zip.

Also, if my account needs to be reconstructed, how do I go about reconstructing it from the zip? Do I have to do it manually, or can I run a command to automate the process?

The reason for this is that both LeechFTP and WS_FTP_LE require some sort of ASCII/ binary toggle to transfer files correctly. This is okay for a small number of file transfers, but gets quite tedious for backing up an account. Also, what happens if the zip backup exceeds the account limit? Is the process killed? Do I get an ugly e-mail? If the zip is created, would it be okay if it exceeded the allotted storage just long enough for me to transfer it, then delete it from my account?
-Matt

[Tibbits]

If you're sending a zip file by FTP (although a tar.gz would make more sense) then the ascii/binary problem ceases to be.. the archive is a binary and the contents are preserved as-is.

I think it wouldn't have mattered in the past if you temporarily want slightly over your limit, but FQ are tightening up. You'll have to wait for them to get back to you on this.

To recontruct just untar the archive and the filestructure will be recreated, although I'd think you'd have to redo the file permissions.

[Matt]

Would a tar.gz preserve the permissions?

[PaulKroll]

"man tar" should reveal that tar can be told to preserve permissions: it's the -p option. :)

And bzip2 generally gives the best compression: the tradeoff there is I'm not aware of any common Windows programs that support it.
[This message has been edited by PaulKroll (edited 02-07-01@5:27 pm)]

[Tibbits]

I don't believe so.. but someone feel free to prove me wrong..
tarballs seem to be used more on Linux than zips, which are used more on windows. I think that tarring and gzipping gives smaller archives than zipping, but again I could be wrong.

[colulus]

http://powerarchiver.efront.com/

A Windoze program that supports bzip2.

Freeware.[nbsp][nbsp]I use it instead of Winzip.

[Jeff]

Now this would be an excellent add-on to the CNC:

A one button "Backup Account" which created a highly compressed single file which preserves the permissions, symbolic links, and avoids ascii/binary problems.

What would be even better is if it would create the backup file for me to download in a "temporary quota buffer" so that you could backup say 40 megs of a 50 meg package without exceeding your quota or having to purchase extra space just to use as storage for the backup file.

If it was really sophisticated, it could even delete the backup file in 6 hours automatically to allow a modem user to download the backup at night but have it deleted as soon as the download was complete to save space.

Terra may have already mentioned something like this - I can't remember but it sounds familiar now that I've typed this.[nbsp][nbsp]Still sounds like a nice feature :)

[Terra]

I will take your idea under advisement, however:
1) I am not too keen on having to eat the disk space consumed...[nbsp][nbsp](NOTE 1)
2) Queuing system will need to be established with no more than 2 backup processes running at any one time on any one server...
3) Controls to avoid abuse or neglect (NOTE 2)

Note 1: The site owner should be responsible for any and all disk space used...[nbsp][nbsp]After all - we are in the service business to provide value-added services... Value-Added and Free do not always go hand-in-hand and 'Free' could be detrimental to our longetivity...[nbsp][nbsp];)

Note 2: It could be perceived as a 'toy' with little regard to the server resources required to perform such task, such is evident with the Disk_Usage module...[nbsp][nbsp]Neglect in the area that 'tinkering' would lead to backups never downloaded, and/or the fiddling with the backup options while trying to figure out exactly what (and/or how) they want to backup...

--
Terra
--Free Lunch usually means soup beans again--
FutureQuest

[ zTAG ] CNC Backup Module

[Terra]

<TAG: NCC-1701D>

Aiyyeeeeeeee![nbsp][nbsp]Someone restrain me, because I could write a book right now...[nbsp][nbsp];)[nbsp][nbsp]

Why can't anything be simple... In short, what I am designing will increase the complexity of our architecture, however the paradoxical result will be simplistic methods of procedure that will allow currently impossible tasks...

After reading your post, it got me brainstorming as this issue had several missing pieces to the puzzle...[nbsp][nbsp]In a matter of minutes a new design for a 5 layer/stage backup and online/offline storage system was pieced together that would actually work in a realistic manner...[nbsp][nbsp]:)

I will answer your 'Big Assumptions' for what parts I can right now:
backup would capture both files and mysql dbs. You would most likely have the choice of either or both...

Backup would work even under your next generation server changes, if/when/how those are done (could get complicated with HA servers, Tux or Boa handling static, etc.) It will have to wait until then, as it currently falls into the next to impossible realm right now...

Backup would either have to halt the users site while backup was going on or user would have to deal with the possibility that content being generated while the backup was going on was going to get lost/corrupted/worse. Worst Case)[nbsp][nbsp]I would have to rely on the paranoid quality of the programmer and hope that they are using Linux file locking ( flock() )on any current 'open-for-write' files...[nbsp][nbsp]At least then I can try to:
if grab_lock:
[nbsp][nbsp]lock exclusive
[nbsp][nbsp]backup the file
[nbsp][nbsp]release lock
else
[nbsp][nbsp]add file to list of locked files (untouchables)
[nbsp][nbsp]continue onward
fi
alarm(120) retry 'grab_lock untouchable'

END{walk untouchables at end, grab_lock || force_backup untouchable anyways}

Best Case) My ultimate goal is to have (very low level) snapshot capability which would peer into the kernel structures and see if the file is open for writing/append/truncation...[nbsp][nbsp]This would remove the whole flock() mess above as I see it for what it really is in realtime and can adjust operations accordingly...[nbsp][nbsp]This would be the safest mode overall, but very very difficult to implement without inducing race conditions...

And then there are the folks who connect via modem or otherwise just can't practically download huge piles of data. This one is pretty simple to solve...[nbsp][nbsp]Why not just get a Offsite Storage Locker:
http://directory.google.com/Top/Computers/Internet/WWW/Web_Applications/Virtual_Disk_Drives/
Then you would not have to worry about slow downloads...[nbsp][nbsp]The only caveat to offline backups is the bandwidth used in transferring the archive from our servers to either your computer, or to a virtual storage locker...

I wish I could dive deeper in the architecture as I've got a box of 64 Crayola Crayons and beginning to fill in the lines created with my Etch-A-Sketch...[nbsp][nbsp]What I can say is that our current setup is child's play in comparison to what I'm envisioning...[nbsp][nbsp]I'm at the point to where many of the missing pieces are now appearing out of nowhere and shattering prior obstacles and hurdles...[nbsp][nbsp]It may take me quite awhile to piece it all together, but it's already devised to be an incremental forward modular retrofit...

I really must stop now![nbsp][nbsp]:P

--
Terra
--I'm told that if I pull the tape too quickly, that it will rip my lips off--
FutureQuest

<EDIT: missing quote>
[This message has been edited by ccTech (edited 02-20-01@2:28 pm)]

[Paul]

Uh oh..[nbsp][nbsp]:P

Paul
- I can see all obstacles in my way. .. Gone are the dark clouds that made me crash -[nbsp]

[PaulKroll]

And then there are the folks who connect via modem or otherwise just can't practically download huge piles of data.

So...

$xx extra for nightly backup to a &quot;backup-server&quot; that just keeps the archives?

I'd pay... let's see... more than a dime for that. :) OK, OK, $25/month (for my Gold account) easy for the peace of mind. For a Platinum or larger account, it could be 3 megs/$1, so that's $50 for Platinum. Seems (from my perspective) fair given the extra storage, CPU time, and safety of having a seperate machine(s) keeping the data without my having to download it every day.

Obvious Big Assumptions here include: backup would capture both files and mysql dbs. Backup would work even under your next generation server changes, if/when/how those are done (could get complicated with HA servers, Tux or Boa handling static, etc.) Backup would either have to halt the users site while backup was going on or user would have to deal with the possibility that content being generated while the backup was going on was going to get lost/corrupted/worse.

Why can't anything be simple... :(