climateprediction.net (CPDN) home page
Thread 'CPDN hogging disk'

Thread 'CPDN hogging disk'

Questions and Answers : Preferences : CPDN hogging disk
Message board moderation

To post messages, you must log in.

Previous · 1 · 2

AuthorMessage
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 46105 - Posted: 29 Apr 2013, 2:13:51 UTC - in response to Message 46100.  

Richard Rodway's automatic backup was for the BBC project's models, and the
server which hosted that project has long since been decommissioned, and the files weren't saved.
These days those people that make backups "do it themselves".

Some thoughts on making one of your own.

Use a batch or script file.

It'll need to:
Check that each climate model is nowhere near a checkpoint.
Suspend each model if safe to do so as in line above.
Suspend BOINC.
Exit from BOINC.
Copy entire BOINC (data) folder set to a "save" location. Possibly labelled with the date.
Re-start BOINC.
Un-suspend BOINC.


Backups: Here
ID: 46105 · Report as offensive     Reply Quote
Belfry

Send message
Joined: 19 Apr 08
Posts: 179
Credit: 4,306,992
RAC: 0
Message 46106 - Posted: 29 Apr 2013, 3:06:30 UTC

When I crunched on Windows I used UK_Nick's vbs backup (unfortunately I can't link to climateprediction.net), which nimbly waited for each task to checkpoint, paused them, then stopped BOINC. Of course it didn't do anything with crashed models.

Here's my Linux Bash backup script. It won't wait for checkpoints, but when combined with a Cron job like "30 10 * * * PATH=$PATH:/sbin /etc/scripts/boinc-backup.sh" it will make a backup every fourth day at 10:30 AM.

#!/bin/bash

/bin/pidof boinc > /dev/null 2>&1 || exit

[ $(( ($(date +%s)/86400) % 4 )) -ne 0 ] && exit  # run only every fourth day
find /var/lib/boinc-backup/*.tgz -mtime +5 | xargs rm  # delete 2nd to last backup

/etc/init.d/boinc-client stop

nice -n 1 tar -czpf /var/lib/boinc-backup/boinc-client_backup-$(date +%d%m%Y).tgz -C /var/lib/boinc-client $(ls /var/lib/boinc-client)

/etc/init.d/boinc-client start
ID: 46106 · Report as offensive     Reply Quote
Belfry

Send message
Joined: 19 Apr 08
Posts: 179
Credit: 4,306,992
RAC: 0
Message 46107 - Posted: 29 Apr 2013, 3:14:30 UTC - in response to Message 46105.  


...
It'll need to:
Check that each climate model is nowhere near a checkpoint.

I don't think this is necessary, although it's nice to pause them for electricty considerations. With eight or more CPDN tasks running something is always near checkpointing.
ID: 46107 · Report as offensive     Reply Quote
Les Bayliss
Volunteer moderator

Send message
Joined: 5 Sep 04
Posts: 7629
Credit: 24,240,330
RAC: 0
Message 46108 - Posted: 29 Apr 2013, 4:03:58 UTC

Unfortunately, the old php board is now defunct, and unlikely to be back.
I'll try to see if I can retrieve Nick's files.

Re: checkpoints - there's a lot of files to save at each checkpoint, which takes time to do.
If the model is paused/stopped in the middle of this, then they become corrupted, and the backup for that one will fail to start.

As for lots of models, that's why I wait for each to get clear of a checkpoint if necessary before suspending it. Then I move on to the next one.

My method for making a backup without exiting, by using xcopy:
Suspend BOINC first.
options: /D/E/C/I/K/O/Y


Backups: Here
ID: 46108 · Report as offensive     Reply Quote
Previous · 1 · 2

Questions and Answers : Preferences : CPDN hogging disk

©2024 cpdn.org