This forum uses cookies
This forum makes use of cookies to store your login information if you are registered, and your last visit if you are not. Cookies are small text documents stored on your computer; the cookies set by this forum can only be used on this website and pose no security risk. Cookies on this forum also track the specific topics you have read and when you last read them. Please confirm whether you accept or reject these cookies being set.

A cookie will be stored in your browser regardless of choice to prevent you being asked this question again. You will be able to change your cookie settings at any time using the link in the footer.

Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
CumulusMX - Automated Backups
#1
After a bit of research, reading and digging around this week, I was able to setup automated backups of CumulusMX to an external host on the network, using the cumulusmx.sh bash script and the built-in cron scheduler.

Prerequisite for this task is that you already have CumulusMX installed and successfully running on your Raspberry Pi. In this case, I am running a Raspberry Pi 3 with Raspbian (Jessie) with Noobs v1.9.0 and I'm using the built-in wireless network interface on the Pi3. I'll give brief outlines below only, and presume that you have some basic understanding and knowledge of Linux and Windows.

Mountpoint to External Host

First step is to setup a mountpoint to an external host on your network. This can be a NAS device, a server or a workstation on your network. On the selected host, create a new user and password, something like 'Raspi' and p@ssw0rD for example. Then create a new share, something like "backup", and give the new user full RW access to the share. Then create a folder in the new share called Backup_CumulusMX, to hold your backups.

We'll go with the examples above, and presume the host is called "server" and its IP address is 192.168.1.1.

You'll need cifs on your Pi, and it seems to be there preinstalled on the later releases of Raspbian. Check if it is installed with:
dpkg -s cifs-util

If cifs is not installed, fetch it with:
sudo apt-get install cifs-utils

Now you need to create a new directory for the mountpoint on your Pi. Open a shell and do the following:
cd /mnt
sudo mkdir server
sudo mkdir /server/backup

Now you need to create the new mountpoint in the fstab file for cifs.
sudo nano /etc/fstab

Add the following line to the bottom of the file (note this is all one line):
//192.168.1.1/backup /mnt/server/backup username=raspi,password=p@ssw0rD,uid=1000,gid=1000,iocharset=utf8 0 0

Now mount the new drive share:
sudo mount –a

Give the mount a few seconds to complete, and if all was well you should be able to browse down your MNT path into the backup share that you created on your network host:
cd /mnt/server/backup/Backup_CumulusMX

At this point, if you copy a file to the share on that network host, from your Windows PC, you should see it on your Raspberry Pi now (ls -l).

To test permissions are correct on the share, on the Raspberri Pi create a test file:
cd /mnt/server/backup/Backup_CumulusMX
sudo nano test1.txt

Save and exit nono editor, and you should see the file in the share, on the host device. If you got this far, all is well with your mountpoint to the external host

Setup Backups in CumulusMX

Go back to your home drive on the Pi, with cd ~
Then CD into the CumulusMX directory. Edit the CumulusMX config file:
sudo nano cumulusmxsh.conf

Add the cifs share as the backup path:
BACKUPFOLDER= "/mnt/server/backup/Backup_CumulusMX/"

Now you can run quick manual (one off) test backup:
sudo ./cumulusmx.sh -b -t

If this worked ok you should have a TAR backup in your network share, and your good to setup the scheduled task next.

Schedule Automated Nightly Backups

To acheive the automated backups, you'll add a scheduled task to the CRONTAB to do the nightly backups:
sudo crontab –e

if asked, select 'nano' as your editor (easiest)

Insert a new line at the bottom of the crontab file:

00 01 * * * sudo /home/pi/CumulusMX/cumulusmx.sh –b –t >> /dev/null 2>&1

Alt-X, Y to save, Enter to select the filename

This example will backup every night at 1am. By default, the cumulusmx.sh script will keep the last five backups, in a rolling window, so your backup share should never fill over time. You can edit the number of kept backups in the cumulusmxsh.conf file.

Hope this info helps. If I've made any typos or something doesn't like or work right, post below and together we'll work it out.

Cheers for now...

Mark R
Reply
#2
Excellent stuff Wizza. I'll be setting that up for sure (once I get to that stage). Thanks again.
Reply
#3
Thumbs Up 
Perfect, Mark.
Best Regards
Zdenek

[Image: banner.php]
My outdoor AQM-I: here
Reply
#4
Nice write up Wizza.
I reckon I'll be sticking with my clickity click click OS though Smile
Reply
#5
If you are using a Raspberry Pi, it's also a good idea to take an image of the entire SD card as a backup once in a while. The Raspberry Pis have a nasty habit of occasionally toasting their SD cards - in my experience, this is more likely if the power fails while the computer is writing to the card. It's not hard to set up a 5v "UPS" for the Raspberry Pi and well worth doing.
Reply
#6
(05-05-2016, 14:09)uncle_bob Wrote: Nice write up Wizza.
I reckon I'll be sticking with my clickity click click OS though Smile

Hello my dear friend uncle_bob, I confess, me too! Smile
MeteoCercal - Air Quality Data
Click here to watch at my ThingSpeak channel



Reply
#7
(05-05-2016, 19:24)Shred Wrote: If you are using a Raspberry Pi, it's also a good idea to take an image of the entire SD card as a backup once in a while. The Raspberry Pis have a nasty habit of occasionally toasting their SD cards - in my experience, this is more likely if the power fails while the computer is writing to the card. It's not hard to set up a 5v "UPS" for the Raspberry Pi and well worth doing.

What can be another good idea, and which can prevent some of the problems with the SD card, is run Cumulus from an SSD disk.
I have one 32Gb SSD disk (less than 20€) inside a USB2 case (5€), formatted in NTFS, and I'm running Cumulus MX from there. If needed it's easy to unplug the disk from the PI, and copy the files to a Windows system. Of course an automated backup system as described by Mark can also be used with the SSD.
MeteoCercal - Air Quality Data
Click here to watch at my ThingSpeak channel



Reply
#8
(05-05-2016, 19:55)Werk_AG Wrote: What can be another good idea, and which can prevent some of the problems with the SD card, is run Cumulus from an SSD disk.
I have one 32Gb SSD disk (less than 20€) inside a USB2 case (5€), formatted in NTFS, and I'm running Cumulus MX from there. ...

Werk_AG
Which SSD brand are you using? I've been playing with a Kingston SSD and an old OCZ SSD but have encountered problems with both. The Kingston has trouble auto-mounting on boot and a bit of research indicated that this is a standard problem with Kingston devices on RPi. I then tried the OCZ but it has intermittent problems. It is supplied from a powered hub so I don't think that power is the problem. I set up a touch cronjob to make sure that it didn't go to sleep but that didn't solve things either. I'm keen to perservere but knowing what brand you were running would be a useful first step (also anyone else that is successfully running with an SSD).

Thanks
Reply
#9
Hi,

I have two, both are: KingSpec SATA III 3.0 2.5" 32GB MLC Digital SSD Solid State Drive
They cost around 17€ on eBay

No problems with auto-mounting on boot with PI 2B and PI 3
MeteoCercal - Air Quality Data
Click here to watch at my ThingSpeak channel



Reply
#10
It helps to use a Micro SD card that implements wear levelling. The manufacturers don't make it easy to identify these. My research indicates that the Sandisk Extreme ones do this - but you pay dearly for it.

For those running from an external USB SSD drive, the Raspberry Pi 3 supports booting directly from USB (and from an NFS server):
https://www.raspberrypi.org/blog/pi-3-bo...rage-boot/

I mount my main linux server via NFS and use a cron job to backup every night. I also write weather data to a MySQL database on the main server to minimise wear on the Raspberry Pi MicroSD card.
Reply




Users browsing this thread: 1 Guest(s)