Feb 17, 2015

Duply with a powernapping system

I've started backing up one of my systems to S3. The instructions from the Phusion blog worked almost perfectly, except my TARGET line was

TARGET='s3+http://<my-bucket-name>'

Also on the AWS side, I set up a lifecycle rule to archive the backups to Glacier after 7 days.

I did run into some issues getting the backups to work together with powernap, which was configured to put the system to sleep after a few minutes of inactivity.

Powernap was causing a problem on two fronts. First, the system was going to sleep mid-backup since full backups take longer than the powernap inactivity timeout. Second, the backups were scheduled for the middle of the night when the system would normally already be asleep.

To get around the mid-backup sleep issue, I made a /usr/local/bin/duply-nightly script which shuts down powernap before calling duply and restarted it afterwards.

To get around the system-already-asleep issue, I'm using an RTC wakeup in /usr/local/bin/duply-nightly to set the system to wake a few minutes before the cron job kicks off (but not early enough for powernap to put the system to bed again...)

The first night I ran the backup, I had to prime the /sys/class/rtc/rtc0/wakalarm time manually, but since then the script has set the wakeup time for the next day

The final /usr/local/bin/duply-nightly script is below

#!/bin/sh +x

/usr/bin/logger "Running nightly backup from $0"

# Disable powernap during the backup
service powernap stop

/usr/bin/duply nightly backup

# Wakeup the system at 3:00am tomorrow
echo 0 > /sys/class/rtc/rtc0/wakealarm
echo `date '+%s' -d '3am next day'` > /sys/class/rtc/rtc0/wakealarm

# Enable powernap again.
service powernap start

The cron job that kicks on /usr/local/bin/duply-nightly is below

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

5 3 * * * root env HOME=<myhomedir> /usr/local/bin/duply-nightly > /var/log/duply-nightly.log

Aug 09, 2013

The case of the late night system boots

A few weeks back, at around 3:00AM I was woken up by the sounds of a computer booting. It turned out to be an old Ubuntu desktop in the study. Since it was storming at the time, I figured it due to a power outage so I shut down the machine and went back to bed.

A couple nights later the same thing happened. This time I made sure the BIOS on the machine was set to restore the machine to its previous state after an outage instead of turning it on. It turned out it was already set to restore the previous state. Odd, but it was 3:00AM so back to bed I went.

The third time it woke me up was soon during the time Thinkhost/Dreamhost dropkicked the old site in the gonads, and I start freaking out a little - I wasn't getting email, my corny vanity site had disappeared and I had a machine turning itself on in the middle of the night. It was coming out of hibernation when it started, so when I noticed it, I was already logged into the console and my 3:00AM self didn't notice any weird activity on the machine. I poked around on the BIOS again to make sure the Wake-on-LAN options were not set and powered down the computer. Still, I started pulling Ethernet cables out before going to bed until I had a chance to really figure out what was going on.

Then over the weekend I had started the machine up and then started some Laundry [*]. When I eventually got back to study, the machine it was still one. I'd been gone long enough that it should have gone into hibernation by then. I did a 'sudo /usr/sbin/pm-hibernate' manually and it went through its normal hibernate routine, but after it powered down, it immediately booted back up again !

Then it all made sense - the machine had been trying to hibernate and rebooting, then waiting some number of minutes with no keyboard activity and repeating the cycle. If I was sleeping lightly enough at the time , the beeps during the restart would wake me up, otherwise I was sleeping through all the restarts. Eventually someone in the house would notice the machine was on and shut it down gracefully. I have no clue how long hibernate had been failing like this. Weeks at least.

But onto the solution. I didn't find much on the Internet about the problem (hence this post), but the combination of the basic-pm-debugging.txt and the pm-hibernate man page led me to try adding an '/etc/pm/config.d/hibernate file with the contents:

HIBERNATE_MODE=shutdown

With that in place, pm-hibernate went back to working like it used to, and I haven't been woken up by late night restarts since.

[*]For purposes of this blog, 'Laundry' is any non-computer indoor activity and 'Going for a Walk' is any non-computer outdoor activity.