Feb 17, 2015

Duply with a powernapping system

I've started backing up one of my systems to S3. The instructions from the Phusion blog worked almost perfectly, except my TARGET line was

TARGET='s3+http://<my-bucket-name>'

Also on the AWS side, I set up a lifecycle rule to archive the backups to Glacier after 7 days.

I did run into some issues getting the backups to work together with powernap, which was configured to put the system to sleep after a few minutes of inactivity.

Powernap was causing a problem on two fronts. First, the system was going to sleep mid-backup since full backups take longer than the powernap inactivity timeout. Second, the backups were scheduled for the middle of the night when the system would normally already be asleep.

To get around the mid-backup sleep issue, I made a /usr/local/bin/duply-nightly script which shuts down powernap before calling duply and restarted it afterwards.

To get around the system-already-asleep issue, I'm using an RTC wakeup in /usr/local/bin/duply-nightly to set the system to wake a few minutes before the cron job kicks off (but not early enough for powernap to put the system to bed again...)

The first night I ran the backup, I had to prime the /sys/class/rtc/rtc0/wakalarm time manually, but since then the script has set the wakeup time for the next day

The final /usr/local/bin/duply-nightly script is below

#!/bin/sh +x

/usr/bin/logger "Running nightly backup from $0"

# Disable powernap during the backup
service powernap stop

/usr/bin/duply nightly backup

# Wakeup the system at 3:00am tomorrow
echo 0 > /sys/class/rtc/rtc0/wakealarm
echo `date '+%s' -d '3am next day'` > /sys/class/rtc/rtc0/wakealarm

# Enable powernap again.
service powernap start

The cron job that kicks on /usr/local/bin/duply-nightly is below

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

5 3 * * * root env HOME=<myhomedir> /usr/local/bin/duply-nightly > /var/log/duply-nightly.log

Aug 31, 2014

Count your YAML dashes

I spent way too long the other day thrying to figure out why some Hiera variable wasn't available in some puppet manifest.

It turns out there was a typo in the YAML file where the first line of the file only had two dashes instead of three.

In this cases, the Ruby 1.8.7 yaml parser corrupts the first entry in the yaml file, adding the two dashes to the beginning of the key, instead of throwing a parser error.

irb(main):001:0> require 'yaml'
=> true
irb(main):002:0> a=YAML.load_file("only-two-dashes.yaml")
=> {"-- first"=>1, "second"=>2}
irb(main):003:0> b=YAML.load_file("correctly-including-three-dashes.yaml")
=> {"first"=>2, "first"=>1}

I couldn't find an existing bug for this, but I didn't look to harrd since this has been fixed in Ruby 1.9

irb(main):001:0> require 'yaml'
=> true
irb(main):002:0> a=YAML.load_file("only-two-dashes.yaml")
Psych::SyntaxError: (t1.yaml): couldn't parse YAML at line 1 column 1
        from /usr/lib/ruby/1.9.1/psych.rb:154:in `parse'
        from /usr/lib/ruby/1.9.1/psych.rb:154:in `parse_stream'
        from /usr/lib/ruby/1.9.1/psych.rb:125:in `parse'
        from /usr/lib/ruby/1.9.1/psych.rb:112:in `load'
        from /usr/lib/ruby/1.9.1/psych.rb:229:in `load_file'
        from (irb):2
        from /usr/bin/irb1.9.1:12:in `<main>'.

So if you're using Ruby 1.8.7, and it looks like the first item in your YAML file is being dropped for some reason, check the first line of the file has 3 dashes.

Oct 03, 2013

Deooglization

Here are the steps I've taken recently to cut down on the amount of free data I've been passing on to Google, and by proxy any unsupervised NSA contractors who may be running amok. I've taken to calling this project 'Deooglization'.

  • Uninstalled Chrome , switched back to Firefox for daily browsing
  • Switched the search bar to use DDG via the DuckDuckGo Plus addon
  • Shuttered the G+ account associated with my identity
  • Migrated away from gmail - Implemented most of Daniel Patterson's Hacker's Replacement for GMail (minus notmuch since I don't live in Emacs that deeply)
  • On StackOveflow, I stopped using Google for an OpenID source and switched to openid.stackexchange.com
  • Removed as many Google apps from my phone as I could.

Outside of Google, I also shut down the Flickr account associated with my identity. I haven't had a Facebook in ages so there wsa nothing to shut down there. I haven't shut down my Twitter account yet, but it's just a matter of time, I suppose.

I've also found that diversifying my passwords has helped a lot: in a few cases where I'd gone to google out of habit, it's been such a PITA to retrieve my password that I just do something else, like take a walk or catch up on the laundry.

Next steps: setting up a Dropbox replacement.

Aug 09, 2013

The case of the late night system boots

A few weeks back, at around 3:00AM I was woken up by the sounds of a computer booting. It turned out to be an old Ubuntu desktop in the study. Since it was storming at the time, I figured it due to a power outage so I shut down the machine and went back to bed.

A couple nights later the same thing happened. This time I made sure the BIOS on the machine was set to restore the machine to its previous state after an outage instead of turning it on. It turned out it was already set to restore the previous state. Odd, but it was 3:00AM so back to bed I went.

The third time it woke me up was soon during the time Thinkhost/Dreamhost dropkicked the old site in the gonads, and I start freaking out a little - I wasn't getting email, my corny vanity site had disappeared and I had a machine turning itself on in the middle of the night. It was coming out of hibernation when it started, so when I noticed it, I was already logged into the console and my 3:00AM self didn't notice any weird activity on the machine. I poked around on the BIOS again to make sure the Wake-on-LAN options were not set and powered down the computer. Still, I started pulling Ethernet cables out before going to bed until I had a chance to really figure out what was going on.

Then over the weekend I had started the machine up and then started some Laundry [*]. When I eventually got back to study, the machine it was still one. I'd been gone long enough that it should have gone into hibernation by then. I did a 'sudo /usr/sbin/pm-hibernate' manually and it went through its normal hibernate routine, but after it powered down, it immediately booted back up again !

Then it all made sense - the machine had been trying to hibernate and rebooting, then waiting some number of minutes with no keyboard activity and repeating the cycle. If I was sleeping lightly enough at the time , the beeps during the restart would wake me up, otherwise I was sleeping through all the restarts. Eventually someone in the house would notice the machine was on and shut it down gracefully. I have no clue how long hibernate had been failing like this. Weeks at least.

But onto the solution. I didn't find much on the Internet about the problem (hence this post), but the combination of the basic-pm-debugging.txt and the pm-hibernate man page led me to try adding an '/etc/pm/config.d/hibernate file with the contents:

HIBERNATE_MODE=shutdown

With that in place, pm-hibernate went back to working like it used to, and I haven't been woken up by late night restarts since.

[*]For purposes of this blog, 'Laundry' is any non-computer indoor activity and 'Going for a Walk' is any non-computer outdoor activity.