Oct 08, 2017

Elixir on Centos 7

I'm at the point with a Elixir/Phoneix side project that I'm thinking about deployment.

The first big stumbling block was that development envioronment (Ubuntu 16.04) wasn't the same as my deployment environment (Centos 7). For Ubuntu, I could pull the latest Elixir packages from Erlang Solutions, but they don't host Centos Elixir packages, and the version of Elixir on EPEL is over 4 years old - old enough that 'mix deps' on my project was erroring out.

I found the posts below about installing Elixir on Centos 7 but, they involve cloning the Elixir repo and building it from source, but I don't want development tools like git and gcc on my production machines.

I also found https://www.vultr.com/docs/how-to-install-the-phoenix-framework-on-centos-7 but it involves downloading a precompiled mystery meat bundle off Github. However https://github.com/elixir-lang/elixir doesn't mention that these precompiled bundles are available or how the bundles were built. The precompiled bundle is mentioned on the elixir-lang.org install page, but still, that's too mysterious for me.

Maybe one of those links will work for you, but what I want is a way to build a more recent version of the Elixir rpm than what was available in EPEL. That way I can recompile and package Elixir on a dev machine, and then only copy the rpm up to my production machine.

It looks like Elixir is stuck in EPEL waiting for the Erlang 18 release to get promoted in EPEL, so maybe I can take the existing Elixir packaging source and build it against the latest Erlang packages from Erlang Solutions...

I found the packaging source at https://src.fedoraproject.org/rpms/elixir and after poking around a bit I came up with Vagrantfile below. It seems to be working OK so far. :

Lessons Learned

  • spectool is my new favorite utility.
  • Make sure your Elixir development environment is as close as possible to your deployment environment.

Next Steps

  • Convert the Vagrantfile into a Dockerfile or start using vagrant-lxc

Sep 16, 2017

Using pm-utils to save/restore VMs on workstation suspend/restore

I use Ubuntu (16.04 for now) and Vagrant (1.9.0 for now) on a bunch of my projects, and I've been running into somethinng like this power management bug for a while now, where after restoring from suspension, my vagrant sessions would be dead and I'd have to 'vagrant halt' and 'vagrant up' before another 'vagrant ssh' would succeed.

To work around this, I came up with some /etc/pm/sleep.d scripts which would save any running vagrant boxes when suspending the workstation and then resume the VMs when resuming the workstation.

Now if I'm in a 'vagrant ssh' session and Ubuntu suspends/resumes, instead of coming back to a frozen session, I'll see I've been disconnected from the ssh session, and I can do another 'vagrant ssh' without having to halt/re-up the VM. That's better than nothing, but the next step here is to start using something like screen or tmux in my vagrant sessions so I can restore right back to where I left off.

So why bother with two scripts when you could have 1 script with a single case statement ? I wanted saving the running vagrant boxes to happen when all the usual services and userspace infrastructure was still running, so I wanted that script in the 00 -49 range from as per the 'Sleep Hook Ordering Convention' portion of 'man 8 pm-action`. However I don't restoration to happen until all the services restarted, so I pushed to the end of the service handling hook range. I may want to revisit this, and rename it to 75_vagrant.

Note in the resume script, the command is pushed into the background since I didn't want want to wait for the VMs to be restored before resuming Ubuntu. I'm usually checking email or the web for a bit before going back to my VMs so I'm OK if that's ready immediately.

Here are some other lessons I learned from these scripts:

The first script is /etc/pm/sleep.d/01_vagrant:

#!/bin/bash

YOURNAME="your normal nonroot user name"

case "$1" in
    suspend)
        timestamp=`date --rfc-3339=seconds`
        echo "${timestamp}: $0 output" >> /var/log/pm-suspend-vagrant.log
        (/sbin/runuser -u ${YOURNAME} /usr/bin/vagrant global-status | grep running | awk '{ print $1; }' | xargs -L1 -I % runuser -u ${YOURNAME} vagrant suspend % ) >> /var/log/pm-suspend-vagrant.log
        ;;
    *)
        ;;
esac

# Don't let errors above stop suspension
true

The second script is /etc/pm/sleep.d/74_vagrant.sh

#!/bin/bash

YOURNAME="your normal nonroot user name"

case "$1" in
     resume)
        # Push the restoration into the background so it doesn't slow down
        timestamp=`date --rfc-3339=seconds`
        ((/sbin/runuser -u ${YOURNAME} /usr/bin/vagrant global-status | grep saved | awk '{ print $1; }' | xargs -L1 -I % runuser -u ${YOURNAME} vagrant resume % ) >> /var/log/pm-resume-vagrant.log) &
        ;;
    *)
        ;;
esac

# Don't let errors above stop restoration
true

Sources: - http://manpages.ubuntu.com/manpages/xenial/man8/pm-action.8.html

Jul 12, 2015

Using Lua macros in an RPM specfile

I've found using Lua macros in rpm spec files to be pretty useful. I haven't found many examples of their use online, so here's how I ended up using them.

I had a situation where I needed to make a subpackage from all the files in a number of different subdirectories. The structure of the files in the subdirectories was fixed, but the number of subdirectories could change over time, and I didn't want to have to update the spec file each time a new subdirectory was added or removed.

Feb 17, 2015

Duply with a powernapping system

I've started backing up one of my systems to S3. The instructions from the Phusion blog worked almost perfectly, except my TARGET line was

TARGET='s3+http://<my-bucket-name>'

Also on the AWS side, I set up a lifecycle rule to archive the backups to Glacier after 7 days.

I did run into some issues getting the backups to work together with powernap, which was configured to put the system to sleep after a few minutes of inactivity.

Powernap was causing a problem on two fronts. First, the system was going to sleep mid-backup since full backups take longer than the powernap inactivity timeout. Second, the backups were scheduled for the middle of the night when the system would normally already be asleep.

To get around the mid-backup sleep issue, I made a /usr/local/bin/duply-nightly script which shuts down powernap before calling duply and restarted it afterwards.

To get around the system-already-asleep issue, I'm using an RTC wakeup in /usr/local/bin/duply-nightly to set the system to wake a few minutes before the cron job kicks off (but not early enough for powernap to put the system to bed again...)

The first night I ran the backup, I had to prime the /sys/class/rtc/rtc0/wakalarm time manually, but since then the script has set the wakeup time for the next day

The final /usr/local/bin/duply-nightly script is below

#!/bin/sh +x

/usr/bin/logger "Running nightly backup from $0"

# Disable powernap during the backup
service powernap stop

/usr/bin/duply nightly backup

# Wakeup the system at 3:00am tomorrow
echo 0 > /sys/class/rtc/rtc0/wakealarm
echo `date '+%s' -d '3am next day'` > /sys/class/rtc/rtc0/wakealarm

# Enable powernap again.
service powernap start

The cron job that kicks on /usr/local/bin/duply-nightly is below

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

5 3 * * * root env HOME=<myhomedir> /usr/local/bin/duply-nightly > /var/log/duply-nightly.log

Aug 31, 2014

Count your YAML dashes

I spent way too long the other day thrying to figure out why some Hiera variable wasn't available in some puppet manifest.

It turns out there was a typo in the YAML file where the first line of the file only had two dashes instead of three.

In this cases, the Ruby 1.8.7 yaml parser corrupts the first entry in the yaml file, adding the two dashes to the beginning of the key, instead of throwing a parser error.

irb(main):001:0> require 'yaml'
=> true
irb(main):002:0> a=YAML.load_file("only-two-dashes.yaml")
=> {"-- first"=>1, "second"=>2}
irb(main):003:0> b=YAML.load_file("correctly-including-three-dashes.yaml")
=> {"first"=>2, "first"=>1}

I couldn't find an existing bug for this, but I didn't look to harrd since this has been fixed in Ruby 1.9

irb(main):001:0> require 'yaml'
=> true
irb(main):002:0> a=YAML.load_file("only-two-dashes.yaml")
Psych::SyntaxError: (t1.yaml): couldn't parse YAML at line 1 column 1
        from /usr/lib/ruby/1.9.1/psych.rb:154:in `parse'
        from /usr/lib/ruby/1.9.1/psych.rb:154:in `parse_stream'
        from /usr/lib/ruby/1.9.1/psych.rb:125:in `parse'
        from /usr/lib/ruby/1.9.1/psych.rb:112:in `load'
        from /usr/lib/ruby/1.9.1/psych.rb:229:in `load_file'
        from (irb):2
        from /usr/bin/irb1.9.1:12:in `<main>'.

So if you're using Ruby 1.8.7, and it looks like the first item in your YAML file is being dropped for some reason, check the first line of the file has 3 dashes.

Oct 03, 2013

Deooglization

Here are the steps I've taken recently to cut down on the amount of free data I've been passing on to Google, and by proxy any unsupervised NSA contractors who may be running amok. I've taken to calling this project 'Deooglization'.

  • Uninstalled Chrome , switched back to Firefox for daily browsing
  • Switched the search bar to use DDG via the DuckDuckGo Plus addon
  • Shuttered the G+ account associated with my identity
  • Migrated away from gmail - Implemented most of Daniel Patterson's Hacker's Replacement for GMail (minus notmuch since I don't live in Emacs that deeply)
  • On StackOveflow, I stopped using Google for an OpenID source and switched to openid.stackexchange.com
  • Removed as many Google apps from my phone as I could.

Outside of Google, I also shut down the Flickr account associated with my identity. I haven't had a Facebook in ages so there wsa nothing to shut down there. I haven't shut down my Twitter account yet, but it's just a matter of time, I suppose.

I've also found that diversifying my passwords has helped a lot: in a few cases where I'd gone to google out of habit, it's been such a PITA to retrieve my password that I just do something else, like take a walk or catch up on the laundry.

Next steps: setting up a Dropbox replacement.

Aug 09, 2013

The case of the late night system boots

A few weeks back, at around 3:00AM I was woken up by the sounds of a computer booting. It turned out to be an old Ubuntu desktop in the study. Since it was storming at the time, I figured it due to a power outage so I shut down the machine and went back to bed.

A couple nights later the same thing happened. This time I made sure the BIOS on the machine was set to restore the machine to its previous state after an outage instead of turning it on. It turned out it was already set to restore the previous state. Odd, but it was 3:00AM so back to bed I went.

The third time it woke me up was soon during the time Thinkhost/Dreamhost dropkicked the old site in the gonads, and I start freaking out a little - I wasn't getting email, my corny vanity site had disappeared and I had a machine turning itself on in the middle of the night. It was coming out of hibernation when it started, so when I noticed it, I was already logged into the console and my 3:00AM self didn't notice any weird activity on the machine. I poked around on the BIOS again to make sure the Wake-on-LAN options were not set and powered down the computer. Still, I started pulling Ethernet cables out before going to bed until I had a chance to really figure out what was going on.

Then over the weekend I had started the machine up and then started some Laundry [*]. When I eventually got back to study, the machine it was still one. I'd been gone long enough that it should have gone into hibernation by then. I did a 'sudo /usr/sbin/pm-hibernate' manually and it went through its normal hibernate routine, but after it powered down, it immediately booted back up again !

Then it all made sense - the machine had been trying to hibernate and rebooting, then waiting some number of minutes with no keyboard activity and repeating the cycle. If I was sleeping lightly enough at the time , the beeps during the restart would wake me up, otherwise I was sleeping through all the restarts. Eventually someone in the house would notice the machine was on and shut it down gracefully. I have no clue how long hibernate had been failing like this. Weeks at least.

But onto the solution. I didn't find much on the Internet about the problem (hence this post), but the combination of the basic-pm-debugging.txt and the pm-hibernate man page led me to try adding an '/etc/pm/config.d/hibernate file with the contents:

HIBERNATE_MODE=shutdown

With that in place, pm-hibernate went back to working like it used to, and I haven't been woken up by late night restarts since.

[*]For purposes of this blog, 'Laundry' is any non-computer indoor activity and 'Going for a Walk' is any non-computer outdoor activity.