NixOS Installation Stumbling Blocks

Posted on Mon 04 September 2017 in lessons • Tagged with lessons, nixos

Here are some issues I ran into installing NixOS and how I eventually got around them.

Setting up a static IP since DHCP wasn't available.

My VM was hosted in an oVirt cluster where DHCP wasn't working/configured, so the installation CD booted without a network. Here's how I manually configred a static IP:

ifconfig enp0s3 <my-static-ip> netmask <my-netmask>
route add default gw <gateway-ip>
echo "nameserver 8.8.8.8" >> /etc/resolv.conf

Partitioning the disk

I spent a lot of time messing with various partitioning schemes until I stumbled across one that worked. I didn't need disk encryption, and I didn't want to bother trying UEFI with ovirt, so here's what I ended up with.

  • A 20G disk split into /dev/sda1 and /dev/sda2
  • /dev/sda1 is a 400MB 'WIN VFAT32' partition (type 'b', not type '4' !!)
  • /dev/sda2 is a LVM partition with the rest of the space
  • For the LVM, /dev/vg/swap is an 8G swap partition and /dev/vg/root has the rest of the LVM parition

In retrospect, I think a lot of my partitioning pain may have been caused by trying to have /dev/sda1 set as a BIOS Parition (type '4'), since I suspect the BIOS partition has to be under 32M.

Also in retrospect, I see only 23M is actually used on the current /boot parition, so maybe 400MB was way too much and I should have gone with /dev/sda1 being 32M and type '4'. ¯\_(ツ)_/¯

I think I also ran into problems using fsck on the boot partition instead of fsck.vfat.

When the boot partition wasn't working, grub would fall into rescue mode and the various 'set prefix / set root / insmod' fixes like this one or this other one didn't work.

What did work here was booting the system with the install CD again, mounting /mnt/boot manually and seeing that failed, or that /mnt/boot contained gibberish after mounting, and then unmounting /mnt/boot and using testdisk to fix the partition type. Testdisk really saved the day.

Mounting the boot partition

Before running nixos-install, I had to also mount the boot partition under /mnt/boot:

> mount /dev/vg/root /mnt
> mkdir -p /mnt/boot
> mount /dev/sda1 /mnt/boot
> nixos-install

Verify the /mnt/etc/nixos/hardware-configuration.nix device paths

When I was messing with the disk partitioning, I rebuilt the /dev/sda1 partition a couple times. Apparently when you do that, you get new UUID for the device.

This meant the "/boot" file system in /mnt/etc/nixos/hardware-configuration.nix was using a device path that was no longer valid. I updated the file to point to the current /boot device and reran 'nixos-install'.

It looks like nixos-install isn't verifying the device paths are valid, since nixos-install ran OK with the invalid device paths.

Configuring a static IP in /mnt/etc/nixos/configuration.nix

Here's what I ended up adding to the configuration.nix file to set up static IP:

networking = {
    hostName = '<my hostname>';
    usePredictableInterfacenames = false;
    interfaces.eth0.ip4 = [{
        address= "<my ipv4 address>";
        prefixLength = <my netmask prefix>;
    }];
    defaultGateway = "<my gateway>"
    nameservers = [ "8.8.8.8" ];
 };

I also adding this boot setting:

boot.load.grub.device = "/dev/sda";

Blizzard Chow

Posted on Mon 25 January 2016 in recipes • Tagged with recipes

I came up with the receipe below while making the most of my blizzard food stash.

Ingrediants

  • 1 lb ground turkey
  • ~4 medium celery stalks, diced
  • ~4 medium carrots, diced
  • 1 onion, chopped
  • ~6 garlic cloves, chopped
  • 1tsp Chinese five spice
  • 1tsp basil
  • 1tsp oregano

Steps

  1. Brown ground turkey and leave on medium heat.
  2. Heat the garlic and onion until the onion is clear.
  3. Add garlic, onion, and spices to the turkey.
  4. Heat the rest of the veggies in a separate pan.
  5. When the veggies are tender, add half of them to the ground turkey.
  6. Put the otiher half in a blender and blend them into a sauce.
  7. Add the veggie sauce to the ground turkey.
  8. Adjust sauciness by adding a cup or 2 or water to the ground turkey.
  9. Continue heating until you're convinced the turkey is finished cooking.
  10. Salt & pepper to taste.

I only blended half the veggies because the pan I was using was too small to cook all the veggies at once, but it turned out really nice, especially with some chipotle hot sauce.


Certmaster lessons learned

Posted on Fri 18 December 2015 in lessons • Tagged with lessons, certmaster

I'm started occassional series about lessons I've learned after finishing with a project. The kick-off article is about a certmaster fork I've been working on. I've been using certmaster at $work for a few years now, but when we wanted to start using multiple certificate authorities, we had to spin up different instances of certmaster, with each instance using on its own IP/port. It would be better if a single instance of certmaster could serve multiple CA's by adding a '--ca' flag. This is the functionality that my fork of certmaster provides, and here are the lessons I learned while working on this:

bats versus shunit2

certmaster doesn't include any tests, so I wanted to write some functional tests to verify my changes worked as expected.

I started out working with bats_, but it fell down when I needed to push a command into the background - it just wouldn't do it. I tried the 'gotcha' suggestions from this engine yard post but to no avail. I switched the tests to shunit2 and had no trouble pushing commands into the background.

Assigning here documents to variables

variable=$(cat <<EOF
this text will get assigned to variable.
EOF

Using Lua macros in an RPM specfile

Posted on Sun 12 July 2015 in hints • Tagged with hint, lua, rpmbuild, rpm, specilfe

I've found using Lua macros in rpm spec files to be pretty useful. I haven't found many examples of their use online, so here's how I ended up using them.

I had a situation where I needed to make a subpackage from all the files in a number of different subdirectories. The structure of the files in the subdirectories was fixed, but the number of subdirectories could change over time, and I didn't want to have to update the spec file each time a new subdirectory was added or removed.


Duply with a powernapping system

Posted on Tue 17 February 2015 in hints • Tagged with hints, ubuntu, duply, powernap

I've started backing up one of my systems to S3. The instructions from the Phusion blog worked almost perfectly, except my TARGET line was

TARGET='s3+http://<my-bucket-name>'

Also on the AWS side, I set up a lifecycle rule to archive the backups to Glacier after 7 days.

I did run into some issues getting the backups to work together with powernap, which was configured to put the system to sleep after a few minutes of inactivity.

Powernap was causing a problem on two fronts. First, the system was going to sleep mid-backup since full backups take longer than the powernap inactivity timeout. Second, the backups were scheduled for the middle of the night when the system would normally already be asleep.

To get around the mid-backup sleep issue, I made a /usr/local/bin/duply-nightly script which shuts down powernap before calling duply and restarted it afterwards.

To get around the system-already-asleep issue, I'm using an RTC wakeup in /usr/local/bin/duply-nightly to set the system to wake a few minutes before the cron job kicks off (but not early enough for powernap to put the system to bed again...)

The first night I ran the backup, I had to prime the /sys/class/rtc/rtc0/wakalarm time manually, but since then the script has set the wakeup time for the next day

The final /usr/local/bin/duply-nightly script is below

#!/bin/sh +x

/usr/bin/logger "Running nightly backup from $0"

# Disable powernap during the backup
service powernap stop

/usr/bin/duply nightly backup

# Wakeup the system at 3:00am tomorrow
echo 0 > /sys/class/rtc/rtc0/wakealarm
echo `date '+%s' -d '3am next day'` > /sys/class/rtc/rtc0/wakealarm

# Enable powernap again.
service powernap start

The cron job that kicks on /usr/local/bin/duply-nightly is below

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

5 3 * * * root env HOME=<myhomedir> /usr/local/bin/duply-nightly > /var/log/duply-nightly.log

Count your YAML dashes

Posted on Sun 31 August 2014 in hints • Tagged with hints, ruby, yaml

I spent way too long the other day thrying to figure out why some Hiera variable wasn't available in some puppet manifest.

It turns out there was a typo in the YAML file where the first line of the file only had two dashes instead of three.

In this cases, the Ruby 1.8.7 yaml parser corrupts the first entry in the yaml file, adding the two dashes to the beginning of the key, instead of throwing a parser error.

irb(main):001:0> require 'yaml'
=> true
irb(main):002:0> a=YAML.load_file("only-two-dashes.yaml")
=> {"-- first"=>1, "second"=>2}
irb(main):003:0> b=YAML.load_file("correctly-including-three-dashes.yaml")
=> {"first"=>2, "first"=>1}

I couldn't find an existing bug for this, but I didn't look to harrd since this has been fixed in Ruby 1.9

irb(main):001:0> require 'yaml'
=> true
irb(main):002:0> a=YAML.load_file("only-two-dashes.yaml")
Psych::SyntaxError: (t1.yaml): couldn't parse YAML at line 1 column 1
        from /usr/lib/ruby/1.9.1/psych.rb:154:in `parse'
        from /usr/lib/ruby/1.9.1/psych.rb:154:in `parse_stream'
        from /usr/lib/ruby/1.9.1/psych.rb:125:in `parse'
        from /usr/lib/ruby/1.9.1/psych.rb:112:in `load'
        from /usr/lib/ruby/1.9.1/psych.rb:229:in `load_file'
        from (irb):2
        from /usr/bin/irb1.9.1:12:in `<main>'.

So if you're using Ruby 1.8.7, and it looks like the first item in your YAML file is being dropped for some reason, check the first line of the file has 3 dashes.


hgr

Posted on Sat 09 November 2013 in opsec • Tagged with opsec, hgr

I've implementing a version of Daniel Patterson's Hacker's GMail Replacement, and finally got around to publishing it as a Puppet module. Even if you don't know puppet, it shouldn't be too hard to walk through the manifests and see what setting it up involved.

I don't live inside Emacs as much as Mr Patterson apparently does, so I left out the notmuch and afew content. Instead I added radicale support for publishing my calendar and todo lists, and I've started moving towards setting up LDAP for my address book. That may be overkill - maintaining an LDAP address book for personal use looks like a PITA.

So far, it's working well with my phone using K9 for mail and aCal for a calender. On the non-phones, Thunderbird/Lightning is working out well.

I've set up gmail to forward to my non-gmail account. Pretty much the only time I log into gmail (or google for that matter) in the past couple months has been to delete content or wipe profile data.

It was an interesting exercise. The next steps for this project would be to update the manifests to follow the Puppet Forge documentation standards, and to include some rspec-puppet tests, and then to maybe like into setting up Packer to build a VM that applies the module.


Deooglization

Posted on Thu 03 October 2013 in hints • Tagged with hints, deooglization

Here are the steps I've taken recently to cut down on the amount of free data I've been passing on to Google, and by proxy any unsupervised NSA contractors who may be running amok. I've taken to calling this project 'Deooglization'.

  • Uninstalled Chrome , switched back to Firefox for daily browsing
  • Switched the search bar to use DDG via the DuckDuckGo Plus addon
  • Shuttered the G+ account associated with my identity
  • Migrated away from gmail - Implemented most of Daniel Patterson's Hacker's Replacement for GMail (minus notmuch since I don't live in Emacs that deeply)
  • On StackOveflow, I stopped using Google for an OpenID source and switched to openid.stackexchange.com
  • Removed as many Google apps from my phone as I could.

Outside of Google, I also shut down the Flickr account associated with my identity. I haven't had a Facebook in ages so there wsa nothing to shut down there. I haven't shut down my Twitter account yet, but it's just a matter of time, I suppose.

I've also found that diversifying my passwords has helped a lot: in a few cases where I'd gone to google out of habit, it's been such a PITA to retrieve my password that I just do something else, like take a walk or catch up on the laundry.

Next steps: setting up a Dropbox replacement.


The case of the late night system boots

Posted on Fri 09 August 2013 in hints • Tagged with hints, ubuntu, hibernation

A few weeks back, at around 3:00AM I was woken up by the sounds of a computer booting. It turned out to be an old Ubuntu desktop in the study. Since it was storming at the time, I figured it due to a power outage so I shut down the machine and went back to bed.

A couple nights later the same thing happened. This time I made sure the BIOS on the machine was set to restore the machine to its previous state after an outage instead of turning it on. It turned out it was already set to restore the previous state. Odd, but it was 3:00AM so back to bed I went.

The third time it woke me up was soon during the time Thinkhost/Dreamhost dropkicked the old site in the gonads, and I start freaking out a little - I wasn't getting email, my corny vanity site had disappeared and I had a machine turning itself on in the middle of the night. It was coming out of hibernation when it started, so when I noticed it, I was already logged into the console and my 3:00AM self didn't notice any weird activity on the machine. I poked around on the BIOS again to make sure the Wake-on-LAN options were not set and powered down the computer. Still, I started pulling Ethernet cables out before going to bed until I had a chance to really figure out what was going on.

Then over the weekend I had started the machine up and then started some Laundry [*]. When I eventually got back to study, the machine it was still one. I'd been gone long enough that it should have gone into hibernation by then. I did a 'sudo /usr/sbin/pm-hibernate' manually and it went through its normal hibernate routine, but after it powered down, it immediately booted back up again !

Then it all made sense - the machine had been trying to hibernate and rebooting, then waiting some number of minutes with no keyboard activity and repeating the cycle. If I was sleeping lightly enough at the time , the beeps during the restart would wake me up, otherwise I was sleeping through all the restarts. Eventually someone in the house would notice the machine was on and shut it down gracefully. I have no clue how long hibernate had been failing like this. Weeks at least.

But onto the solution. I didn't find much on the Internet about the problem (hence this post), but the combination of the basic-pm-debugging.txt and the pm-hibernate man page led me to try adding an '/etc/pm/config.d/hibernate file with the contents:

HIBERNATE_MODE=shutdown

With that in place, pm-hibernate went back to working like it used to, and I haven't been woken up by late night restarts since.

[*]For purposes of this blog, 'Laundry' is any non-computer indoor activity and 'Going for a Walk' is any non-computer outdoor activity.

A Password Diversification Plan

Posted on Wed 24 July 2013 in security • Tagged with security

(TLDR)

You shouldn't be using the same password everywhere You know this. Everytime some popular site gets hacked and millions of passwords are lost there's a flurry of news stories telling you never use the same password twice. There's even a xkcd comic about it.

But let's face it, if you are a reuser, this can be a pretty daunting task. It's like getting in shape or losing some weight: it's not something that's going to change overnight. It's going to take some time to switching all your passwords.

With that in mind, here's a plan you can use to move away from password reuse at yuor own pace. As a side effect, it should also get you in the habit of using stronger passwords and make it easier to remember to occasional change your passwords..

Pick a password manager

Instead of trying to remember multiple passwords, you should use a password manager. I like keepass, but there are others available. Pick one and start using it. If you don't like it switch to another one. Here are the important features:

  • Cross platform - it should work on all your devices which are likely to prompt for passwords
  • cut&pasting passwords - the plan is to eventually start using long passwords you can't possibly remember, so being able to copy the passwords into your clipboard and paste them into the password field is very useful
  • random password generators

Pick a password for the password manager's database. Use a different one from your normal password, since you're getting out of the habit of using the same password everywhere. Try to make it longer than your normal password.

Then enter your current reused password as the first entry. The manager most likely has a 'copy and paste password to the clipboard' feature. For a day or two, get into the habit of copy&pasting your password from the manager instead of typing it manually.

Set up Dropbox account

You can use Dropbox (or an equivilent service) to distribute your password database across multiple machines and devices.

If you're setting up a new account, don't reuse your default password. If you already have a Dropbox account, change your password. Add the new or updated password to the password manager.

Copy your password database into the shared directory. Make sure it's not in a publicly accessible folder. Now you can access your password database from any machine or device you can access with Dropbox

If your password manager supports keys, add key support, but don't store the keys on Dropbox. This way if Dropbox is hacked and your database is compromised, your database is still safe since the meenie weenies will not have your key.

Start Changing your passwords when visiting sites

If you visit a site and can log into a site with your default password, find the place on the site that'll let you change the pasword and then set it to a strong password and it to your password manager.

Try to change one or two accounts a day in the beginning, and then ramp up as you get more comfortable with the process.

When adding the new entries in the pasword manager an expiration date of something like 3 months. This is to also get yourself into the habit of changing your passwords as well.

Learn to love the password recovery pages

The first couple times you add a password into the manager, you may screw it up and need to recover your password for the site. This is a hassle, but for most sites it's not really that big a deal so that let that scare you off from changing your passwords.

Still, until you've changed a couple passwords, it might be good to hold off on switching the passwords for important services like email, which would be more disruptive if you had to have the pasword reset.

Backup your password database

After the first week or so, backup your password database to a USB drive or burn it to the CD or export the database to a CSV file, print that out and store it in the safe deposit box in your fallout shelter with your lifetime supply of canned creamed corn. Whatever your comfortable with. The important thing is you make the backup.

Also set up some sort of weekly reminder to update the backup.

Start using different usernames for new sites

If you're also using the same username for all your accounts, you could also start using differnet usernames as well to make it more difficult for folks to track you between sites. Your password manager should have a place to store your username, so you could cut&paste that field in addition to your password.

Start deactivating unused account

You probably have lots of accounts that you never use. Instead of changing the passwords for these sites, I suggest closing out the account entirely. Pull down whatever data you want to save from the site, then change your password to something random,and don't bother adding it to your password manager. Then deactivate the account.

Bacn is a good source of reminders about accounts you may want to close out.

After a site's been hacked

When an online service you're using is hacked, you should update your password on the site right away, making it more random if you already aren't using totally random passwrds.

You should also use this oppurtunity to look around and see if there's some other more secure site that you could be using instead, or if your technically inclined, if it's now possible for you to run a local version of this service on your own servers.

Change passwords that are expiring

Since you've been adding expiration dates to the password entries, when the manager starts warning you of an impending expiration, go ahead and change the password on the site. By this time you will have been using the manager for a few months, so try making the password longer and more cryptic. The password manager probably has the option of creating random passwords for you, so try that out. You should be very used to cut&pasting your passwords, so it shouldn't matter to you much by now that you can't remember the password.

Summary

  • Don't use the same password everywhere
  • Use a password manager - back it up regularly
  • Move away from using your reused password slowly at your own pace
  • Close out unused sites that are using the reused password
  • Set expirations on the passwords so you'll remember to change them regularly
  • Moving to more secure sites / self-hosting after sites are hacked.