Sep 24, 2017

Lessons from Wriing a Pylint Plugin

At work there's a python coding convention that I tend to overlook a lot. So when I post merge requests, there's a pretty good chance someone's going to call me out on this, which leads to a followup commit and another round of peer review. This can lead to an extra delay of a few hours until I notice the comments, switch context back to that merge request, making the changes, update the merge request and wait for another round of reviews. If I could find a way to check my code for this convention before posting the merge requsts, I could get my code merged in a few hours faster....

The Convention

The coding convention I cannot internalize is as follows: In python, the format method for strings will call the __format__ method on its arguments for you, so any code that looks like:

"interpolate these: {} {}".format(str(a), str(b))

Need only look like:

"interpolate me: {} {}".format(a, b)

The Pylint Plugin

So googling around led my to this Ned Batchelder post from a few years back. That post also led to a couple pylint plugins here. Looking at pylint's own format checker reminded me that I should also be handling keyword arguments.

From the post and sample code, it looked like I needed to define a checker class with a visit_callfunc method that would check when the 'format' method was used, and then check all the arguments to the format call and throw an error if any of them where a function call to str().

Here's what I eventually ended up.

To come up with this I used an embarassing amount of exploratory programming to figure out astroid. I wrote an initial visit_callfunc() method based on the sample code that didn't do much more than dump out all the data about the node argument via dir(node) and node.__dict__. Then I would call pylint with the plugin against some sample source with the error I was trying to plugin to report.

I run the plugin against the existing code and found one lingering case where the reviewers had allowed one of my unneccessary str() call into the codebase. It's been removed now.

Lessons Learned

  • pylint plugins are pretty powerful and I wouldn't shy away from writing another one. I'm on the lookout for other excuses to write another one.
  • https://greentreesnakes.readthedocs.io is a useful 'missing manual' for the python AST.
  • format() can take both positional and keyword arguments. My original pass at the plugin only supported positional arguments.
  • The bandit project exists and looks useful. I stumbled acros it while looking for other pylint plugins.

Sep 04, 2017

NixOS Installation Stumbling Blocks

Here are some issues I ran into installing NixOS and how I eventually got around them.

Setting up a static IP since DHCP wasn't available.

My VM was hosted in an oVirt cluster where DHCP wasn't working/configured, so the installation CD booted without a network. Here's how I manually configred a static IP:

ifconfig enp0s3 <my-static-ip> netmask <my-netmask>
route add default gw <gateway-ip>
echo "nameserver 8.8.8.8" >> /etc/resolv.conf

Partitioning the disk

I spent a lot of time messing with various partitioning schemes until I stumbled across one that worked. I didn't need disk encryption, and I didn't want to bother trying UEFI with ovirt, so here's what I ended up with.

  • A 20G disk split into /dev/sda1 and /dev/sda2
  • /dev/sda1 is a 400MB 'WIN VFAT32' partition (type 'b', not type '4' !!)
  • /dev/sda2 is a LVM partition with the rest of the space
  • For the LVM, /dev/vg/swap is an 8G swap partition and /dev/vg/root has the rest of the LVM parition

In retrospect, I think a lot of my partitioning pain may have been caused by trying to have /dev/sda1 set as a BIOS Parition (type '4'), since I suspect the BIOS partition has to be under 32M.

Also in retrospect, I see only 23M is actually used on the current /boot parition, so maybe 400MB was way too much and I should have gone with /dev/sda1 being 32M and type '4'. ¯\_(ツ)_/¯

I think I also ran into problems using fsck on the boot partition instead of fsck.vfat.

When the boot partition wasn't working, grub would fall into rescue mode and the various 'set prefix / set root / insmod' fixes like this one or this other one didn't work.

What did work here was booting the system with the install CD again, mounting /mnt/boot manually and seeing that failed, or that /mnt/boot contained gibberish after mounting, and then unmounting /mnt/boot and using testdisk to fix the partition type. Testdisk really saved the day.

Mounting the boot partition

Before running nixos-install, I had to also mount the boot partition under /mnt/boot:

> mount /dev/vg/root /mnt
> mkdir -p /mnt/boot
> mount /dev/sda1 /mnt/boot
> nixos-install

Verify the /mnt/etc/nixos/hardware-configuration.nix device paths

When I was messing with the disk partitioning, I rebuilt the /dev/sda1 partition a couple times. Apparently when you do that, you get new UUID for the device.

This meant the "/boot" file system in /mnt/etc/nixos/hardware-configuration.nix was using a device path that was no longer valid. I updated the file to point to the current /boot device and reran 'nixos-install'.

It looks like nixos-install isn't verifying the device paths are valid, since nixos-install ran OK with the invalid device paths.

Configuring a static IP in /mnt/etc/nixos/configuration.nix

Here's what I ended up adding to the configuration.nix file to set up static IP:

networking = {
    hostName = '<my hostname>';
    usePredictableInterfacenames = false;
    interfaces.eth0.ip4 = [{
        address= "<my ipv4 address>";
        prefixLength = <my netmask prefix>;
    }];
    defaultGateway = "<my gateway>"
    nameservers = [ "8.8.8.8" ];
 };

I also adding this boot setting:

boot.load.grub.device = "/dev/sda";

Dec 18, 2015

Certmaster lessons learned

I'm started occassional series about lessons I've learned after finishing with a project. The kick-off article is about a certmaster fork I've been working on. I've been using certmaster at $work for a few years now, but when we wanted to start using multiple certificate authorities, we had to spin up different instances of certmaster, with each instance using on its own IP/port. It would be better if a single instance of certmaster could serve multiple CA's by adding a '--ca' flag. This is the functionality that my fork of certmaster provides, and here are the lessons I learned while working on this:

bats versus shunit2

certmaster doesn't include any tests, so I wanted to write some functional tests to verify my changes worked as expected.

I started out working with bats_, but it fell down when I needed to push a command into the background - it just wouldn't do it. I tried the 'gotcha' suggestions from this engine yard post but to no avail. I switched the tests to shunit2 and had no trouble pushing commands into the background.

Assigning here documents to variables

variable=$(cat <<EOF
this text will get assigned to variable.
EOF