Jul 13, 2020

A workaround for running deja-dup as root in Ubuntu 20.04

I found a workaround for the Duplicity fails to start issue where 'sudo deja-dup' would fail with the python stacktrace mentioned in the launchpad ticket.

The ticket was not very useful, so I started looking at the various files in the stacktrace and saw line from /usr/lib/python3/dist-packages/duplicity/backends/giobackend.py was within an "if u'DBUS_SESSION_BUS_ADDRESS' not in os.environ" block.

So I wondered what would happen if I let that environment variable pass into the sudo environment. I tried 'sudo -E deja-dup' as per preserve the environment. This didn't result in a stacktrace, but it ended up running the backup as the normal non-root user, probably because the preserved environment included the USER and HOME variables along with the DBUS_SESSION_BUS_ADDRESS variable.

Then I tried preserving just DBUS_SESSION_BUS_ADDRESS with 'sudo --preserve-env=DBUS_SESSION_BUS_ADDRESS deja-dup', it worked as expected.

So the hint here is that when presented with a stacktrace don't be afraid to "Use the Source, Jean Luc".

A image of Patrick Stewart playing Gurney Halleck from David Lynch's Dune film, with the meme text 'Use the Source, Jean Luc'.

Jul 13, 2020

How I orgmode

This post covers how I'm keeping my orgmode notebooks synced between my desktop and phone. There are also some tips on how I resolve merge conflicts in my notebooks and other hints for using Orgzly.

Setup

uml diagram

(The Pelican PlantUML plugin is pretty nice.)

Owncloud Server

Set up an owncloud server somewhere where both my desktop and phone can reach it.

Desktop: Owncloud client and git repo

On the desktop, install the owncloud client and configured ~/owncloud to be shared with the server. Also set up an ~/owncloud/org directory and add an ~/org symlink that points to ~/owncloud/org. This is mostly to make it easier to access the org files.

Update your emacs and/or vi configs to support orgmode. I'll update the org files in the editor that's more convenient at the time, though I'll use emacs if I need to update the deadline of a task.

I've also added .git to the owncloud and did a 'git init' in the ~/owncloud/org directory. This repo will be used in the 'Handling Conflicts' section below.

Create or move your notebook files into the ~/owncloud/org directory and make sure they have a '.org' extension. Verify that the org files are being synced to the owncloud server.

Make sure your owncloud client is set to start when your desktop reboots.

Phone: Owncloud client and Orgzly

Install the owncloud client on your phone. Configure it to sync with the owncloud server and verify it's syncing the notebook files from the server.

Also install and configure Orgzly. For the syncing the repositories use WebDAV with the URL https://<your-owncloud-server>/remote.php/webdav/org. I originally tried using local storage, but I couldn't get that to work.

Then for each of the notebooks, set the link to https://<your-owncloud-server>/remote.php/webdav/org. Update one of your notebooks with Orgzly and verify the change is synced to your desktop.

Day-to-day Usage

Throughout the day, I'm usually using Orgzly, checking the Agenda tab for overdue and upcoming tasks, and adding new tasks and notes.

I'm access the Notebooks on the desktop less frequently, mostly archiving completed tasks, and adding links and notes to research tasks. I also tend to move tasks between notebooks from the desktop.

Handling Recurring Events

I ignore the scheduled time setting in Orgzly, and only set the deadlines with warning times. Then I treat the notification that's generated from warning time passing as the time I should start on a task.

For repeating events, I use the '++' modifier for tasks. This way if I miss a few iterations of a task it will add the next iteration in the future.

I'm also try to set an appropriate warning period when setting tasks.

It took me a while to figure out I could tap on the date itself to bring up a calender instead of being limited to the 'today', 'tomorrow', and 'next week' options. <shrug>.

Handling Conflicts

Sometimes when I'm updating the notebooks on both my desktop and phone, orgzly will say there's a conflict.

When this happens I go to my desktop and make a checkpoint commit of any outstanding changes on the ~/owncloud/org repo. Then I push the Notebook from Orgzly (the Cloud with the 'up' arrow.

Then on the desktop, I do a 'git diff' and adjust Notebook as needed.

Usually this includes adding some new notes or adjusting some deadlines.

Jun 21, 2020

Importing Stock Transactions Into Gnucash With Python

I use GnuCash, but keeping my 401(k) accounts up to date has always been a tedious, manual process.

I came up with the python snippet below to handle the import, but it was a pain to write since I couldn't find any examples for setting up stock or mutual fund transactions.

The SetSharePriceAndAmount method ended up being key, and I only found that by searching through the gnucash source.

The GncNumeric class also ended up being more of a pain to use than I expected. There's probably a better way to use it, but the 'multiple values by 1000000/100000' approach is working for me now.

I'm using the stock GnuCash and python-gnucash version 2.6.19 available in Ubuntu 18.04, so this stuck using python 2.7.

#!/usr/bin/python2.7

import csv
from datetime import datetime

import gnucash

session = gnucash.Session("xml://yourfile.gnucash")
book = session.book
root_account = book.get_root_account()

usd = book.get_table().lookup('ISO4217','USD')

# There's probably a better way to use 'Your:Retirement:Contributions' instead ....
contrib_acct = root_account.lookup_by_name("Your").lookup_by_name("Retirement").lookup_by_name("Contributions")

parent_acct = root_account.lookup_by_name("401k")

with open('your_transactions.csv', 'rb') as trans_csv:
  trans_reader = csv.reader(trans_csv, delimiter=',')

  # Skip over the first row since it's headers
  header = next(trans_reader)

  for description, date, fund_name, share_price_str, share_amount_str, amount_str in trans_reader:
    child_account = parent_acct.lookup_by_name(fund_name)

    posting_date = datetime.strptime(date,"%m/%d/%y")

    tx = gnucash.Transaction(book)
    tx.BeginEdit()

    tx.SetCurrency(usd)

    tx.SetDatePostedTS(posting_date)
    tx.SetDescription(description)

    sp1 = gnucash.Split(book)
    sp1.SetParent(tx)
    sp1.SetAccount(child_account)

    # GncNumeric(n,d) represents numbers as fractions of the form n/d, so GncNumeric(1234567/1000000) = 1.234567
    # There's probably a better way to do this...
    share_price = gnucash.GncNumeric(float(share_price_str)*(10**6), 10**6)
    share_amount = gnucash.GncNumeric(float(share_amount_str)*(10**6), 10**6)

    # share_price * share_amount == amount, so I could have used that instead using the value from the csv
    amount = gnucash.GncNumeric(float(amount_str)*(10**6), 10**6)

    # ( ˘▽˘)っ♨  This is the secret sauce for setting the number of shares and the price.
    sp1.SetSharePriceAndAmount(share_price, share_amount)

    sp2 = gnucash.Split(book)
    sp2.SetParent(tx)
    sp2.SetAccount(contrib_acct)
    sp2.SetValue(amount.neg())

    tx.CommitEdit()
session.save()
session.end()

Special thanks to this post for providing most of the code above.

Oct 08, 2017

Elixir on Centos 7

I'm at the point with a Elixir/Phoneix side project that I'm thinking about deployment.

The first big stumbling block was that development envioronment (Ubuntu 16.04) wasn't the same as my deployment environment (Centos 7). For Ubuntu, I could pull the latest Elixir packages from Erlang Solutions, but they don't host Centos Elixir packages, and the version of Elixir on EPEL is over 4 years old - old enough that 'mix deps' on my project was erroring out.

I found the posts below about installing Elixir on Centos 7 but, they involve cloning the Elixir repo and building it from source, but I don't want development tools like git and gcc on my production machines.

I also found https://www.vultr.com/docs/how-to-install-the-phoenix-framework-on-centos-7 but it involves downloading a precompiled mystery meat bundle off Github. However https://github.com/elixir-lang/elixir doesn't mention that these precompiled bundles are available or how the bundles were built. The precompiled bundle is mentioned on the elixir-lang.org install page, but still, that's too mysterious for me.

Maybe one of those links will work for you, but what I want is a way to build a more recent version of the Elixir rpm than what was available in EPEL. That way I can recompile and package Elixir on a dev machine, and then only copy the rpm up to my production machine.

It looks like Elixir is stuck in EPEL waiting for the Erlang 18 release to get promoted in EPEL, so maybe I can take the existing Elixir packaging source and build it against the latest Erlang packages from Erlang Solutions...

I found the packaging source at https://src.fedoraproject.org/rpms/elixir and after poking around a bit I came up with Vagrantfile below. It seems to be working OK so far. :

Lessons Learned

  • spectool is my new favorite utility.
  • Make sure your Elixir development environment is as close as possible to your deployment environment.

Next Steps

  • Convert the Vagrantfile into a Dockerfile or start using vagrant-lxc

Sep 16, 2017

Using pm-utils to save/restore VMs on workstation suspend/restore

I use Ubuntu (16.04 for now) and Vagrant (1.9.0 for now) on a bunch of my projects, and I've been running into somethinng like this power management bug for a while now, where after restoring from suspension, my vagrant sessions would be dead and I'd have to 'vagrant halt' and 'vagrant up' before another 'vagrant ssh' would succeed.

To work around this, I came up with some /etc/pm/sleep.d scripts which would save any running vagrant boxes when suspending the workstation and then resume the VMs when resuming the workstation.

Now if I'm in a 'vagrant ssh' session and Ubuntu suspends/resumes, instead of coming back to a frozen session, I'll see I've been disconnected from the ssh session, and I can do another 'vagrant ssh' without having to halt/re-up the VM. That's better than nothing, but the next step here is to start using something like screen or tmux in my vagrant sessions so I can restore right back to where I left off.

So why bother with two scripts when you could have 1 script with a single case statement ? I wanted saving the running vagrant boxes to happen when all the usual services and userspace infrastructure was still running, so I wanted that script in the 00 -49 range from as per the 'Sleep Hook Ordering Convention' portion of 'man 8 pm-action`. However I don't restoration to happen until all the services restarted, so I pushed to the end of the service handling hook range. I may want to revisit this, and rename it to 75_vagrant.

Note in the resume script, the command is pushed into the background since I didn't want want to wait for the VMs to be restored before resuming Ubuntu. I'm usually checking email or the web for a bit before going back to my VMs so I'm OK if that's ready immediately.

Here are some other lessons I learned from these scripts:

The first script is /etc/pm/sleep.d/01_vagrant:

#!/bin/bash

YOURNAME="your normal nonroot user name"

case "$1" in
    suspend)
        timestamp=`date --rfc-3339=seconds`
        echo "${timestamp}: $0 output" >> /var/log/pm-suspend-vagrant.log
        (/sbin/runuser -u ${YOURNAME} /usr/bin/vagrant global-status | grep running | awk '{ print $1; }' | xargs -L1 -I % runuser -u ${YOURNAME} vagrant suspend % ) >> /var/log/pm-suspend-vagrant.log
        ;;
    *)
        ;;
esac

# Don't let errors above stop suspension
true

The second script is /etc/pm/sleep.d/74_vagrant.sh

#!/bin/bash

YOURNAME="your normal nonroot user name"

case "$1" in
     resume)
        # Push the restoration into the background so it doesn't slow down
        timestamp=`date --rfc-3339=seconds`
        ((/sbin/runuser -u ${YOURNAME} /usr/bin/vagrant global-status | grep saved | awk '{ print $1; }' | xargs -L1 -I % runuser -u ${YOURNAME} vagrant resume % ) >> /var/log/pm-resume-vagrant.log) &
        ;;
    *)
        ;;
esac

# Don't let errors above stop restoration
true

Sources: - http://manpages.ubuntu.com/manpages/xenial/man8/pm-action.8.html

Jul 12, 2015

Using Lua macros in an RPM specfile

I've found using Lua macros in rpm spec files to be pretty useful. I haven't found many examples of their use online, so here's how I ended up using them.

I had a situation where I needed to make a subpackage from all the files in a number of different subdirectories. The structure of the files in the subdirectories was fixed, but the number of subdirectories could change over time, and I didn't want to have to update the spec file each time a new subdirectory was added or removed.

Feb 17, 2015

Duply with a powernapping system

I've started backing up one of my systems to S3. The instructions from the Phusion blog worked almost perfectly, except my TARGET line was

TARGET='s3+http://<my-bucket-name>'

Also on the AWS side, I set up a lifecycle rule to archive the backups to Glacier after 7 days.

I did run into some issues getting the backups to work together with powernap, which was configured to put the system to sleep after a few minutes of inactivity.

Powernap was causing a problem on two fronts. First, the system was going to sleep mid-backup since full backups take longer than the powernap inactivity timeout. Second, the backups were scheduled for the middle of the night when the system would normally already be asleep.

To get around the mid-backup sleep issue, I made a /usr/local/bin/duply-nightly script which shuts down powernap before calling duply and restarted it afterwards.

To get around the system-already-asleep issue, I'm using an RTC wakeup in /usr/local/bin/duply-nightly to set the system to wake a few minutes before the cron job kicks off (but not early enough for powernap to put the system to bed again...)

The first night I ran the backup, I had to prime the /sys/class/rtc/rtc0/wakalarm time manually, but since then the script has set the wakeup time for the next day

The final /usr/local/bin/duply-nightly script is below

#!/bin/sh +x

/usr/bin/logger "Running nightly backup from $0"

# Disable powernap during the backup
service powernap stop

/usr/bin/duply nightly backup

# Wakeup the system at 3:00am tomorrow
echo 0 > /sys/class/rtc/rtc0/wakealarm
echo `date '+%s' -d '3am next day'` > /sys/class/rtc/rtc0/wakealarm

# Enable powernap again.
service powernap start

The cron job that kicks on /usr/local/bin/duply-nightly is below

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

5 3 * * * root env HOME=<myhomedir> /usr/local/bin/duply-nightly > /var/log/duply-nightly.log

Aug 31, 2014

Count your YAML dashes

I spent way too long the other day thrying to figure out why some Hiera variable wasn't available in some puppet manifest.

It turns out there was a typo in the YAML file where the first line of the file only had two dashes instead of three.

In this cases, the Ruby 1.8.7 yaml parser corrupts the first entry in the yaml file, adding the two dashes to the beginning of the key, instead of throwing a parser error.

irb(main):001:0> require 'yaml'
=> true
irb(main):002:0> a=YAML.load_file("only-two-dashes.yaml")
=> {"-- first"=>1, "second"=>2}
irb(main):003:0> b=YAML.load_file("correctly-including-three-dashes.yaml")
=> {"first"=>2, "first"=>1}

I couldn't find an existing bug for this, but I didn't look to harrd since this has been fixed in Ruby 1.9

irb(main):001:0> require 'yaml'
=> true
irb(main):002:0> a=YAML.load_file("only-two-dashes.yaml")
Psych::SyntaxError: (t1.yaml): couldn't parse YAML at line 1 column 1
        from /usr/lib/ruby/1.9.1/psych.rb:154:in `parse'
        from /usr/lib/ruby/1.9.1/psych.rb:154:in `parse_stream'
        from /usr/lib/ruby/1.9.1/psych.rb:125:in `parse'
        from /usr/lib/ruby/1.9.1/psych.rb:112:in `load'
        from /usr/lib/ruby/1.9.1/psych.rb:229:in `load_file'
        from (irb):2
        from /usr/bin/irb1.9.1:12:in `<main>'.

So if you're using Ruby 1.8.7, and it looks like the first item in your YAML file is being dropped for some reason, check the first line of the file has 3 dashes.

Oct 03, 2013

Deooglization

Here are the steps I've taken recently to cut down on the amount of free data I've been passing on to Google, and by proxy any unsupervised NSA contractors who may be running amok. I've taken to calling this project 'Deooglization'.

  • Uninstalled Chrome , switched back to Firefox for daily browsing
  • Switched the search bar to use DDG via the DuckDuckGo Plus addon
  • Shuttered the G+ account associated with my identity
  • Migrated away from gmail - Implemented most of Daniel Patterson's Hacker's Replacement for GMail (minus notmuch since I don't live in Emacs that deeply)
  • On StackOveflow, I stopped using Google for an OpenID source and switched to openid.stackexchange.com
  • Removed as many Google apps from my phone as I could.

Outside of Google, I also shut down the Flickr account associated with my identity. I haven't had a Facebook in ages so there wsa nothing to shut down there. I haven't shut down my Twitter account yet, but it's just a matter of time, I suppose.

I've also found that diversifying my passwords has helped a lot: in a few cases where I'd gone to google out of habit, it's been such a PITA to retrieve my password that I just do something else, like take a walk or catch up on the laundry.

Next steps: setting up a Dropbox replacement.

Aug 09, 2013

The case of the late night system boots

A few weeks back, at around 3:00AM I was woken up by the sounds of a computer booting. It turned out to be an old Ubuntu desktop in the study. Since it was storming at the time, I figured it due to a power outage so I shut down the machine and went back to bed.

A couple nights later the same thing happened. This time I made sure the BIOS on the machine was set to restore the machine to its previous state after an outage instead of turning it on. It turned out it was already set to restore the previous state. Odd, but it was 3:00AM so back to bed I went.

The third time it woke me up was soon during the time Thinkhost/Dreamhost dropkicked the old site in the gonads, and I start freaking out a little - I wasn't getting email, my corny vanity site had disappeared and I had a machine turning itself on in the middle of the night. It was coming out of hibernation when it started, so when I noticed it, I was already logged into the console and my 3:00AM self didn't notice any weird activity on the machine. I poked around on the BIOS again to make sure the Wake-on-LAN options were not set and powered down the computer. Still, I started pulling Ethernet cables out before going to bed until I had a chance to really figure out what was going on.

Then over the weekend I had started the machine up and then started some Laundry [*]. When I eventually got back to study, the machine it was still one. I'd been gone long enough that it should have gone into hibernation by then. I did a 'sudo /usr/sbin/pm-hibernate' manually and it went through its normal hibernate routine, but after it powered down, it immediately booted back up again !

Then it all made sense - the machine had been trying to hibernate and rebooting, then waiting some number of minutes with no keyboard activity and repeating the cycle. If I was sleeping lightly enough at the time , the beeps during the restart would wake me up, otherwise I was sleeping through all the restarts. Eventually someone in the house would notice the machine was on and shut it down gracefully. I have no clue how long hibernate had been failing like this. Weeks at least.

But onto the solution. I didn't find much on the Internet about the problem (hence this post), but the combination of the basic-pm-debugging.txt and the pm-hibernate man page led me to try adding an '/etc/pm/config.d/hibernate file with the contents:

HIBERNATE_MODE=shutdown

With that in place, pm-hibernate went back to working like it used to, and I haven't been woken up by late night restarts since.

[*]For purposes of this blog, 'Laundry' is any non-computer indoor activity and 'Going for a Walk' is any non-computer outdoor activity.