do-rsync.sh
cache
content/pages/resume.pdf
+.mypy_cache/
--- /dev/null
+#########################################################
+A workaround for running deja-dup as root in Ubuntu 20.04
+#########################################################
+
+:date: 2020-07-13
+:tags: hint, deja-dup, ubuntu
+:category: hints
+:author: Jude N
+
+I found a workaround for the `Duplicity fails to start`_ issue where :code:`'sudo deja-dup'` would fail with the python stacktrace mentioned in the launchpad ticket.
+
+The ticket was not very useful, so I started looking at the various files in the stacktrace and saw line from /usr/lib/python3/dist-packages/duplicity/backends/giobackend.py was within an :code:`"if u'DBUS_SESSION_BUS_ADDRESS' not in os.environ"` block.
+
+So I wondered what would happen if I let that environment variable pass into the sudo environment. I tried :code:`'sudo -E deja-dup'` as per `preserve the environment`_. This didn't result in a stacktrace, but it ended up running the backup as the normal non-root user, probably because the preserved environment included the USER and HOME variables along with the DBUS_SESSION_BUS_ADDRESS variable.
+
+Then I tried preserving just DBUS_SESSION_BUS_ADDRESS with :code:`'sudo --preserve-env=DBUS_SESSION_BUS_ADDRESS deja-dup'`, it worked as expected.
+
+So the hint here is that when presented with a stacktrace don't be afraid to "Use the Source, Jean Luc".
+
+.. image:: {static}/images/use-the-source-jean-luc.png
+ :align: center
+ :alt: A image of Patrick Stewart playing Gurney Halleck from David Lynch's Dune film, with the meme text 'Use the Source, Jean Luc'.
+
+
+.. _Duplicity fails to start: https://bugs.launchpad.net/ubuntu/+source/duplicity/+bug/1855736
+.. _preserve the environment: https://www.petefreitag.com/item/877.cfm
--- /dev/null
+################
+Expired CA Notes
+################
+
+:date: 2022-10-31
+:tags: hint, openssl, x509, expiration
+:category: hints
+:author: Jude N
+
+Recently, I ran some tests to see what would happen when my root CA cert expired, and what I'd need to do to update the cert.
+
+Spoiler alert: Updating the CA cert was not that hard...
+
+First I created a CA that expired in 2 hours using the new-ca.py code below:
+
+.. code-block:: python
+
+ from OpenSSL import crypto
+
+ #Following script will create a self signed root ca cert.
+ from OpenSSL import crypto, SSL
+ from os.path import join
+ import random
+
+ CN='expired-ca-test'
+ pubkey = "%s.crt" % CN #replace %s with CN
+ privkey = "%s.key" % CN # replcate %s with CN
+
+ pubkey = join(".", pubkey)
+ privkey = join(".", privkey)
+
+ k = crypto.PKey()
+ k.generate_key(crypto.TYPE_RSA, 2048)
+
+ # create a self-signed cert
+ cert = crypto.X509()
+ cert.get_subject().CN = CN
+ cert.set_serial_number(0)
+ cert.gmtime_adj_notBefore(0)
+ cert.gmtime_adj_notAfter(7200) # CA is only good for 2 hours
+ cert.set_issuer(cert.get_subject())
+ cert.set_subject(cert.get_subject())
+ cert.set_version(2)
+ xt = crypto.X509Extension(b'basicConstraints',1,b'CA:TRUE')
+ cert.add_extensions((xt,))
+
+ cert.set_pubkey(k)
+ cert.sign(k, 'sha512')
+ pub=crypto.dump_certificate(crypto.FILETYPE_PEM, cert)
+ priv=crypto.dump_privatekey(crypto.FILETYPE_PEM, k)
+ open(pubkey,"wt").write(pub.decode("utf-8"))
+ open(privkey, "wt").write(priv.decode("utf-8") )
+
+This block is based on how the ancient certmaster program `created its CA`_.
+
+Then I created a expired-ca-test.srl with contents "01"
+
+.. code-block:: bash
+
+ > echo 01 > expired-ca-test.srl
+
+Then I issued a cert against this CA:
+
+.. code-block:: bash
+
+ > openssl genrsa -out pre-expired-example.key 4096
+ > openssl req -new -key pre-expired-example.key -out pre-expired-example.csr
+ > openssl x509 -req -days 365 -in pre-expired-example.csr -CA expired-ca-test.crt -CAkey expired-ca-test.key -CAserial expired-ca-test.srl -out pre-expired-example.crt
+ > openssl x509 -in pre-expired-example.crt -text
+ > openssl verify -verbose -CAfile expired-ca-test.crt pre-expired-example.crt
+ pre-expired-example.crt: OK
+
+Then I `waited 2 hours`_ and went back to check the certs:
+
+.. code-block:: bash
+
+ > openssl x509 -in expired-ca-test.crt -noout -enddate
+ notAfter=Oct 2 17:41:11 2022 GMT
+
+Then I tested what would happen if I tried verifying the cert signed with the expired CA:
+
+.. code-block:: bash
+
+ > openssl verify -verbose -CAfile expired-ca-test.crt pre-expired-example.crt
+ CN = expired-ca-test
+ error 10 at 1 depth lookup: certificate has expired
+ error pre-expired-example.crt: verification failed
+
+
+THIS FAILED. I thought previously signed keys would continue to verify against the expired CA but new certs wouldn't be created. Instead previously signed certs won't validate against the expired CA.
+
+Then I tried signing a new cert with the expired CA. Certainly this fail, right ?
+
+.. code-block:: bash
+
+ > openssl genrsa -out expired-example.key 4096
+ > openssl req -new -key expired-example.key -out expired-example.csr
+ > openssl x509 -req -days 365 -in expired-example.csr -CA expired-ca-test.crt -CAkey expired-ca-test.key -CAserial expired-ca-test.srl -out expired-example.crt
+
+THIS WORKED in that it created the cert, though verification still fails:
+
+.. code-block:: bash
+
+ > openssl verify -verbose -CAfile expired-ca-test.crt expired-example.crt
+ CN = expired-ca-test
+ error 10 at 1 depth lookup: certificate has expired
+ error expired-example.crt: verification failed
+
+
+Now lets see what happens if we update the CA cert with the update-ca.py script below.
+
+This is almost the same as the new-ca.py script above, except **the original CA key is reused instead of generating a new key**. Also **the CN and serial number need to be the same as the original expired CA cert**.
+
+Verification will fail if the CN or serial number values are not the same as the original CA, but unfortuanetly I didn't save the errors from when I tried using 'updated-ca-test' as the CN, or when I tried bumping up the serial number to 1.
+
+.. code-block:: python
+
+ from OpenSSL import crypto
+
+ #Following script will create a self signed root ca cert.
+ from OpenSSL import crypto, SSL
+ from os.path import join
+ import random
+
+ CN='updated-ca-test'
+ pubkey = "%s.crt" % CN #replace %s with CN
+ privkey = "%s.key" % CN # replcate %s with CN
+
+ pubkey = join(".", pubkey)
+ privkey = join(".", privkey)
+
+ # Instead of creating a new key, use the old CA's key
+ # nope: k = crypto.PKey()
+ # nope: k.generate_key(crypto.TYPE_RSA, 2048)
+ st_key=open('expired-ca-test.key', 'rt').read()
+ k = crypto.load_privatekey(crypto.FILETYPE_PEM, st_key)
+
+ # create a self-signed cert
+ cert = crypto.X509()
+ cert.get_subject().CN = 'expired-ca-test' # keep the same CN as the old CA cert
+ cert.set_serial_number(0) # keep the same serial number as the old CA cert
+ cert.gmtime_adj_notBefore(0)
+ cert.gmtime_adj_notAfter(63072000) # CA is only good for 2 years
+ cert.set_issuer(cert.get_subject())
+ cert.set_subject(cert.get_subject())
+ cert.set_version(2)
+ xt = crypto.X509Extension(b'basicConstraints',1,b'CA:TRUE')
+ cert.add_extensions((xt,))
+
+ cert.set_pubkey(k)
+ cert.sign(k, 'sha512')
+ pub=crypto.dump_certificate(crypto.FILETYPE_PEM, cert)
+ priv=crypto.dump_privatekey(crypto.FILETYPE_PEM, k)
+ open(pubkey,"wt").write(pub.decode("utf-8"))
+ open(privkey, "wt").write(priv.decode("utf-8") )
+
+Note that this code creates a updated-ca-test.key that's the same as expired-ca-test.key, so I could have continued using expired-ca-test.key in the cert creation below.
+
+.. code-block:: bash
+
+ > diff expired-ca-test.key updated-ca-test.key
+ > echo $?
+ 0
+
+
+Next I created an updated-ca-test.srl file. I could have continuned using expired-ca-test.srl
+
+.. code-block:: bash
+
+ > cp expired-ca-test.srl updated-ca-test.srl
+
+Now let's see if the new CA can be used to create a new cert:
+
+.. code-block:: bash
+
+ > openssl genrsa -out post-expired-example.key 4096
+ > openssl req -new -key post-expired-example.key -out post-expired-example.csr
+ > openssl x509 -req -days 365 -in post-expired-example.csr -CA updated-ca-test.crt -CAkey updated-ca-test.key -CAserial updated-ca-test.srl -out post-expired-example.crt
+ > openssl x509 -in post-expired-example.crt -text
+ > openssl verify -verbose -CAfile updated-ca-test.crt post-expired-example.crt
+ post-expired-example.crt: OK
+
+Now verify the old cert verifies using the new CA:
+
+.. code-block:: bash
+
+ > openssl verify -verbose -CAfile updated-ca-test.crt pre-expired-example.crt
+ pre-expired-example.crt: OK
+
+THIS WORKED. The updated CA could be used to verify both new and previous created certs Hurray !!
+
+Conclusion
+==========
+
+An expired/expiring root CA may be a hassle, but it's not catastrophic. The biggest pain should be pushingout the updated root CA everywhere the cert is being used in your environment. If you're using an orchestration/CM tool like Salt or Ansible, updating the root CA cert shouldn't be too bad, but remember to reload or restart any services using the cert to force the updated CA cert to read.
+
+Sources
+=======
+- https://serverfault.com/questions/306345/certification-authority-root-certificate-expiry-and-renewal
+- https://gist.github.com/mohanpedala/468cf9cef473a8d7610320cff730cdd1
+
+
+.. _`created its CA`: https://github.com/jude/certmaster/blob/master/certmaster/certs.py#L92
+.. _`waited 2 hours`: https://store.steampowered.com/app/1366540/Dyson_Sphere_Program/
+
+
--- /dev/null
+#############
+How I orgmode
+#############
+
+:date: 2020-07-13
+:tags: hint, orgmode, orgzly, owncloud
+:category: hints
+:author: Jude N
+
+This post covers how I'm keeping my `orgmode`_ notebooks synced between my desktop and phone. There are also some tips on how I resolve merge conflicts in my notebooks and other hints for using Orgzly.
+
+Setup
+=====
+
+.. uml::
+
+ node workstation {
+ folder "orgs" as workstation_orgs
+ }
+
+
+ node laptop {
+ folder "orgs" as laptop_orgs
+ }
+
+ cloud ownCloud {
+ folder "orgs" as owncloud_orgs
+ }
+
+ node phone {
+ node Orgzly {
+ folder notebooks
+ }
+ }
+
+ workstation_orgs - owncloud_orgs
+ owncloud_orgs - notebooks
+ owncloud_orgs -- laptop_orgs
+
+(The `Pelican PlantUML plugin`_ is pretty nice.)
+
+Owncloud Server
+---------------
+
+Set up an `owncloud`_ server somewhere where both my desktop and phone can reach it.
+
+Desktop: Owncloud client and git repo
+-------------------------------------
+
+On the desktop, install the owncloud client and configured ~/owncloud to be shared with the server. Also set up an ~/owncloud/org directory and add an ~/org symlink that points to ~/owncloud/org. This is mostly to make it easier to access the org files.
+
+Update your emacs and/or vi configs to support orgmode. I'll update the org files in the editor that's more convenient at the time, though I'll use emacs if I need to update the deadline of a task.
+
+I've also added .git to the owncloud and did a 'git init' in the ~/owncloud/org directory. This repo will be used in the 'Handling Conflicts' section below.
+
+Create or move your notebook files into the ~/owncloud/org directory and make sure they have a '.org' extension. Verify that the org files are being synced to the owncloud server.
+
+Make sure your owncloud client is set to start when your desktop reboots.
+
+Phone: Owncloud client and Orgzly
+---------------------------------
+
+Install the `owncloud client`_ on your phone. Configure it to sync with the owncloud server and verify it's syncing the notebook files from the server.
+
+Also install and configure `Orgzly`_. For the syncing the repositories `use WebDAV` with the URL https://<your-owncloud-server>/remote.php/webdav/org. I originally tried using local storage, but I couldn't get that to work.
+
+Then for each of the notebooks, `set the link` to https://<your-owncloud-server>/remote.php/webdav/org. Update one of your notebooks with Orgzly and verify the change is synced to your desktop.
+
+
+Day-to-day Usage
+================
+
+Throughout the day, I'm usually using Orgzly, checking the Agenda tab for overdue and upcoming tasks, and adding new tasks and notes.
+
+I'm access the Notebooks on the desktop less frequently, mostly archiving completed tasks, and adding links and notes to research tasks. I also tend to move tasks between notebooks from the desktop.
+
+
+Handling Recurring Events
+-------------------------
+
+I ignore the scheduled time setting in Orgzly, and only set the deadlines with warning times. Then I treat the notification that's generated from warning time passing as the time I should start on a task.
+
+For repeating events, I use the '++' modifier for tasks. This way if I miss a few iterations of a task it will add the next iteration in the future.
+
+I'm also try to set an appropriate warning period when setting tasks.
+
+It took me a while to figure out I could tap on the date itself to bring up a calender instead of being limited to the 'today', 'tomorrow', and 'next week' options. <shrug>.
+
+Handling Conflicts
+------------------
+Sometimes when I'm updating the notebooks on both my desktop and phone, orgzly will say there's a conflict.
+
+When this happens I go to my desktop and make a checkpoint commit of any outstanding changes on the ~/owncloud/org repo. Then I push the Notebook from Orgzly (the Cloud with the 'up' arrow.
+
+Then on the desktop, I do a 'git diff' and adjust Notebook as needed.
+
+Usually this includes adding some new notes or adjusting some deadlines.
+
+.. _orgmode: https://orgmode.org/
+.. _orgzly: http://www.orgzly.com/
+.. _owncloud: https://owncloud.org/
+.. _owncloud client: https://owncloud.com/apps/
+.. _set the link: http://www.orgzly.com/help#4268ee7
+.. _use WebDAV: http://www.orgzly.com/help#sync-repo-webdav
+.. _Pelican PlantUML plugin: https://github.com/getpelican/pelican-plugins/tree/master/plantuml
--- /dev/null
+################################################
+An Exploration into Jinja2 Unit Tests Wth Pytest
+################################################
+
+:date: 2023-11-20
+:tags: lessons,jinja2,pytest,salt
+:category: lessons
+:author: Jude N
+
+This post covers an approach I've used for add pytest-style unit tests to my Salt-related projects.
+
+The content below may be specific to Salt, but the testing techniques should work on any project using Jinja2.
+
+Project Directory Structure
+===========================
+
+In my existing salt project repo, I added a tests subdirectory, a setup.cfg with the contents below, and added a test target to the project's Makefile.
+
+I also installed the pytest and jinja2 eggs in a virtualenv in my working directory.
+
+::
+
+ ├─ test_project repo
+ ├─── .git
+ ├─── init.sls
+ ├─── map.jinja
+ ├─── templates
+ ├─────── some_template.cfg
+ ├─── tests
+ ├─────── conftest.py
+ ├─────── test_init.py
+ ├─────── test_map.py
+ ├─────── test_some_template.py
+ ├─── setup.cfg
+ ├─── Makefile
+ ├─── env
+ ├─────── ... pytest
+ ├─────── ... jinja2
+
+Here's a snippet of the Makefile for kicking off the tests:
+
+Note that since the tests are python code, you should run them through whatever linters and style-checkers you're using on the rest of your python code.
+
+::
+
+ test:
+ py.test-3
+ pycoderstyle ./tests/*.py
+
+
+And here's the setup.cfg:
+
+Without the extra '--tb=native' argument, pytest would sometimes through an internal error when jinja ended up throwing exception, as we'll see later below.
+
+::
+
+ [tool:pytest]
+ python_files = test_*.py tests/__init__.py tests/*/__init__.py
+ #uncomment the line below for full unittest diffs
+ addopts =
+ # Any --tb option except native (including no --tb option) throws an internal pytest exception
+ # jinja exceptions are thrown
+ --tb=native
+ # Uncomment the next line for verbose output
+ # -vv
+
+ [pycodestyle]
+ max-line-length=999
+ ignore=E121,E123,E126,E226,E24,E704,E221,E127,E128,W503,E731,E131,E402
+
+
+Note there is a test_*.py file for each file that includes Jinja2 markup.
+
+tests/conftest.py
+=================
+
+The conftest.py contains the common fixtures used by the tests. I've tried adding docstring
+comments to explain how to use the fixtures, but also see the examples.
+
+::
+
+ import pytest
+ from unittest.mock import Mock
+
+ import jinja2
+ from jinja2 import Environment, FileSystemLoader, ChoiceLoader, DictLoader, StrictUndefined
+
+
+ class RaiseException(Exception):
+ """ Exception raised when using raise() in the mocked Jinja2 context"""
+ pass
+
+ @pytest.fixture()
+ def mocked_templates(request):
+ """ A dict of template names to template content.
+ Use this to mock out Jinja 'import "template" as tmpl" lines.
+ """
+ mocked_templates = {}
+ return mocked_templates
+
+
+ @pytest.fixture(autouse=True)
+ def jinja_env(request, mocked_templates):
+ """ Provide a Jinja2 environment for loading templates.
+ The ChoiceLoader will first check when mocking any 'import' style templates,
+ Then the FileSystemLoader will check the local file system for templates.
+
+ The DictLoader is first so the Jinja won't end up using the FileSystemLoader for
+ templates pulled in with an import statement which doesn't include the 'with context'
+ modifier.
+
+ Setting undefined to StrictUndefined throws exceptions when the templates use undefined variables.
+ """
+
+ test_loader=ChoiceLoader([
+ DictLoader(mocked_templates),
+ FileSystemLoader('.'),
+ ])
+
+ env = Environment(loader=test_loader,
+ undefined=StrictUndefined,
+ extensions=['jinja2.ext.do', 'jinja2.ext.with_', 'jinja2.ext.loopcontrols'])
+
+ return env
+
+
+ @pytest.fixture(scope='session', autouse=True)
+ def salt_context():
+ """ Provide a set of common mocked keys.
+ Currently this is only the 'raise' key for mocking out the raise() calls in the templates,
+ and an empty 'salt' dict for adding salt-specific mocks.
+ """
+
+ def mocked_raise(err):
+ raise RaiseException(err)
+
+ context = {
+ 'raise': mocked_raise,
+ 'salt': {}
+ }
+
+ return context
+
+init.sls
+========
+For purposes of the sections below, here's what the init.sls looks like:
+
+::
+
+ #!jinja|yaml
+ # {% set version = salt['pillar.get']('version', 'latest') %}
+ # version: {{ version }}
+
+ # {% if version == 'nope' %}
+ # {{ raise("OH NO YOU DIDN'T") }}
+ # {% endif %}
+
+
+
+Mock out the Jinja Context
+==========================
+
+Let's test out that rendering init.sls should return a version key with some value.
+
+Being able to mock out the salt pillar.get() function was a big breakthrough with respect to being able to write any sort of unit tests for the Salt states.
+
+::
+
+ @pytest.fixture
+ def poc_context(self, salt_context):
+ """ Provide a proof-of-concept context for mocking out salt[function](args) calls """
+ poc_context = salt_context.copy()
+
+ def mocked_pillar_get(key,default):
+ """ Mocked salt['pillar.get'] function """
+ pillar_data = {
+ 'version' : '1234'
+ }
+ return pillar_data.get(key, default)
+
+ # This is the super sauce:
+ # We can mock out the ``salt['function'](args)`` calls in the salt states by
+ # defining a 'salt' dict in the context, who's keys are the functions, and the values of mocked functions
+ poc_context['salt']['pillar.get'] = mocked_pillar_get
+
+ return poc_context
+
+
+ def test_jinja_template_poc(self, jinja_env, poc_context):
+ """ Render a template and check it has the expected content """
+
+ # This assumes the tests are run from the root of the project.
+ # The conftest.py file is setting the jinja_env to look for files under the 'latest' directory
+ template = jinja_env.get_template('init.sls')
+
+ # Return a string of the rendered template.
+ result = template.render(poc_context)
+
+ # Now we can run assertions on the returned rendered template.
+ assert "version: 1234" in result
+
+
+Mocking a raise() error
+=======================
+
+Now, let's see how we can test triggering the raise() error based on the pillar data:
+
+::
+
+ @pytest.fixture
+ def bad_context(self, salt_context):
+ """ Lets see what happens if the template triggers a raise() """
+
+ # The base salt_context from conftest.py includes a 'raise' entry that raises a RaiseException
+ bad_context = salt_context.copy()
+ bad_context['salt']['pillar.get'] = lambda k, d: 'nope'
+ return bad_context
+
+ def test_raise_poc(self, jinja_env, bad_context):
+ """ Try rendering a template that should fail with some raise() exception """
+
+ with pytest.raises(RaiseException) as exc_info:
+ template = jinja_env.get_template('init.sls')
+ result = template.render(bad_context)
+
+ raised_exception = exc_info.value
+ assert str(raised_exception) == "OH NO YOU DIDN'T"
+
+
+Mocking imported templates
+==========================
+
+Sometimes the Jinja templates may try import other templates which are either out of scope with respect to the current project, or the import doesn't include the 'with context' modifier, so the Jinja context isn't available when rendering the template.
+
+In this case we can used with DictLoader portion of the jinja_env to mock out importing the template.
+
+In this example, lets assume the following template file exists in the templates directory:
+
+::
+
+ {%- import 'missing.tmpl' as missing -%}
+ Can we mock out missing/out of scope imports ?
+
+ Mocked: {{ missing.perhaps }}
+ Macro Call: {{ missing.lost('forever' }}
+
+Now here is a test that can mock out the missing.tmpl contents, including the lost() macro call:
+
+::
+
+ def test_missing_template(self, jinja_env, mocked_templates, salt_context):
+ """
+ In this example, templates/missing-import.tmpl tries to import a non-available 'missing.tmpl' template.
+ The ChoiceLoader checks DictLoader loader, which checks mocked_templates and finds a match
+ """
+
+ mocked_templates['missing.tmpl'] = """
+ {% set perhaps="YES" %}
+ {% macro lost(input) %}MOCKED_LOST{% endmacro %}
+ """
+ missing_template = jinja_env.get_template('templates/missing-import.tmpl')
+ missing_result = missing_template.render(salt_context)
+ assert "Mocked: YES" in missing_result
+ assert "Macro Call: MOCKED_LOST" in missing_result
+
+
+Mocking a macro call
+====================
+
+Let's say I have a Jinja2 macro defined below:
+
+::
+
+ #!jinja|yaml
+
+ # {% macro test_macro(input) %}
+ # {% if input == 'nope' %}
+ # {{ raise("UNACCEPTABLE") }}
+ # {% endif %}
+ # {% set version = salt['pillar.get']('version', 'latest') %}
+ "macro sez {{ input }}":
+ test.show_notification:
+ - text: "{{ input }}"
+
+ "version sez {{ version }}":
+ test.show_notifications:
+ - text: "{{ version }}"
+
+ # {% endmacro %}
+
+Testing out this macro is a little more involved, since first we have to include calls to the macro after rendering the template. Note we're reusing the poc_context fixture defined earlier so the pillar.get() call is still mocked out to return 1234 for the version"
+
+::
+
+ def test_get_pillar_from_macro(self, jinja_env, poc_context):
+ """
+ If we want to reference the mocked context in the macros, we need
+ to render the source + macro call within a context.
+ """
+
+ # The '[0]' is because get source returns a (source,filename,up-to-date) tuple.
+ template_source = jinja_env.loader.get_source(jinja_env, 'macro.sls')[0]
+ new_template = jinja_env.from_string(template_source + "{{ test_macro('hello') }}")
+ result = new_template.render(poc_context)
+
+ assert "macro sez hello" in result
+ assert "version sez 1234" in result
+
+It's also possible to check that the macro raises an error based on the input:
+
+::
+
+ def test_raise_from_macro(self, jinja_env, salt_context):
+ """
+ In this test, try forcing a raise() from within a macro
+ """
+
+ with pytest.raises(RaiseException) as exc_info:
+ template_source = jinja_env.loader.get_source(jinja_env, 'macro.sls')[0]
+ new_template = jinja_env.from_string(template_source + "{{ test_macro('nope') }}")
+ result = new_template.render(salt_context)
+
+ raised_exception = exc_info.value
+ assert str(raised_exception) == "UNACCEPTABLE"
+
+FECUNDITY: Checking for undefined variables during template rendering
+=====================================================================
+Back in the day I learned that one of the virtues of a scientific theory was 'fecundity', or the ability for the theory to predict new behavior the original theory hadn't considered.
+
+It looks like this may be called `fruitfulness`_ now, but still whenever I stumble across something like this, I shout out 'FECUNDITY' internally to myself. :shrug:
+
+While I was working on this project, I noticed the jinja Environment constructor has an `undefined`_ argument that defaulted to Undefined. I also noticed `StrictUndefined`_ was another value that the undefined argument could use.
+
+It would be useful if the tests could throw exceptions when they ran into undefined variables. This could happen from typos in the templates, or possibly not mocking out all the globals variables used in a template.
+
+So I tried making an jinja Environment with undefined=StrictUndefined, and wrote a test with template with a typo in a variable name to see if the test would raise an exception, and it did !
+
+This is much more useful than the default jinja behavior where Jinja would give the typo a value of None and would likely surface in the output as a empty string.
+
+It's also more useful than setting undefined to `DebugUndefined`_, which sometimes raised an exception, but sometimes left the un-modified '{{ whatever }}' strings in the rendered templates. Bleh.
+
+Here's the sample template I used, called unexpected_variable.sls. It's the same as the original init.sls, but with a 'verion' typo:
+
+::
+
+ #!jinja|yaml
+
+ # {% set version = salt['pillar.get']('role:echo-polycom:version', 'latest') %}
+ # version: {{ version }}
+
+ # {% if verion == 'nope' %}
+ # {{ raise("OH NO YOU DIDN'T") }}
+ # {% endif %}
+
+And let's try adding this test, which is the same as the earlier test_jinja_template_poc() test, but with the buggy template:
+
+::
+
+ def test_unexpected_variable(self, jinja_env, poc_context):
+ """ Render a template and check it has the expected content """
+
+ # This assumes the tests are run from the root of the project.
+ # The conftest.py file is setting the jinja_env to look for files under the 'latest' directory
+ template = jinja_env.get_template('unexpected_variable.sls')
+
+ # Return a string of the rendered template.
+ result = template.render(poc_context)
+
+ # Now we can run assertions on the returned rendered template.
+ assert "version: 1234" in result
+
+This test will fail with an undefined error exception below !
+Cool. I can fix the typo, and rerun the test to get it passing again ! FECUNDITY !
+
+::
+
+ ==================================================== FAILURES =======================================================
+ _________________________________________ TestJinja.test_unexpected_variable __________________________________________
+ Traceback (most recent call last):
+ File "/my/working/dir/test_jinja_template_poc.py", line 150, in test_unexpected_variable
+ result = template.render(poc_context)
+ File "/usr/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
+ return original_render(self, *args, **kwargs)
+ File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
+ return self.environment.handle_exception(exc_info, True)
+ File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
+ reraise(exc_type, exc_value, tb)
+ File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
+ raise value.with_traceback(tb)
+ File "unexpected_variable.sls", line 6, in top-level template code
+ # {% if verion == 'nope' %}
+ jinja2.exceptions.UndefinedError: 'verion' is undefined
+ ========================================= 1 failed, 5 passed in 0.89 seconds ==========================================
+
+
+Running the tests
+=================
+
+The tests are kicked off via 'pytest' like any other python project using pytest.
+
+.. code-block:: shell-session
+
+ workstation:~/projects/test_project.git# source ./env/bin/activate
+ (env) workstation:~/projects/test_project.git# pytest
+ ===================================================================== test session starts =====================================================================
+ platform linux -- Python 3.6.8, pytest-2.9.2, py-1.4.32, pluggy-0.3.1
+ rootdir: /vagrant, inifile:
+ plugins: catchlog-1.2.2
+ collected 5 items
+
+ latest/tests/test_jinja_template_poc.py .....
+
+Credit
+======
+
+I based this work on some ideas from the blog post `A method of unit testing Jinja2 templates`_ by `alexharv074`_.
+
+
+.. _`A method of unit testing Jinja2 templates` : https://alexharv074.github.io/2020/01/18/a-method-of-unit-testing-jinja2-templates.html
+.. _`alexharv074` : https://alexharv074.github.io/
+.. _`fruitfulness` : https://link.springer.com/article/10.1007/s11229-017-1355-6
+.. _`undefined`: https://jinja.palletsprojects.com/en/3.1.x/api/#undefined-types
+.. _`StrictUndefined`: https://jinja.palletsprojects.com/en/3.1.x/api/#jinja2.StrictUndefined
+.. _`DebugUndefined`: https://jinja.palletsprojects.com/en/3.1.x/api/#jinja2.DebugUndefined
--- /dev/null
+#####################################
+Podman/Testinfra/Salt Lessons Learned
+#####################################
+
+:date: 2022-11-13
+:tags: lessons,podman,testinfra,salt
+:category: lessons
+:author: Jude N
+
+Oh my, less than two years between posts. I'm on a roll !
+
+I've been looking into using `Podman`_ and `Testinfra`_ to test `Salt`_ states.
+
+I'd like to add some unit tests and a Containerfile to an existing repo of salt states, where running 'pytest' in the repo's workspace would spin up the container and run the tests against it, and then tear down the container.
+
+The tests would run 'salt state.apply' commands against the container, applying different sets of pillar data depending on the test.
+
+Project Directory Structure
+===========================
+
+First let's set up a directory structure for the project that includes the states, their tests, and any needed test data. In the case of salt states, the test dtaa will be pillar files and files served by ext_pillar. The directory structure below is what I ended up using:
+
+::
+
+ ├─ test_project repo
+ ├─── .git
+ ├─── env
+ ├────── ... testinfra egg
+ ├─── Containerfile
+ ├─── setup.cfg
+ ├─── tests
+ ├───── test_*.py
+ ├───── data
+ ├──────── ext_pillar
+ ├──────── pillar
+ ├────────── top.sls
+ ├────────── test_zero.sls
+ ├────────── test_one.sls
+ ├────────── ...
+ ├──────── top.sls
+ ├─── test_project
+ ├───── *.sls
+ ├───── *.jinja
+ ├───── templates
+ ├──────── *.jinja
+ ├───── files
+ ├───── ...
+
+
+Assuming all these files are stored in git, there's a .git directory from when you cloned the repo
+
+The 'env' directory is a python virtualenv under 'env', where the testinfra egg has been installed. You can skip the virtualenv if you're pulling in testinfra from a global package.
+
+Containerfile is, well a Podman Containerfile, and setup.cfg contains some pytest-specific settings.
+
+The tests directory is where the testinfra test\_\*.py files are stored.
+
+The tests/data/pillar directory will end up be mapped to the /srv/pillar directory in the test container. Similarly tests/data/ext_pillar will be mapped to /srv/ext_pillar.
+
+The salt-states directory includes the \*.sls and \*.jinja files, and any other salt-related subdirectories like 'templates', 'files', 'macros', etc. This directory will be mapped to /srv/salt/project in the container.
+
+Containerfile
+-------------
+
+The Containerfile I'm using for this project is below.
+
+.. code-block:: docker
+
+ # I'm using Ubuntu 20.4 for this project-under-test so pull in the stock Ubuntu image for that version
+ FROM ubuntu:focal
+ RUN apt-get update
+
+ # The stock image doesn't include curl, so install it and bootstrap salt
+ # Longterm, I would host the bootstrapping script internally in case that site disappeared.
+ RUN apt-get install -y curl
+ RUN curl -L https://bootstrap.saltproject.io | sh -s --
+
+ # Configure salt run as a masterless minion
+ RUN echo "file_client: local" > /etc/salt/minion.d/masterless.conf
+ RUN printf "local" > /etc/salt/minion_id
+
+ # Set up the /srv/salt environment
+ RUN mkdir -p /srv/salt
+ RUN mkdir -p /srv/ext_pillar/hosts/local/files
+ RUN printf "ext_pillar:\n - file_tree:\n root_dir: /srv/ext_pillar\n" >> /etc/salt/minion.d/masterless.conf
+
+ # Delay setting up /srv/salt/top.sls until the container starts. so PROJECT can be sent in as a ENV
+ RUN printf "printf \"base:\\n '*':\\n - \${PROJECT}\\n\" > /srv/salt/top.sls" >> /root/.bashrc
+
+ # Create a local user
+ RUN useradd local_user
+
+ # The Salt git states apparently assume git is already installed on the host, so install it.
+ RUN apt-get install -y git
+
+Building and verifying the saltmasterless:latest image
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Using this Containerfile, I built a saltmasterless:latest image:
+
+.. code-block:: shell-session
+
+ workstation:~/projects/test_project.git# podman build -t saltmasterless:latest .
+
+Then with this image, I can start a container that includes volumes mapping the tests/data/pillar ro /srv/pillar, tests/data/ext_pillar to /srv/ext_pillar, and test_project to /srv/salt:
+
+.. code-block:: shell-session
+
+ workstation:~/projects/test_project.git# podman run -it --env "PROJECT=test_project" -v ${PWD}/test_project:/srv/salt/test_project -v ${PWD}/tests/data/pillar:/srv/pillar -v ${PWD}/tests/data/ext_pillar:/srv/ext_pillar/hosts/local/files --name test_box --hostname local saltmasterless:latest
+ root@local:/#
+ root@local:/# find /srv
+ root@local:/# exit
+ workstation:~/projects/test_project.git# podman rm -f test_box
+
+setup.cfg
+---------
+The setup.cfg file is mostly used to tell pytest to ignore the salt states directory:
+
+.. code-block
+
+ [tool:pytest]
+ norecursedirs = test_project/files/*
+ addopts = -s
+ log_cli=true
+ log_level=NOTSET
+
+tests/data/pillar/top.sls
+-------------------------
+As mentioned above the tests/data/pillar directory will be mapped to /srv/pillar in the container, but let's look at the top.sls a little closer. From the Containerfile, /etc/salt/minion_id was set to 'local', so normally the top.sls file will end up using /srv/pillar/test_zero.sls for it's pillar data.
+
+But lets say we want to run a test with some other pillar data. In that case , in the test we'll use the salt-call '-id' argument to run the command as a different minion id. So with the top.sls file below, running 'salt-call --local -id=test_one state.apply' will use the test_one.sls pillar data instead of test_zero.sls
+
+.. code-block:: yaml+jinja
+
+ {{ saltenv }}:
+
+ '*':
+ - match: glob
+ - ignore_missing: True
+
+ 'local':
+ - test_zero
+
+ 'test_one':
+ - test_one
+
+ 'missing_mandatory_pillar_item':
+ - missing_mandatory_pillar_item
+
+
+tests/test_project.py host fixture
+----------------------------------
+
+The tests/test_project.py file includes a host fixture based on https://testinfra.readthedocs.io/en/latest/examples.html#test-docker-images. Note that the podman_cmd is pretty much the same as the command used above when testing the container. The cwd-related logic is because the -v args required full path names.
+
+.. code-block:: python
+
+ # scope='session' uses the same container for all the tests;
+ # scope='function' uses a new container per test function.
+ @pytest.fixture(scope='session')
+ def host(request):
+
+ cwd = os.getcwd()
+
+ podman_cmd = "podman run -d -it --env PROJECT=test_project -v ${PWD}/test_project:/srv/salt/test_project -v ${PWD}/tests/data/pillar:/srv/pillar -v ${PWD}/tests/data/ext_pillar:/srv/ext_pillar/hosts/local/files --name test_box --hostname local saltmasterless:latest bash"
+ podman_cmd = podman_cmd.replace("${PWD}",cwd)
+ podman_cmd_list = podman_cmd.split(' ')
+
+ # run a container
+ podman_id = subprocess.check_output(podman_cmd_list).decode().strip()
+ # return a testinfra connection to the container
+ yield testinfra.get_host("podman://" + podman_id)
+
+ # at the end of the test suite, destroy the container
+ subprocess.check_call(['podman', 'rm', '-f', podman_id])
+
+tests/test_project.py full salt run test
+----------------------------------------
+
+Here's a test that does a full salt state.apply on the container. This test is slow, since the container starts are with just salt and git installed, and the project-under-test is making a lot of changes. Note theuse of the '--local' argument to tell salt to try to pull data from a saltmaster.
+
+.. code-block:: python
+
+ def test_full_salt_run(host):
+ print('running salt-call state.apply. This will take a few minutes')
+ cmd_output = host.run('salt-call --state-output=terse --local state.apply')
+
+ print('cmd.stdout: ' + cmd_output.stdout)
+
+ assert cmd_output.rc == 0
+ assert cmd_output.stderr == ''
+
+tests/test_project.py alternative pillar data test
+---------------------------------------------------
+
+In this example, suppose ./test_project/map.jinja included a check like below:
+
+.. code-block:: jinja
+
+ {% if not salt['pillar.get']('mandatory_pillar_item') %}
+ {{ raise('mandatory_pillar_item is mandatory') }}
+ {% endif %}
+
+And then there's a 'missing_mandatory_pillar_item' in the ./test/data/pillar/top.sls as per above, and a ./test/data/pillar/missing_mandatory_pillar_item.sls file exists that's missing the mandatory pillar item.
+
+Then a test like below could force a salt run that uses this pillar data by using the '--id' argument as per below, and an assertion could check the error was raised.
+
+.. code-block:: python
+
+ def test_missing_mandatory_pillar_itemn(host):
+ print('running another salt-call state.apply with bad pillar data.')
+ cmd_output = host.run('salt-call --state-output=terse --local --id=missing_mandatory_pillar_item state.apply')
+ assert "mandatory_pillar_item is mandatory" in cmd_output.stderr
+ assert cmd_output.rc != 0
+
+Running the tests
+=================
+
+The tests are kicked off via 'pytest' like any other python project using pytest.`
+
+.. code-block:: shell-session
+
+ workstation:~/projects/test_project.git# source ./env/bin/activate
+ (env) workstation:~/projects/test_project.git# pytest
+ ...
+ ================================================================================ 3 passed in 333.88s (0:05:33) ================================================================================
+
+
+What's Next
+===========
+
+- Set up the salt bootstrapping so it'll work without having to reach out to bootstrap.saltproject.io
+- Move the host fixture out of /tests/test_project.py to ./tests/conftest.py
+- Speed up the tests. As mentioned above, a full 'salt state.apply' for a project can take a few minutes on my workstation
+
+.. _Podman: https://podman.io/
+.. _Testinfra: https://github.com/pytest-dev/pytest-testinfra
+.. _Salt: https://saltproject.io/
+
--- /dev/null
+TS Stands For Test Station
+##########################
+
+:date: 2022-12-17
+:tags: now-you-know,infrastructure,test-stations
+:category: lessons
+:author: Jude N
+
+Every now and then someone will come along and spraypaint yellow "TS"'s on the sidewalks around the neighorhood with arrows next to them. The arrows they lead to little square metal covers with a hole in the middle.
+
+From `99% Invisible`_, I figured it had something to do with the gas line since it was yellow, and that they was probably some sort of access point under the covers, but I couldn't figure out why they were using 'TS' instead something like 'GL' or 'GAS'.
+
+Recently I found one of the TS's pointing to a more informative cover:
+
+.. image:: {static}/images/test-station.png
+ :align: center
+ :alt: A yellow spraypainted 'TS' pointing to a metal cover including the text 'Test Station'.
+
+Apparently it's the test station for a `Cathodic Protection System`_.
+
+.. _99% Invisible: https://99percentinvisible.org/article/colorful-language-decoding-utility-markings-spray-painted-on-city-streets/
+.. _Cathodic Protection System: https://www.usbr.gov/tsc/training/webinars-corrosion/2014-02_TestingCathodicProtectionSystems/2014-02_TestingCathodicProtectionSystems_slides_508.pdf
--- /dev/null
+TakeMyBlood lessons learned
+###########################
+
+:date: 2022-08-22
+:tags: lessons
+:category: lessons
+:author: Jude N
+:status: draft
+
+I'm a frequent blood donor, but I don't like the `Red Cross donation site`_, and don't get me started on their Android app. I could go off on this for a while. but lets just say I've
+rage-quit setting up a donation appointment more than once.
+
+When I came across the Julia Evans `How to use undocumented web APIs`, I looked into whether I could come up with some way of finding donation sites that worked better for me.
+
+
+
+
+
+.. _Red Cross donation site: https://www.redcrossblood.org/give.html
+.. _How to use undocumented web APIs: https://jvns.ca/blog/2022/03/10/how-to-use-undocumented-web-apis/
Jude Nagurney
#############
-.. :date: 2019-07-14
+.. :date: 2024-05-02
.. :tags: resume
.. :category: resume
.. :author: Jude N
-.. :status: hidden
+.. :status: published
-| 1136 Forest Edge Drive
-| Reston, Virginia, 20190
-| Phone: (703) 829-0744
+| Northern Virginia
| jude.nagurney@gmail.com
=======
Technical Skills:
=================
-- **Languages** : Python, C/C++, Ruby, Perl, Java, SQL, Bash, lua, Expect, Tcl/Tk, UML OCL, COBOL
-- **Tools** : Puppet, Salt, Cobbler, Jenkins, emacs, vi, Jira, git, Gitlab, Docker, Mercurial, Subversion, Jira, SELinux
-- **Frameworks** : Django, Angular, Pylons, Rails, SqlAlchemy
-- **Operating Systems** : Linux (Ubuntu, Debian, RedHat, CentOS, Raspbian), Microsoft Windows, vxWorks, Solaris
-- **Databases** : PostgresSQL, MySQL, Oracle
-- **Standards Expertise** : SONET, SDH, TL1, LMP
+- **Languages** : Python, C/C++, Ruby, SQL, Bash, lua, Javascript, Typescript
+- **Tools** : Salt, SELinux, Jenkins, emacs, vi, Jira, git, Gitlab, Docker, podman, Automate Android app, Nagios, Zabbix
+- **Frameworks** : Django, Angular, Rails, SqlAlchemy, AngularJS
+- **Operating Systems** : Linux (Ubuntu, Debian, RedHat, CentOS, Rocky, Raspbian), Microsoft Windows
+- **Databases** : PostgresSQL, MySQL, Oracle, sqlite
----
===============
------------------
-Layer 2 Technology
+L2T, LLC
------------------
-| **Reston Virginia**
+| **Herndon Virginia**
| **October 2016 - Present**
------------------
-Software Engineer
------------------
+-----------------------
+Staff Software Engineer
+-----------------------
+| **February 2023 - Present**
+
+Solution-oriented Staff Engineer adept at unraveling complex technical challenges, and fostering a culture of proactive problem-solving across projects.
+
+- Created a series of project-based orientation presentations to facilitate onboarding of new employees.
+
+- Assumed responsibility for a mission-critical, dormant project, ensuring users had a dedicated point of contact.
+
+- Assumed responsibility for tagging new software releases and pushing the releases into production.
+
+- Worked with the leadership at my current company to recognize the staff engineer role.
+
+- Reviewed software changes to ensure code quality standards were upheld, and that knowledge was shared across the teams.
+
+------------------------
+Senior Software Engineer
+------------------------
+| **October 2016 - February 2023**
Developed and maintained Python-based software projects
-- Developed an SMS-based solution for the Netgear LTE Mobile Horspot Router
- This included a deep dive into the AT modem commands used for sending and receiving SMS messages.
+- Developed an SMS-based solution for the Netgear LTE Mobile Hotspot Router
+ Conducted a deep dive into AT modem commands essential for sending and receiving SMS messages.
- Developed SMS-based solutions using services such as Twilio, Plivo, Nexmo, and Vitelity.
+- Developed and maintained Salt states and Puppet manifests for various projects.
+
+- Developed flows with the Automate app for controlling and monitoring Android phones.
+
- Wrote a Errbot plugin for reporting open merge requests that were waiting for peer reviews.
-- Worked on porting projects to Raspbian to run on a Raspberry Pi 3 Model B.
+- Worked on porting projects to Raspbian to run on a Raspberry Pi 3 Model B.
This included rebuilding packages for the arm7 architecture.
-- Developed and maintained Salt states and Puppet manifests for various projects.
-
-- Developed and maintained Jenkins continuous integration jobs for various projects.
+- Developed and maintained Jenkins continuous integration jobs for multiple projects.
Also proactively tracked down the root causes of build failures when the jobs failed.
+- Developed and monitored Nagios monitoring for the development network.
+
+- Provided Production support and troubleshooting.
+
+- Provided training and support for junior engineers.
+
---------------------
Applied Security Inc.
---------------------
- Extended a project to dynamically allocate AWS hosts based on system usage.
- Wrote Python code for sending and receiving SMS messages through Plivo and Twilio
-
+
- Developed and maintained Puppet manifests for development projects
..................................
Extended devops practices to cover security reviews
-- Introduced an SELinux strict policy workflow allowing developers to do most of the work associated with setting up a policy.
+- Introduced an SELinux strict policy workflow allowing developers to do most of the work associated with setting up a policy.
Previously all policy work was done by a single engineer. Now policy work can be distributed across the development team.
- Continued supporting puppet infrastructure for both the dev and ops environments, especially with respect to security-related changes.
TRANSLATION_FEED_ATOM = None
FEED_RSS = 'feeds/rss.xml'
FEED_ALL_RSS = 'feeds/all.rss.xml'
-CATEGORY_FEED_RSS = 'feeds/cat.%s.rss.xml'
-TAG_FEED_RSS = 'feeds/tag.%s.rss.xml'
+CATEGORY_FEED_RSS = 'feeds/cat.{slug}.rss.xml'
+TAG_FEED_RSS = 'feeds/tag.{slug}.rss.xml'
FEED_MAX_ITEMS = 20
# Blogroll
TAG_CLOUD_STEPS = 4
TAG_CLOUD_MAX_ITEMS = 100
-THEME = "../pelican-themes/pelican-mockingbird"
+#THEME = "../pelican-themes/pelican-mockingbird"
+THEME = "../pelican-themes/Flex"
+THEME_COLOR = 'dark'
+USE_LESS = False
+MAIN_MENU = True
+SITELOGO = None
+SITELOGO = '/blog/images/profile.png'
+FAVICON = '/blog/images/favicon-16x16.png'
+SUMMARY_MAX_LENGTH = 0
DISPLAY_PAGES_ON_MENU = True
+PLUGIN_PATHS = ['../pelican-plugins']
+PLUGINS = ['plantuml']
+
+MENUITEMS = (
+ ("Archives", "/blog/archives.html"),
+ ("Categories", "/blog/categories.html"),
+ ("Tags", "/blog/tags.html"),
+)
+
+SOCIAL = (
+ ("github", "https://github.com/jude"),
+ ("rss", "blog/feeds/all.rss.xml"),
+ ("mastodon", "https://aleph.land/@pwan")
+)
TRANSLATION_FEED_ATOM = None
FEED_RSS = 'feeds/rss.xml'
FEED_ALL_RSS = 'feeds/all.rss.xml'
-CATEGORY_FEED_RSS = 'feeds/cat.%s.rss.xml'
-TAG_FEED_RSS = 'feeds/tag.%s.rss.xml'
+CATEGORY_FEED_RSS = 'feeds/cat.{slug}.rss.xml'
+TAG_FEED_RSS = 'feeds/tag.{slug}.rss.xml'
FEED_MAX_ITEMS = 20
+DISPLAY_PAGES_ON_MENU = True
+
DELETE_OUTPUT_DIRECTORY = True
+PLUGIN_PATHS = ['../pelican-plugins']
+PLUGINS = ['plantuml']
+
+
# Following items are often useful when publishing
#DISQUS_SITENAME = ""
projects / pwan.org.git / commitdiff
