Podman/Testinfra/Salt Lessons Learned

Posted on Sun 13 November 2022 in lessons

Oh my, less than two years between posts. I'm on a roll !

I've been looking into using Podman and Testinfra to test Salt states.

I'd like to add some unit tests and a Containerfile to an existing repo of salt states, where running 'pytest' in the repo's workspace would spin up the container and run the tests against it, and then tear down the container.

The tests would run 'salt state.apply' commands against the container, applying different sets of pillar data depending on the test.

Project Directory Structure

First let's set up a directory structure for the project that includes the states, their tests, and any needed test data. In the case of salt states, the test dtaa will be pillar files and files served by ext_pillar. The directory structure below is what I ended up using:

├─ test_project repo
├─── .git
├─── env
├────── ... testinfra egg
├─── Containerfile
├─── setup.cfg
├─── tests
├───── test_*.py
├───── data
├──────── ext_pillar
├──────── pillar
├────────── top.sls
├────────── test_zero.sls
├────────── test_one.sls
├────────── ...
├──────── top.sls
├─── test_project
├───── *.sls
├───── *.jinja
├───── templates
├──────── *.jinja
├───── files
├───── ...

Assuming all these files are stored in git, there's a .git directory from when you cloned the repo

The 'env' directory is a python virtualenv under 'env', where the testinfra egg has been installed. You can skip the virtualenv if you're pulling in testinfra from a global package.

Containerfile is, well a Podman Containerfile, and setup.cfg contains some pytest-specific settings.

The tests directory is where the testinfra test_*.py files are stored.

The tests/data/pillar directory will end up be mapped to the /srv/pillar directory in the test container. Similarly tests/data/ext_pillar will be mapped to /srv/ext_pillar.

The salt-states directory includes the *.sls and *.jinja files, and any other salt-related subdirectories like 'templates', 'files', 'macros', etc. This directory will be mapped to /srv/salt/project in the container.

Containerfile

The Containerfile I'm using for this project is below.

# I'm using Ubuntu 20.4 for this project-under-test so pull in the stock Ubuntu image for that version
FROM ubuntu:focal
RUN apt-get update

# The stock image doesn't include curl, so install it and bootstrap salt
# Longterm, I would host the bootstrapping script internally in case that site disappeared.
RUN apt-get install -y curl
RUN curl -L https://bootstrap.saltproject.io | sh -s --

# Configure salt run as a masterless minion
RUN echo "file_client: local" > /etc/salt/minion.d/masterless.conf
RUN printf "local" > /etc/salt/minion_id

# Set up the /srv/salt environment
RUN mkdir -p /srv/salt
RUN mkdir -p /srv/ext_pillar/hosts/local/files
RUN printf "ext_pillar:\n  - file_tree:\n      root_dir: /srv/ext_pillar\n" >>  /etc/salt/minion.d/masterless.conf

# Delay setting up /srv/salt/top.sls until the container starts. so PROJECT can be sent in as a ENV
RUN printf "printf \"base:\\n    '*':\\n      - \${PROJECT}\\n\" > /srv/salt/top.sls" >> /root/.bashrc

# Create a local user
RUN useradd local_user

# The Salt git states apparently assume git is already installed on the host, so install it.
RUN apt-get install -y git

Building and verifying the saltmasterless:latest image

Using this Containerfile, I built a saltmasterless:latest image:

workstation:~/projects/test_project.git# podman build -t saltmasterless:latest .

Then with this image, I can start a container that includes volumes mapping the tests/data/pillar ro /srv/pillar, tests/data/ext_pillar to /srv/ext_pillar, and test_project to /srv/salt:

workstation:~/projects/test_project.git# podman run -it --env "PROJECT=test_project" -v ${PWD}/test_project:/srv/salt/test_project -v ${PWD}/tests/data/pillar:/srv/pillar -v ${PWD}/tests/data/ext_pillar:/srv/ext_pillar/hosts/local/files --name test_box --hostname local saltmasterless:latest
root@local:/#
root@local:/# find /srv
root@local:/# exit
workstation:~/projects/test_project.git# podman rm -f test_box

setup.cfg

The setup.cfg file is mostly used to tell pytest to ignore the salt states directory:

tests/data/pillar/top.sls

As mentioned above the tests/data/pillar directory will be mapped to /srv/pillar in the container, but let's look at the top.sls a little closer. From the Containerfile, /etc/salt/minion_id was set to 'local', so normally the top.sls file will end up using /srv/pillar/test_zero.sls for it's pillar data.

But lets say we want to run a test with some other pillar data. In that case , in the test we'll use the salt-call '-id' argument to run the command as a different minion id. So with the top.sls file below, running 'salt-call --local -id=test_one state.apply' will use the test_one.sls pillar data instead of test_zero.sls

{{ saltenv }}:

  '*':
  - match: glob
  - ignore_missing: True

  'local':
  - test_zero

  'test_one':
  - test_one

  'missing_mandatory_pillar_item':
  - missing_mandatory_pillar_item

tests/test_project.py host fixture

The tests/test_project.py file includes a host fixture based on https://testinfra.readthedocs.io/en/latest/examples.html#test-docker-images. Note that the podman_cmd is pretty much the same as the command used above when testing the container. The cwd-related logic is because the -v args required full path names.

# scope='session' uses the same container for all the tests;
# scope='function' uses a new container per test function.
@pytest.fixture(scope='session')
def host(request):

    cwd = os.getcwd()

    podman_cmd = "podman run -d -it --env PROJECT=test_project -v ${PWD}/test_project:/srv/salt/test_project -v ${PWD}/tests/data/pillar:/srv/pillar -v ${PWD}/tests/data/ext_pillar:/srv/ext_pillar/hosts/local/files --name test_box --hostname local saltmasterless:latest bash"
    podman_cmd = podman_cmd.replace("${PWD}",cwd)
    podman_cmd_list = podman_cmd.split(' ')

    # run a container
    podman_id = subprocess.check_output(podman_cmd_list).decode().strip()
    # return a testinfra connection to the container
    yield testinfra.get_host("podman://" + podman_id)

    # at the end of the test suite, destroy the container
    subprocess.check_call(['podman', 'rm', '-f', podman_id])

tests/test_project.py full salt run test

Here's a test that does a full salt state.apply on the container. This test is slow, since the container starts are with just salt and git installed, and the project-under-test is making a lot of changes. Note theuse of the '--local' argument to tell salt to try to pull data from a saltmaster.

def test_full_salt_run(host):
    print('running salt-call state.apply.  This will take a few minutes')
    cmd_output = host.run('salt-call --state-output=terse --local state.apply')

    print('cmd.stdout: ' + cmd_output.stdout)

    assert cmd_output.rc == 0
    assert cmd_output.stderr == ''

tests/test_project.py alternative pillar data test

In this example, suppose ./test_project/map.jinja included a check like below:

{% if not salt['pillar.get']('mandatory_pillar_item') %}
  {{ raise('mandatory_pillar_item is mandatory') }}
{% endif %}

And then there's a 'missing_mandatory_pillar_item' in the ./test/data/pillar/top.sls as per above, and a ./test/data/pillar/missing_mandatory_pillar_item.sls file exists that's missing the mandatory pillar item.

Then a test like below could force a salt run that uses this pillar data by using the '--id' argument as per below, and an assertion could check the error was raised.

def test_missing_mandatory_pillar_itemn(host):
    print('running another salt-call state.apply with bad pillar data.')
    cmd_output = host.run('salt-call --state-output=terse --local --id=missing_mandatory_pillar_item state.apply')
    assert "mandatory_pillar_item is mandatory" in cmd_output.stderr
    assert cmd_output.rc != 0

Running the tests

The tests are kicked off via 'pytest' like any other python project using pytest.`

workstation:~/projects/test_project.git# source ./env/bin/activate
(env) workstation:~/projects/test_project.git# pytest
...
================================================================================ 3 passed in 333.88s (0:05:33) ================================================================================

What's Next

  • Set up the salt bootstrapping so it'll work without having to reach out to bootstrap.saltproject.io
  • Move the host fixture out of /tests/test_project.py to ./tests/conftest.py
  • Speed up the tests. As mentioned above, a full 'salt state.apply' for a project can take a few minutes on my workstation