An Exploration into Jinja2 Unit Tests Wth Pytest

Project Directory Structure

In my existing salt project repo, I added a tests subdirectory, a setup.cfg with the contents below, and added a test target to the project's Makefile.

I also installed the pytest and jinja2 eggs in a virtualenv in my working directory.

├─ test_project repo
├─── .git
├─── init.sls
├─── map.jinja
├─── templates
├─────── some_template.cfg
├─── tests
├─────── conftest.py
├─────── test_init.py
├─────── test_map.py
├─────── test_some_template.py
├─── setup.cfg
├─── Makefile
├─── env
├─────── ... pytest
├─────── ... jinja2

Here's a snippet of the Makefile for kicking off the tests:

Note that since the tests are python code, you should run them through whatever linters and style-checkers you're using on the rest of your python code.

test:
    py.test-3
    pycoderstyle ./tests/*.py

And here's the setup.cfg:

Without the extra '--tb=native' argument, pytest would sometimes through an internal error when jinja ended up throwing exception, as we'll see later below.

[tool:pytest]
python_files = test_*.py tests/__init__.py tests/*/__init__.py
#uncomment the line below for full unittest diffs
addopts =
        # Any --tb option except native (including no --tb option) throws an internal pytest exception
        # jinja exceptions are thrown
        --tb=native
        # Uncomment the next line for verbose output
        # -vv

[pycodestyle]
max-line-length=999
ignore=E121,E123,E126,E226,E24,E704,E221,E127,E128,W503,E731,E131,E402

Note there is a test_*.py file for each file that includes Jinja2 markup.

tests/conftest.py

The conftest.py contains the common fixtures used by the tests. I've tried adding docstring comments to explain how to use the fixtures, but also see the examples.

import pytest
from unittest.mock import Mock

import jinja2
from jinja2 import Environment, FileSystemLoader, ChoiceLoader, DictLoader, StrictUndefined


class RaiseException(Exception):
    """ Exception raised when using raise() in the mocked Jinja2 context"""
    pass

@pytest.fixture()
def mocked_templates(request):
    """ A dict of template names to template content.
        Use this to mock out Jinja 'import "template" as tmpl" lines.
    """
    mocked_templates = {}
    return mocked_templates


@pytest.fixture(autouse=True)
def jinja_env(request, mocked_templates):
    """  Provide a Jinja2 environment for loading templates.
         The ChoiceLoader will first check when mocking any 'import' style templates,
         Then the FileSystemLoader will check the local file system for templates.

         The DictLoader is first so the Jinja won't end up using the FileSystemLoader for
         templates pulled in with an import statement which doesn't include the 'with context'
         modifier.

         Setting undefined to StrictUndefined throws exceptions when the templates use undefined variables.
    """

    test_loader=ChoiceLoader([
       DictLoader(mocked_templates),
       FileSystemLoader('.'),
    ])

    env = Environment(loader=test_loader,
                      undefined=StrictUndefined,
                      extensions=['jinja2.ext.do', 'jinja2.ext.with_', 'jinja2.ext.loopcontrols'])

    return env


@pytest.fixture(scope='session', autouse=True)
def salt_context():
    """ Provide a set of common mocked keys.
        Currently this is only the 'raise' key for mocking out the raise() calls in the templates,
        and an empty 'salt' dict for adding salt-specific mocks.
    """

    def mocked_raise(err):
        raise RaiseException(err)

    context = {
        'raise': mocked_raise,
        'salt': {}
    }

    return context

init.sls

For purposes of the sections below, here's what the init.sls looks like:

#!jinja|yaml
# {% set version = salt['pillar.get']('version', 'latest') %}
# version: {{ version }}

# {% if version == 'nope' %}
#     {{ raise("OH NO YOU DIDN'T") }}
# {% endif %}

Mock out the Jinja Context

Let's test out that rendering init.sls should return a version key with some value.

Being able to mock out the salt pillar.get() function was a big breakthrough with respect to being able to write any sort of unit tests for the Salt states.

  @pytest.fixture
  def poc_context(self, salt_context):
      """ Provide a proof-of-concept context for mocking out salt[function](args) calls """
      poc_context = salt_context.copy()

      def mocked_pillar_get(key,default):
          """ Mocked salt['pillar.get'] function """
          pillar_data = {
              'version' : '1234'
          }
          return pillar_data.get(key, default)

      # This is the super sauce:
      # We can mock out the ``salt['function'](args)`` calls in the salt states by
      # defining a 'salt' dict in the context, who's keys are the functions, and the values of mocked functions
      poc_context['salt']['pillar.get'] = mocked_pillar_get

      return poc_context


def test_jinja_template_poc(self, jinja_env, poc_context):
    """ Render a template and check it has the expected content """

    # This assumes the tests are run from the root of the project.
    # The conftest.py file is setting the jinja_env to look for files under the 'latest' directory
    template = jinja_env.get_template('init.sls')

    # Return a string of the rendered template.
    result = template.render(poc_context)

    # Now we can run assertions on the returned rendered template.
    assert "version: 1234" in result

Mocking a raise() error

Now, let's see how we can test triggering the raise() error based on the pillar data:

@pytest.fixture
def bad_context(self, salt_context):
    """ Lets see what happens if the template triggers a raise() """

    # The base salt_context from conftest.py includes a 'raise' entry that raises a RaiseException
    bad_context = salt_context.copy()
    bad_context['salt']['pillar.get'] = lambda k, d: 'nope'
    return bad_context

def test_raise_poc(self, jinja_env, bad_context):
    """ Try rendering a template that should fail with some raise() exception """

    with pytest.raises(RaiseException) as exc_info:
        template = jinja_env.get_template('init.sls')
        result = template.render(bad_context)

    raised_exception = exc_info.value
    assert str(raised_exception) == "OH NO YOU DIDN'T"

Mocking imported templates

Sometimes the Jinja templates may try import other templates which are either out of scope with respect to the current project, or the import doesn't include the 'with context' modifier, so the Jinja context isn't available when rendering the template.

In this case we can used with DictLoader portion of the jinja_env to mock out importing the template.

In this example, lets assume the following template file exists in the templates directory:

{%- import 'missing.tmpl' as missing -%}
Can we mock out missing/out of scope imports ?

Mocked: {{ missing.perhaps }}
Macro Call: {{ missing.lost('forever' }}

Now here is a test that can mock out the missing.tmpl contents, including the lost() macro call:

def test_missing_template(self, jinja_env, mocked_templates, salt_context):
    """
    In this example, templates/missing-import.tmpl tries to import a non-available 'missing.tmpl' template.
    The ChoiceLoader checks DictLoader loader, which checks mocked_templates and finds a match
    """

    mocked_templates['missing.tmpl'] = """
       {% set perhaps="YES" %}
       {% macro lost(input) %}MOCKED_LOST{% endmacro %}
    """
    missing_template = jinja_env.get_template('templates/missing-import.tmpl')
    missing_result = missing_template.render(salt_context)
    assert "Mocked: YES" in missing_result
    assert "Macro Call: MOCKED_LOST" in missing_result

Mocking a macro call

Let's say I have a Jinja2 macro defined below:

#!jinja|yaml

# {% macro test_macro(input) %}
#   {% if input == 'nope' %}
#     {{ raise("UNACCEPTABLE") }}
#   {% endif %}
#   {% set version =  salt['pillar.get']('version', 'latest') %}
"macro sez {{ input }}":
  test.show_notification:
  - text: "{{ input }}"

"version sez {{ version }}":
  test.show_notifications:
  - text: "{{ version }}"

# {% endmacro %}

Testing out this macro is a little more involved, since first we have to include calls to the macro after rendering the template. Note we're reusing the poc_context fixture defined earlier so the pillar.get() call is still mocked out to return 1234 for the version"

def test_get_pillar_from_macro(self, jinja_env, poc_context):
    """
    If we want to reference the mocked context in the macros, we need
    to render the source + macro call within a context.
    """

    # The '[0]' is because get source returns a (source,filename,up-to-date) tuple.
    template_source = jinja_env.loader.get_source(jinja_env, 'macro.sls')[0]
    new_template = jinja_env.from_string(template_source + "{{ test_macro('hello') }}")
    result = new_template.render(poc_context)

    assert "macro sez hello" in result
    assert "version sez 1234" in result

It's also possible to check that the macro raises an error based on the input:

def test_raise_from_macro(self, jinja_env, salt_context):
    """
    In this test, try forcing a raise() from within a macro
    """

    with pytest.raises(RaiseException) as exc_info:
         template_source = jinja_env.loader.get_source(jinja_env, 'macro.sls')[0]
         new_template = jinja_env.from_string(template_source + "{{ test_macro('nope') }}")
         result = new_template.render(salt_context)

    raised_exception = exc_info.value
    assert str(raised_exception) == "UNACCEPTABLE"

FECUNDITY: Checking for undefined variables during template rendering

Back in the day I learned that one of the virtues of a scientific theory was 'fecundity', or the ability for the theory to predict new behavior the original theory hadn't considered.

It looks like this may be called fruitfulness now, but still whenever I stumble across something like this, I shout out 'FECUNDITY' internally to myself. :shrug:

While I was working on this project, I noticed the jinja Environment constructor has an undefined argument that defaulted to Undefined. I also noticed StrictUndefined was another value that the undefined argument could use.

It would be useful if the tests could throw exceptions when they ran into undefined variables. This could happen from typos in the templates, or possibly not mocking out all the globals variables used in a template.

So I tried making an jinja Environment with undefined=StrictUndefined, and wrote a test with template with a typo in a variable name to see if the test would raise an exception, and it did !

This is much more useful than the default jinja behavior where Jinja would give the typo a value of None and would likely surface in the output as a empty string.

It's also more useful than setting undefined to DebugUndefined, which sometimes raised an exception, but sometimes left the un-modified '{{ whatever }}' strings in the rendered templates. Bleh.

Here's the sample template I used, called unexpected_variable.sls. It's the same as the original init.sls, but with a 'verion' typo:

#!jinja|yaml

# {% set version = salt['pillar.get']('role:echo-polycom:version', 'latest') %}
# version: {{ version }}

# {% if verion == 'nope' %}
#     {{ raise("OH NO YOU DIDN'T") }}
# {% endif %}

And let's try adding this test, which is the same as the earlier test_jinja_template_poc() test, but with the buggy template:

def test_unexpected_variable(self, jinja_env, poc_context):
    """ Render a template and check it has the expected content """

    # This assumes the tests are run from the root of the project.
    # The conftest.py file is setting the jinja_env to look for files under the 'latest' directory
    template = jinja_env.get_template('unexpected_variable.sls')

    # Return a string of the rendered template.
    result = template.render(poc_context)

    # Now we can run assertions on the returned rendered template.
    assert "version: 1234" in result

This test will fail with an undefined error exception below ! Cool. I can fix the typo, and rerun the test to get it passing again ! FECUNDITY !

==================================================== FAILURES =======================================================
_________________________________________ TestJinja.test_unexpected_variable __________________________________________
Traceback (most recent call last):
  File "/my/working/dir/test_jinja_template_poc.py", line 150, in test_unexpected_variable
    result = template.render(poc_context)
  File "/usr/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
     return original_render(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
     return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
     reraise(exc_type, exc_value, tb)
   File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
     raise value.with_traceback(tb)
   File "unexpected_variable.sls", line 6, in top-level template code
     # {% if verion == 'nope' %}
 jinja2.exceptions.UndefinedError: 'verion' is undefined
========================================= 1 failed, 5 passed in 0.89 seconds ==========================================

Running the tests

The tests are kicked off via 'pytest' like any other python project using pytest.

workstation:~/projects/test_project.git# source ./env/bin/activate
(env) workstation:~/projects/test_project.git# pytest
===================================================================== test session starts =====================================================================
platform linux -- Python 3.6.8, pytest-2.9.2, py-1.4.32, pluggy-0.3.1
rootdir: /vagrant, inifile:
plugins: catchlog-1.2.2
collected 5 items

latest/tests/test_jinja_template_poc.py .....

Credit

I based this work on some ideas from the blog post A method of unit testing Jinja2 templates by alexharv074.