An Exploration into Jinja2 Unit Tests Wth Pytest
Posted on Mon 20 November 2023 in lessons • Tagged with lessons, jinja2, pytest, salt
This post covers an approach I've used for add pytest-style unit tests to my Salt-related projects.
The content below may be specific to Salt, but the testing techniques should work on any project using Jinja2.
Project Directory Structure
In my existing salt project repo, I added a tests subdirectory, a setup.cfg with the contents below, and added a test target to the project's Makefile.
I also installed the pytest and jinja2 eggs in a virtualenv in my working directory.
├─ test_project repo ├─── .git ├─── init.sls ├─── map.jinja ├─── templates ├─────── some_template.cfg ├─── tests ├─────── conftest.py ├─────── test_init.py ├─────── test_map.py ├─────── test_some_template.py ├─── setup.cfg ├─── Makefile ├─── env ├─────── ... pytest ├─────── ... jinja2
Here's a snippet of the Makefile for kicking off the tests:
Note that since the tests are python code, you should run them through whatever linters and style-checkers you're using on the rest of your python code.
test:
py.test-3
pycoderstyle ./tests/*.py
And here's the setup.cfg:
Without the extra '--tb=native' argument, pytest would sometimes through an internal error when jinja ended up throwing exception, as we'll see later below.
[tool:pytest]
python_files = test_*.py tests/__init__.py tests/*/__init__.py
#uncomment the line below for full unittest diffs
addopts =
# Any --tb option except native (including no --tb option) throws an internal pytest exception
# jinja exceptions are thrown
--tb=native
# Uncomment the next line for verbose output
# -vv
[pycodestyle]
max-line-length=999
ignore=E121,E123,E126,E226,E24,E704,E221,E127,E128,W503,E731,E131,E402
Note there is a test_*.py file for each file that includes Jinja2 markup.
tests/conftest.py
The conftest.py contains the common fixtures used by the tests. I've tried adding docstring comments to explain how to use the fixtures, but also see the examples.
import pytest
from unittest.mock import Mock
import jinja2
from jinja2 import Environment, FileSystemLoader, ChoiceLoader, DictLoader, StrictUndefined
class RaiseException(Exception):
""" Exception raised when using raise() in the mocked Jinja2 context"""
pass
@pytest.fixture()
def mocked_templates(request):
""" A dict of template names to template content.
Use this to mock out Jinja 'import "template" as tmpl" lines.
"""
mocked_templates = {}
return mocked_templates
@pytest.fixture(autouse=True)
def jinja_env(request, mocked_templates):
""" Provide a Jinja2 environment for loading templates.
The ChoiceLoader will first check when mocking any 'import' style templates,
Then the FileSystemLoader will check the local file system for templates.
The DictLoader is first so the Jinja won't end up using the FileSystemLoader for
templates pulled in with an import statement which doesn't include the 'with context'
modifier.
Setting undefined to StrictUndefined throws exceptions when the templates use undefined variables.
"""
test_loader=ChoiceLoader([
DictLoader(mocked_templates),
FileSystemLoader('.'),
])
env = Environment(loader=test_loader,
undefined=StrictUndefined,
extensions=['jinja2.ext.do', 'jinja2.ext.with_', 'jinja2.ext.loopcontrols'])
return env
@pytest.fixture(scope='session', autouse=True)
def salt_context():
""" Provide a set of common mocked keys.
Currently this is only the 'raise' key for mocking out the raise() calls in the templates,
and an empty 'salt' dict for adding salt-specific mocks.
"""
def mocked_raise(err):
raise RaiseException(err)
context = {
'raise': mocked_raise,
'salt': {}
}
return context
init.sls
For purposes of the sections below, here's what the init.sls looks like:
#!jinja|yaml
# {% set version = salt['pillar.get']('version', 'latest') %}
# version: {{ version }}
# {% if version == 'nope' %}
# {{ raise("OH NO YOU DIDN'T") }}
# {% endif %}
Mock out the Jinja Context
Let's test out that rendering init.sls should return a version key with some value.
Being able to mock out the salt pillar.get() function was a big breakthrough with respect to being able to write any sort of unit tests for the Salt states.
@pytest.fixture
def poc_context(self, salt_context):
""" Provide a proof-of-concept context for mocking out salt[function](args) calls """
poc_context = salt_context.copy()
def mocked_pillar_get(key,default):
""" Mocked salt['pillar.get'] function """
pillar_data = {
'version' : '1234'
}
return pillar_data.get(key, default)
# This is the super sauce:
# We can mock out the ``salt['function'](args)`` calls in the salt states by
# defining a 'salt' dict in the context, who's keys are the functions, and the values of mocked functions
poc_context['salt']['pillar.get'] = mocked_pillar_get
return poc_context
def test_jinja_template_poc(self, jinja_env, poc_context):
""" Render a template and check it has the expected content """
# This assumes the tests are run from the root of the project.
# The conftest.py file is setting the jinja_env to look for files under the 'latest' directory
template = jinja_env.get_template('init.sls')
# Return a string of the rendered template.
result = template.render(poc_context)
# Now we can run assertions on the returned rendered template.
assert "version: 1234" in result
Mocking a raise() error
Now, let's see how we can test triggering the raise() error based on the pillar data:
@pytest.fixture
def bad_context(self, salt_context):
""" Lets see what happens if the template triggers a raise() """
# The base salt_context from conftest.py includes a 'raise' entry that raises a RaiseException
bad_context = salt_context.copy()
bad_context['salt']['pillar.get'] = lambda k, d: 'nope'
return bad_context
def test_raise_poc(self, jinja_env, bad_context):
""" Try rendering a template that should fail with some raise() exception """
with pytest.raises(RaiseException) as exc_info:
template = jinja_env.get_template('init.sls')
result = template.render(bad_context)
raised_exception = exc_info.value
assert str(raised_exception) == "OH NO YOU DIDN'T"
Mocking imported templates
Sometimes the Jinja templates may try import other templates which are either out of scope with respect to the current project, or the import doesn't include the 'with context' modifier, so the Jinja context isn't available when rendering the template.
In this case we can used with DictLoader portion of the jinja_env to mock out importing the template.
In this example, lets assume the following template file exists in the templates directory:
{%- import 'missing.tmpl' as missing -%}
Can we mock out missing/out of scope imports ?
Mocked: {{ missing.perhaps }}
Macro Call: {{ missing.lost('forever' }}
Now here is a test that can mock out the missing.tmpl contents, including the lost() macro call:
def test_missing_template(self, jinja_env, mocked_templates, salt_context):
"""
In this example, templates/missing-import.tmpl tries to import a non-available 'missing.tmpl' template.
The ChoiceLoader checks DictLoader loader, which checks mocked_templates and finds a match
"""
mocked_templates['missing.tmpl'] = """
{% set perhaps="YES" %}
{% macro lost(input) %}MOCKED_LOST{% endmacro %}
"""
missing_template = jinja_env.get_template('templates/missing-import.tmpl')
missing_result = missing_template.render(salt_context)
assert "Mocked: YES" in missing_result
assert "Macro Call: MOCKED_LOST" in missing_result
Mocking a macro call
Let's say I have a Jinja2 macro defined below:
#!jinja|yaml
# {% macro test_macro(input) %}
# {% if input == 'nope' %}
# {{ raise("UNACCEPTABLE") }}
# {% endif %}
# {% set version = salt['pillar.get']('version', 'latest') %}
"macro sez {{ input }}":
test.show_notification:
- text: "{{ input }}"
"version sez {{ version }}":
test.show_notifications:
- text: "{{ version }}"
# {% endmacro %}
Testing out this macro is a little more involved, since first we have to include calls to the macro after rendering the template. Note we're reusing the poc_context fixture defined earlier so the pillar.get() call is still mocked out to return 1234 for the version"
def test_get_pillar_from_macro(self, jinja_env, poc_context):
"""
If we want to reference the mocked context in the macros, we need
to render the source + macro call within a context.
"""
# The '[0]' is because get source returns a (source,filename,up-to-date) tuple.
template_source = jinja_env.loader.get_source(jinja_env, 'macro.sls')[0]
new_template = jinja_env.from_string(template_source + "{{ test_macro('hello') }}")
result = new_template.render(poc_context)
assert "macro sez hello" in result
assert "version sez 1234" in result
It's also possible to check that the macro raises an error based on the input:
def test_raise_from_macro(self, jinja_env, salt_context):
"""
In this test, try forcing a raise() from within a macro
"""
with pytest.raises(RaiseException) as exc_info:
template_source = jinja_env.loader.get_source(jinja_env, 'macro.sls')[0]
new_template = jinja_env.from_string(template_source + "{{ test_macro('nope') }}")
result = new_template.render(salt_context)
raised_exception = exc_info.value
assert str(raised_exception) == "UNACCEPTABLE"
FECUNDITY: Checking for undefined variables during template rendering
Back in the day I learned that one of the virtues of a scientific theory was 'fecundity', or the ability for the theory to predict new behavior the original theory hadn't considered.
It looks like this may be called fruitfulness now, but still whenever I stumble across something like this, I shout out 'FECUNDITY' internally to myself. :shrug:
While I was working on this project, I noticed the jinja Environment constructor has an undefined argument that defaulted to Undefined. I also noticed StrictUndefined was another value that the undefined argument could use.
It would be useful if the tests could throw exceptions when they ran into undefined variables. This could happen from typos in the templates, or possibly not mocking out all the globals variables used in a template.
So I tried making an jinja Environment with undefined=StrictUndefined, and wrote a test with template with a typo in a variable name to see if the test would raise an exception, and it did !
This is much more useful than the default jinja behavior where Jinja would give the typo a value of None and would likely surface in the output as a empty string.
It's also more useful than setting undefined to DebugUndefined, which sometimes raised an exception, but sometimes left the un-modified '{{ whatever }}' strings in the rendered templates. Bleh.
Here's the sample template I used, called unexpected_variable.sls. It's the same as the original init.sls, but with a 'verion' typo:
#!jinja|yaml
# {% set version = salt['pillar.get']('role:echo-polycom:version', 'latest') %}
# version: {{ version }}
# {% if verion == 'nope' %}
# {{ raise("OH NO YOU DIDN'T") }}
# {% endif %}
And let's try adding this test, which is the same as the earlier test_jinja_template_poc() test, but with the buggy template:
def test_unexpected_variable(self, jinja_env, poc_context):
""" Render a template and check it has the expected content """
# This assumes the tests are run from the root of the project.
# The conftest.py file is setting the jinja_env to look for files under the 'latest' directory
template = jinja_env.get_template('unexpected_variable.sls')
# Return a string of the rendered template.
result = template.render(poc_context)
# Now we can run assertions on the returned rendered template.
assert "version: 1234" in result
This test will fail with an undefined error exception below ! Cool. I can fix the typo, and rerun the test to get it passing again ! FECUNDITY !
==================================================== FAILURES =======================================================
_________________________________________ TestJinja.test_unexpected_variable __________________________________________
Traceback (most recent call last):
File "/my/working/dir/test_jinja_template_poc.py", line 150, in test_unexpected_variable
result = template.render(poc_context)
File "/usr/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
return original_render(self, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
return self.environment.handle_exception(exc_info, True)
File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
raise value.with_traceback(tb)
File "unexpected_variable.sls", line 6, in top-level template code
# {% if verion == 'nope' %}
jinja2.exceptions.UndefinedError: 'verion' is undefined
========================================= 1 failed, 5 passed in 0.89 seconds ==========================================
Running the tests
The tests are kicked off via 'pytest' like any other python project using pytest.
workstation:~/projects/test_project.git# source ./env/bin/activate
(env) workstation:~/projects/test_project.git# pytest
===================================================================== test session starts =====================================================================
platform linux -- Python 3.6.8, pytest-2.9.2, py-1.4.32, pluggy-0.3.1
rootdir: /vagrant, inifile:
plugins: catchlog-1.2.2
collected 5 items
latest/tests/test_jinja_template_poc.py .....
Credit
I based this work on some ideas from the blog post A method of unit testing Jinja2 templates by alexharv074.
