1 ################################################
2 An Exploration into Jinja2 Unit Tests Wth Pytest
3 ################################################
6 :tags: lessons,jinja2,pytest,salt
10 This post covers an approach I've used for add pytest-style unit tests to my Salt-related projects.
12 The content below may be specific to Salt, but the testing techniques should work on any project using Jinja2.
14 Project Directory Structure
15 ===========================
17 In my existing salt project repo, I added a tests subdirectory, a setup.cfg with the contents below, and added a test target to the project's Makefile.
19 I also installed the pytest and jinja2 eggs in a virtualenv in my working directory.
28 ├─────── some_template.cfg
33 ├─────── test_some_template.py
40 Here's a snippet of the Makefile for kicking off the tests:
42 Note that since the tests are python code, you should run them through whatever linters and style-checkers you're using on the rest of your python code.
48 pycoderstyle ./tests/*.py
51 And here's the setup.cfg:
53 Without the extra '--tb=native' argument, pytest would sometimes through an internal error when jinja ended up throwing exception, as we'll see later below.
58 python_files = test_*.py tests/__init__.py tests/*/__init__.py
59 #uncomment the line below for full unittest diffs
61 # Any --tb option except native (including no --tb option) throws an internal pytest exception
62 # jinja exceptions are thrown
64 # Uncomment the next line for verbose output
69 ignore=E121,E123,E126,E226,E24,E704,E221,E127,E128,W503,E731,E131,E402
72 Note there is a test_*.py file for each file that includes Jinja2 markup.
77 The conftest.py contains the common fixtures used by the tests. I've tried adding docstring
78 comments to explain how to use the fixtures, but also see the examples.
83 from unittest.mock import Mock
86 from jinja2 import Environment, FileSystemLoader, ChoiceLoader, DictLoader, StrictUndefined
89 class RaiseException(Exception):
90 """ Exception raised when using raise() in the mocked Jinja2 context"""
94 def mocked_templates(request):
95 """ A dict of template names to template content.
96 Use this to mock out Jinja 'import "template" as tmpl" lines.
99 return mocked_templates
102 @pytest.fixture(autouse=True)
103 def jinja_env(request, mocked_templates):
104 """ Provide a Jinja2 environment for loading templates.
105 The ChoiceLoader will first check when mocking any 'import' style templates,
106 Then the FileSystemLoader will check the local file system for templates.
108 The DictLoader is first so the Jinja won't end up using the FileSystemLoader for
109 templates pulled in with an import statement which doesn't include the 'with context'
112 Setting undefined to StrictUndefined throws exceptions when the templates use undefined variables.
115 test_loader=ChoiceLoader([
116 DictLoader(mocked_templates),
117 FileSystemLoader('.'),
120 env = Environment(loader=test_loader,
121 undefined=StrictUndefined,
122 extensions=['jinja2.ext.do', 'jinja2.ext.with_', 'jinja2.ext.loopcontrols'])
127 @pytest.fixture(scope='session', autouse=True)
129 """ Provide a set of common mocked keys.
130 Currently this is only the 'raise' key for mocking out the raise() calls in the templates,
131 and an empty 'salt' dict for adding salt-specific mocks.
134 def mocked_raise(err):
135 raise RaiseException(err)
138 'raise': mocked_raise,
146 For purposes of the sections below, here's what the init.sls looks like:
151 # {% set version = salt['pillar.get']('version', 'latest') %}
152 # version: {{ version }}
154 # {% if version == 'nope' %}
155 # {{ raise("OH NO YOU DIDN'T") }}
160 Mock out the Jinja Context
161 ==========================
163 Let's test out that rendering init.sls should return a version key with some value.
165 Being able to mock out the salt pillar.get() function was a big breakthrough with respect to being able to write any sort of unit tests for the Salt states.
170 def poc_context(self, salt_context):
171 """ Provide a proof-of-concept context for mocking out salt[function](args) calls """
172 poc_context = salt_context.copy()
174 def mocked_pillar_get(key,default):
175 """ Mocked salt['pillar.get'] function """
179 return pillar_data.get(key, default)
181 # This is the super sauce:
182 # We can mock out the ``salt['function'](args)`` calls in the salt states by
183 # defining a 'salt' dict in the context, who's keys are the functions, and the values of mocked functions
184 poc_context['salt']['pillar.get'] = mocked_pillar_get
189 def test_jinja_template_poc(self, jinja_env, poc_context):
190 """ Render a template and check it has the expected content """
192 # This assumes the tests are run from the root of the project.
193 # The conftest.py file is setting the jinja_env to look for files under the 'latest' directory
194 template = jinja_env.get_template('init.sls')
196 # Return a string of the rendered template.
197 result = template.render(poc_context)
199 # Now we can run assertions on the returned rendered template.
200 assert "version: 1234" in result
203 Mocking a raise() error
204 =======================
206 Now, let's see how we can test triggering the raise() error based on the pillar data:
211 def bad_context(self, salt_context):
212 """ Lets see what happens if the template triggers a raise() """
214 # The base salt_context from conftest.py includes a 'raise' entry that raises a RaiseException
215 bad_context = salt_context.copy()
216 bad_context['salt']['pillar.get'] = lambda k, d: 'nope'
219 def test_raise_poc(self, jinja_env, bad_context):
220 """ Try rendering a template that should fail with some raise() exception """
222 with pytest.raises(RaiseException) as exc_info:
223 template = jinja_env.get_template('init.sls')
224 result = template.render(bad_context)
226 raised_exception = exc_info.value
227 assert str(raised_exception) == "OH NO YOU DIDN'T"
230 Mocking imported templates
231 ==========================
233 Sometimes the Jinja templates may try import other templates which are either out of scope with respect to the current project, or the import doesn't include the 'with context' modifier, so the Jinja context isn't available when rendering the template.
235 In this case we can used with DictLoader portion of the jinja_env to mock out importing the template.
237 In this example, lets assume the following template file exists in the templates directory:
241 {%- import 'missing.tmpl' as missing -%}
242 Can we mock out missing/out of scope imports ?
244 Mocked: {{ missing.perhaps }}
245 Macro Call: {{ missing.lost('forever' }}
247 Now here is a test that can mock out the missing.tmpl contents, including the lost() macro call:
251 def test_missing_template(self, jinja_env, mocked_templates, salt_context):
253 In this example, templates/missing-import.tmpl tries to import a non-available 'missing.tmpl' template.
254 The ChoiceLoader checks DictLoader loader, which checks mocked_templates and finds a match
257 mocked_templates['missing.tmpl'] = """
258 {% set perhaps="YES" %}
259 {% macro lost(input) %}MOCKED_LOST{% endmacro %}
261 missing_template = jinja_env.get_template('templates/missing-import.tmpl')
262 missing_result = missing_template.render(salt_context)
263 assert "Mocked: YES" in missing_result
264 assert "Macro Call: MOCKED_LOST" in missing_result
270 Let's say I have a Jinja2 macro defined below:
276 # {% macro test_macro(input) %}
277 # {% if input == 'nope' %}
278 # {{ raise("UNACCEPTABLE") }}
280 # {% set version = salt['pillar.get']('version', 'latest') %}
281 "macro sez {{ input }}":
282 test.show_notification:
283 - text: "{{ input }}"
285 "version sez {{ version }}":
286 test.show_notifications:
287 - text: "{{ version }}"
291 Testing out this macro is a little more involved, since first we have to include calls to the macro after rendering the template. Note we're reusing the poc_context fixture defined earlier so the pillar.get() call is still mocked out to return 1234 for the version"
295 def test_get_pillar_from_macro(self, jinja_env, poc_context):
297 If we want to reference the mocked context in the macros, we need
298 to render the source + macro call within a context.
301 # The '[0]' is because get source returns a (source,filename,up-to-date) tuple.
302 template_source = jinja_env.loader.get_source(jinja_env, 'macro.sls')[0]
303 new_template = jinja_env.from_string(template_source + "{{ test_macro('hello') }}")
304 result = new_template.render(poc_context)
306 assert "macro sez hello" in result
307 assert "version sez 1234" in result
309 It's also possible to check that the macro raises an error based on the input:
313 def test_raise_from_macro(self, jinja_env, salt_context):
315 In this test, try forcing a raise() from within a macro
318 with pytest.raises(RaiseException) as exc_info:
319 template_source = jinja_env.loader.get_source(jinja_env, 'macro.sls')[0]
320 new_template = jinja_env.from_string(template_source + "{{ test_macro('nope') }}")
321 result = new_template.render(salt_context)
323 raised_exception = exc_info.value
324 assert str(raised_exception) == "UNACCEPTABLE"
326 FECUNDITY: Checking for undefined variables during template rendering
327 =====================================================================
328 Back in the day I learned that one of the virtues of a scientific theory was 'fecundity', or the ability for the theory to predict new behavior the original theory hadn't considered.
330 It looks like this may be called `fruitfulness`_ now, but still whenever I stumble across something like this, I shout out 'FECUNDITY' internally to myself. :shrug:
332 While I was working on this project, I noticed the jinja Environment constructor has an `undefined`_ argument that defaulted to Undefined. I also noticed `StrictUndefined`_ was another value that the undefined argument could use.
334 It would be useful if the tests could throw exceptions when they ran into undefined variables. This could happen from typos in the templates, or possibly not mocking out all the globals variables used in a template.
336 So I tried making an jinja Environment with undefined=StrictUndefined, and wrote a test with template with a typo in a variable name to see if the test would raise an exception, and it did !
338 This is much more useful than the default jinja behavior where Jinja would give the typo a value of None and would likely surface in the output as a empty string.
340 It's also more useful than setting undefined to `DebugUndefined`_, which sometimes raised an exception, but sometimes left the un-modified '{{ whatever }}' strings in the rendered templates. Bleh.
342 Here's the sample template I used, called unexpected_variable.sls. It's the same as the original init.sls, but with a 'verion' typo:
348 # {% set version = salt['pillar.get']('role:echo-polycom:version', 'latest') %}
349 # version: {{ version }}
351 # {% if verion == 'nope' %}
352 # {{ raise("OH NO YOU DIDN'T") }}
355 And let's try adding this test, which is the same as the earlier test_jinja_template_poc() test, but with the buggy template:
359 def test_unexpected_variable(self, jinja_env, poc_context):
360 """ Render a template and check it has the expected content """
362 # This assumes the tests are run from the root of the project.
363 # The conftest.py file is setting the jinja_env to look for files under the 'latest' directory
364 template = jinja_env.get_template('unexpected_variable.sls')
366 # Return a string of the rendered template.
367 result = template.render(poc_context)
369 # Now we can run assertions on the returned rendered template.
370 assert "version: 1234" in result
372 This test will fail with an undefined error exception below !
373 Cool. I can fix the typo, and rerun the test to get it passing again ! FECUNDITY !
377 ==================================================== FAILURES =======================================================
378 _________________________________________ TestJinja.test_unexpected_variable __________________________________________
379 Traceback (most recent call last):
380 File "/my/working/dir/test_jinja_template_poc.py", line 150, in test_unexpected_variable
381 result = template.render(poc_context)
382 File "/usr/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
383 return original_render(self, *args, **kwargs)
384 File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
385 return self.environment.handle_exception(exc_info, True)
386 File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
387 reraise(exc_type, exc_value, tb)
388 File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
389 raise value.with_traceback(tb)
390 File "unexpected_variable.sls", line 6, in top-level template code
391 # {% if verion == 'nope' %}
392 jinja2.exceptions.UndefinedError: 'verion' is undefined
393 ========================================= 1 failed, 5 passed in 0.89 seconds ==========================================
399 The tests are kicked off via 'pytest' like any other python project using pytest.
401 .. code-block:: shell-session
403 workstation:~/projects/test_project.git# source ./env/bin/activate
404 (env) workstation:~/projects/test_project.git# pytest
405 ===================================================================== test session starts =====================================================================
406 platform linux -- Python 3.6.8, pytest-2.9.2, py-1.4.32, pluggy-0.3.1
407 rootdir: /vagrant, inifile:
408 plugins: catchlog-1.2.2
411 latest/tests/test_jinja_template_poc.py .....
416 I based this work on some ideas from the blog post `A method of unit testing Jinja2 templates`_ by `alexharv074`_.
419 .. _`A method of unit testing Jinja2 templates` : https://alexharv074.github.io/2020/01/18/a-method-of-unit-testing-jinja2-templates.html
420 .. _`alexharv074` : https://alexharv074.github.io/
421 .. _`fruitfulness` : https://link.springer.com/article/10.1007/s11229-017-1355-6
422 .. _`undefined`: https://jinja.palletsprojects.com/en/3.1.x/api/#undefined-types
423 .. _`StrictUndefined`: https://jinja.palletsprojects.com/en/3.1.x/api/#jinja2.StrictUndefined
424 .. _`DebugUndefined`: https://jinja.palletsprojects.com/en/3.1.x/api/#jinja2.DebugUndefined