Resume typos

[pwan.org.git] / content / lessons / JinjaUnitTests.rst

################################################
An Exploration into Jinja2 Unit Tests Wth Pytest
################################################

:date: 2023-11-20
:tags: lessons,jinja2,pytest,salt
:category: lessons
:author: Jude N

This post covers an approach I've used for add pytest-style unit tests to my Salt-related projects.

The content below may be specific to Salt, but the testing techniques should work on any project using Jinja2.

Project Directory Structure
===========================

In my existing salt project repo, I added a tests subdirectory, a setup.cfg with the contents below, and added a test target to the project's Makefile.

I also installed the pytest and jinja2 eggs in a virtualenv in my working directory.

::

   ├─ test_project repo
   ├─── .git
   ├─── init.sls
   ├─── map.jinja
   ├─── templates
   ├─────── some_template.cfg
   ├─── tests
   ├─────── conftest.py
   ├─────── test_init.py
   ├─────── test_map.py
   ├─────── test_some_template.py
   ├─── setup.cfg
   ├─── Makefile
   ├─── env
   ├─────── ... pytest
   ├─────── ... jinja2

Here's a snippet of the Makefile for kicking off the tests:

Note that since the tests are python code, you should run them through whatever linters and style-checkers you're using on the rest of your python code.

::

  test:
      py.test-3
      pycoderstyle ./tests/*.py


And here's the setup.cfg:

Without the extra '--tb=native' argument, pytest would sometimes through an internal error when jinja ended up throwing exception, as we'll see later below.

::

  [tool:pytest]
  python_files = test_*.py tests/__init__.py tests/*/__init__.py
  #uncomment the line below for full unittest diffs
  addopts =
          # Any --tb option except native (including no --tb option) throws an internal pytest exception
          # jinja exceptions are thrown
          --tb=native
          # Uncomment the next line for verbose output
          # -vv
  
  [pycodestyle]
  max-line-length=999
  ignore=E121,E123,E126,E226,E24,E704,E221,E127,E128,W503,E731,E131,E402


Note there is a test_*.py file for each file that includes Jinja2 markup.

tests/conftest.py
=================

The conftest.py contains the common fixtures used by the tests.  I've tried adding docstring
comments to explain how to use the fixtures, but also see the examples.

::

  import pytest
  from unittest.mock import Mock

  import jinja2
  from jinja2 import Environment, FileSystemLoader, ChoiceLoader, DictLoader, StrictUndefined
  
  
  class RaiseException(Exception):
      """ Exception raised when using raise() in the mocked Jinja2 context"""
      pass
  
  @pytest.fixture()
  def mocked_templates(request):
      """ A dict of template names to template content.
          Use this to mock out Jinja 'import "template" as tmpl" lines.
      """
      mocked_templates = {}
      return mocked_templates
  
  
  @pytest.fixture(autouse=True)
  def jinja_env(request, mocked_templates):
      """  Provide a Jinja2 environment for loading templates.
           The ChoiceLoader will first check when mocking any 'import' style templates,
           Then the FileSystemLoader will check the local file system for templates.
  
           The DictLoader is first so the Jinja won't end up using the FileSystemLoader for
           templates pulled in with an import statement which doesn't include the 'with context'
           modifier.

           Setting undefined to StrictUndefined throws exceptions when the templates use undefined variables.
      """
  
      test_loader=ChoiceLoader([
         DictLoader(mocked_templates),
         FileSystemLoader('.'),
      ])
  
      env = Environment(loader=test_loader,
                        undefined=StrictUndefined,
                        extensions=['jinja2.ext.do', 'jinja2.ext.with_', 'jinja2.ext.loopcontrols'])
  
      return env
  
  
  @pytest.fixture(scope='session', autouse=True)
  def salt_context():
      """ Provide a set of common mocked keys.
          Currently this is only the 'raise' key for mocking out the raise() calls in the templates,
          and an empty 'salt' dict for adding salt-specific mocks.
      """
  
      def mocked_raise(err):
          raise RaiseException(err)
  
      context = {
          'raise': mocked_raise,
          'salt': {}
      }
  
      return context

init.sls
========
For purposes of the sections below, here's what the init.sls looks like:

::

  #!jinja|yaml
  # {% set version = salt['pillar.get']('version', 'latest') %}
  # version: {{ version }}
  
  # {% if version == 'nope' %}
  #     {{ raise("OH NO YOU DIDN'T") }}
  # {% endif %}



Mock out the Jinja Context
==========================

Let's test out that rendering init.sls should return a version key with some value.

Being able to mock out the salt pillar.get() function was a big breakthrough with respect to being able to write any sort of unit tests for the Salt states.

::

      @pytest.fixture
      def poc_context(self, salt_context):
          """ Provide a proof-of-concept context for mocking out salt[function](args) calls """
          poc_context = salt_context.copy()
  
          def mocked_pillar_get(key,default):
              """ Mocked salt['pillar.get'] function """
              pillar_data = {
                  'version' : '1234'
              }
              return pillar_data.get(key, default)
  
          # This is the super sauce:
          # We can mock out the ``salt['function'](args)`` calls in the salt states by
          # defining a 'salt' dict in the context, who's keys are the functions, and the values of mocked functions
          poc_context['salt']['pillar.get'] = mocked_pillar_get
  
          return poc_context
  
  
    def test_jinja_template_poc(self, jinja_env, poc_context):
        """ Render a template and check it has the expected content """

        # This assumes the tests are run from the root of the project.
        # The conftest.py file is setting the jinja_env to look for files under the 'latest' directory
        template = jinja_env.get_template('init.sls')
  
        # Return a string of the rendered template.
        result = template.render(poc_context)
  
        # Now we can run assertions on the returned rendered template.
        assert "version: 1234" in result


Mocking a raise() error
=======================

Now, let's see how we can test triggering the raise() error based on the pillar data:

::

    @pytest.fixture
    def bad_context(self, salt_context):
        """ Lets see what happens if the template triggers a raise() """
  
        # The base salt_context from conftest.py includes a 'raise' entry that raises a RaiseException
        bad_context = salt_context.copy()
        bad_context['salt']['pillar.get'] = lambda k, d: 'nope'
        return bad_context
  
    def test_raise_poc(self, jinja_env, bad_context):
        """ Try rendering a template that should fail with some raise() exception """
  
        with pytest.raises(RaiseException) as exc_info:
            template = jinja_env.get_template('init.sls')
            result = template.render(bad_context)
  
        raised_exception = exc_info.value
        assert str(raised_exception) == "OH NO YOU DIDN'T"


Mocking imported templates
==========================

Sometimes the Jinja templates may try import other templates which are either out of scope with respect to the current project, or the import doesn't include the 'with context' modifier, so the Jinja context isn't available when rendering the template.

In this case we can used with DictLoader portion of the jinja_env to mock out importing the template.

In this example, lets assume the following template file exists in the templates directory:

::

  {%- import 'missing.tmpl' as missing -%}
  Can we mock out missing/out of scope imports ?
  
  Mocked: {{ missing.perhaps }}
  Macro Call: {{ missing.lost('forever' }}

Now here is a test that can mock out the missing.tmpl contents, including the lost() macro call:

::

    def test_missing_template(self, jinja_env, mocked_templates, salt_context):
        """
        In this example, templates/missing-import.tmpl tries to import a non-available 'missing.tmpl' template.
        The ChoiceLoader checks DictLoader loader, which checks mocked_templates and finds a match
        """
  
        mocked_templates['missing.tmpl'] = """
           {% set perhaps="YES" %}
           {% macro lost(input) %}MOCKED_LOST{% endmacro %}
        """
        missing_template = jinja_env.get_template('templates/missing-import.tmpl')
        missing_result = missing_template.render(salt_context)
        assert "Mocked: YES" in missing_result
        assert "Macro Call: MOCKED_LOST" in missing_result


Mocking a macro call
====================

Let's say I have a Jinja2 macro defined below:

::

  #!jinja|yaml

  # {% macro test_macro(input) %}
  #   {% if input == 'nope' %}
  #     {{ raise("UNACCEPTABLE") }}
  #   {% endif %}
  #   {% set version =  salt['pillar.get']('version', 'latest') %}
  "macro sez {{ input }}":
    test.show_notification:
    - text: "{{ input }}"
  
  "version sez {{ version }}":
    test.show_notifications:
    - text: "{{ version }}"
  
  # {% endmacro %}

Testing out this macro is a little more involved, since first we have to include calls to the macro after rendering the template.  Note we're reusing the poc_context fixture defined earlier so the pillar.get() call is still mocked out to return 1234 for the version"

::

    def test_get_pillar_from_macro(self, jinja_env, poc_context):
        """
        If we want to reference the mocked context in the macros, we need
        to render the source + macro call within a context.
        """
  
        # The '[0]' is because get source returns a (source,filename,up-to-date) tuple.
        template_source = jinja_env.loader.get_source(jinja_env, 'macro.sls')[0]
        new_template = jinja_env.from_string(template_source + "{{ test_macro('hello') }}")
        result = new_template.render(poc_context)
  
        assert "macro sez hello" in result
        assert "version sez 1234" in result

It's also possible to check that the macro raises an error based on the input:

::

    def test_raise_from_macro(self, jinja_env, salt_context):
        """
        In this test, try forcing a raise() from within a macro
        """
  
        with pytest.raises(RaiseException) as exc_info:
             template_source = jinja_env.loader.get_source(jinja_env, 'macro.sls')[0]
             new_template = jinja_env.from_string(template_source + "{{ test_macro('nope') }}")
             result = new_template.render(salt_context)
  
        raised_exception = exc_info.value
        assert str(raised_exception) == "UNACCEPTABLE"

FECUNDITY: Checking for undefined variables during template rendering
=====================================================================
Back in the day I learned that one of the virtues of a scientific theory was 'fecundity', or the ability for the theory to predict new behavior the original theory hadn't considered.

It looks like this may be called `fruitfulness`_ now, but still whenever I stumble across something like this, I shout out 'FECUNDITY' internally to myself. :shrug:

While I was working on this project, I noticed the jinja Environment constructor has an `undefined`_ argument that defaulted to Undefined.  I also noticed `StrictUndefined`_ was another value that the undefined argument could use.

It would be useful if the tests could throw exceptions when they ran into undefined variables.  This could happen from typos in the templates, or possibly not mocking out all the globals variables used in a template.

So I tried making an jinja Environment with undefined=StrictUndefined, and wrote a test with template with a typo in a variable name to see if the test would raise an exception, and it did !

This is much more useful than the default jinja behavior where Jinja would give the typo a value of None and would likely surface in the output as a empty string.

It's also more useful than setting undefined to  `DebugUndefined`_, which sometimes raised an exception, but sometimes left the un-modified '{{ whatever }}' strings in the rendered templates.  Bleh.

Here's the sample template I used, called unexpected_variable.sls.  It's the same as the original init.sls, but with a 'verion' typo:

::

   #!jinja|yaml
   
   # {% set version = salt['pillar.get']('role:echo-polycom:version', 'latest') %}
   # version: {{ version }}
   
   # {% if verion == 'nope' %}
   #     {{ raise("OH NO YOU DIDN'T") }}
   # {% endif %}

And let's try adding this test, which is the same as the earlier test_jinja_template_poc() test, but with the buggy template:

::

    def test_unexpected_variable(self, jinja_env, poc_context):
        """ Render a template and check it has the expected content """
  
        # This assumes the tests are run from the root of the project.
        # The conftest.py file is setting the jinja_env to look for files under the 'latest' directory
        template = jinja_env.get_template('unexpected_variable.sls')
  
        # Return a string of the rendered template.
        result = template.render(poc_context)
  
        # Now we can run assertions on the returned rendered template.
        assert "version: 1234" in result

This test will fail with an undefined error exception below !
Cool. I can fix the typo, and rerun the test to get it passing again ! FECUNDITY !

::

   ==================================================== FAILURES =======================================================
   _________________________________________ TestJinja.test_unexpected_variable __________________________________________
   Traceback (most recent call last):
     File "/my/working/dir/test_jinja_template_poc.py", line 150, in test_unexpected_variable
       result = template.render(poc_context)
     File "/usr/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
        return original_render(self, *args, **kwargs)
     File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
        return self.environment.handle_exception(exc_info, True)
     File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
        reraise(exc_type, exc_value, tb)
      File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
        raise value.with_traceback(tb)
      File "unexpected_variable.sls", line 6, in top-level template code
        # {% if verion == 'nope' %}
    jinja2.exceptions.UndefinedError: 'verion' is undefined
   ========================================= 1 failed, 5 passed in 0.89 seconds ==========================================


Running the tests
=================

The tests are kicked off via 'pytest' like any other python project using pytest.

.. code-block:: shell-session

  workstation:~/projects/test_project.git# source ./env/bin/activate
  (env) workstation:~/projects/test_project.git# pytest
  ===================================================================== test session starts =====================================================================
  platform linux -- Python 3.6.8, pytest-2.9.2, py-1.4.32, pluggy-0.3.1
  rootdir: /vagrant, inifile:
  plugins: catchlog-1.2.2
  collected 5 items

  latest/tests/test_jinja_template_poc.py .....

Credit
======

I based this work on some ideas from the blog post `A method of unit testing Jinja2 templates`_ by `alexharv074`_.


.. _`A method of unit testing Jinja2 templates` : https://alexharv074.github.io/2020/01/18/a-method-of-unit-testing-jinja2-templates.html
.. _`alexharv074` : https://alexharv074.github.io/
.. _`fruitfulness` :  https://link.springer.com/article/10.1007/s11229-017-1355-6
.. _`undefined`: https://jinja.palletsprojects.com/en/3.1.x/api/#undefined-types
.. _`StrictUndefined`: https://jinja.palletsprojects.com/en/3.1.x/api/#jinja2.StrictUndefined
.. _`DebugUndefined`: https://jinja.palletsprojects.com/en/3.1.x/api/#jinja2.DebugUndefined