Is Python the Perfect JSON/YAML Templating Engine?
If you work within the infrastructure provisioning/DevOps space, you may be familiar with the use of YAML configuration files. It seems everything is moving to YAML and away from JSON in order to gain readability, the ability to use comments, and the ease to just start typing producing concise declarative configuration files easily. Or so we thought.
I first experienced YAML templating when using SaltStack and thought how much nicer and easier things felt compared to Chef and other configuration management tools which implemented their own DSLs. SaltStack and Ansible being Python based, naturally leaned into Jinja templating for YAML configuration files, and with the rise of Kubernetes, and more modern CI/CD tools such as Concourse, Azure Pipelines etc. the YAML templating space has exploded with various tooling all trying to deal with the reality that YAML configuration is rarely declarative at all. I even did a small, study comparing spruce,ytt and jinja2, found here, only to conclude that there really was no perfect templating engine and current solutions are only getting more and more convoluted. But then I tried just pure Python and was really surprised.
How it Starts
“We just need a little Interpolation…”
The gateway drug for YAML templating has always been variable interpolation. For example, you have a configuration and within it, it will have a few variables which need to be environment specific, so you start interpolating some values in YAML with things such as key: $(BACKEND_SERVICE) or key: ((grab $BACKEND_SERVICE )). You think “Ok, it’s still YAML and I can understand this.”
“But what if…”
But before long , you need something a bit more. You come across a scenario where a block of YAML should exist in one environment but not the others. Well we can just sprinkle in some if statements…
or…
Copy. Paste. Copy Paste. “A loop would be nice…”
Ah! here we go…
or…
The Seventh Level
Before you know it you have macros and inputs all over the place, you’re keeping track of indents and so on.
You are now so far down the dark path that you are probably using tools like ytt, jinja, spruce, or even kustomize or who knows what. But this has to be the way, the big dogs use this, everyone is doing it this way and then it hits you.
You have an epiphany
In your relentless search for the perfect templating engine you come across this article…
“If you’re starting to template yaml, ask yourself the question: why am I not *generating* json?”
… and you ask yourself, why the F$*k am I templating YAML?
Clicking into that article it eventually introduces you to jsonnet which looks interesting, it even has bindings for Python but wait a minute…
… NO. NO. NO! Yet another tool or language to learn. Passing a snippet into an interpreter within an interpreter… I’m already in Python so why not just try to generate some YAML or JSON with, I don’t know, Python?
Python, I Choose You!
Python has been around for a long, long, long time. It has a great community, it’s readable, it’s powerful and it can happily dump JSON/YAML all day long. It is a language and thus it has EVERYTHING you could ever want to generate some YAML/JSON. Read vars from env? check. Concat two strings, two lists, merge two maps? Check, Check, Check. Let’s try it out. Just as a note, this is going to use python 3.9 because it introduces the new | dictionary merge operator.
A Concourse Pipeline Experiment
For my full-time job, I work with a lot of very large, complex, YAML configurations for our concourse pipelines which deploy cloud foundry. These pipelines can easily get into the hundreds and thousands of lines of YAML. We use a combination of spruce and some bash to template or merge our configs. They aren’t the most approachable configurations and any updates requires a lot of find-and-replace type workflows.
A typical pipeline which we operate is built around provisioning orgs, spaces, and users into Cloud Foundry using this tool, cf-mgmt.
It produces a pipeline like this.
I cut out about a dozen jobs which are similar but not quite the same which this pipeline also includes. All in all, the complete YAML config is about 500 lines of code. Yes, you can use YAML’s built in tricks to reference anchors and point to them etc. But I found many people aren’t aware that YAML can do that, and getting into pointers, anchors, and references are kinda wacky to use and sometimes produce unexpected results. I omitted such features for this example.
Each one of these jobs have the following subtle differences, noted in side comments.
Infrastructure as Actual Code
The magic I envisioned which I implemented was if some-config.py is executed I expect a some-config.yml file to be spit out right next to it. BUT, you can just spit it out to std.out, it’s up to you, you have all of Python to make that choice. I chose a file for now as it was easier to develop this example.
I accomplished that by starting to write a generate(config) function to do this, and it uses a bit of Python magic to figure out the original calling python script and use that as the base for the yaml file name.
I stuck this within a module called configer so I could use this and other helper functions in this config and in the future.
My cloud foundry org-management-pipeline as code so far:
python org-management-pipeline.py and I now have an org-management-pipeline.yml which now contains the basic structure of a concourse-pipeline:
resources: []
jobs: []Dealing with Environment Variables
Next, let’s add a single resource.
Oh, I need to dip into the environment for some variables, no problem. os.getenv() to the rescue. And look at that, let’s just concat the return value to a string. Easy. But even os.getenv() can be a little simpler on the eyes, so let’s wrap it in a helper.
Sourcing Variables from External Files
When working with configurations which slightly change from environment to environment, or just reference some common set of data needed in many configs, it’s common to source in some sort of external inputs to pluck out these variables.
One thing I never liked about overlay and merge tools is that I can never reason where the data will come from or if any data will come at all. The merge tools pretty much smoosh it all together and you are wondering where certain values come from.
Looking to the Zen of Python…
Explicit is better than implicit.
So let’s explicitly declare all the source files our configuration will need.
What’s going on here? Well we’re in Python! A complete OOP language, so I simply created a small class which takes in all file paths you want to source from and maps them to a convenience label to use for referencing them.
Using an object makes sense here because if we reference multiple values from the same file, we only want to do the File IO once and just store the contents for future lookups.
The other issue to solve for is we don’t want to simply lookup values in a dictionary as you could run into all sorts of errors like KeyNotFound and IndexOutOfRang etc. and you want know what went wrong without a big stack trace, so let’s just write a grabmethod for the Sources class.
Using it within the config:
The org-management-pipeline.yml generated so far:
If we typo a lookup for grab we now get a nicer message.
Error- meta.docker_host does not exist in file: vars.ymlLets do something similar for referencing the contents of an entire file, like a cert CA.
and use it as such:
Looping
Our resources are looking good, now moving into jobs we saw from earlier that concourse pipelines can be pretty verbose and often repeat similar, yet subtlety different. Lets configure a single job
which produces:
The org-management pipeline essentially uses the same job 20 times, with the only difference being the CF_MGMT_COMMAND parameter passed into the task.
Instead of declaring 20 blocks of yaml we’re going to just see how we can use this one block as a base to meet all the nuances of the 20 jobs.
Let’s start with introducing a loop to give us 20 jobs, one for each command. We can easily do this using Python’s in-line list for loop pattern.
[ item + "-foo" for item in ["x", "y", "z"] will give us ["x-foo", "y-foo", "z-foo"]
outputs:
You can also do things like this to combine looping and non-looping blocks using list concatenation.
What if you need an if?
So far everything seen is pretty straight forward. We define some configs using dictionaries and lists and if we need to inject a value we call out to a function and use the return value as that value. The looping also plays well in-line to the configuration so far. But the trickiest bit is dealing with conditional elements or blocks within the configuration.
Templating engines usually handle this fairly well like such
But in Python we can’t really do something that easily. For example, this will absolutely not work.
However, we can use functions to facilitate similar behavior. Here I’ve made a class called Include with one static method called when(). This simply implements the if statement for us so we can use it within data structures. Classes with static methods are great ways to easily name-space methods and make things readable.
To render whether a key should exist at all within a dictionary we can use the new Python 3.9 syntax for merging two dictionaries. {dict_a} | {dict_b} and do something like this.
{**dict1, **dict2}This will give us the following because the Include.when() will return an empty dictionary unless otherwise specified.
foo: bar
biz: bazWe can also toggle values in-line within a dictionary if the key will always exist but it’s value is conditional.
Used in this way, the Include.when() takes a conditional, and the value to return if true and the value to return as the else.
lastly, the Include.when() method can be used in-line within a list like such.
The prune_empty() helper just removes all empty values from a list and is just a wrapper around filter().
The above, combined with the prune_empty() will give us the following:
foo: bar
biz: baz
stuff:
- a
- c
- dApplying Conditionals within the Pipeline Configuration
Now that we have a method to implement conditional blocks and values, we can account for all the differences between the 20 jobs for this pipeline. Here is what the entire job config will look like now.
Here is a snippet of the output. Notice the differences in passed and the use of the time-trigger.
Custom Functions and Classes
The above will fully generate this pipeline configuration, and it turns the 500–600 lines of YAML into about 155 lines!
Looking at it though, too much of a good thing can often be bad and the constant chaining of Include.when() can be a little hard to rationalize when chained so many times in a row. Another way of approaching this configuration is creating custom methods and classes which can help us organize similar types of configuration blocks.
Here is a simple class with a static method which organizes task blocks.
applied:
This will still need 20 Jobs, but a lot less lines. In addition we can also pass in parameters into functions which isn’t possible in YAML anchors and pointers.
We can make this even slimmer by defining a custom class to handle the get resource blocks required in each plan as those are repetitive as well.
all together:
This solution is a best of both worlds approach. It forgoes the looping so you can see each job as it would be declared within a normal config file, but the common blocks have been implemented using helper classes and methods to keep things as concise as possible. Overall, this solution would implement all 20 jobs in about 280 lines of code, including the class definitions.
Wrapping Up
This entire example implementation of generic-code (not including custom classes specific to concourse) is about ~100 lines of code once it’s cleaned up, polished and refined (it’s not right now). You can find this little configer.py module below along with the final configs and YAML, but first let’s conclude this article.
After trying this experiment out it dawned on me that we’re going down a really bad trend with our obsession to find the right templating and merging tools. They’re often sold as a declarative solution, but it’s far from that. At some point you have to ask yourself if it’s easier to define data structures within a programing language or a programing language within data structures?
I also want to touch on a few points:
- Why Python? I like python. I think it’s easy to reason about, and people have gotten over the indent aware nature these days because we live in YAML world now, anyway. I don’t see any other reason why you could not implement this within Ruby, or Javascript as well. In-fact JS may be really interesting to see due to it’s functional programing style.
- Why not Go? Yes all the cool things in infra-tooling are Go based. The issue with Go is that in cases like this, strongly typed languages may be a bit too ridged and verbose. Also compiling configs to generate them seems like it isn’t the right tool for the job. But if you like this idea of generating and like Go, jsonnet does have a Go implementation.
- Isn’t too much Power a bad thing? Having the entire Python library available to generate some JSON/YAML seems crazy, but as you can see, without getting too much in the weeds of OOP, lambdas, or more, it’s fairly easy to build a solution which works for you and your team.
- Build Upon What You know: We all become better developers and software engineers when we build upon the core tools we already know: Python, Ruby, JS etc. Constantly reaching for a new tool which promises to make a specific niche easier when we have perfectly good programing languages, with vibrant communities, and overflowing stacks of information available at our fingertips, is making less and less sense. Doing this little exercise I learned about merging dictionaries, got to dig deeper into that funky in-line for loop a bit more, and re-invented the if statement using
Include.when()which made me just a tad better of a Python developer at the end of the day.
Configer and Final Config
Configer “Templating Engine”
org-management.py
org-management.yml
