I’ve written before about having sentinel fmt in multiple spots.. Today I want to go more into depth about that topic – where and why we have sentinel fmt running throughout our SDLC.
Impact of not running sentinel fmt
Rather than treating this as a foregone conclusion, it’s important to understand why this step is needed. sentinel fmt does more for us than just canonically formatting your Sentinel policies for readability and standardization – it also serves as an early indicator of any egregious errors, almost a “linter lite” of sorts.
So what? Fun fact: if you have a policy that has an error which prevents it from executing properly (e.g.: unresolved open brackets), and if that policy is in hard-mandatory, CONGRATULATIONS!!! You’ve just shut down your org’s Terraform pipelines. So, yes, it’s important to get a quick and easy way to ensure that your code has had at least a cursory exam.
Real testing is better though, right? Yes. That’s the topic of a future blog post though ;)
Obvious place: In-Pipeline
It should come as no surprise that one of the first steps in our CI/CD pipelines for Sentinel policies includes a sentinel fmt stage. What might come as a surprise though is that my pipelines check mocks as well as the policies. This might seem a bit strange, but much in the same way that live testing is great, it’s really great to ensure that all unit tests are also free of defects. After all, a bad test gives a false sense of safety!
Something else peculiar: at the time of this writing, it is impermissible to let automation write back to source code management tooling. That is, I can let CI tooling run sentinel fmt, but it can’t commit that modified version back. Further, since regulated environments (this one included) require that all deployed artifacts are reproducible I’m left with 2 options:
- Package up the artifacts and upload to the company’s artifact repository
- Any changes/unformatted code fail the pipeline (uses
sentinel fmt -check=false$policyName)
Downsides of Option 1:
- Changes are untracked for people to notice and fix; the same fixes would be applied every pipeline run
- Harder to notice if there’s a problem with the code
- Troubleshooting would require downloading both copies and comparing… sounds awful
Downsides of Option 2:
- Can feel like the machine knows what to do, why do we fail, run
sentinel fmt, re-commit, etc.?
Solving Option 2’s Downsides:**
Git pre-commit hooks are pretty cool, and this seemed like a great use case for them. Further, our CI solution is pretty slow sometimes – especially when trying to format 50+ policies and hundreds of mock files with loads of data. Any shifting left and closer to the developer is welcome. SO, we put sentinel fmt into a pre-commit hook to get a bit faster feedback on our code.
Our local development environment is pretty fast, but still our pre-commit checks were continuing to take longer and longer over time. Knowing that everything would get fully checked in the pipeline later, we felt comfortable moving the pre-commit sentinel fmt checks to only the changed files in a given commit. This made our commit speed super fast again, and of course our team now runs far fewer executions on the CI system, so I’m sure that load reduction (while overall minimal) is a nice bonus.
Summary
We have reduced outages, increased confidence in our policy deployments, and gained some efficiencies along the way by running sentinel fmt as a pre-commit hook on modified files in the git commit process followed by running sentinel fmt -check=false in the CI pipeline. I highly recommend considering implementing something similar in your organization.
