Conjtest user guide

Introduction

Conjtest is a command-line utility heavily inspired and partially based on conftest. It allows you to write policies against structured configuration data using a robust and practical language. You can, for example, write policies against your EDN files, Kubernetes configurations, Terraform code, or against other common configuration formats.

Who is this for?

This tool is suited for you if:

You want to validate your configuration files using a Policy-as-Code tool as part of a CI/CD pipeline.
Catch problems or security issues before they become incidents & enforce compliance.
You want maintain the policies using Clojure/Babashka.

Installation

Installer script

Download & run the installer script using bash.

bash < <(curl -s https://raw.githubusercontent.com/ilmoraunio/conjtest/master/install)

By default the script will install the binary to /usr/local/bin (you may need to use sudo).

You can install the binary to another location using --install-dir.

curl -sO https://raw.githubusercontent.com/ilmoraunio/conjtest/master/install
chmod u+x install
./install --install-dir .

To install a specific version, you may provide --version.

./install --version 0.3.0

The full list of versions can be found from here.

GitHub releases

Download the binaries for the MacOS (arm64) and Linux (amd64) platforms from the repository’s latest release and install the binary to usr/local/bin.

MacOS (arm64)

curl -sL https://github.com/ilmoraunio/conjtest/releases/download/v0.3.0/conjtest-0.3.0-macos-arm64.zip -o conjtest.zip
unzip conjtest.zip conjtest
sudo mv conjtest /usr/local/bin

Linux (amd64)

curl -sL https://github.com/ilmoraunio/conjtest/releases/download/v0.3.0/conjtest-0.3.0-linux-x86_64.tar.gz -o conjtest.tar.gz
tar -xvzf conjtest.zip conjtest
sudo mv conjtest /usr/local/bin

Linux (arm64)

curl -sL https://github.com/ilmoraunio/conjtest/releases/download/v0.3.0/conjtest-0.3.0-linux-arm64.tar.gz -o conjtest.tar.gz
tar -xvzf conjtest.tar.gz conjtest
sudo mv conjtest /usr/local/bin

Getting started

Basic example

Here’s how to get started with a simple deny policy.

First, let’s initialize conjtest configuration. This will create a default conjtest.edn file to your current directory.

conjtest init

After that, add an nginx-based Kubernetes ingress definition defined using YAML.

cat <<EOF > my-ingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/cors-allow-origin: '*'
  name: service-a
  namespace: foobar
EOF

Now add a policy called policy.clj containing a rule to fail if the field nginx.ingress.kubernetes.io/cors-allow-origin contains a "*".

cat <<EOF > policy.clj
(ns policy)

(defn deny-*-cors
  [input]
  (when (= "*" (get-in input
                       [:metadata
                        :annotations
                        :nginx.ingress.kubernetes.io/cors-allow-origin]))
    "CORS is too permissive"))
EOF

Putting these together, you should expect to see conjtest fail due to the forbidden asterisk "*".

$ conjtest test my-ingress.yaml -p policy.clj
FAIL - my-ingress.yaml - deny-*-cors - CORS is too permissive

1 tests, 0 passed, 0 warnings, 1 failures

$ echo $?
1

Once you’ve seen your policy fail, it’s time to make it pass.

cat <<EOF > my-ingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/cors-allow-origin: 'https://safe.site.com'
  name: service-a
  namespace: foobar
EOF

Your policy should now pass without failure.

$ conjtest test my-ingress.yaml -p policy.clj
1 tests, 1 passed, 0 warnings, 0 failures

$ echo $?
0

And that’s how you create a policy! Your next steps can be to attach the run commands as part of your CI pipeline or to your githooks and, of course, to create more policies.

Declarative policy example

You can also get started using declarative policies which are just malli schemas.

cat <<EOF > my-ingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/cors-allow-origin: '*'
  name: service-a
  namespace: foobar
EOF

cat <<EOF > policy.clj
(ns policy)

(def allow-non-*-cors
  [:map
   [:metadata
    [:map
     [:annotations
      [:map
       [:nginx.ingress.kubernetes.io/cors-allow-origin [:not= {:error/message "CORS is too permissive"} "*"]]]]]]])
EOF

conjtest init
conjtest test my-ingress.yaml -p policy.clj

Output:

$ conjtest test my-ingress.yaml -p policy.clj
FAIL - my-ingress.yaml - allow-non-*-cors - {:metadata {:annotations {:nginx.ingress.kubernetes.io/cors-allow-origin ["CORS is too permissive"]}}}

1 tests, 0 passed, 0 warnings, 1 failures

See also chapter on Declarative policies

Usage

Policies

There are two kinds of policies:

Function-based policies (referred to as policies)
Declarative policies (based on malli schemas)

Policies are single-arity functions which must return either of the following values:

true (or truthy)
false (or falsey)
string, eg. "error message"
a collection, eg. ["error message 1" "error message 2"]

Returning a string or a non-empty collection from the function will always result in a policy failure. For truthy/falsey values, the outcome depends on the policy’s type. An empty collection is regarded as a falsey value.

For deny policies, function must return false (or falsey) to succeed. For allow policies, function must return true (or truthy) to succeed. For warn policies, function must return false (or falsey) to succeed.

In conjtest, policy names (ie. defined using def or defn) must start with either deny-, allow-, or warn-.

When conjtest is provided a policy file, it will gather every public var in that file starting with deny-, allow-, or warn- and evaluate them. Functions using this naming conventions will be evaluated, functions not using this convention will be filtered out and not evaluated.

Policy type Policy name (begins with) Policy success

Policy type	Policy name (begins with)	Policy success
Deny	`deny-`	`false` or falsey
Allow	`allow-`	`true` or truthy
Warn	`warn-`	`false` or falsey

Deny

deny-

false or falsey

Allow

allow-

true or truthy

Warn

warn-

false or falsey

Deny policy example

(ns policy)

(defn- deployment?
  [input]
  (= "Deployment" (:kind input)))

(defn deny-should-not-run-as-root
  [input]
  (let [name (-> input :metadata :name)]
    (when (and (deployment? input)
               (not (true? (get-in input
                                   [:spec
                                    :template
                                    :spec
                                    :securityContext
                                    :runAsNonRoot]))))
      (format "Containers must not run as root in Deployment \"%s\"" name))))

Allow policy example

(ns policy)

(defn is-allowlisted?
  [allowlist x]
  (assert (coll? allowlist))
  (some? ((into #{} allowlist) x)))

(def ^:private allowlist ["hello-kubernetes"])

(defn allow-allowlisted-selector-only
  [input]
  (and (= "Service" (:kind input))
       (is-allowlisted? allowlist (get-in input [:spec :selector :app]))))

Warn policy example

(ns policy)

(defn warn-when-not-port-80
  [input]
  (and (= "v1" (:apiVersion input))
       (= "Service" (:kind input))
       (not= 80 (-> input :spec :ports first :port))))

Declarative policies

It’s possible to also write declarative policies using malli schemas.

Declarative policies work much the same way as function-based policies except that they are defined as malli schemas and have different rules for policy success. Malli schemas are evaluated using malli.core/validate after which they are processed for any errors using malli.core/explain and malli.error/humanize.

Policy type Policy name (begins with) Policy success

Policy type	Policy name (begins with)	Policy success
Deny	`deny-`	`malli.core/validate` returns `false`
Allow	`allow-`	`malli.core/validate` returns `true`
Warn	`warn-`	`malli.core/validate` returns `false`

Deny

deny-

malli.core/validate returns false

Allow

allow-

malli.core/validate returns true

Warn

warn-

malli.core/validate returns false

It’s recommended to write allow policies when using declarative policies. Deny and warn declarative policies are still supported. However, custom error messages are not supported with deny or warn policies.

Declarative policy YAML example

(ns policy)

(def allow-declarative-example
  [:map
   [:kind [:= "Deployment"]]
   [:metadata
    [:map
     [:name :string]]]
   [:spec
    [:map
     [:selector
      [:map
       [:matchLabels
        [:map
         [:app :string]
         [:release :string]]]]]
     [:template
      [:map
       [:spec
        [:map
         [:securityContext {:optional true}
          [:map
           [:runAsNonRoot [:not= {:error/message "Containers must not run as root"}
                           true]]]]]]]]]]])

Declarative policy JSON example using regular expressions

(ns policy)

(def allow-declarative-example
  [:map
   [:dependencies [:map-of :keyword [:re
                                     {:error/fn (fn [{:keys [value]} _]
                                                  (format "caret ranges not allowed, version found: %s" value))}
                                     "^[0-9~]|latest|beta|>="]]]])

Declarative policy EDN example using Fn schemas

(def allow-declarative-example
  [:and
   [:map
    [:db
     [:map
      [:user :string]
      [:pwd :string]
      [:host :string]
      [:db :string]
      [:port :int]]]
    [:myapp [:map
             [:port :int]
             [:features [:set
                         [:enum :admin-panel :keyboard-shortcuts]]]
             [:foo
              [:map
               [:hostname :string]
               [:api-keys [:vector :string]]
               [:recheck-frequency :string]]]
             [:forever-date inst?]
             [:process-pool :int]]]
    [:log :keyword]
    [:env [:enum :production]]]
   [:fn {:error/message "Applications in the production environment should have error only logging"}
    (fn [input]
      (and (= :production (:env input))
           (= :error (:log input))))]])

More declarative policy examples here.

Policy metadata

You can define policy name, top-level error message (if the rule otherwise fails), and the policy type using metadata.

Supported keys Supported values Superseded by

Supported keys	Supported values	Superseded by
`rule/type`	`:deny`, `:allow`, `:warn`	Policy name (begins with `deny-`, `allow-`, or `warn-`)
`rule/message`	string	Function returns an error message

rule/type

:deny, :allow, :warn

Policy name (begins with deny-, allow-, or warn-)

rule/message

string

Function returns an error message

Policy with custom name and error message

(defn ^{:rule/type :deny
        :rule/message "port should be 80"}
      differently-named-deny-rule
  [input]
  (and (= "v1" (:apiVersion input))
       (= "Service" (:kind input))
       (not= 80 (-> input :spec :ports first :port))))

Running policies

Once you have a configuration file and a policy, you can perform policy testing using the following command syntax:

conjtest test <configuration_file> [configuration_file [...]] [flags]

You can provide file(s) directly, use globs, or directories for both configuration files and policies.

Basic example

conjtest test infra/deployment.yaml --policy policies/policy.clj

Multiple files

conjtest test infra/deployment.yaml infra/my-other-deployment.yml --policy policies/policy.clj --policy other_policies/another-policy.clj

Glob support

conjtest test infra/**/*.{yaml,yml} --policy **/*.clj

Directories support

conjtest test infra/ --policy policies/

Exit codes

conjtest test command normally returns exit code 0 on success. Failing deny or allow policies result in exit code 1. Warn policies will result in exit code 0.

--fail-on-warn

If the option --fail-on-warn is provided to the conjtest test command, then failing warn policies result in error code 1. Failing deny policies will result in exit code 2.

Supported runtime libraries

Policies are run using Babashka/Clojure runtime. Conjtest runs policies inside a SCI sandbox which places limitations on code that can be run.

Currently, only the following namespaces (and their contents) can be required:

clojure.core
clojure.set
clojure.edn
clojure.repl
clojure.string
clojure.walk
clojure.template
Locally defined namespaces (see: Local file requires)

Keyworded keys

Since 0.2.0 this behavior is provided by default when running conjtest init.

Some configuration formats support keyworded keys by default.

CLI invocation example

$ conjtest parse deps.edn

{:deps
 {org.clojure/clojure #:mvn{:version "1.12.0"},
  ilmoraunio/conjtest-clj #:local{:root "../conjtest-clj"}}}

To return keyworded keys for all configuration formats, you can provide the keywordize? option via --config.

conjtest.edn

{:keywordize? true}

CLI invocation example

$ conjtest parse examples/hcl2/terraform.tf --config conjtest.edn
{:resource
 {:aws_alb_listener
  {:my-alb-listener [{:port "80", :protocol "HTTP"}]},
  :aws_db_security_group {:my-group [{}]},
  :aws_s3_bucket
  {:valid
   [{:acl "private",
     :bucket "validBucket",
     :tags {:environment "prod", :owner "devops"}}]},
  :aws_security_group_rule
  {:my-rule [{:cidr_blocks ["0.0.0.0/0"], :type "ingress"}]},
  :azurerm_managed_disk
  {:source [{:encryption_settings [{:enabled false}]}]}}}

$ conjtest test examples/hcl2/terraform.tf -p examples/hcl2/policy.clj --config conjtest.edn
FAIL - examples/hcl2/terraform.tf - deny-fully-open-ingress - ASG rule ':my-rule' defines a fully open ingress
FAIL - examples/hcl2/terraform.tf - deny-http - ALB listener ':my-alb-listener' is using HTTP rather than HTTPS
FAIL - examples/hcl2/terraform.tf - deny-missing-tags - AWS resource: :aws_alb_listener named ':my-alb-listener' is missing required tags: #{:environment :owner}
FAIL - examples/hcl2/terraform.tf - deny-missing-tags - AWS resource: :aws_db_security_group named ':my-group' is missing required tags: #{:environment :owner}
FAIL - examples/hcl2/terraform.tf - deny-missing-tags - AWS resource: :aws_security_group_rule named ':my-rule' is missing required tags: #{:environment :owner}
FAIL - examples/hcl2/terraform.tf - deny-unencrypted-azure-disk - Azure disk ':source' is not encrypted

4 tests, 0 passed, 0 warnings, 4 failures

Local file requires

It’s possible to require your locally defined namespaces by passing a config file via --config and :paths key.

CLI invocation example

conjtest test infra/deployment.yaml --policy policies/policy.clj --config conjtest.edn

conjtest.edn

{:paths ["policies/"]}

Conjtest will will recursively include namespaces of all .clj, .cljc, and .bb files.

Now you can require any local Clojure namespaces.

Policy example using local require

(ns conjtest.example-local-require
  (:require [conjtest.util.core :as util]))

(def ^:private allowlist ["hello-kubernetes"])

(defn allow-allowlisted-selector-only
  [input]
  (and (= "Service" (:kind input))
       (util/is-allowlisted? allowlist (get-in input [:spec :selector :app]))))

Parsing configuration files

Conjtest uses multiple parsers under-the-hood to provide wide support for different configuration file formats, using either Clojure-based parsers or Go-based (ie. Conftest-based) parsers.

Conjtest will try to parse using Clojure-based parsers if a suitable parser can be found, otherwise it will use a Go-based parser. If a suitable parser cannot be found, exit code 1 is returned.

Conjtest supports returning keys using keywords for Clojure-based parsers. For Go-based parsers, keys are returned as a string by default, but can be changed via config file (see: Keyworded keys).

If necessary, you can force Conjtest to use Go-based parsers via --go-parsers-only.

Supported configuration file formats

Debugging

Conjtest supports multiple ways to develop & debug policies.

Parsing files

To debug configuration file contents, you can use conjtest parse.

Example

$ conjtest parse examples/yaml/kubernetes/deployment.yaml
{:apiVersion "apps/v1",
 :kind "Deployment",
 :metadata
 {:name "hello-kubernetes",
  :labels
  #:app.kubernetes.io{:name "mysql",
                      :version "5.7.21",
                      :component "database",
                      :part-of "wordpress",
                      :managed-by "helm"}},
 :spec
 {:replicas 3,
  :selector {:matchLabels {:app "hello-kubernetes"}},
  :template
  {:metadata {:labels {:app "hello-kubernetes"}},
   :spec
   {:containers
    ({:name "hello-kubernetes",
      :image "paulbouwer/hello-kubernetes:1.5",
      :ports ({:containerPort 8080})})}}}}

You can copy-paste this output and pass it as an argument to your policy function.

--trace

If you need to understand policy behavior under-the-hood, you can use the --trace flag to debug the policy more closely.

$ conjtest test test-resources/invalid.yaml --policy test-resources/conjtest/example_deny_rules.clj --trace

Policy argument(s): test-resources/conjtest/example_deny_rules.clj
Filenames parsed: test-resources/invalid.yaml
Policies used: test-resources/conjtest/example_deny_rules.clj
TRACE:
---
Rule name: deny-my-absolute-bare-rule
Input file: test-resources/invalid.yaml
Parsed input: {:apiVersion "v1",
 :kind "Service",
 :metadata {:name "hello-kubernetes"},
 :spec
 {:type "LoadBalancer",
  :ports ({:port 81, :targetPort 8080}),
  :selector {:app "bad-hello-kubernetes"}}}
Result: true

...
...
...

FAIL - test-resources/invalid.yaml - deny-my-absolute-bare-rule - :conjtest/rule-validation-failed
FAIL - test-resources/invalid.yaml - deny-my-bare-rule - port should be 80
FAIL - test-resources/invalid.yaml - deny-my-rule - port should be 80
FAIL - test-resources/invalid.yaml - differently-named-deny-rule - port should be 80

4 tests, 0 passed, 0 warnings, 4 failures

REPL

It’s possible to connect to a network REPL using babashka.nrepl.server.

CLI example

$ conjtest repl
Started nREPL server at 0.0.0.0:1667

Executing this command will open up an nREPL connection to port 1667.

This allows you to develop your policy within the SCI sandbox.

REPL example

$ clj -Sdeps '{:deps {nrepl/nrepl {:mvn/version "0.5.3"}}}' -m nrepl.cmdline -c --host 127.0.0.1 --port 1667
...
user=> (require '[pod-ilmoraunio-conjtest.api :as api])
nil
user=> (defn deny-my-policy
         [input]
         (when ((into #{} (:paths input)) "evil-dir")
           "evil-dir found!"))
#'user/deny-my-policy
user=> (deny-my-policy (first (vals (api/parse "deps.edn"))))
nil
user=> (deny-my-policy (update (first (vals (api/parse "deps.edn")))
                               :paths
                               conj
                               "evil-dir"))
"evil-dir found!"
user=> (conjtest/test! [(update (first (vals (api/parse "deps.edn")))
                                :paths
                                conj
                                "evil-dir")] #'deny-my-policy)
{:summary {:total 1, :passed 0, :warnings 0, :failures 1}, :summary-report "1 tests, 0 passed, 0 warnings, 1 failures\n", :result ({:message "evil-dir found!", :name "deny-my-policy", :rule-type :deny, :failure? true})}

Use via Babashka

Conjtest supports usage via Babashka directly with conjtest-clj library and pod-ilmoraunio-conjtest.

This can be useful if you want to create your own scripting using Babashka directly.

Example

This is a simple example using babashka tasks.

Assume you have the following bb.edn:

{:paths ["."]
 :deps {org.conjtest/conjtest-clj {:mvn/version "0.4.0"}}
 :pods {ilmoraunio/conjtest {:version "0.1.1"}}
 :tasks
 {:requires ([main])
  test (apply main/test *command-line-args*)}}

In your main.clj you can define the test task with custom reporting behavior (I’ve provided a very barebones example here).

(ns main
  (:require [conjtest.core :as conjtest]
            [pod-ilmoraunio-conjtest.api :as parser]
            [policy]))

(defn test
  [& args]
  (let [inputs (apply parser/parse args)]
    (let [{:keys [summary-report failure-report]}
          (conjtest/test inputs 'policy)]
      (cond
        failure-report (do (println failure-report) (System/exit 1))
        summary-report (do (println summary-report) (System/exit 0))))))

Running this against an EDN configuration file, sample_config.edn, using the following policy gives you the output below.

$ bb test sample_config.edn
FAIL - sample_config.edn - deny-incorrect-log-level-production - Applications in the production environment should have error only logging

2 tests, 1 passed, 0 warnings, 1 failures

$ echo $?
1

Contributing

Contributions are welcome!

Join the #conjtest channel in the Clojurians Slack for discussion.

You can also make an issue or PR to the following project repositories:

Documentation

To contribute to this user guide, make an issue and/or PR to the repository.

License

Licensed under CC BY-SA 4.0: https://creativecommons.org/licenses/by-sa/4.0

With attributions to: