Kubernetes Custom Resources

Timoni allows defining Kubernetes Custom Resources (CRs) in modules and can ensure that these are validated against their Kubernetes Custom Resource Definitions (CRDs). For waiting on custom resources that don’t follow the kstatus conventions, see custom health checks. To enable validation for custom resources, you have to generate the CUE schemas from the Kubernetes CRDs OpenAPI validation spec with the timoni mod vendor crds command.

Example

To demonstrate this feature, we’ll use the Prometheus Operator CRDs, and we’ll add a ServiceMonitor custom resource to a Timoni module.

Vendor Prometheus Operator CRDs

From the root dir of your module, run the timoni mod vendor crds command, and pass the URL to the YAML file which contains the Prometheus Operator CRDs:

timoni mod vendor crds -f https://github.com/prometheus-operator/prometheus-operator/releases/latest/download/stripped-down-crds.yaml

The above command will generate the CUE schemas corresponding to the Kubernetes CRDs inside the cue.mod/gen directory:

cue.mod/gen/
└── monitoring.coreos.com
    ├── alertmanager
    ├── alertmanagerconfig
    ├── podmonitor
    ├── probe
    ├── prometheus
    ├── prometheusagent
    ├── prometheusrule
    ├── scrapeconfig
    ├── servicemonitor
    └── thanosruler

CEL validation rulesThe generated files embed the original CRD in a hidden _crd field. This lets timoni mod vet enforce the CEL rules set with x-kubernetes-validations, such as mutually exclusive fields or constraints across field values, applying the same strict validation as the Kubernetes API server.

Select kinds and versions

By default, all the CRDs and all their versions found in the YAML file are vendored. To vendor only the custom resources used by the module, select them with the --kind and --version flags:

timoni mod vendor crds -f stripped-down-crds.yaml --kind ServiceMonitor,PodMonitor --version v1

The --kind flag accepts the kind name or the fully qualified Kind.group name. The qualified form selects a single CRD when the same kind is defined in multiple API groups, e.g. --kind ProviderConfig.aws.upbound.io. The --version flag applies to all the selected kinds and matches the version names declared in the CRDs, e.g. --version v1,v1beta1. Both flags match case-insensitively. The command fails when a selected kind or version is not found in the YAML file, listing the kinds and versions available. To list the kinds and versions found in a YAML file without vendoring them, use the --list flag:

$ timoni mod vendor crds -f stripped-down-crds.yaml --list --kind ServiceMonitor,PodMonitor
KIND                                  VERSION
PodMonitor.monitoring.coreos.com      v1
ServiceMonitor.monitoring.coreos.com  v1

Prune vendored schemas

The command never removes the previously vendored schemas, so narrowing the selection or vendoring a new release of the CRDs leaves the stale definitions in cue.mod/gen. To remove them, use the --prune flag:

timoni mod vendor crds -f stripped-down-crds.yaml --kind ServiceMonitor --prune

Pruning is scoped to the API groups found in the YAML file: every vendored definition under those groups that was not generated by the current run is removed, while the definitions of other API groups, such as the ones vendored with timoni mod vendor k8s or from another CRD file, are left untouched. Only the files generated by Timoni are removed, hand-written CUE files placed under the same API group are kept.

Create the `ServiceMonitor` template

In the templates directory, create a servicemonitor.cue file with the following content:

package templates

import (
	promv1 "monitoring.coreos.com/servicemonitor/v1"
)

#ServiceMonitor: promv1.#ServiceMonitor & {
	#config:  #Config
	metadata: #config.metadata
	spec: {
		endpoints: [{
			// Change this to match the Service port where
			// your app exposes the /metrics endpoint
			port:     "http-metrics"
			path:     "/metrics"
			interval: "\(#config.monitoring.interval)s"
		}]
		namespaceSelector: matchNames: [#config.metadata.namespace]
		selector: matchLabels: #config.selector.labels
	}
}

Make sure to replace the port and path values with the ones used by your app. The port name must match one of the ports exposed in the Kubernetes Service template.

API Version and KindNote that for Kubernetes custom resources, you don’t need to specify the apiVersion and kind, these fields are set by Timoni in the generated schema.

Add the `monitoring` configuration

In the templates/config.cue file, add the monitoring configuration:

#Config: {

	// Promethues service monitor (optional)
	monitoring: {
		enabled:  *false | bool
		interval: *15 | int & >=5 & <=3600
	}

}

Add the `ServiceMonitor` to the instance

In the templates/config.cue file, add the ServiceMonitor resource to the instance objects:

#Instance: {
	config: #Config

	if config.monitoring.enabled {
		objects: servicemonitor: #ServiceMonitor & {#config: config}
	}

}

Document the `monitoring` configuration

Finally, document the monitoring configuration in the README.md file, so that users know how to enable monitoring if they have Prometheus Operator installed.

Custom resource validation

The timoni mod vet command validates the custom resources rendered by the module against their CRD schemas with the same checks as the Kubernetes API server:

OpenAPI schema validation
embedded resource metadata validation
uniqueness of x-kubernetes-list-type: map and set items
CEL validation rules set by x-kubernetes-validations

The CRD schemas are taken from:

the CUE packages imported by the module, generated with timoni mod vendor crds
the CRDs included in the module’s output

When both sources define the same kind and version, the CRD included in the module takes precedence. Custom resources that pass all checks are reported as valid custom resource. Any violation is reported with the field path and the message, and the command fails:

$ timoni mod vet --debug
vetting with debug values
HTTPRoute/default/app spec.endpoints: spec.rules[0].filters[0]: filter.requestRedirect must be specified for RequestRedirect filter.type
validation failed, 1 invalid custom resource(s)

The same checks run in timoni build, timoni bundle build and timoni bundle apply, where only the violations are reported and --validate=false disables the validation. In a bundle, the custom resources of an instance are also validated against the CRDs rendered by the other instances, with bundle apply by the instances applied before it, see bundle validation.

CUE Language

Module Development

Module Distribution

Kubernetes Custom Resources

Example

Vendor Prometheus Operator CRDs

Select kinds and versions

Prune vendored schemas

Create the `ServiceMonitor` template

Add the `monitoring` configuration

Add the `ServiceMonitor` to the instance

Document the `monitoring` configuration

Custom resource validation

​Example

​Vendor Prometheus Operator CRDs

​Select kinds and versions

​Prune vendored schemas

​Create the ServiceMonitor template

​Add the monitoring configuration

​Add the ServiceMonitor to the instance

​Document the monitoring configuration

​Custom resource validation

Example

Vendor Prometheus Operator CRDs

Select kinds and versions

Prune vendored schemas

Create the `ServiceMonitor` template

Add the `monitoring` configuration

Add the `ServiceMonitor` to the instance

Document the `monitoring` configuration

Custom resource validation