Flux AIO Distribution
Flux All-In-One is a lightweight Flux CD distribution made with Timoni for running the GitOps Toolkit controllers as a single deployable unit.
Helm charts interoperability
Flux AIO can be used as a bridge between Timoni and Helm, enabling Timoni to orchestrate Helm chart deployments by leveraging Flux's declarative Helm APIs. For more information see the Helm interoperability guide.
Specifications
This distribution is optimized for running Flux on:
- Edge clusters with limited CPU and memory resources
- Bare clusters without a CNI plugin installed
- Clusters where plain HTTP communication is not allowed between pods
- Serverless clusters for cost optimisation (EKS Fargate, GKE Autopilot)
Timoni modules part of this distribution:
- flux-aio - A module for deploying Flux core components such as source-controller, helm-controller, kustomize-controller and notification-controller.
- flux-oci-sync - A module for configuring Flux to deploy workloads from OCI artifacts.
- flux-git-sync - A module for configuring Flux to deploy workloads from Git repositories.
- flux-helm-release - A module for configuring Flux to deploy Helm releases from charts hosted on Helm HTTP/S and OCI repositories.
- flux-tenant - A module for onboarding tenants with limited access on Flux-managed clusters.
Flux installation
To deploy Flux on Kubernetes clusters, you'll be using the flux-aio module and a Bundle file where you'll define the configuration of the Flux controllers and their settings.
Install Flux with default settings on a Kubernetes cluster:
bundle: {
apiVersion: "v1alpha1"
name: "flux-aio"
instances: {
"flux": {
module: url: "oci://ghcr.io/stefanprodan/modules/flux-aio"
namespace: "flux-system"
values: {
controllers: {
helm: enabled: true
kustomize: enabled: true
notification: enabled: true
}
hostNetwork: false
securityProfile: "privileged"
}
}
}
}
Apply the bundle with:
timoni bundle apply -f flux-aio.cue
You can fine tune the Flux installation using various options listed in the flux-aio module readme.
To install Flux on a bare metal cluster, you need to set hostNetwork: true in the
flux-aio module values.
For example, to deploy Flux and configure it to install the latest version of the Cilium CNI:
bundle: {
apiVersion: "v1alpha1"
name: "flux-aio"
instances: {
"flux": {
module: url: "oci://ghcr.io/stefanprodan/modules/flux-aio"
namespace: "flux-system"
values: {
hostNetwork: true
securityProfile: "privileged"
}
}
"cilium": {
module: url: "oci://ghcr.io/stefanprodan/modules/flux-helm-release"
namespace: "flux-system"
values: {
repository: url: "https://helm.cilium.io"
chart: {
name: "cilium"
version: "*"
}
helmValues: {
operator: replicas: 1
ipam: mode: "kubernetes"
}
sync: targetNamespace: "kube-system"
}
}
}
}
The Flux pod binds to the following ports on the host network:
9292notification-controller webhook receiver endpoint9690notification-controller events receiver endpoint9790source-controller storage endpoint9791-9799metrics, liveness and readiness endpoints
Note that on clusters without kube-proxy, you'll have to add the following env vars to values:
values: env: {
"KUBERNETES_SERVICE_HOST": "<host>"
"KUBERNETES_SERVICE_PORT": "<port>"
}
When installing Flux on a managed Kubernetes cluster, the host network can be disabled if the cloud vendor has already setup a CNI for you. You can also configure persistent storage for Flux artifacts cache to speed up the startup after a pod eviction.
To grant Flux access to cloud resources such as container registries (for pulling OCI artifacts)
or KMS (for secretes decryption), you can use Kubernetes Workload Identity to bind the flux
service account from the flux-system namespace to an IAM role.
For example, on an EKS cluster with IRSA enabled, grant Flux access to ECR by specified an AWS role ARN:
bundle: {
apiVersion: "v1alpha1"
name: "flux-aio"
instances: {
"flux": {
module: url: "oci://ghcr.io/stefanprodan/modules/flux-aio"
namespace: "flux-system"
values: {
hostNetwork: false
securityProfile: "privileged"
workload: {
identity: "arn:aws:iam::111122223333:role/my-role"
provider: "aws"
}
persistence: {
enabled: true
storageClass: "standard"
size: "8Gi"
}
}
}
}
}
For Azure Workload Identity, the type must be set to azure and the identity set to the Azure Client ID.
For Google Cloud, the type must be set to gcp and the identity set to the GCP Identity Name.
Flux upgrade
To upgrade Flux to the latest version, you can rerun the timoni bundle apply -f flux-aio.cue command,
and Timoni will check if a newer version of the flux-aio module is available and apply the changes.
To upgrade Flux to a specific version, you can specify the version in the module.version field:
module: {
url: "oci://ghcr.io/stefanprodan/modules/flux-aio"
version: "2.2.3-0"
}
Flux AIO versioning
The versioning of the AIO distribution follows semver with the following format:
<flux version>-<distribution release number>, e.g. 2.2.3-0.
To list all available versions of the flux-aio module, you can use the timoni mod ls command,
or you can check the flux-aio release page.
Flux OCI sync configuration
To configure Flux to deploy workloads from OCI artifacts hosted on container registries, you'll be using the flux-oci-sync module.
This module generates Flux OCIRepository and Kustomization objects and allows
the configuration of the OCI artifact URL, auth credentials, tag, interval, substitutions, health checks.
To configure Flux to sync with a public OCI repository, you have to specify the OCI address of the repository, the tag or semver range, and optionally you can enable waiting for the workloads to become ready.
For example, to sync the podinfo
Kustomize overlay to the default namespace:
bundle: {
apiVersion: "v1alpha1"
name: "podinfo"
instances: {
"podinfo": {
module: url: "oci://ghcr.io/stefanprodan/modules/flux-oci-sync"
namespace: "flux-system"
values: {
artifact: {
url: "oci://ghcr.io/stefanprodan/manifests/podinfo"
semver: ">=1.0.0"
}
sync: {
targetNamespace: "default"
wait: true
}
}
}
}
}
Apply the bundle with:
timoni bundle apply -f podinfo.cue
You can fine tune the sync using the options listed in the flux-oci-sync module readme.
To configure Flux to sync from a private container registry, you can specify the registry credentials
that will be persisted in the cluster as a Kubernetes Secret of type dockerconfigjson.
To avoid storing sensitive information in your bundle files, Timoni can read values from environment variable.
For example, to sync podinfo from your own private repo:
bundle: {
apiVersion: "v1alpha1"
name: "podinfo"
instances: {
"podinfo": {
module: url: "oci://ghcr.io/stefanprodan/modules/flux-oci-sync"
namespace: "flux-system"
values: {
artifact: {
auth: credentials: {
username: "flux"
password: string @timoni(runtime:string:GCHR_TOKEN)
}
url: "oci://ghcr.io/stefanprodan/manifests/podinfo"
tag: "latest"
}
sync: targetNamespace: "default"
}
}
}
}
Assuming the GCHR_TOKEN is set in your environment, apply the bundle
using the --runtime-from-env flag and Timoni will fill in the password value:
timoni bundle apply -f podinfo.cue --runtime-from-env
Flux Git sync configuration
To configure Flux to deploy workloads from a Git repository, you'll be using the flux-git-sync module.
This module generates Flux GitRepository and Kustomization objects and allows
the configuration of the Git HTTPS URL, auth token, branch, path, interval, health checks.
To configure Flux to sync with a public Git repository, you have to specify the Git HTTPS address of the repository, the Git ref (can be a branch or tag) and the path to the Kubernetes manifests or to the Kustomize overlay.
For example, to sync the podinfo
Kustomize overlay to the default namespace:
bundle: {
apiVersion: "v1alpha1"
name: "podinfo"
instances: {
"podinfo": {
module: url: "oci://ghcr.io/stefanprodan/modules/flux-git-sync"
namespace: "flux-system"
values: {
git: {
url: "https://github.com/stefanprodan/podinfo"
ref: "refs/heads/master"
path: "./kustomize"
}
sync: targetNamespace: "default"
}
}
}
}
Apply the bundle with:
timoni bundle apply -f podinfo.cue
You can fine tune the sync using the options listed in the flux-git-sync module readme.
To configure Flux to sync with a private Git repository, you can specify a Git token (GitHub PAT, GitLab deploy token, BitBucket token, etc).
To avoid storing sensitive information in your bundle files, Timoni can read values from environment variable.
For example, to sync podinfo from your own private repo:
bundle: {
apiVersion: "v1alpha1"
name: "podinfo"
instances: {
"podinfo": {
module: url: "oci://ghcr.io/stefanprodan/modules/flux-git-sync"
namespace: "flux-system"
values: {
git: {
token: string @timoni(runtime:string:GITHUB_TOKEN)
url: "https://github.com/my-org/my-private-repo"
ref: "refs/heads/master"
path: "./kustomize"
}
sync: targetNamespace: "default"
}
}
}
}
Assuming the GITHUB_TOKEN is set in your environment, apply the bundle
using the --runtime-from-env flag and Timoni will fill in the token value:
timoni bundle apply -f podinfo.cue --runtime-from-env
When using a self-hosted Git server such as GitLab, the self-signed TLS Certificate Authority must be specified for Flux to be able to sync the repository.
For example, to sync podinfo from a self-hosted private repo:
bundle: {
apiVersion: "v1alpha1"
name: "podinfo"
instances: {
"podinfo": {
module: url: "oci://ghcr.io/stefanprodan/modules/flux-git-sync"
namespace: "flux-system"
values: {
git: {
ca: string @timoni(runtime:string:GIT_CA)
token: string @timoni(runtime:string:GIT_TOKEN)
url: "https://gitlab.internal/my-group/my-repo"
ref: "refs/heads/master"
path: "./kustomize"
}
sync: targetNamespace: "default"
}
}
}
}
Export the GIT_TOKEN and GIT_CA env vars, then apply the bundle:
export GIT_CA=$(cat ca.crt)
export GIT_TOKEN="your-token"
timoni bundle apply -f podinfo.cue --runtime-from-env
Note that the CA certificate must be PEM-encoded.
Flux multi-tenancy configuration
To enable Flux multi-tenancy lockdown,
you can set securityProfile: "restricted" in the flux-aio values.
With the restricted profile, Flux Kustomizations and HelmReleases
can't create cluster-wide resources (CRDs, Namespaces, ClusterRoleBindings, etc)
unless they are deployed in the flux-system namespace.
The flux-system namespace, like kube-system, is reserved to cluster admins.
On-board tenants
To configure Flux to deploy workloads from a tenant repository,
you'll be using the flux-tenant and flux-git-sync Timoni modules.
The flux-tenant module generates the tenant's Kubernetes namespace
and RBAC (service account & role binding) that constrains Flux to be able
to deploy applications only in that namespace.
The flux-git-sync module configures Flux to reconcile the tenant's Kubernetes
resources from their Git repository while impersonating the restricted service account.
bundle: {
apiVersion: "v1alpha1"
name: "dev-team"
instances: {
"dev-team": {
module: url: "oci://ghcr.io/stefanprodan/modules/flux-tenant"
namespace: "dev-team-apps"
values: {
role: "namespace-admin"
resourceQuota: {
kustomizations: 100
helmreleases: 100
}
}
}
"dev-team-apps": {
module: url: "oci://ghcr.io/stefanprodan/modules/flux-git-sync"
namespace: "dev-team-apps"
values: {
git: {
token: string @timoni(runtime:string:DEVTEAM_TOKEN)
url: "https://github.com/org/dev-team-apps"
ref: "refs/heads/main"
path: "./deploy"
}
sync: targetNamespace: namespace
}
}
}
}
On-board the tenant with:
export DEVTEAM_TOKEN=<GH TOKEN>
timoni bundle apply -f dev-team.cue --runtime-from-env
Off-board tenants
Off-board the tenant and remove all their workloads with:
timoni bundle delete dev-team