# ResourceSet CRD
**ResourceSet** is a declarative API for generating a group of Kubernetes objects
based on a matrix of input values and a set of templated resources.
## Example
The following example shows a ResourceSet that generates an application instance consisting of a
Flux HelmRelease and OCIRepository for each tenant with a specific version and replica count.
```yaml
apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSet
metadata:
name: podinfo
namespace: default
annotations:
fluxcd.controlplane.io/reconcile: "enabled"
fluxcd.controlplane.io/reconcileEvery: "30m"
fluxcd.controlplane.io/reconcileTimeout: "5m"
spec:
commonMetadata:
labels:
app.kubernetes.io/name: podinfo
inputs:
- tenant: "team1"
app:
version: "6.7.x"
replicas: 2
- tenant: "team2"
app:
version: "6.6.x"
replicas: 3
resources:
- apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
name: podinfo-<< inputs.tenant >>
namespace: default
spec:
interval: 10m
url: oci://ghcr.io/stefanprodan/charts/podinfo
ref:
semver: << inputs.app.version | quote >>
- apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: podinfo-<< inputs.tenant >>
namespace: default
spec:
interval: 1h
releaseName: podinfo-<< inputs.tenant >>
chartRef:
kind: OCIRepository
name: podinfo-<< inputs.tenant >>
values:
replicaCount: << inputs.app.replicas | int >>
```
You can run this example by saving the manifest into `podinfo.yaml`.
**1.** Apply the ResourceSet on the cluster:
```shell
kubectl apply -f podinfo.yaml
```
**2.** Wait for the ResourceSet to reconcile the generated resources:
```shell
kubectl wait resourceset/podinfo --for=condition=ready --timeout=5m
```
**3.** Run `kubectl get resourceset` to see the status of the resource:
```console
$ kubectl get resourceset
NAME AGE READY STATUS
podinfo 59s True Reconciliation finished in 52s
```
**4.** Run `kubectl describe resourceset` to see the reconciliation status conditions and events:
```console
$ kubectl describe resourceset podinfo
Status:
Conditions:
Last Transition Time: 2024-09-24T09:58:53Z
Message: Reconciliation finished in 52s
Observed Generation: 1
Reason: ReconciliationSucceeded
Status: True
Type: Ready
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ApplySucceeded 72s flux-operator HelmRelease/default/podinfo-team1 created
HelmRelease/default/podinfo-team2 created
OCIRepository/default/podinfo-team1 created
OCIRepository/default/podinfo-team2 created
Normal ReconciliationSucceeded 72s flux-operator Reconciliation finished in 52s
```
**5.** Run `kubectl events` to see the events generated by the flux-operator:
```shell
kubectl events --for resourceset/podinfo
```
**6.** Run `kubectl delete` to remove the ResourceSet and its generated resources:
```shell
kubectl delete resourceset podinfo
```
## Writing a ResourceSet spec
As with all other Kubernetes config, a ResourceSet needs `apiVersion`,
`kind`, `metadata.name` and `metadata.namespace` fields.
The name of a ResourceSet object must be a valid DNS subdomain name.
A ResourceSet also needs a `.spec` section.
### Inputs configuration
The `.spec.inputs` field is optional and specifies a list of input values
to be used in the resources templates.
An input value is a key-value pair of strings and structs, where the key is the input name
which can be referenced in the resource templates using the `<< inputs.name >>` syntax.
Example of static inputs:
```yaml
spec:
inputs:
- tenant: team1
role: restricted
- tenant: team2
role: privileged
```
The `.spec.inputsFrom` field is optional and specifies a list of
[ResourceSetInputProvider](/docs/crd/resourcesetinputprovider/)
references to objects that provide input values to the ResourceSet.
It has the following subfields (exactly one of them must be set):
- `apiVersion`: The API version of the referenced object. Must be
`fluxcd.controlplane.io/v1`. Optional.
- `kind`: The kind of the referenced object. Must be
`ResourceSetInputProvider`. Optional.
- `.name`: The name of a `ResourceSetInputProvider` object in the same namespace
as the `ResourceSet`. Optional.
- `.selector`: A label selector to select multiple `ResourceSetInputProvider`
objects in the same namespace as the `ResourceSet`. Optional.
Example of inputs generated from GitHub Pull Requests:
```yaml
spec:
inputsFrom:
- apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSetInputProvider
name: podinfo-pull-requests
```
Example of inputs generated from multiple `ResourceSetInputProvider` objects via
Label Selectors:
```yaml
spec:
inputsFrom:
- apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSetInputProvider
selector:
matchLabels:
app: podinfo
matchExpressions:
- key: environment
operator: In
values:
- dev
- staging
```
At runtime, the operator will fetch the input values every time the `ResourceSetInputProvider`
reconciler detects a change in the upstream source.
When both `.spec.inputs` and `.spec.inputsFrom` are set, the resulting inputs are the
flattened concatenation of the `.spec.inputs` input sets with the input sets from each
selected `ResourceSetInputProvider` object.
#### Input strategy
By default, the resulting inputs are the flattened concatenation of the `.spec.inputs` input sets
with the input sets from each selected `ResourceSetInputProvider` object. The `.spec.inputStrategy`
field can be set to change this default behavior.
The `.spec.inputStrategy` field has the following subfields:
- `.name`: The name of the input strategy. If `.spec.inputStrategy` is not set, the behavior
matches setting `.spec.inputStrategy.name` to `Flatten`. Supported values are `Flatten` and
`Permute`.
- `.includeEmptyProviders`: Only applies when `.name` is `Permute`. When set to `true`, if any
input provider exports zero inputs the resulting permutation set is empty (mathematically
correct Cartesian product behavior). When `false` or unset (default), providers with zero
inputs are silently skipped and the remaining providers still permute among themselves.
When `.spec.inputStrategy.name` is set to `Permute`, the resulting inputs are the Cartesian product
of the input sets from each selected `ResourceSetInputProvider` object, and from the `ResourceSet`
object itself if `.spec.inputs` is set. Therefore, the total amount of permutations, i.e. input sets,
is the product of the number of input sets from each source, i.e. `A_1 x A_2 x ... x A_n`, where `A_i`
is the number of input sets from the `i`-th source.
**Note**: The combinatorial explosion of the `Permute` strategy can lead to a very large
number of permutations and can impact the performance of the operator. It is recommended
to use this strategy together with `ResourceSetInputProvider` filters to export only a
single input set per `ResourceSetInputProvider` object. If the number of permutations
exceeds `10000`, the operator will fail the reconciliation of the `ResourceSet` with
a `Stalled` condition.
When merging the input sets from each object, the `Permute` strategy places each input
set under a key that is the normalized name of the object providing the input set. The
normalization process is as follows:
1. Uppercase letters are converted to lowercase.
2. Spaces and punctuation (including `-`) are converted to underscores (`_`).
3. All characters not in the set [a-z0-9_] are removed.
4. The remaining string is split by underscores and the resulting non-empty words are rejoined with underscores.
The name normalization applied by the `Permute` strategy ensures that the resulting key is
compatible with the templating engine used in the resources templates.
Example of the `Permute` strategy with two `ResourceSetInputProvider` objects,
one selecting a single Git tag and the other selecting a single OCI tag:
```yaml
apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSet
metadata:
name: my-rset
namespace: default
spec:
inputStrategy:
name: Permute
inputs:
- id: id1
someField: foo
- id: id2
someField: bar
inputsFrom:
- kind: ResourceSetInputProvider
name: git-tag
- kind: ResourceSetInputProvider
name: oci-tag
resources:
- apiVersion: v1
kind: ConfigMap
metadata:
name: my-cm-<< inputs.id >>
namespace: default
data:
rsetID: << inputs.my_rset.id | quote >>
rsipGitID: << inputs.git_tag.id | quote >>
rsipOCIID: << inputs.oci_tag.id | quote >>
someField: << inputs.my_rset.someField | quote >>
sha: << inputs.git_tag.sha | quote >>
digest: << inputs.oci_tag.digest | quote >>
```
In the example above, the resulting input sets are similar to the following:
```yaml
- id: "768965678"
my_rset:
id: id1
someField: foo
git_tag:
id: "8765674567"
sha: bf5d6e01cf802734853f6f3417b237e3ad0ba35d
oci_tag:
id: "9876543210"
digest: sha256:d4ec9861522d4961b2acac5a070ef4f92d732480dff2062c2f3a1dcf9a5d1e91
- id: "234567654"
my_rset:
id: id2
someField: bar
git_tag:
id: "8765674567"
sha: bf5d6e01cf802734853f6f3417b237e3ad0ba35d
oci_tag:
id: "9876543210"
digest: sha256:d4ec9861522d4961b2acac5a070ef4f92d732480dff2062c2f3a1dcf9a5d1e91
```
This will generate two `ConfigMap` resources, one whose name will be `my-cm-768965678`,
and the other `my-cm-234567654`, each containing the rendered `.data` according to the
input set.
### Resources configuration
The `.spec.resources` field is optional and specifies the list of Kubernetes resource
to be generated and reconciled on the cluster.
The resources can be templated using the `<< inputs.name >>` syntax. The templating engine
is based on Go text template. The `<< >>` delimiters are used instead of `{{ }}` to avoid
conflicts with Helm templating and allow ResourceSets to be included in Helm charts.
Example of templated resources:
```yaml
spec:
inputs:
- tenant: team1
role: admin
- tenant: team2
role: cluster-admin
resources:
- apiVersion: v1
kind: Namespace
metadata:
name: << inputs.tenant >>
- apiVersion: v1
kind: ServiceAccount
metadata:
name: flux
namespace: << inputs.tenant >>
- apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: flux
namespace: << inputs.tenant >>
subjects:
- kind: ServiceAccount
name: flux
namespace: << inputs.tenant >>
roleRef:
kind: ClusterRole
name: << inputs.role >>
apiGroup: rbac.authorization.k8s.io
```
The above example will generate a `Namespace`, `ServiceAccount` and `RoleBinding` for each tenant
with the specified role.
#### Templating functions
The templating engine supports slim-sprig functions.
It is recommended to use the `quote` function when templating strings to avoid issues with
special characters e.g. `<< inputs.version | quote >>`.
When templating integers, use the `int` function to convert the string to an integer
e.g. `<< inputs.replicas | int >>`.
When using integer or boolean inputs as metadata label values, use the `quote` function to convert
the value to a string e.g. `<< inputs.enabled | quote >>`.
When templating nested fields, use the `toYaml` and `nindent` functions
to properly format the string e.g.:
```yaml
spec:
inputs:
- tenant: team1
layerSelector:
mediaType: "application/vnd.cncf.helm.chart.content.v1.tar+gzip"
operation: copy
resources:
- apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
name: << inputs.tenant >>
spec:
layerSelector: << inputs.layerSelector | toYaml | nindent 4 >>
```
To assign a default value when an input field is not specified, use `get` combined with `default`.
E.g. using the namespace input as a default value for name:
```yaml
spec:
inputs:
- namespace: team1
resources:
- apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
name: << get inputs "name" | default inputs.namespace >>
namespace: << inputs.namespace >>
```
In addition to the slim-sprig functions, a `slugify` function is available to normalize a string for use in a Kubernetes
label value
e.g. `<< inputs.tenant | slugify >>`.
#### Resource deduplication
The flux-operator deduplicates resources based on the
`apiVersion`, `kind`, `metadata.name` and `metadata.namespace` fields.
This allows defining shared resources that are applied only once, regardless of the number of inputs.
Example of a shared Flux source:
```yaml
spec:
inputs:
- tenant: "team1"
replicas: "2"
- tenant: "team2"
replicas: "3"
resources:
- apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
name: podinfo
namespace: default
spec:
interval: 10m
url: oci://ghcr.io/stefanprodan/charts/podinfo
ref:
semver: '*'
- apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: podinfo-<< inputs.tenant >>
namespace: default
spec:
interval: 1h
releaseName: podinfo-<< inputs.tenant >>
chartRef:
kind: OCIRepository
name: podinfo
values:
replicaCount: << inputs.replicas | int >>
```
In the above example, the `OCIRepository` resource is created only once
and referred by all `HelmRelease` resources.
#### Copying data from existing ConfigMaps and Secrets
To generate resources with data copied from existing ConfigMaps and Secrets,
the `fluxcd.controlplane.io/copyFrom: namespace/name` annotation must be
set in the ConfigMap or Secret template.
Example of copying data from an existing ConfigMap and Secret:
```yaml
spec:
inputs:
- tenant: "team1"
- tenant: "team2"
resources:
- apiVersion: v1
kind: Namespace
metadata:
name: << inputs.tenant >>
- apiVersion: v1
kind: ConfigMap
metadata:
name: runtime-info
namespace: << inputs.tenant >>
annotations:
fluxcd.controlplane.io/copyFrom: "flux-system/runtime-info"
- apiVersion: v1
kind: Secret
metadata:
name: docker-auth
namespace: << inputs.tenant >>
annotations:
fluxcd.controlplane.io/copyFrom: "flux-system/docker-auth"
```
In the above example, a ConfigMap and a Secret are generated for each tenant
with the data copied from the `runtime-info` and `docker-auth` ConfigMap and Secret
from the `flux-system` namespace. If the source data changes, the operator will
update the generated resources accordingly on the next
[reconciliation interval](#reconciliation-configuration).
Note that on [multi-tenant clusters](#role-based-access-control), the service account
used by the ResourceSet must have the necessary permissions to read the ConfigMaps
and Secrets from the source namespace.
To trigger a reconciliation of the ResourceSet when changes occur in
the source ConfigMap or Secret, you can set the following label on the
source ConfigMap or Secret:
```yaml
metadata:
labels:
reconcile.fluxcd.io/watch: Enabled
```
Note that neither this label nor any other metadata is copied to the generated
ConfigMaps and Secrets, you must label every ConfigMap or Secret that you want
to watch for changes.
An alternative to labeling every ConfigMap or Secret is
setting the `--watch-configs-label-selector=owner!=helm`
flag in flux-operator, which allows watching all ConfigMaps
and Secrets except for Helm storage Secrets.
#### Converting kubeconfig data from Secrets
To generate ConfigMap resources with the API server address and CA certificate
extracted from a kubeconfig stored in a Secret, the
`fluxcd.controlplane.io/convertKubeConfigFrom` annotation must be set on the
ConfigMap template.
The annotation value must be in the format `namespace/name` or `namespace/name:key`:
- `namespace/name` - looks for the kubeconfig data under the `kubeconfig` key first,
then falls back to the `value` key in the referenced Secret.
- `namespace/name:key` - uses the specified custom key to read the kubeconfig data
from the referenced Secret.
Example of converting kubeconfig data from a CAPI Secret:
```yaml
spec:
inputs:
- cluster: "staging"
- cluster: "production"
resources:
- apiVersion: v1
kind: ConfigMap
metadata:
name: << inputs.cluster >>-cluster-info
namespace: flux-system
annotations:
fluxcd.controlplane.io/convertKubeConfigFrom: "flux-system/<< inputs.cluster >>-kubeconfig"
data:
cluster: << inputs.cluster >>
```
In the above example, the operator reads the kubeconfig from the
`flux-system/-kubeconfig` Secret, extracts the API server address
and the CA certificate from the first cluster entry, and populates the
`address` and `ca.crt` fields in the generated ConfigMap.
If the ConfigMap template already contains `address` or `ca.crt` fields,
the existing values are preserved and not overwritten.
To trigger an immediate reconciliation of the ResourceSet when the referenced
kubeconfig Secret changes, you can set the `reconcile.fluxcd.io/watch: Enabled`
label on the Secret.
Example using a custom Secret key:
```yaml
spec:
inputs:
- cluster: "staging"
resources:
- apiVersion: v1
kind: ConfigMap
metadata:
name: << inputs.cluster >>-cluster-info
namespace: flux-system
annotations:
fluxcd.controlplane.io/convertKubeConfigFrom: "flux-system/<< inputs.cluster >>-kubeconfig:my-custom-key"
data:
cluster: << inputs.cluster >>
```
#### Computing checksums from ConfigMaps and Secrets
To trigger a rolling update of a workload when the data of a referenced
ConfigMap or Secret changes, the `fluxcd.controlplane.io/checksumFrom` annotation
can be placed on any `metadata.annotations` map in the ResourceSet template.
The annotation value must be a comma-separated list of references in the format
`Kind/namespace/name`, where `Kind` is either `ConfigMap` or `Secret`.
During reconciliation, the operator resolves each reference, computes a SHA256
digest over the combined data, and writes the result into a sibling
`fluxcd.controlplane.io/checksum` annotation as `sha256:`. The original
`checksumFrom` annotation is preserved.
Example of using `checksumFrom` to trigger a Pod rollout on a Deployment
whenever the referenced ConfigMap or Secret changes:
```yaml
spec:
inputs:
- name: podinfo
resources:
- apiVersion: apps/v1
kind: Deployment
metadata:
name: << inputs.name >>
namespace: << inputs.provider.namespace >>
spec:
selector:
matchLabels:
app.kubernetes.io/name: << inputs.name >>
template:
metadata:
annotations:
fluxcd.controlplane.io/checksumFrom: |
ConfigMap/<< inputs.provider.namespace >>/<< inputs.name >>-config,
Secret/<< inputs.provider.namespace >>/<< inputs.name >>-secret
labels:
app.kubernetes.io/name: << inputs.name >>
spec:
containers:
- name: podinfo
image: ghcr.io/stefanprodan/podinfo:6.0.0
envFrom:
- configMapRef:
name: << inputs.name >>-config
- secretRef:
name: << inputs.name >>-secret
```
After reconciliation, the Deployment's pod template will carry both annotations:
```yaml
template:
metadata:
annotations:
fluxcd.controlplane.io/checksumFrom: "ConfigMap/apps/podinfo-config,Secret/apps/podinfo-secret"
fluxcd.controlplane.io/checksum: "sha256:abc123def456..."
```
When the data of any referenced ConfigMap or Secret changes, the computed
checksum changes, the pod template hash changes, and Kubernetes performs a
rolling update of the Deployment.
References to ConfigMaps or Secrets generated by the same ResourceSet are
resolved from the pending apply. For references to resources outside the ResourceSet,
the checksum refreshes on the next reconciliation of the ResourceSet. To trigger
an immediate refresh when the referenced resource changes, you can set the
`reconcile.fluxcd.io/watch: Enabled` label on the referenced ConfigMap or Secret.
Note that on [multi-tenant clusters](#role-based-access-control), the service account
used by the ResourceSet must have the necessary permissions to read the ConfigMaps
and Secrets referenced by `checksumFrom`, if those are outside the ResourceSet itself.
#### Conditional resource exclusion
To exclude a resource based on input values, the `fluxcd.controlplane.io/reconcile` annotation can be set
to `disabled` on the resource metadata. This will prevent the resource from being reconciled by the operator.
Example of excluding a resource based on an input value:
```yaml
spec:
inputs:
- tenant: "team1"
- tenant: "team2"
resources:
- apiVersion: v1
kind: Namespace
metadata:
name: << inputs.tenant >>
- apiVersion: v1
kind: ServiceAccount
metadata:
name: flux
namespace: << inputs.tenant >>
annotations:
fluxcd.controlplane.io/reconcile: << if eq inputs.tenant "team1" >>enabled<< else >>disabled<< end >>
```
In the above example, the `ServiceAccount` resource is generated only for the `team1` tenant.
#### Built-in input fields
When computing all the input sets for generating the resource matrix, the operator
adds a few built-in fields to each input set. Users cannot override these fields.
Every input set contains a built-in `inputs.id` field that is a unique identifier
for the input set amongst all the input sets generated for the ResourceSet. This
field can be used in the resource templates to generate unique resource names.
Every input set also contains the reference of the object providing those inputs:
- `inputs.provider.apiVersion`: The API version of the object providing the inputs.
- `inputs.provider.kind`: The kind of the object providing the inputs.
- `inputs.provider.name`: The name of the object providing the inputs.
- `inputs.provider.namespace`: The namespace of the object providing the inputs.
In the case of inline inputs provided directly in the `ResourceSet` object through the
`.spec.inputs` field, `inputs.provider.apiVersion` is `fluxcd.controlplane.io/v1` and
`inputs.provider.kind` is `ResourceSet`.
In the case of inputs provided through a `ResourceSetInputProvider` referenced in the
`.spec.inputsFrom` field of a `ResourceSet`, `inputs.provider.apiVersion` is
`fluxcd.controlplane.io/v1` and `inputs.provider.kind` is `ResourceSetInputProvider`.
If the `.spec.inputStrategy.name` [field](#input-strategy) is set to `Permute`, the
built-in fields above are nested under a key that is the normalized name of the object
providing the inputs, i.e.:
- `inputs..id`
- `inputs..provider.apiVersion`
- `inputs..provider.kind`
- `inputs..provider.name`
- `inputs..provider.namespace`
And `inputs.id` will be derived from the permutation configuration, making each
permutation have a unique ID.
### Resources template
The `.spec.resourcesTemplate` field is optional and offers an alternative to the `.spec.resources`.
The `.spec.resourcesTemplate` is a single string that contains the multi-document YAML of the resources
definitions. This field can be used for complex templating scenarios with the trade-off of reduced readability.
Note that when both `.spec.resources` and `.spec.resourcesTemplate` are set, the resulting resources
are the union of the two. If duplicate resources are defined in both fields, the resources from
`.spec.resources` take precedence.
Example of a template containing conditional and repeated blocks:
```yaml
spec:
inputs:
- bundle: addons
decryption: false
components:
- ingress-nginx
- cert-manager
- bundle: apps
decryption: true
components:
- frontend
- backend
resourcesTemplate: |
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
name: << inputs.bundle >>
namespace: flux-system
spec:
interval: 10m
url: oci://registry.example.com/<< inputs.bundle >>
<<- range $component := inputs.components >>
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: << $component >>
namespace: flux-system
spec:
interval: 1h
prune: true
<<- if inputs.decryption >>
decryption:
provider: sops
secretRef:
name: << inputs.bundle >>-sops
<<- end >>
sourceRef:
kind: OCIRepository
name: << inputs.bundle >>
path: ./<< $component >>
<<- end >>
```
The above example generates two `OCIRepository` resources (one for each bundle) and four
`Kustomization` resources (one for each component in each bundle).
### Common metadata
The `.spec.commonMetadata` field is optional and specifies common metadata to be applied to all resources.
It has two optional fields:
- `labels`: A map used for setting labels
on an object. Any existing label will be overridden if it matches with a key in
this map.
- `annotations`: A map used for setting annotations
on an object. Any existing annotation will be overridden if it matches with a key
in this map.
Example common metadata:
```yaml
spec:
commonMetadata:
labels:
app.kubernetes.io/name: podinfo
annotations:
fluxcd.controlplane.io/prune: disabled
```
In the above example, all resources generated by the ResourceSet
will not be pruned by the [garbage collection](#garbage-collection) process as
the `fluxcd.controlplane.io/prune` annotation is set to `disabled`.
### Dependency management
`.spec.dependsOn` is an optional list used to refer to Kubernetes
objects that the ResourceSet depends on. If specified, then the ResourceSet
is reconciled after the referred objects exist in the cluster.
A dependency is a reference to a Kubernetes object with the following fields:
- `apiVersion`: The API version of the referred object (required).
- `kind`: The kind of the referred object (required).
- `name`: The name of the referred object (required).
- `namespace`: The namespace of the referred object (optional).
- `ready`: A boolean indicating if the referred object must have the `Ready` status condition set to `True` (optional, default is `false`).
- `readyExpr`: A [CEL expression](#cel-readiness-expressions) that evaluates to a boolean indicating if the referred object is ready (optional).
Example of conditional reconciliation based on the existence of CustomResourceDefinitions
and the readiness of a ResourceSet:
```yaml
spec:
dependsOn:
- apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
name: helmreleases.helm.toolkit.fluxcd.io
ready: true
- apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
name: servicemonitors.monitoring.coreos.com
- apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSet
name: cluster-addons
namespace: flux-system
ready: true
```
Note that is recommended to define dependencies on CustomResourceDefinitions if the ResourceSet
deploys Flux HelmReleases which contain custom resources.
When the dependencies are not met, the flux-operator will reevaluate the requirements
every five seconds and reconcile the ResourceSet when the dependencies are satisfied.
Failed dependencies are reported in the ResourceSet `Ready` [status condition](#ResourceSet-Status),
in log messages and Kubernetes events.
#### CEL readiness expressions
The `readyExpr` field allows for more complex readiness checks and
can be used for gating the reconciliation of a ResourceSet based on the evaluation
of the CEL expression.
The expression is evaluated in the context of the referred object and has access to all the fields of the object,
including the status conditions and the status subfields. The expression must evaluate to a boolean value, any syntax
or runtime errors will be reported in the ResourceSet status conditions.
Example readiness expression:
```yaml
spec:
dependsOn:
- apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
name: my-cluster
namespace: dev
ready: true
readyExpr: |
metadata.generation == status.observedGeneration &&
status.controlPlaneReady == true
- apiVersion: v1
kind: Secret
name: my-gate
namespace: dev
ready: true
readyExpr: |
string(base64.decode(data.gate)) == 'opened'
```
For testing the CEL expressions, you can use the CEL playground.
### Reconciliation configuration
The reconciliation behavior of a ResourceSet can be configured using the following annotations:
- `fluxcd.controlplane.io/reconcile`:
Enable or disable the reconciliation loop. Default is `enabled`, set to `disabled` to pause the reconciliation.
- `fluxcd.controlplane.io/reconcileEvery`:
Set the reconciliation interval used for drift detection and correction. Default is `1h`.
- `fluxcd.controlplane.io/reconcileTimeout`:
Set the reconciliation timeout including health checks. Default is `5m`.
- `fluxcd.controlplane.io/force`:
When set to `enabled`, the controller will replace the generated resources that contain immutable field changes.
This annotation can also be used on individual resources to force their reconciliation.
### Health check configuration
The `.spec.wait` field is optional and instructs the flux-operator to perform
a health check on all applied resources and waits for them to become ready. The health
check is disabled by default and can be enabled by setting the `.spec.wait` field to `true`.
The health check is performed for the following resources types:
- Kubernetes built-in kinds: Deployment, DaemonSet, StatefulSet,
PersistentVolumeClaim, Service, Ingress, CustomResourceDefinition.
- Flux kinds: HelmRelease, OCIRepository, Kustomization, GitRepository, etc.
- Custom resources that are compatible with kstatus.
By default, the wait timeout is `5m` and can be changed with the
`fluxcd.controlplane.io/reconcileTimeout` annotation, set on the ResourceSet object.
### Role-based access control
The `.spec.serviceAccountName` field is optional and specifies the name of the
Kubernetes ServiceAccount used by the flux-operator to reconcile the ResourceSet.
The ServiceAccount must exist in the same namespace as the ResourceSet
and must have the necessary permissions to create, update and delete
the resources defined in the ResourceSet.
On multi-tenant clusters, it is recommended to use a dedicated ServiceAccount per tenant namespace
with the minimum required permissions. To enforce a ServiceAccount for all ResourceSets,
the `--default-service-account=flux-operator` flag can be set in the flux-operator container arguments.
With this flag set, only the ResourceSets created in the same namespace as the flux-operator
will run with cluster-admin permissions.
When installing the Flux Operator with Helm, you can change the default service account name with:
```shell
helm install flux-operator oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator \
--namespace flux-system \
--create-namespace \
--set multitenancy.enabled=true \
--set multitenancy.defaultServiceAccount=flux-operator
```
When installing the Flux Operator on OpenShift from OperatorHub, the default service account name
can be changed by setting the `DEFAULT_SERVICE_ACCOUNT` environment variable using the OLM
Subscription
`.spec.config.env` field.
### Garbage collection
The operator performs garbage collection of the resources previously generated by a ResourceSet
that are no longer present in the current revision. Garbage collection is also performed
when a ResourceSet object is deleted, triggering a removal of all Kubernetes
objects previously applied on the cluster.
The garbage collection process removes stale resources in stages, first it deletes the Flux
custom resources and waits for the Flux Kustomizations and HelmReleases to be finalized by
the controllers. After the Flux resources are removed (max wait is one minute), the operator
proceeds with the deletion of the remaining Kubernetes objects. This ensures that the
Flux controllers have a chance to clean up the resources they manage before the operator
deletes the Kubernetes ServiceAccount and RoleBinding used by Flux impersonation.
After the garbage collection process is completed, the operator issues a Kubernetes event
containing the list of removed resources and the duration of the cleanup.
The garbage collection is enabled by default and can be disabled for certain resources
by setting the `fluxcd.controlplane.io/prune` annotation to `disabled`. To fully disable
the garbage collection for a ResourceSet, the annotation must be set on all resources
using the [`.spec.commonMetadata`](#common-metadata) field.
## ResourceSet Status
### Conditions
A ResourceSet enters various states during its lifecycle, reflected as Kubernetes Conditions.
It can be [reconciling](#reconciling-resourceset) while applying the resources on the cluster,
it can be [ready](#ready-resourceset),
it can [fail during reconciliation](#failed-resourceset),
or it can [fail due to misconfiguration](#stalled-resourceset).
The ResourceSet API is compatible with the **kstatus** specification,
and reports `Reconciling` and `Stalled` conditions where applicable to
provide better (timeout) support to solutions polling the ResourceSet to
become `Ready`.
#### Reconciling ResourceSet
The flux-operator marks a ResourceSet as _reconciling_ when it starts
the reconciliation of the same. The Condition added to the ResourceSet's
`.status.conditions` has the following attributes:
- `type: Reconciling`
- `status: "True"`
- `reason: Progressing` | `reason: ProgressingWithRetry`
The Condition `message` is updated during the course of the reconciliation to
report the action being performed at any particular moment such as
building manifests, detecting drift, etc.
The `Ready` Condition's `status` is also marked as `Unknown`.
#### Ready ResourceSet
The flux-operator marks a ResourceSet as _ready_ when the resources were
built and applied on the cluster and all health checks are observed to be passing.
When the ResourceSet is "ready", the flux-operator sets a Condition with the
following attributes in the ResourceSet’s `.status.conditions`:
- `type: Ready`
- `status: "True"`
- `reason: ReconciliationSucceeded`
#### Failed ResourceSet
The flux-operator may get stuck trying to reconcile and apply a
ResourceSet without completing. This can occur due to some of the following factors:
- The dependencies are not ready.
- The templating of the resources fails.
- The resources are invalid and cannot be applied.
- Garbage collection fails.
- Running health checks fails.
When this happens, the flux-operator sets the `Ready` Condition status to False
and adds a Condition with the following attributes to the ResourceSet’s
`.status.conditions`:
- `type: Ready`
- `status: "False"`
- `reason: DependencyNotReady | BuildFailed | ReconciliationFailed | HealthCheckFailed`
The `message` field of the Condition will contain more information about why
the reconciliation failed.
While the ResourceSet has one or more of these Conditions, the flux-operator
will continue to attempt a reconciliation with an
exponential backoff, until it succeeds and the ResourceSet is marked as [ready](#ready-resourceset).
#### Stalled ResourceSet
The flux-operator may fail the reconciliation of a ResourceSet object terminally due
to a misconfiguration. When this happens, the flux-operator adds the `Stalled` Condition
to the ResourceSet’s `.status.conditions` with the following attributes:
- `type: Stalled`
- `status: "True"`
- `reason: InvalidCELExpression | BuildFailed`
Misconfigurations can include:
- The `.spec.dependsOn[].readyExpr` expression is invalid. In this case the condition reason is
`InvalidCELExpression`.
- The templating of the resources fails. In this case the condition reason is `BuildFailed`.
When this happens, the flux-operator will not attempt to reconcile the ResourceSet
until the misconfiguration is fixed. The `Ready` Condition status is also set to `False`.
### History
With `.status.history` the operator tracks the reconciliation attempts over time, providing insights
into the ResourceSet's behavior which can be used for audit, anomaly detection and debugging purposes.
The history is stored as a list of snapshots, ordered by last reconciliation time. Each snapshot contains:
- `digest`: A SHA256 digest that uniquely identifies the set of generated resources being reconciled
- `firstReconciled`: The timestamp when this particular configuration was first reconciled
- `lastReconciled`: The timestamp of the most recent reconciliation attempt for this configuration
- `lastReconciledDuration`: How long the most recent reconciliation attempt took
- `lastReconciledStatus`: The status of the most recent reconciliation (e.g., `ReconciliationSucceeded`, `BuildFailed`, `ReconciliationFailed`)
- `totalReconciliations`: The total number of reconciliations for this configuration
- `metadata`: Additional information about the reconciliation, including the number of generated resources and inputs processed
The operator deduplicates entries based on the digest and status.
The history is automatically truncated to keep only the 5 most recent entries.
Example:
```yaml
status:
history:
- digest: sha256:43ad78c94b2655429d84f21488f29d7cca9cd45b7f54d2b27e16bbec8eff9228
firstReconciled: "2025-07-15T10:11:00Z"
lastReconciled: "2025-07-15T14:30:00Z"
lastReconciledDuration: 2.818583s
lastReconciledStatus: ReconciliationSucceeded
totalReconciliations: 5
metadata:
inputs: "2"
resources: "4"
- digest: sha256:ec8dbfe61777b65001190260cf873ffe454451bd2e464bd6f9a154cffcdcd7e5
firstReconciled: "2025-07-14T13:10:00Z"
lastReconciled: "2025-07-15T10:10:00Z"
lastReconciledDuration: 4.813292s
lastReconciledStatus: ReconciliationFailed
totalReconciliations: 120
metadata:
inputs: "1"
resources: "2"
```
Note that for `BuildFailed` errors, the digest is calculated from the resource templates, as the final resources
are not available.
### Inventory
In order to perform operations such as drift detection, garbage collection, upgrades, etc.,
the flux-operator needs to keep track of all Kubernetes objects that are
reconciled as part of a ResourceSet. To do this, it maintains an inventory
containing the list of Kubernetes resource object references that have been
successfully applied and records it in `.status.inventory`. The inventory
records are in the format `Id: ___, V: `.
Example:
```yaml
status:
inventory:
entries:
- id: apps_podinfo_helm.toolkit.fluxcd.io_HelmRelease
v: v2
- id: apps_podinfo_source.toolkit.fluxcd.io_OCIRepository
v: v1
```
## ResourceSet Metrics
The Flux Operator exports Prometheus metrics for the ResourceSet objects
that can be used to monitor the reconciliation status.
Metrics:
```text
flux_resourceset_info{uid, kind, name, exported_namespace, ready, suspended, revision}
```
Labels:
- `uid`: The Kubernetes unique identifier of the resource.
- `kind`: The kind of the resource (e.g. `ResourceSet`).
- `name`: The name of the resource (e.g. `podinfo`).
- `exported_namespace`: The namespace where the resource is deployed (e.g. `apps`).
- `ready`: The readiness status of the resource (e.g. `True`, `False` or `Unknown`).
- `reason`: The reason for the readiness status (e.g. `ReconciliationSucceeded`, `BuildFailed`, `HealthCheckFailed`, etc.).
- `suspended`: The suspended status of the resource (e.g. `True` or `False`).
- `resources`: The number of resources generated by the ResourceSet.
- `revision`: The revision last applied on the cluster (e.g. `sha256:75aa209c6a...`).