Framework documentation

# Framework Documentation

[Contribute on GitHub](https://github.com/kbst/terraform-kubestack)[Join #kubestack on Slack](https://slack.k8s.io/)

Topics

-   [Introduction](/documentation/)
-   [Platform Architecture](/documentation/platform-architecture/)
-   [GitOps Process](/documentation/gitops-process/)
-   [Inheritance Model](/documentation/inheritance-model/)
-   [Repository layout](/documentation/repository-layout/)
-   [Module Types](/documentation/module-types/)
-   [Clusters](/documentation/clusters/)
-   [Node-Pools](/documentation/node-pools/)
-   [Platform Features](/documentation/platform-features/)
-   [Extending Kubestack](/documentation/extending-kubestack/)

## Extending Kubestack

**TL;DR:**

-   Use custom OpenTofu modules to extend your platform with on-premise or cloud resources.
-   Integrate custom modules into the framework by adopting Kubestack metadata and configuration inheritance.
-   Provision any Kubernetes YAML using the kustomization overlay module.

### Introduction

You can extend a Kubestack platform to integrate with existing infrastructure or to provision resources that Kubestack does not provide modules for. There are two main use-cases.

1.  For infrastructure requirements you can write [custom OpenTofu modules](#opentofu-modules) and integrate them with the environments and outputs from the Kubestack modules.
2.  For [Kubernetes manifests](#kubernetes-manifests) you want to provision as part of your platform, Kubestack provides the kustomization overlay module. Using the overlay module, you can integrate and configure any upstream YAML the same way as all platform feature modules.

### Cluster Module Outputs

To integrate with cloud resources that are created by Kubestack modules, the modules provide [OpenTofu outputs](https://opentofu.org/docs/language/values/outputs/). You can combine the outputs with OpenTofu [data sources](https://developer.hashicorp.com/terraform/language/data-sources), to query information about created resources that you may need as inputs to other resources.

#### Output `current_config`

The `current_config` output has the config used for the cluster in the current environment.

**Amazon**

```
# module.eks_gc0_eu-west-1.current_config
eks_gc0_eu-west-1_current_config = {
  "base_domain"                = "kubestack.example.com"
  "cluster_availability_zones" = ["eu-west-1a", "eu-west-1b", "eu-west-1c"]
  "cluster_version"            = "1.30"
  "default_node_pool" = {
    "desired_capacity" = 3
    "instance_types"   = ["t3a.xlarge"]
    "max_size"         = 9
    "min_size"         = 3
  }
  "name_prefix" = "gc0"
}
```

#### Output `current_metadata`

The `current_metadata` output holds all the information that the metadata module generated for the cluster.

**Amazon**

```
# module.eks_gc0_eu-west-1.current_metadata
eks_gc0_eu-west-1_current_metadata = {
  "domain"          = "aws.kubestack.example.com"
  "fqdn"            = "gc0-ops-eu-west-1.aws.kubestack.example.com"
  "label_namespace" = "kubestack.com/"
  "labels" = {
    "kubestack.com/cluster_domain"          = "aws.kubestack.example.com"
    "kubestack.com/cluster_fqdn"            = "gc0-ops-eu-west-1.aws.kubestack.example.com"
    "kubestack.com/cluster_name"            = "gc0-ops-eu-west-1"
    "kubestack.com/cluster_provider_name"   = "aws"
    "kubestack.com/cluster_provider_region" = "eu-west-1"
    "kubestack.com/cluster_workspace"       = "ops"
  }
  "name" = "gc0-ops-eu-west-1"
  "tags" = [
    "gc0-ops-eu-west-1",
    "ops",
    "aws",
    "eu-west-1",
  ]
}
```

#### Output `kubeconfig`

The `kubeconfig` output, provides a KUBECONFIG file that can be used to connect and authenticate to the Kubernetes API server of the respective cluster. This output is used to configure the `kustomization` providers used for platform feature modules.

The following examples show how to use it with different OpenTofu providers focused on Kubernetes.

```
provider "kustomization" {
  alias = "gke_gc0_europe-west1"

  kubeconfig_raw = module.gke_gc0_europe-west1.kubeconfig
}
```

```
locals {
  gke_gc0_europe-west1_kubeconfig = yamldecode(module.gke_gc0_europe-west1.kubeconfig)
}

provider "kubernetes" {
  alias = "gke_gc0_europe-west1"

  host                   = local.gke_gc0_europe-west1_kubeconfig["clusters"][0]["cluster"]["server"]
  cluster_ca_certificate = base64decode(local.gke_gc0_europe-west1_kubeconfig["clusters"][0]["cluster"]["certificate-authority-data"])
  token                  = local.gke_gc0_europe-west1_kubeconfig["users"][0]["user"]["token"]
}
```

### OpenTofu Modules

Kubestack modules, are just OpenTofu modules. And consequently they can coexist alongside your custom modules in the same root module.

#### Query Using Data Sources

OpenTofu data sources can be used to query information about an existing resource. The primary resource is usually the cluster. Once you have the data source, you can access the attributes of the respective cluster resource.

The following examples show how to query information about an AKS, EKS or GKE clusters using a data source and the Kubestack module outputs.

**Amazon**

```
data "aws_eks_cluster" "eks_gc0_eu-west-1" {
  provider = aws.eks_gc0_eu-west-1

  name = module.eks_gc0_eu-west-1.current_metadata.name
}
```

Attributes provided by the `aws_eks_cluster` data source are [documented upstream](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/eks_cluster).

#### Unique Names per Environment

Kubestack uses OpenTofu workspaces for its environments. When you create cloud resources, names usually have to be unique for one account or even globally. To make sure names are unique per environment, you can include the environment name in the resource name. The easiest way to achieve that is to include the `terraform.workspace` in the name.

```
locals {
  example_name = "example-${terraform.workspace}"
}
```

Kubestack itself goes a step further and includes a `name_prefix`, the `terraform.workspace` and the `region` in names. This provides names that are unique per environment, region and cloud provider.

The module that implements this, is used by all other Kubestack modules. And you can use the common metadata module to generate names and other metadata in your custom modules aswell.

```
module "example_metadata" {
  source = "github.com/kbst/terraform-kubestack//common/metadata?ref=<version>"

  name_prefix = "example"
  base_domain = var.base_domain

  provider_name   = "gcp" # kubestack uses aws, gcp or azure
  provider_region = module.gke_gc0_europe-west1.current_config["region"]

  # it's good practice to namespace labels
  label_namespace = "example.com/"
}
```

You can then access the same metadata that all Kubestack cluster modules provide. See the section about the `current_metadata` OpenTofu output above for more details.

```
output "example_metadata_name" {
  value = module.example_metadata.name
}

output "example_metadata_labels" {
  value = module.example_metadata.labels
}
```

#### Inheritance Based Configuration

To build custom modules that implement the same [inheritance based configuration](/documentation/inheritance-model/) as Kubestack modules, use the common configuration module.

```
module "example_configuration" {
  source = "github.com/kbst/terraform-kubestack//common/configuration?ref=<version>"

  configuration = var.configuration
  base_key      = var.configuration_base_key
}
```

The configuration module expects to be given the `configuration`, as a map of maps. Where the keys of the outer map, are the environment names. And the `base_key` sets which environment all others inherit from. Most commonly you'll use this module inside another module, and the `configuration` and the `base_key` are passed in using variables when calling your module.

### Kubernetes Manifests

Kubestack provides the kustomization overlay module to apply any Kubernetes YAML. The overlay module accepts the same configuration attributes as [platform feature modules](/documentation/platform-features/#configuration). You can use it to integrate any Kubernetes YAML into your Kubestack platform the same way as any platform feature.

#### Usage example

To include custom YAML manifests:

1.  place the YAML files in a directory under `manifests/`
2.  call the overlay module and reference the YAML files as `resources`

Below example uses the overlay module to apply fictitious upstream YAML. It also sets an `env` label for all resources with the current OpenTofu workspace as the value.

If the Kubernetes service you need provides a helm chart, you can use `helm template` to vendor the rendered YAML in a file and apply them using the overlay module.

```
module "example_custom_manifests" {
  providers = {
    kustomization = kustomization.gke_gc0_europe-west1
  }

  source = "github.com/kbst/terraform-kubestack//kustomization/overlay?ref=<version>"

  configuration = {
    apps = {
      namespace = "example-${terraform.workspace}"

      resources = [
        "${path.module}/manifests/example/upstream.yaml"
      ]

      common_labels = {
        "env" = terraform.workspace
      }
    }

    ops = {}
  }
}
```

#### Overlay Configuration Attributes

The overlay module configuration attributes are identical to [platform feature module configuration attributes](/documentation/platform-features/#configuration). Use `resources` to specify a Kustomization or the individual YAML files to provision.

```
module "example_custom_manifests" {
  providers = {
    kustomization = kustomization.example_cluster_alias
  }

  # kustomization overlay module, use the framework version
  # that all other modules in your repository use
  source = "github.com/kbst/terraform-kubestack//kustomization/overlay?ref=<version>"

  configuration = {
    apps = {
      # list of paths to YAML files or Kustomizations to deploy (required)
      resources = [
        # kustomization
        "${path.module}/manifests/example-kustomization",

        # or individual YAML files
        "${path.module}/manifests/example/namespace.yaml",
        "${path.module}/manifests/example/deployment.yaml",
        "${path.module}/manifests/example/service.yaml",
      ]

      # set annotations on all Kubernetes resources
      # (optional), defaults to null
      common_annotations = {
        "example-annotation" = var.example_annotation
      }

      # set labels on all Kubernetes resources
      # (optional), defaults to null
      common_labels = {
        "example-label" = var.example_label
      }

      # generate Kubernetes configMaps
      # (optional), defaults to null
      config_map_generator = [{
        # name (required)
        # Sets 'metadata.name' of the configMap resource
        name = "example"

        # namespace (required)
        # Sets 'metadata.namespace' of the configMap resource
        namespace = "example"

        # behavior (optional)
        # Valid values: 'create', 'replace' or 'merge'. Defaults to 'create'.
        behavior = "create"  

        # literals (optional)
        # List of 'KEY=VALUE' strings. Sets 'data[KEY] = VALUE' in the configMap.
        literals = [
          "KEY=VALUE"
        ]

        # envs (optional)
        # List of paths (strings) to env files (one KEY=VALUE pair per line).
        # Sets 'data[KEY] = VALUE' in the configMap per line in the env file.
        envs = [
          "${path.module}/manifests/env"
        ]

        # files (optional)
        # List of paths (strings) to files.
        # Sets 'data[KEY] = file_content'. KEY defaults to the file's name.
        # Overwrite by prefixing 'customkey='.
        files = [
          "${path.module}/manifests/cfg.ini",          # data["cfg.ini"] = file_content
          "prod.ini=${path.module}/manifests/cfg.ini"  # data["prod.ini"] = file_content
        ]

        # options (optional)
        # Same as top level 'generator_options' but specific to this configMap.
        options = {
          # see generator_options
        }
      }]

      # generate Kubernetes secrets
      # (optional), defaults to null
      secret_generator = [{
        # name (required)
        # Sets 'metadata.name' of the secret resource
        name = "example"

        # namespace (required)
        # Sets 'metadata.namespace' of the secret resource
        namespace = "example"

        # behavior (optional)
        # Valid values: 'create', 'replace' or 'merge'. Defaults to 'create'.
        behavior = "create"

        # type (optional)
        # 'type' attribute of the secret to generate.
        type = "generic"

        # literals (optional)
        # List of 'KEY=VALUE' strings. Sets 'data[KEY] = VALUE' in the secret.
        literals = [
          "KEY=VALUE"
        ]

        # envs (optional)
        # List of paths (strings) to env files (one KEY=VALUE pair per line).
        # Sets 'data[KEY] = VALUE' in the secret per line in the env file.
        envs = [
          "${path.module}/manifests/env"
        ]

        # files (optional)
        # List of paths (strings) to files.
        # Sets 'data[KEY] = file_content'. KEY defaults to the file's name.
        # Overwrite by prefixing 'customkey='.
        files = [
          "${path.module}/manifests/cfg.ini",          # data["cfg.ini"] = file_content
          "prod.ini=${path.module}/manifests/cfg.ini"  # data["prod.ini"] = file_content
        ]

        # options (optional)
        # Same as top level 'generator_options' but specific to this secret.
        options = {
          # see generator_options
        }
      }]

      # set options for all configMap and secret generators
      # (optional), defaults to null
      generator_options = {
        # annotations (optional)
        # Sets 'metadata.annotations' on the generated resources. Defaults to '{}'.
        annotations = {
          example-annotation = "example"
        }
        
        # labels (optional)
        # Sets 'metadata.labels' on the generated resources. Defaults to '{}'.
        labels = {
          example-label = "example"
        }

        # disable_name_suffix_hash (optional)
        # Disables hash suffix of generated resource's 'metadata.name'.
        # Defaults to 'false'.
        disable_name_suffix_hash = true
      }

      # patch image names, tags or digests
      # (optional), defaults to null
      images = [{
        # Refers to the 'pod.spec.container.name' to modify the 'image' attribute of.
        name     = "busybox"
        
        # Customize the 'registry/name' part of the image. The part before the ':'
        new_name = "new_name"
        
        # Customize the 'tag' part of the image. The part after the ':'.
        new_tag  = "new_tag"

        # Replace the 'tag' part of an image with a 'digest'.
        digest   = "sha256:..."
      }]

      # prefix or suffix to add to resource names
      # (optional), default to null
      name_prefix = "prefix-"
      name_suffix = "-suffix"

      # namespace to set for all resources
      # (optional), default to null
      namespace = "example"

      # patches to apply to resources
      # (optional), default to null
      patches = [{
        # path (optional)
        # Path to a file that defines a strategic merge or JSON patch.
        path = "${path.module}/manifests/patch.yaml"

        # patch (optional)
        # Inline string that defines a strategic merge or JSON patch.
        patch = <<-EOF
          - op: replace
            path: /metadata/name
            value: newname
        EOF

        # target (optional)
        # Target one or multiple resources to be patched.
        target = {
          group               = ""
          version             = "v1"
          kind                = "ConfigMap"
          name                = "example"
          namespace           = "example"
          label_selector      = "key=value"
          annotation_selector = "key=value"
        }
      }]

      # set replicas attribute of deployments and similar resources
      # (optional), default to null
      replicas = [{
        # Refers to the 'metadata.name' of the resource to scale
        name = "example"
        
        # Sets the desired number of replicas.
        count = 5
      }]
    }

    ops  = {}
  }
}
```

##### low-level arguments

In addition to the convenience kustomization attributes documented above, platform feature modules also pass the following low-level attributes through to the `kustomization` provider. These Kustomization attributes are less useful in the OpenTofu context and as such are not documented here.

-   `components` (optional) List of paths to Kustomize [components](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/components/).
-   `crds` (optional) List of paths to Kustomize [CRDs](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/crds/).
-   `generators` (optional) List of paths to Kustomize generators.
-   `transformers` (optional) List of paths to Kustomize transformers.
-   `vars` (optional) List of objects to define Kustomize [vars](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/vars/).

[PreviousPlatform Features](/documentation/platform-features/ "Platform Features")

On this page

-   [Extending Kubestack](#extending-kubestack)
-   [Introduction](#introduction)
-   [Cluster Module Outputs](#cluster-module-outputs)
-   [OpenTofu Modules](#open-tofu-modules)
-   [Kubernetes Manifests](#kubernetes-manifests)

For your agent

-   [SKILL.md](/SKILL.md)
-   [This page as Markdown](/documentation/extending-kubestack/index.md)