Edit this page

CNI with Cilium

Cilium CNI is a powerful networking plugin for Kubernetes which provides enhanced security and networking capabilities for containerised applications. It leverages the power of eBPF (extended Berkeley Packet Filter), a highly efficient and programmable kernel-level technology, to deliver transparent network security and traffic monitoring features.

Deploying Charmed Kubernetes with Cilium

Note:

If you are deploying Charmed Kubernetes + Cilium on AWS or OpenStack, please set the following model configurations to disable FAN networking, as this conflicts with the network interfaces created by Cilium: `juju model-config container-networking-method=local fan-config=`

To deploy a cluster with Cilium, deploy the charmed-kubernetes bundle with the Cilium overlay:

juju deploy charmed-kubernetes --overlay cilium-overlay.yaml

You can apply any additional customisation overlays to this deployment as well.

Upgrading Charmed Kubernetes with Cilium

Note:

Upgrades should be executed stepping through major versions.

Determine which version of cilium is currently installed on the cluster using juju status cilium

App                       Version Status  Scale  Charm                     Channel        Rev  Exposed  Message
cilium                    1.12.5  active      5  cilium                    latest/stable   29  no       Ready

Upgrading will require changing the release config of the charm with juju config cilium release=<version>, but choosing which version requires seeing what the charm currently supports.

juju run cilium/leader list-versions
cilium-versions: |
   1.12.5
   1.13.16
   1.14.11
hubble-versions: |-
   1.12.5
   1.13.16
   1.14.11

If the units are all at 1.12.5, The next available step would be to 1.13.16. Cilium recommends the cni be upgrading through each minor release. To get to the latest current supported charm, follow the following steps.

juju config cilium release=1.13.16
juju-wait
kubectl get pods -A
# ensuring that all cilium pods restart on the new image versions
juju exec cilium/leader -- cilium status
# ensure that cilium reflects a successful deployment

Then repeat the steps with 1.14.11.

Cilium configuration options

A full list of Cilium configuration options and their descriptions can be found in the Cilium charm page on Charmhub.

Checking the current configuration

To check the current configuration settings for Cilium, run the command:

juju config cilium

Setting a config option

To set an option, simply run the config command with an additional <key>=<value> argument. For example, to configure the port-forward service for Hubble:

juju config cilium port-forward-hubble=true

The configuration settings that require further explanation are provided below.

Configuring the default Cluster Pool CIDR

The Cilium charm uses the Cluster Scope IPAM mode as its default method for IP allocation. To define the IP address range that can be assigned to Kubernetes Pods within the cluster, two configuration settings can be adjusted: cluster-pool-ipv4-cidr and cluster-pool-ipv4-mask-size.

The cluster-pool-ipv4-cidr configuration specifies the IPv4 CIDR block for the cluster pool. This determines the range of IP addresses that the Kubernetes Pods can use. On the other hand, the cluster-pool-ipv4-mask-size configuration setting specifies the number of bits in the subnet mask to be used for the IP addresses in the cluster pool. It defines the number of IP addresses that are available in the node from the cluster pool.

For instance, if cluster-pool-ipv4-cidr is set to 10.1.0.0/16 and cluster-pool-ipv4-mask-size is set to 24, then the available IP addresses for the Kubernetes Pods in the one of the nodes will fall in the range of 10.1.0.1 to 10.1.0.254.

Note:

It is vital to choose an appropriate IP address range and subnet mask size that corresponds to the size of your cluster and the number of Pods you expect to deploy.

If you need to configure Cilium with a different CIDR pool during deployment, simply create a file named cluster-pool-overlay.yaml and include the following contents:

applications:
  cilium:
    options:
      cluster-pool-ipv4-cidr: 10.1.0.0/16
      cluster-pool-ipv4-mask-size: 24

After creating the cluster-pool-overlay.yaml file, add it as an overlay during the deployment of Charmed Kubernetes. Keep in mind that the order of the overlays is important:

juju deploy charmed-kubernetes --overlay cilium-overlay.yaml --overlay cluster-pool-overlay.yaml

Configuring Hubble

Hubble is a fully distributed networking and security observability platform that complements the Cilium networking solution. It provides a comprehensive view of the Kubernetes networking infrastructure, allowing users to monitor and troubleshoot network traffic, microservices, and policies.

Since Hubble is developed as a separate daemon that operates alongside the Cilium agent, it must be enabled manually. To enable this component in the Cilium charm, execute the following command:

juju config cilium enable-hubble=true

This command will deploy Hubble in your cluster, allowing you to take advantage of its networking and security observability features.

CLI Tools

The Cilium charm includes the cilium-cli and hubble CLI tools which enable you to manage, troubleshoot, and visualise insights about your Kubernetes cluster running Cilium. These CLI tools are accessible in each Kubernetes Control Plane and Worker units. The upstream documentation provides guidance on how to use these CLI tools. Please refer to it for further details.

Hubble Port Forward

To access the Hubble API, you need to port-forward the Hubble service, which enables you to connect the Hubble client to the service and access the Hubble Relay in your Kubernetes cluster. You can do this by running the following command:

juju config port-forward-hubble=true

Once you have done this, you should be able to reach Hubble from your units. Run the following command to verify the connection and check the status:

hubble status

…which will return the current stats, for example:

Healthcheck (via localhost:4245): Ok
Current/Max Flows: 20,475/20,475 (100.00%)
Flows/s: 17.18
Connected Nodes: 5/5

For further information and details on how to use the Hubble client, please consult the relevant documentation.

Observability Configuration

Cilium and Hubble provide valuable metrics about the cluster and its workloads. However, by default, the Cilium charm disables metric exposure. Once you have enabled metric exposure for Cilium and Hubble, you can use the Canonical Observability Stack or another observability stack of your choosing to scrape and visualize the metrics.

To enable metrics for Cilium and Hubble, follow the instructions below:

Cilium

To enable metrics for the Cilium CNI components, execute the following command:

juju config cilium enable-cilium-metrics=true

By executing this command, you will be able to access metrics for Cilium CNI components such as cilium-agent and cilium-operator. For more detailed information, please refer to the upstream documentation.

Hubble

Hubble provides several sets of metrics that can be enabled independently. To activate specific metrics, you can pass a list of metrics to enable as a space-separated string. Here is an example:

juju config cilium enable-hubble-metrics="dns icmp"

This command will enable both the dns and icmp metrics. For additional information about the available metrics that can be enabled, refer to the Hubble metrics section in the upstream documentation.

COS Lite Integration

The Cilium charm integrates seamlessly with the Canonical Observability Stack (COS). You can refer to the COS observability stack documentation for detailed instructions on how to deploy the required components: Deploy the COS Lite observability stack on MicroK8s

Prometheus

Prometheus, from the COS Lite stack, can automatically scrape metrics from the Cilium and Hubble components. To set this up, you can integrate both Prometheus and Cilium via a Cross-Model Integration (CMI). The COS Lite documentation provides detailed instructions on how to create cross-model integrations using overlays. You can also refer to the Juju documentation for more information on how to set up and manage cross-model integrations.

Grafana

The Cilium charm includes the necessary dashboards to visualise the metrics exported by the different Cilium components. For more information about the dashboards bundled in the Cilium charm, please refer to the Cilium repository.

To retrieve the dashboards, the Cilium charm needs to be related via a Cross-Model Integration to Grafana from the COS Lite stack.

Troubleshooting

If there is an issue with connectivity, it can be useful to inspect the Juju logs. To see a complete set of logs for Cilium, use the following command:

juju debug-log --replay --include=cilium

This will provide a detailed log output that can be used to identify and resolve issues. For additional troubleshooting pointers, please see the dedicated troubleshooting page.

We appreciate your feedback on the documentation. You can edit this page or file a bug here.

See the guide to contributing or discuss these docs in our public Mattermost channel.