Raymond Augé

Raymond Augé

My personal thoughts.

OSGi On Kubernetes - Configuration

8 minute read

Configuration is an important aspect in application design and especially in the design of Cloud applications. The need to develop, test, and release code in the ephemeral spaces of the Cloud mean that many important details about environment and behaviour need to be abstracted away from the application logic.

Many years before Cloud environments existed OSGi technology was being leveraged to deploy complex, distributed applications to countless devices. Due in large part to its resilience and innovative design it continues to be used in this fashion today. Relatively early in OSGi’s long history the Configuration Admin Service Specification was defined with much the same concerns for configuration as are present in current environments like today’s Cloud.

Kubernetes is a very popular and widely adopted technology for orchestrating applications in the Cloud and is provided by a growing number of the largest Cloud vendors. It also defines a set of core components that are responsible for keeping applications well fed with configuration.

Notably, Kubernetes configuration principles very closely resemble those of OSGi Configuration making integration quite natural which is the topic of this post.

Defining ConfigMaps

The first Kubernetes configuration component is called ConfigMap and represents the basic building block for providing non-confidential data in key-value pairs.

(Don’t place secure or data that requires encryption in ConfigMaps. Another Kubernetes component we’re going to talk about later is intended for that.)

A ConfigMap is defined in a Kubernetes manifest file (commonly using the YAML syntax) like the following:

apiVersion: v1
kind: ConfigMap
metadata:
  name: osgi-demo-config
data:
  # property-like keys; each key maps to a simple value
  pool_initial_size: "3"
  pool_max_size: "10"

  # file-like keys
  message.processor-email.config: |
    pool.initial.size=i"${env:pool_initial_size}"
	pool.max.size=i"${env:pool_max_size}"
    topic="/email"
  message.processor-log.config: |
    pool.initial.size=i"${env:pool_initial_size}"
	pool.max.size=i"20"
    topic="/log"

(We’ll be seeing a lot of YAML in this and future posts so take heart.)

The example above demonstrates the basic model for ConfigMaps. The first 4 lines are boiler plate and are setup for the data field which holds a series of key-value pairs in one of two flavours;

  • property-like keys - typical key mapping to a single value
  • file like keys - maps a key (typically with a file-like name) to a multi-line chunk of data (that looks a lot like a file)

Take note of the syntax used in the file content. We’ll come back to that later.

The next thing you should note about the value of the file-like-keys flavour is that the format of the value is largely irrelevant. The ConfigMap does not care one way or the other if it’s XML, JSON, Java properties files, more YAML, or any other character based syntax you wish (I supposed it could even be script… but I’d be careful with that). The only real restriction is that a given ConfigMap’s data does not exceed 1 MiB in size.

Consuming ConfigMaps

There are several ways this could be used but we’ll talk about the 2 most common.

Consider the following Pod spec.

apiVersion: v1
kind: Pod
metadata:
  name: osgi-demo-pod
spec:
  containers:
    - name: osgi-demo-container
      image: rotty3000/config-osgi-k8s-demo
      env:
        # Define environment variables
        - name: pool_initial_size
          valueFrom:
            configMapKeyRef:
              name: osgi-demo-config # The ConfigMap this value comes from
              key: pool_initial_size # The key to fetch
        - name: pool_max_size
          valueFrom:
            configMapKeyRef:
              name: osgi-demo-config
              key: pool_max_size
      volumeMounts:
      - name: config
        mountPath: "/app/configs"
        readOnly: true
  volumes:
    - name: config
      configMap:
        # The name of the ConfigMap you want to mount
        name: osgi-demo-config

(Pods are the smallest deployable units of computing in a Kubernetes cluster.)

The example above defines a Pod, the first 5 lines of which are pretty boiler plate, named osgi-demo-pod, has exactly one container named osgi-demo-container. The container specifies the Docker image to be rotty3000/config-osgi-k8s-demo. That alone is the basic information necessary to define a usable Pod. You could deploy this and have a running system:

apiVersion: v1
kind: Pod
metadata:
  name: osgi-demo-pod
spec:
  containers:
    - name: osgi-demo-container
      image: rotty3000/config-osgi-k8s-demo

(Though Naked Pods are discouraged, they are useful for illustration.)

However, this is not sufficient in the majority of cases. Usually some additional input is required, such as resource constraints, volume mounts, configuration, etc.

Let’s start our examination at the bottom of the spec with the volumes field. Volumes are Kubernetes’ mechanism for sharing file systems that supplement a Pod’s own ephemeral file system (ephemeral meaning nothing that is modified in the Pod’s own file system is retained across Pod restarts). There are many different types of Volumes in Kubernetes. The one we’re showing here adds a ConfigMap:

  volumes:
    - name: config
      configMap:
        # The name of the ConfigMap you want to mount
        name: osgi-demo-config

In order for the ConfigMap to be usable it must be mounted as a volume, so we define the volume by first giving it a name (config), by specifying a Volume type in this case a configMap named (osgi-demo-config).

(The Pod and the ConfigMap have to be in the same Namespace.)

Now that the volume is defined we can mount it into the pod using the volumeMounts field under the containers array:

      volumeMounts:
      - name: config
        mountPath: "/app/configs"
        readOnly: true

The mount is set to be located at /app/configs. In this case the image we’re using already listens to this directory using Apache Felix FileInstall; an OSGi bundle that manages OSGi configurations from the file system on demand. A perfect companion for Kubernetes ConfigMaps. FileInstall is configured to do this using the system property felix.fileinstall.dir=/app/configs.

Mounted files

You should note that as configured the volume will cause each key in the ConfigMap to be mounted as a separate file in the mount directory (the value of each pair being the contents of the file). This is great because FileInstall automatically detects and manages files that end with .config (or the older .cfg) identified in it’s scanned directory . (Unknown files will be ignored.)

Mounting specific entries

You can pick and choose which keys of the ConfigMap are loaded as files by adding an items array:

  volumes:
    - name: config
      configMap:
        # The name of the ConfigMap you want to mount
        name: osgi-demo-config
        # An array of keys from the ConfigMap to create as files
        items:
        - key: "message.processor-email.config"
          path: "message.processor-email.config"
        - key: "message.processor-log.config"
          path: "message.processor-log.config"

However; I don’t recommend the items approach in the OSGi Configuration scenario in the default case because it forces a duplication of the same information in two places; the ConfigMap and the volumes definition. I suggest lettings the ConfigMap be the source of truth and I’ll explain why a little later.

Sub-directory scans

By default FileInstall scans sub-directories as well, so create any hierarchy that suites your needs by mounting the ConfigMap in any sub-directory of the path scanned by FileInstall. This is handy for grouping related configuration files.

Environment Variables

Another way to use the key-value pairs of the ConfigMap in the Pod is by defining environment variables. If you go back to the env field of the Pod spec you’ll see the following:

      env:
        # Define environment variables
        - name: pool_initial_size
          valueFrom:
            configMapKeyRef:
              name: osgi-demo-config # The ConfigMap this value comes from
              key: pool_initial_size # The key to use
        - name: pool_max_size
          valueFrom:
            configMapKeyRef:
              name: osgi-demo-config
              key: pool_max_size

Here, two of the keys are cherry-picked from the ConfigMap and turned into environment variables that will be available in the Pod from the moment of startup.

Interestingly, if you recall the note about the syntax of the configuration files content which I’ve replicated next,

  message.processor-email.config: |
    pool.initial.size=i"${env:pool_initial_size}"
	pool.max.size=i"${env:pool_max_size}"
    topic="/email"

you should be aware that FileInstall has built in interpolation for environment variables using the env: prefix. This allows you to compose configuration using the environment with no additional tooling installed.

Advanced interpolation using mounted files

Remember those property-like keys we discussed earlier? As was just stated a ConfigMap volume will cause these to be mounted as individual files. FileInstall will gladly ignore them, however it could be interesting, particularly if the contents of the directory comes from different ConfigMaps, to have a more advanced interpolation mechanism that would allow those to be used as a source of values.

As it turns out the Apache Felix project provides a companion bundle that adds exactly this functionality via a Configuration Plugin. This bundle is called org.apache.felix.configadmin.plugin.interpolation and it’s already part of the base image we’re using in the examples.

We configure the Felix interpolation plugin using system properties.

The first property tells Felix Config Admin (the implementation of OSGi Configuration Admin) to require the plugin before processing any configuration:

felix.cm.config.plugins=org.apache.felix.configadmin.plugin.interpolation

This second property tells the plugin which directory to search for values:

org.apache.felix.configadmin.plugin.interpolation.secretsdir=/app/configs

With that in mind, consider the 2 following data sections from 2 separate ConfigMaps:

## ConfigMap #1
# assume this is mounted as `/app/configs/values`
data:
  player_initial_lives: "3"

## ConfigMap #2
# assume this is mounted as `/app/configs/files`
data:
  game.pid.config: |
    player.initial.lives="$[secret:values/player_initial_lives;type=long]"
    player.maximum.lives=i"5"
    colors="$[secret:colors;type=String[];delimiter=|;default=green|red|blue]"

As you can see with the syntax provided by the Felix interpolation plugin allows us to refer to ConfigMap files as interpolation values.

You’ll also note that Felix interpolation plugin has more advanced capabilities like rich type coercion and the ability to provide defaults for missing values.

Dynamic reaction to configuration changes

Many software systems are capable of gracefully handling changes in configuration without requiring full restarts. For example Apache HTTPD Server can use it’s graceful restart mode to smoothly reload it’s configuration and automatically adjust its behaviour accordingly. NGINX offers a similar functionality and so do many other systems.

Besides the obvious goal of reducing service interruptions this also supports the notion of feature flags, service discovery and other real-time signals required to efficiently and speedily control system behaviour.

ConfigMaps as mounted files vs. environment variables

Given that mounted ConfigMaps are updated automatically without needing to restart the Pod vs. environment variable which require a Pod be restarted in order to be updated means that mounted files are more efficient for systems that are designed to be signalled on configuration file changes.

Conclusion

OSGi Configuration Admin is meant to dynamically propagate configuration changes throughout an OSGi framework. Apache Felix FileInstall adds to that the ability to react to changes in configuration resources on the file system which are subsequently propagated to Configuration Admin. Kubernetes ConfigMaps provide Pods with centrally and dynamically updated configuration resources.

The combination of the three makes for a completely dynamic and fully integrated application configuration system with very little effort.

In a future post I plan to further various aspects of this subject. So stay tuned!

Test Resources

The project demonstrating the ideas in this post can be found in this repository. The docker image that defines the configurable OSGi application can be found here on Docker Hub.

Updated: