上QQ阅读APP看书，第一时间看更新

Designing a more dynamic monitoring solution

How can we improve Prometheus design to suit our purposes better? How can we make it more dynamic and more scheduler friendly?

One improvement we can make is the usage of environment variables. That would save us from having to create a new image every time we need to change its configuration. At the same time, environment variables would remove the need to use a network drive (at least for configuration).

When using containers, environment variables are a preferable way of passing configuration information to the processes running inside them.

We can make a generic solution that will transform any environment variable into a Prometheus configuration entry or an initialization argument.

We'll go through the code I created. It is written in Go and should be relatively straightforward to understand even if you are not a Go developer. The goal of the walkthrough is to get familiar with a potential solution that you could write yourself in your favorite language. That does not mean that you should not adopt Docker Flow Monitor as a ready to go solution and skip the coding part, but, rather, that you should evaluate your options and make your own choice.

To enable Prometheus configuration through environment variables, we need to distinguish those that should be used as command line arguments from those that will serve to create the configuration file. We'll define a naming convention stating that every environment argument with a name that starts with ARG_ is a startup argument.

The code can be as follows:

func Run() error { 
    cmdString := "prometheus" 
    for _, e := range os.Environ() { 
        if key, value := getArgFromEnv(e, "ARG"); len(key) > 0 { 
            cmdString = fmt.Sprintf("%s -%s=%s", cmdString, key, 
            value) 
        } 
    } 
    cmd := exec.Command("/bin/sh", "-c", cmdString) 
    return cmdRun(cmd) 
}

It is a very simple function. It iterates through all the environment variables. If their names start with ARG, they will be added as arguments of the executable Prometheus. Once the iteration is done, binary is launched with arguments.

We made Prometheus more Docker-friendly with only a few lines of code that sits on top of it.

The full source code can be found in the run.go file at https://github.com/vfarcic/docker-flow-monitor/blob/master/prometheus/run.go.

We should do something similar with the configuration file. Specifically, we can make the global section of the configuration use environment variables prefixed with GLOBAL_.

The logic of the code is similar to the Run function we explored. Please go through config.go for more details (https://github.com/vfarcic/docker-flow-monitor/blob/master/prometheus/config.go). The GetGlobalConfig function returns global section of the config while the WriteConfig function writes the configuration to the file.

Please consult Prometheus configuration (https://prometheus.io/docs/operating/configuration/) for more information about the available options.

By using environment variables, we managed to get rid of the network drive. As far as configuration is concerned, it will be fault tolerant. If the service fails and gets rescheduled with Swarm, it will not lose its configuration since it is part of the service definition. There is a downside though. Every time we want to change the configuration, we'll need to execute docker service update command or modify the stack file, and re-execute docker stack deploy. As a result, Docker will stop the currently running replica and start a new one thus producing a short downtime. However, since we are, at the moment, only dealing with global configuration and startup arguments, changes will be very uncommon. We'll deal with more dynamic parts of the configuration later.

Figure 3-5: Prometheus configuration defined through environment variables

I have the code compiled and available as vfarcic/docker-flow-monitor/ (https://hub.docker.com/r/vfarcic/docker-flow-monitor/). Let's give it a spin.

Please note that you can choose not to use Docker Flow Monitor nor to create your own solution. What follows is still equally useful if you decide to use the "original" Prometheus. You will learn how to apply manually all the changes we'll perform in a more dynamic fashion.