Using memory reservations and limits in Prometheus

Metrics obtained through cAdvisor are not restricted to actual usage. We have, among others, metrics based on container specs. We can, for example, retrieve memory limits with the metric container_spec_memory_limit_bytes.

Please type container_spec_memory_limit_bytes{container_label_com_docker_stack_namespace!=""} in the Expression field and click the Execute button. The result should be straight lines that represent memory limits we defined in our stacks.

The usage of the container_label_com_docker_stack_namespace label is important. We used it to filter the metrics so that only those that come from the stacks are included. That way, we excluded root metrics from cAdvisor that provide summarized totals.

In Prometheus, memory limits are not very useful in themselves. However, if we combine them with the actual memory usage, we can get percentages that can provide indications of the health of our system.

Please type container_memory_usage_bytes{container_label_com_docker_stack_namespace!=""} / container_spec_memory_limit_bytes{container_label_com_docker_stack_namespace!=""} * 100 in the Expression field and click the Execute button.

The result consists of percentages based on memory limits and the actual usage. These should all be below 60%. We will leverage this information later when we start working on alerts.