Prometheus Cheat Sheet - How to Join Multiple Metrics (Vector Matching)

PromQL looks neat and powerful. And at first sight, simple. But when you start using it for real, you'll quickly notice that it's far from being trivial. Searching the Internet for query explanation rarely helps - most articles focus on pretty high-level overviews of the language's most basic capabilities. For example, when I needed to match multiple metrics using the common labels, I quickly found myself reading the code implementing binary operations on vectors. Without a solid understanding of the matching rules, I constantly stumbled upon various query execution errors, such as complaints about missing group_left or group_right modifier. Reading the code, feeding my local Prometheus playground with artificial metrics, running test queries, and validating assumptions, finally helped me understand how multiple metrics can be joined together. Below are my findings.

PromQL binary operators

PromQL comes with 15 binary operators that can be divided into three groups by operation type:

• arithmetic + - / * ^ %
• comparison < > <= >= == !=
• logical/set and, unless, or

Binary operations are defined for different types of operands - scalar/scalar, scalar/vector, and vector/vector. And the last pair of operands, vector/vector, is the most puzzling one because the subtle vector matching rules may differ depending on the cardinality of the sides or operation type.

One-to-one vector matching

The following diagram tries to shed some light on:

• how one-to-one vector matching works
• how on and ignoring modifiers reduce the set of labels to be used for matching
• what problem this label set reduction can cause

PromQL one-to-one vector matching - arithmetic and comparison operations (clickable, 1.1 MB).

One-to-many and many-to-one vector matching

One-to-one matching is the most straightforward one. Most of the time, on or ignoring modifiers help to make a query return something reasonable. But the pitfall here is that some query results may show a one-to-one cardinality only by coincidence. For instance, when our assumptions of the potential set of values for a given label are erroneous. So, it's just so happened that the query showed a one-to-one data relationship on a selected time range, but it can be one-to-many in general. Or many-to-one.

Luckily, Prometheus does support many-to-one and one-to-many vector matching. But it has to be specified explicitly by adding either group_left or group_right modifier to a query. Otherwise, the following error might be returned during the query execution:

multiple matches for labels: many-to-one matching must be explicit (group_left/group_right)

Unless logical binary operator and|unless|or is used, Prometheus always considers at least one side of the binary operation as having the cardinality of "one". If during a query execution Prometheus finds a collision (label-wise) on the "one" side, the query will fail with the following error:

found duplicate series for the match group <keys/values> on the <left|right> hand-side of the operation: <op>;
many-to-many matching not allowed: matching labels must be unique on one side

Interesting, that even if the "one" side doesn't have collisions and group_left or group_right is specified, a query can still fail with:

multiple matches for labels: grouping labels must ensure unique matches

It can happen because, for every element on the "many" side, Prometheus should find no more than one element from the "one" side. Otherwise, the query result would become ambiguous. If the requested label matching doesn't allow to build an unambiguous result, Prometheus just fails the query.

PromQL many-to-one and one-to-many vector matching - arithmetic and comparison operations (clickable, 1.2 MB).

Many-to-many vector matching (logical/set operations)

Logical (aka set) binary operators and, unless, and or surprisingly adhere to a simpler vector matching logic. These operations are always many-to-many. Hence no group_left or group_right may be needed. The following diagram focuses on how logical/set operations behave in Prometheus:

PromQL many-to-many vector matching - logical/set operations (clickable, 1.4 MB).

Instead of conclusion

If you find this useful, check out other my Prometheus drawing. And if the idea of querying files such as Nginx or Envoy access logs with PromQL-like syntax sounds interesting to you, please give the pq project a star on GitHub. It really fuels me up:

pq - parse and query files with PromQL-like syntax

Resources

• Prometheus repo on GitHub
• Prometheus docs on operators - turns out it's a very good resource when you already understand the internals. Can't really call it beginner-friendly, though.