This article is part of the Learning Prometheus series:
- Prometheus Is Not a TSDB
- How to learn PromQL with Prometheus Playground
- Prometheus Cheat Sheet - Basics (Metrics, Labels, Time Series, Scraping)
- Prometheus Cheat Sheet - How to Join Multiple Metrics (Vector Matching)
- Prometheus Cheat Sheet - Moving Average, Max, Min, etc (Aggregation Over Time)
PromQL looks neat and powerful. And at first sight, simple. But when you start using it for real, you'll quickly notice that it's far from being trivial. Searching the Internet for query explanation rarely helps - most articles focus on pretty high-level overviews of the language's most basic capabilities. For example, when I needed to match multiple metrics using the common labels, I quickly found myself reading the code implementing binary operations on vectors. Without a solid understanding of the matching rules, I constantly stumbled upon various query execution errors, such as complaints about missing
group_right modifier. Reading the code, feeding my local Prometheus playground with artificial metrics, running test queries, and validating assumptions, finally helped me understand how multiple metrics can be joined together. Below are my findings.
PromQL binary operators
PromQL comes with 15 binary operators that can be divided into three groups by operation type:
+ - / * ^ %
< > <= >= == !=
and, unless, or
Binary operations are defined for different types of operands - scalar/scalar, scalar/vector, and vector/vector. And the last pair of operands, vector/vector, is the most puzzling one because the subtle vector matching rules may differ depending on the cardinality of the sides or operation type.
One-to-one vector matching
The following diagram tries to shed some light on:
- how one-to-one vector matching works
ignoringmodifiers reduce the set of labels to be used for matching
- what problem this label set reduction can cause
NOTE: check out this article to learn why some operations preserve the metric name in the result and some produce nameless series.
One-to-many and many-to-one vector matching
One-to-one matching is the most straightforward one. Most of the time,
ignoring modifiers help to make a query return something reasonable. But the pitfall here is that some query results may show a one-to-one cardinality only by coincidence. For instance, when our assumptions of the potential set of values for a given label are erroneous. So, it's just so happened that the query showed a one-to-one data relationship on a selected time range, but it can be one-to-many in general. Or many-to-one.
Luckily, Prometheus does support many-to-one and one-to-many vector matching. But it has to be specified explicitly by adding either
group_right modifier to a query. Otherwise, the following error might be returned during the query execution:
multiple matches for labels: many-to-one matching must be explicit (group_left/group_right)
Unless logical binary operator
and|unless|or is used, Prometheus always considers at least one side of the binary operation as having the cardinality of "one". If during a query execution Prometheus finds a collision (label-wise) on the "one" side, the query will fail with the following error:
found duplicate series for the match group <keys/values> on the <left|right> hand-side of the operation: <op>; many-to-many matching not allowed: matching labels must be unique on one side
Interesting, that even if the "one" side doesn't have collisions and
group_right is specified, a query can still fail with:
multiple matches for labels: grouping labels must ensure unique matches
It can happen because, for every element on the "many" side, Prometheus should find no more than one element from the "one" side. Otherwise, the query result would become ambiguous. If the requested label matching doesn't allow to build an unambiguous result, Prometheus just fails the query.
PromQL many-to-one and one-to-many vector matching - arithmetic and comparison operations (clickable, 1.2 MB).
Many-to-many vector matching (logical/set operations)
Logical (aka set) binary operators
or surprisingly adhere to a simpler vector matching logic. These operations are always many-to-many. Hence no
group_right may be needed. The following diagram focuses on how logical/set operations behave in Prometheus:
Instead of conclusion
If you find this useful, check out other my Prometheus drawing. And if the idea of querying files such as Nginx or Envoy access logs with PromQL-like syntax sounds interesting to you, please give the
pq project a star on GitHub. It really fuels me up:
- Prometheus repo on GitHub
- Prometheus docs on operators - turns out it's a very good resource when you already understand the internals. Can't really call it beginner-friendly, though.