The container_cpu_usage_seconds_total metric gives us the amount of CPU seconds spent running each container, and comes from the cadvisor exporter. The node_exporter version can be found in the node_exporter_build_info metric. To make things a bit harder, since the container metrics come from cadvisor, the container and pod registered in those metrics are the cadvisor ones and not from the target pods; however, we can find the original container names and pod names in the container_label_io_kubernetes_container_name and container_label_io_kubernetes_pod_name labels, respectively.

The first thing we need to do is get the average number of CPU seconds per second each pod is using on a rolling window of one minute:

sum by (container_label_io_kubernetes_pod_name) (
rate(container_cpu_usage_seconds_total{container_label_io_kubernetes_container_name="node-exporter"}[1m])

Next, we need to create a new label in node_exporter_build_info so that matching works as intended. For this use case, we can use either label_join or label_replace, as we're just reading one label and writing its contents verbatim in another:

label_join(node_exporter_build_info, "container_label_io_kubernetes_pod_name", "", "pod")

Alternatively, we can use the following code:

label_replace(node_exporter_build_info, "container_label_io_kubernetes_pod_name", "$1", "pod", "(.+)")

Finally, we just need to match both metrics through their common label, container_label_io_kubernetes_pod_name, by using on() and then ask for the version label to be joined to the CPU expression's label set by using group_left(). Let's put that all together:

sum by (container_label_io_kubernetes_pod_name) (
rate(container_cpu_usage_seconds_total{container_label_io_kubernetes_container_name="node-exporter"}[1m])
) 
* on (container_label_io_kubernetes_pod_name) 
 group_left (version)
 label_replace(node_exporter_build_info, "container_label_io_kubernetes_pod_name", "$1", "pod", "(.+)")

All of this might seem complex at first but after you try out these concepts for yourself they quickly become much easier to understand, apply and to reason about.