How To Call Kubernetes API from Go - Types and Common Machinery

The official Kubernetes Go client comes loaded with high-level abstractions - Clientset, Informers, Cache, Scheme, Discovery, oh my! When I tried to use it without learning the moving parts first, I ran into an overwhelming amount of new concepts. It was an unpleasant experience, but more importantly, it worsened my ability to make informed decisions in the code.

So, I decided to unravel client-go for myself by taking a thorough look at its components.

But where to start? Before dissecting client-go itself, it's probably a good idea to understand its two main dependencies - k8s.io/api and k8s.io/apimachinery modules. It'll simplify the main task, but that's not the only benefit. These two modules were factored out for a reason - they can be used not only by clients but also on the server-side or by any other piece of software dealing with Kubernetes objects.

API Resources, Kinds, and Objects

First, a quick recap. Familiarity with the following concepts is vital for the success of the further discussion:

• Resource Type - loosely, an entity served by a Kubernetes API endpoint: pods, deployments, configmaps, etc.
• API Group - resource types are organized into versioned logical groups: apps/v1, batch/v1, storage.k8s.io/v1beta1, etc.
• Object - a resource instance - every API endpoint deals with objects of a certain resource type.
• Kind - every object returned or accepted by the API must conform to an object schema - a certain composition of attributes defined by its kind: Pod, Deployment, ConfigMap, etc.

It's also important to differentiate between objects in a broad sense and Kubernetes "first-class" Objects - persistent entities like Pod, Service, or Secret serving as a record of intent for the cluster. While every API object must have an API version and kind attributes for the sake of its serialization and deserialization, not every API object is a "first-class" Kubernetes Object.

Module k8s.io/api

Go is a statically typed programming language. So, where do all the structs corresponding to Pods, ConfigMaps, Secrets, and other first-class Kubernetes Objects live? Right, in k8s.io/api.

Despite the loose naming, the k8s.io/api module seems to be solely for API type definitions. It's full of concrete structs closely resembling those YAML manifests we all know and love:

package main

import (
  "fmt"

  appsv1 "k8s.io/api/apps/v1"
  corev1 "k8s.io/api/core/v1"
)

func main() {
  deployment := appsv1.Deployment{
    Spec: appsv1.DeploymentSpec{
      Template: corev1.PodTemplateSpec{
        Spec: corev1.PodSpec{
          Containers: []corev1.Container{
            { Name:  "web", Image: "nginx:1.21" },
          },
        },
      },
    },
  }

  fmt.Printf("%#v", &deployment)
}

The module defines not only the top-level Kubernetes Objects like the Deployment above but also numerous auxiliary types for their inner attributes:

// PodSpec is a description of a pod.
type PodSpec struct {
  Volumes []Volume `json:"volumes,omitempty" patchStrategy:"merge,retainKeys" patchMergeKey:"name" protobuf:"bytes,1,rep,name=volumes"`
  
  InitContainers []Container `json:"initContainers,omitempty" patchStrategy:"merge" patchMergeKey:"name" protobuf:"bytes,20,rep,name=initContainers"`
    
  Containers []Container `json:"containers" patchStrategy:"merge" patchMergeKey:"name" protobuf:"bytes,2,rep,name=containers"`

  EphemeralContainers []EphemeralContainer `json:"ephemeralContainers,omitempty" patchStrategy:"merge" patchMergeKey:"name" protobuf:"bytes,34,rep,name=ephemeralContainers"`

  RestartPolicy RestartPolicy `json:"restartPolicy,omitempty" protobuf:"bytes,3,opt,name=restartPolicy,casttype=RestartPolicy"`
    
  ...
}

All the structs defined in the k8s.io/api module come with json and protobuf annotations. But be careful:

• Marshaling into JSON is supported.
• Protobuf serialization is discouraged - produced results will likely be incompatible with the existing API servers (see README for more).

Summarizing, the k8s.io/api module is:

• Huge - 1000+ structs describing Kubernetes API objects.
• Simple - almost no algorithms, only "dumb" data structures.
• Useful - its data types are used by clients, servers, controllers, etc.

Module k8s.io/apimachinery

Unlike the narrowly-scoped k8s.io/api module, the k8s.io/apimachinery module is rather manifold. The README describes its purpose as:

This library is a shared dependency for servers and clients to work with Kubernetes API infrastructure without direct type dependencies. Its first consumers are k8s.io/kubernetes, k8s.io/client-go, and k8s.io/apiserver.

It'd be hard to cover all the responsibilities of the apimachinery module in a single post without being too shallow. So instead, I'll talk about packages, types, and functionality from this module I stumble upon in the wild the most often.

Useful Structs and Interfaces

While the k8s.io/api module focuses on the concrete higher-level types like Deployments, Secrets, or Pods, the k8s.io/apimachinery is a home for lower-level but more universal data structures.

For instance, all these common attributes of Kubernetes Object: apiVersion, kind, name, uid, ownerReferences, creationTimestamp, etc. If I were to construct my own Kubernetes Custom Resource, I wouldn't need to define data types for these attributes myself - thanks to the apimachinery module.

The k8s.io/apimachinery/pkg/apis/meta package defines two handy structs - TypeMeta and ObjectMeta that can be embedded into a user-defined struct, making it look much like any other Kubernetes Object.

Additionally, the TypeMeta and ObjectMeta structs implement meta.Type and meta.Object interfaces that can be used to point to any compatible object in a generic way.

Another handy type defined in the apimachinery module is the interface runtime.Object. Due to its simplistic definition, it may look useless:

// pkg/runtime

type Object interface {
  GetObjectKind() schema.ObjectKind
  DeepCopyObject() Object
}

But in reality, it's used a lot! Kubernetes code was written long before Go got the support of true generics. So, the runtime.Object is much like the traditional interface{} workaround - it's a generic interface that is widely type-asserted and type-switched in the codebase. And the actual type can be obtained by checking the kind of the underlying object.

More useful apimachinery types:

• PartialObjectMetadata struct - combination of meta.TypeMeta and meta.ObjectMeta as a generic way to represent any object with metadata.
• APIVersions, APIGroupList, APIGroup structs - remember the API exploration exercise with kubectl get --raw /apis? These and similar structs are used for types that are Kubernetes API resources, but not Kubernetes Objects (i.e., they have kind and apiVersion attributes but no true Object metadata).
• GetOptions, ListOptions, UpdateOptions, etc. - these structs represent arguments for the corresponding client action on resources.
• GroupKind, GroupVersionKind, GroupResource, GroupVersionResource, etc. - simple data transfer objects - tuples containing group, version, kind, or resource strings.

Unstructured struct

Yes, you've heard it right. But jokes aside, it's another important and widely used data type.

type Unstructured struct 🤔

Ok, what's next, Kubernetes? Untyped type? Unsigned sign?

— Ivan Velichko (@iximiuz) January 23, 2022

Working with Kubernetes Objects using concrete k8s.io/api types is convenient, but what if:

• You need to work with Kubernetes Objects in a generic way?
• You don't want to or cannot depend on the api module?
• You need to work with Custom Resources that aren't defined in the api module?

The unstructured.Unstructured struct for the rescue! This struct allows objects that do not have Go structs registered to be manipulated as generic JSON-like objects:

type Unstructured struct {
  // Object is a JSON compatible map with
  // string, float, int, bool, []interface{}, or
  // map[string]interface{} children.
  Object map[string]interface{}
}

// And for the list of objects you can 
// use the UnstructuredList struct.
type UnstructuredList struct {
  Object map[string]interface{}

  Items []Unstructured
}

Under the hood, these two structs are just map[string]interface{}. However, they come with a bunch of handy methods simplifying nested attribute access and JSON marshaling/unmarshaling.

Type Conversion - Unstructured to Typed and vice versa

Naturally, a need for converting an unstructured object into a struct of a concrete k8s.io/api type (or vice versa) can arise. The runtime.UnstructuredConverter interface and its default implementation DefaultUnstructuredConverter can help you with that:

type UnstructuredConverter interface {
  ToUnstructured(obj interface{}) (map[string]interface{}, error)
  FromUnstructured(u map[string]interface{}, obj interface{}) error
}

Object Serialization to JSON, YAML, or Protobuf

Another tedious task when working with an API from a statically typed language is marshaling and unmarshaling data structures into and from their wire representation.

A non-trivial amount of apimachinery code is dedicated to this task:

// pkg/runtime

// Encoder writes objects to a serialized form
type Encoder interface {
  Encode(obj Object, w io.Writer) error
  Identifier() Identifier
}

// Decoder attempts to load an object from data.
type Decoder interface {
  Decode(
    data []byte,
    defaults *schema.GroupVersionKind,
    into Object
  ) (Object, *schema.GroupVersionKind, error)
}

type Serializer interface {
  Encoder
  Decoder
}

Noticed these Object's in the snippet above? Yep, those are runtime.Objects aka Kind-able interface{} instances.

Scheme and RESTMapper

The runtime.Scheme concept pops up here and there when working with client-go, especially when writing controllers (or operators 🤔) that deal with custom resources.

It took me a while to understand its purpose. However, approaching things in the right order helped.

Think about the potential implementation of Unstructured to Typed conversion: there is a JSON-like object, and a corresponding object of some concrete k8s.io/api type needs to be created from it. Probably, the very first step would be to figure out how to create an empty instance of the typed object using the kind string.

A naive approach could look like a huge switch statement over all possible kinds (and API groups, actually):

import (
  appsv1 "k8s.io/api/apps/v1"
  corev1 "k8s.io/api/core/v1"
)

func New(apiVersion, kind string) runtime.Object {
  switch (apiVersion + "/" + kind) {  
  case: "v1/Pod":
    return &corev1.Pod{}
  case: "apps/v1/Deployment":
    return &appsv1.Deployment{}
  }
  ...
}

A smarter approach is to use reflection. Instead of the switch, a map[string]reflect.Type can be maintained for all registered types:

type Registry struct {
  map[string]reflect.Type types
}

func (r *Registry) Register(apiVersion, kind string, typ reflect.Type) {
  r.types[apiVersion + "/" + kind] = typ
}

func (r *Registry) New(apiVersion, kind string) runtime.Object {
  return r.types[apiVersion + "/" + kind].New().(runtime.Object)
}

The advantage of this approach is that it requires no code generation and that new type mappings can be added at runtime.

Now, consider a deserialization problem: a piece of YAML or JSON needs to be converted into a Typed object. The very first step - object creation - will be very similar.

Turns out, creating empty objects by their API Groups and kinds is such a frequent task that it got its own component in the apimachinery module - runtime.Scheme:

// Scheme defines methods for serializing and deserializing API objects, a type
// registry for converting group, version, and kind information to and from Go
// schemas, and mappings between Go schemas of different versions. 
type Scheme struct {
  gvkToType map[schema.GroupVersionKind]reflect.Type
  
  typeToGVK map[reflect.Type][]schema.GroupVersionKind
  
  unversionedTypes map[reflect.Type]schema.GroupVersionKind
  
  unversionedKinds map[string]reflect.Type

  ...
}

The runtime.Scheme struct is such a registry containing the kind to type and type to kind mappings for all kinds of Kubernetes objects.

The runtime.Scheme struct is actually very powerful - it has a whole bunch of methods and implements some foundational interfaces like:

// ObjectTyper contains methods for extracting 
// the APIVersion and Kind of objects.
type ObjectTyper interface {
  ObjectKinds(runtime.Object) ([]schema.GroupVersionKind, bool, error)
  Recognizes(gvk schema.GroupVersionKind) bool
}

// ObjectCreater contains methods for instantiating
// an object by kind and version.
type ObjectCreater interface {
  New(kind schema.GroupVersionKind) (out Object, err error)
}

However, the runtime.Scheme is not almighty. It has mappings from kinds to types, but what if instead of kind only the resource name is known?

That's where the RESTMapper kicks in:

type RESTMapper interface {
  // KindFor takes a partial resource and returns the single match.  Returns an error if there are multiple matches
  KindFor(resource schema.GroupVersionResource) (schema.GroupVersionKind, error)

  // KindsFor takes a partial resource and returns the list of potential kinds in priority order
  KindsFor(resource schema.GroupVersionResource) ([]schema.GroupVersionKind, error)

  ...
  
  ResourceSingularizer(resource string) (singular string, err error)
}

The RESTMapper is also some sort of a registry. However, it maintains mapping of resources to kinds. So, feeding a string like apps/v1/deployments to the mapper gives the API Group apps/v1 and the kind Deployment. The RESTMapper also can deal with resource shortcuts and singularization: po, pod, and pods can be registered as aliases for the same resource.

Field and Label Selectors

Types, creation, and matching logic for fields and labels also live in the apimachinery module. For instance, here is what can be done with the k8s.io/apimachinery/pkg/labels package:

lbl := labels.Set{"foo": "bar"}
sel, _ = labels.Parse("foo==bar")
if sel.Matches(lbl) {
  fmt.Printf("Selector %v matched label set %v\n", sel, lbl)
}

API Error Handling

Working with the Kubernetes API in code is impossible without handling its errors properly. The API server might be completely gone, requests may be unauthorized, objects might be missing, and concurrent updates may conflict. Luckily, the k8s.io/apimachinery/pkg/api/errors package defines some handy utility functions to deal with the API errors. Here is an example:

_, err = client.
  CoreV1().
  ConfigMaps("default").
  Get(
    context.Background(),
    "this_name_definitely_does_not_exist",
    metav1.GetOptions{},
  )
if !errors.IsNotFound(err) {
  panic(err.Error())
}

Miscellaneous Utils

Last but not least, the apimachinery/pkg/util package is full of useful stuff. Here are some examples:

• util/wait package eases the task of waiting for resources to appear or to be gone, with retries and proper backoff/jitter implementation.
• util/yaml helps to unmarshal YAML or convert it into JSON.

Summarizing

The k8s.io/api and k8s.io/apimachinery packages is a good starting place to learn how to work with Kubernetes objects in Go. If you need to write your first controller, jumping straight to client-go, or even to controller-runtime or kubebuilder will likely make the learning experience too rough - there might be way too many knowledge gaps. However, taking a look and playing around with the api and apimachinery packages first will help you keep peace of mind during the rest of the journey 🧘

Stay tuned

It's been three articles, and I haven't touched client-go yet. Next time, it'll be an article about the client, I promise!