Kubernetes operators using Kubebuilder
In this post, we will be going over the fastest no-frills approach to getting your operator off the ground using kubebuilder. The post assumes knowledge of the following:
- Kubernetes and how it works
- Kubernetes custom resource definitions
- Kubernetes Operators and reconciliation loops
- Setting up a local cluster, I use
kindfor my k8s orchestration needs - Golang
The task is to create an operator that operates on a Kubernetes CRD TodoList. It listens on the pods available in the system. If there are any pods with the same name as the TodoList, it marks the status as True.
This operator only operates on the operator-namespace namespace.
The first step is to install kubebuilder using the following command:
curl -L -o kubebuilder https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH) && chmod +x kubebuilder && mv kubebuilder /usr/local/bin/Check if it’s installed properly by running kubebuilder version .
Install the kubernetes cluster using kind
kind create cluster --name operatorsSetup the initial project, APIs, groups and kinds
kubebuilder init --domain sarmag.co --repo sarmag.co/todo
kubebuilder create api --group todo --version v1 --kind TodoListThis will create the required scaffolded project, group, API and kind.
Once that’s done, there are 2 main files that we have to update
- api/v1/todolist_types.go
- internal/controller/todolist_controller.go
In the todolist_types.go file, update the required specification and status of the CRD.
type TodoListSpec struct {
Task string `json:"task,omitempty"`
}
type TodoListStatus struct {
IsCompleted bool `json:"status,omitempty"`
}You can refer to the file here https://github.com/gauravsarma1992/todolist-k8s-operator/blob/main/api/v1/todolist_types.go#L24-L30.
In the todolist_controller.go file, update the reconcilation logic with the following:
func (r *TodoListReconciler) Reconcile(ctx context.Context, req ctrl.Request) (result ctrl.Result, err error) {
var (
todoList todov1.TodoList
podList corev1.PodList
logger logr.Logger
isCompleted bool
)
logger = log.FromContext(ctx)
logger.Info("Reconciling TodoList")
if err = r.Get(ctx, req.NamespacedName, &todoList); err != nil {
logger.Error(err, "Error in fetching Todolist")
err = client.IgnoreNotFound((err))
return
}
if err = r.List(ctx, &podList); err != nil {
logger.Error(err, "Error in fetching pods list")
return
}
for _, item := range podList.Items {
if item.GetName() != todoList.Spec.Task {
continue
}
logger.Info("Pod just became available with", "name", item.GetName())
isCompleted = true
}
todoList.Status.IsCompleted = isCompleted
if err = r.Status().Update(ctx, &todoList); err != nil {
logger.Error(err, "Error in updating TodoList", "status", isCompleted)
return
}
if todoList.Status.IsCompleted == true {
result.RequeueAfter = time.Minute * 2
}
return
}You can refer to the complete file here https://github.com/gauravsarma1992/todolist-k8s-operator/blob/main/internal/controller/todolist_controller.go#L48-L89.
Once the controller is done, you can create and deploy your code to the kubernetes cluster already created.
make manifests
make install
make runThis will run the manager with the required reconciliation logic hooked in to the k8s cluster.
Testing time!!
Create the todolist object
apiVersion: todo.sarmag.co/v1
kind: TodoList
metadata:
name: jack
namespace: operator-namespace
spec:
task: jackThis creates a todolist object called jack in the k8s namespace namedoperator-namespace .
You can refer to the full file here https://github.com/gauravsarma1992/todolist-k8s-operator/blob/main/samples/todo.yml#L1.
apiVersion: v1
kind: Pod
metadata:
name: jack
namespace: operator-namespace
spec:
containers:
- name: ubuntu
image: ubuntu:latest
# Just sleep forever
command: [ "sleep" ]
args: [ "infinity" ]You can refer to the full file here https://github.com/gauravsarma1992/todolist-k8s-operator/blob/main/samples/pod.yml#L1-L12.
Once you create the pod, you will see this specific log line https://github.com/gauravsarma1992/todolist-k8s-operator/blob/main/internal/controller/todolist_controller.go#L48-L89. This means that the operator was able to find a pod with the same name as the task.
Watching on pod events
One problem with the current approach is the operator only listens on the events of the TodoList type, whereas it should also monitor the pod events so that it can update the state accordingly. In order to ensure the reconcilation loops when the pod events change, chain the following method on the manager.
func (r *MyController) SetupWithManager(mgr ctrl.Manager) (err error) {
err = ctrl.NewControllerManagedBy(mgr).
For(&todov1.TodoList{}).
Watches(&source.Kind{Type: &corev1.Pod{}}, &handler.EnqueueRequestForObject{}).
Complete(r)
return
}If you want to ensure that the operator only watch on the pods it has created, you can create the pod and set the OwnerReferences by calling SetControllerReference on the pod.
You can then create the manager by using the Owns method
func (r *MyController) SetupWithManager(mgr ctrl.Manager) (err error) {
err = ctrl.NewControllerManagedBy(mgr).
For(&todov1.TodoList{}).
Owns(&corev1.Pod{}).
Complete(r)
return
}Watching on external events
What if we want the reconcilation loop to run on external events as well?
You can create a goroutine which sends an event to the reconciliation loop every 5 seconds
func (r *TodoListReconciler) startTickerLoop(periodicReconcileCh chan event.GenericEvent) {
var (
ticker *time.Ticker
count int
)
ticker = time.NewTicker(time.Second * 5)
defer ticker.Stop()
for {
select {
case <-ticker.C:
periodicReconcileCh <- event.GenericEvent{Object: &todov1.TodoList{ObjectMeta: metav1.ObjectMeta{Name: "jack", Namespace: "operator-namespace"}}}
count += 1
if count > 100 {
return
}
}
}
}You can then change the manager setup to also watch on the periodReconcileCh channel
func (r *TodoListReconciler) SetupWithManager(mgr ctrl.Manager) (err error) {
var (
periodicReconcileCh chan event.GenericEvent
)
periodicReconcileCh = make(chan event.GenericEvent)
go r.startTickerLoop(periodicReconcileCh)
err = ctrl.NewControllerManagedBy(mgr).
For(&todov1.TodoList{}).
Watches(&source.Kind{Type: &corev1.Pod{}}, &handler.EnqueueRequestForObject{}).
Watches(&source.Channel{Source: periodicReconcileCh}, &handler.EnqueueRequestForObject{}).
Complete(r)
return
}You can hook in the above channel with external events, expose an API which can trigger the loop, etc.
If you want the reconciliation loop to requeue itself after some duration even when it’s successful, you can use RequeueAfter as shown here https://github.com/gauravsarma1992/todolist-k8s-operator/blob/main/internal/controller/todolist_controller.go#L112-L125.
