Chris’ Intro to Kubernetes Christopher Browne GTALUG, January 2020 The Agenda A wee intro to k8s (because kubernetes is really long to type!) Some major components A progression of service evolution Chris’ crazy batch jobs, and my challenges Some useful tools k8s Originally a Google project called “Borg” It’s all about orchestrating the running of services in containers The “Borg” metaphor… Robotic services, lots of them, with sub- and sub-sub-services. Launched automatically. Relaunched on failure. Spread across a series of Borg Kubes. If one Kube blows up, there’s more! Kubernetes is a Greek word meaning “helmsman” or “pilot”; the one that steers a ship It uses Docker (usually); it’s the “opposite side” Major Components Starts with a Container system Docker, containerd, cri-o, rtklet, frakti, AWS Firecracker, gVisor, … Alternatives claim to be faster or more secure… are they??? Etcd - distributed key/value store for configuration DNS server - automatically managing hostnames inside the cluster Kube Proxy - maintains network rules so outside hosts can get at services inside the cluster Less Visible Components Kube API server - used to control node activities Kube scheduler - launches and destroys Pods Node controller - monitors each node (am I alive?) Replication Controller - add/drop Pods based on policy Endpoint Controller - connects Pods to form Services Visible or not, the Real Point of Kubernetes is to have the clusters self-monitoring and self-administering. A Progression of Sorts of Services Pods - a set of related containers sharing storage + IP Multiple containers so you add extra services via extra containers Use fluentd to aggregate collect logs, forward to ElasticSearch CronJob so a process runs periodically Job (run the thing once, more or less) DaemonSet (so each node has an instance of a service pod) ReplicationController - obsolesced ReplicaSet - obsolesced Deployment defining a ReplicaSet StatefulSet (needs stable network or storage) More Sophisticated Services Operators (think: “system operators”) PostgreSQL Operator Deploy multiple replica nodes (sync and async replication) Run backups using pgBackRest (including to S3) Automated failover via distributed consensus subsystem Automated recovery based on replicas or backups Component for managing users and permissions Capability to scale up by cloning DB clusters Sets up pgBouncer for connection pooling Includes health monitoring componentshttps://github.com/operator-framework/awesome-operators Yaml, Yaml Everywhere Manifests about how the pods in your services are configured, are written in YAML Configuration to pass to the pods, also YAML You are in a maze of twisty YAMLs, all nearly the same… I think I want an engine for generating YAML way more automatically (you’ll see why later…) Well, that’s why they have Helm - http://helm.sh Lots of systems use the Go language system for templating {{- if (.Files.Glob “myfile.conf”) }} {{ (.Files.Glob “myfile.conf”).AsConfig | indent 2 }} Configuration Pains Some things are painful… Twisty maze of YAML, Almost the same… Environment Variable Overlay Don’t call a config element PGPORT or PGHOST or PGDATABASE It’s kinda like running within a cron job... My Job Journey The stuff I build tends to be shell scripts running SQL queries Not a great fit for being an “endpoint” or a “service” Best fit appears to be k8s “Jobs” I have activities split into a series of job steps that pretty much need to be run serially, one after another The “best fit” isn’t to run a bunch of k8s Jobs, either No dependency system provided No indication that this is an area to expect much evolution So, I wrote 2 shell scripts as a “batch job controller” Receives a list of job steps to be run, JOBSTEPS Verifies that the steps are all valid (abort if not) Iterates across the steps in JOBSTEPS, in order: echo “start $step” > $LAUNCHMARKFILE Poll, waiting for the step to indicate it is done via seeing “done” in $JOBSTATEFILE job-controller.sh gets run by 1 container in the Job pod, logging as it goes Run-controller.sh Job-controller.sh Batch-job.sh Job-runner.sh There are a couple dozen containers, one per job step. Each runs via: job-runner.sh my-step-name [immediate] job-launcher function in batch-job.sh: Grabs configuration, turns it into a set of environment variable values Wait Loop; watch until $LAUNCHMARKFILE contains “start my-step-name”, It’s my turn now :-) Run the logic for this job step echo “done” > $JOBSTATEFILE to launch the next step A little Column A, a Little Column B... This Job pod has (at the moment) 22 containers Boy, that’s a lotta YAML!!! That’s a lotta containers!!! Some aspects are pretty clean Each job step gets its own distinct STDOUT logging It’s easy to identify which step is which At any given time, only one container is busy; others are lazy There’s only one Pod There’s only one container definition, used by all 22 containers Using one Pod means all the containers have easy access to common data files, no need to resort to GFS/gluster/NFS to share data Not terribly parallel, but that’s OK Helm Loosely, a package manager for k8s Implements Charts, that comprise sets of k8s Pods Provides a way of deploying locally-custom config, that, attached to a Chart, comprises a software Release Templates help you mix your configuration with defaults to provide a complete k8s manifest Requirements document dependencies (“we need a bunch of other Charts installed”) Some support for upgrading to newer versions of software Shell Tools Because k8s encourages long command lines, so you want any help you can get… Command completion - built-in source <(kubectl completion bash) source <(kubectl completion zsh) github.com/derailed/k9s - curses UI to manage clustergithub.com/ahmetb/kubectx - CLI help to smoothly switch between clusters (kubectx) and namespaces (kubens)github.com/jonmosco/kube-ps1 - k8s prompts for bash/zshgithub.com/abrochard/kubel - k8s buffer in Emacs Emacs Toolsgithub.com/abrochard/kubel - k8s buffer in Emacsgithub.com/chrisbarrett/kubernetes-el - k8s “porcelain”github.com/abrochard/kubernetes-helm - run Helm commands from within Emacs