A Meditation On Kubernetes

Dear Reader,
Fair warning; there be nerd dragons ahead. I'm talking deeply nerdy musings on Kubernetes. Enter at your own risk. 

 

 

My past few weeks have been getting back into the swing of things with the wonderful platform that is K8s. There's nothing quite like sitting down in front of a giant pile of hardware boards strapped to a mainboard with the lonesome spaghetti of network cabling leading into the nethers of my house where the core network resides. Call me old fashioned, but I enjoy doing things the hard way with hardware sometimes. What follows is a meditation on Kubernetes. 

 

Kubernetes has taken the corporate world by storm. Pretty much every major org at this point has a container strategy and quite a few have big multi-cluster deployments of various stripes. I actually spent a few years managing these behemoths when the implementations were getting started. Back then, my colleagues and I quickly coined an internal meme: "You are not Google," and used it liberally, even though these were some huge organizations. Even within large enterprises, as architects, we saw a significant deficiency in the understanding of the Kubernetes why, what we will generously call "the value prop."

 

K8s was initially written at, for, and by Google. This in itself is entirely neutral. However, as a tool that was clearly made for enormous scale (and due to this, some of the paradigms that it forced upon the industry) were silly at best and counterproductive at worst. But hey, I'm not here to rag on the merits or failings of K8s. Lots of people happily will do that without me, so let's look at the system as it exists today - what are the use cases, how do we go about using it, and why the title mentions elephants.

 

 

K8s is an abstraction that, at its core, is a resource-and-time-sharing system. If you're old enough to remember what a time-sharing system actually is, you're probably older than I am by a factor of at least two. But in the best case scenario, Kubernetes services are stateless, and either run for a predetermined amount of time and then exit producing a result, or run in a loop forever. The only functional difference between this paradigm and that old, hulking mainframe in the museum is that punch cards are arguably a better man-machine interface than yaml. In fact, if we were to update the analogy a bit, Kubernetes is not a tool, it's an operating system. It has a kernel. It has a job scheduler. It has the concept of processes. Really instead of abstracting away specific hardware components (the memory, the i/o, etc.), it abstracts away compute nodes. The api is the user interface, the UI is, well, the UI. Everything holds.

 

I'm sure folks remember the Unix wars of the 90s or the eventual dominance of NT in the corporate space. Twenty years on, Macs have finally broken into the corporate world and outside of certain applications, linux dominates the server side. The runs-on-hardware side of operating systems hasn't necessarily changed much in the past two decades. Fundamentally, outside of hardware compatibility, there's little difference between Linux 2.4 and 6.1 or NT 8 and 4.x respectively. Kubernetes as an OS is much closer to the Inferno/Plan9 family of OSs in terms of its end goal of spreading resources across multiple machines.

 

Where this lands us is that kubernetes is a simulation of outgrowth of an ancient time-sharing system, running jobs that are themselves encapsulated in abstractions, of what is itself a clone of a time-sharing system (remember the origins of *NIX), running ON TOP of a clone of a time-share system on multiple computers. And I'm not even getting into the levels of abstraction layers below the OS. I can think of at least three.

 

A lot of folks, when talking about abstractions, fall down because they start talking about performance. Performance, for better or for worse, has historically been fungible. There was a time that applications would load extremely slowly because we were running them on literal potatoes. Now, our applications are not performant - not because the hardware is non-performant, but we're either doing something furiously complex, or, more likely, poorly optimized. This is not, in and of itself, a problem that has anything to do with, for, or against K8s. However, if you are to operate on and work with a modern enterprise system(tm), you are faced with the prospect of going back to working on essentially a shared mainframe. This has its own architectural implications that I feel are not necessarily grasped by man - causing a gross misunderstanding of the paradigm that they are operating under.

 

So let's break down the bits that make running your software on a massive time-sharing system a bit weird - or at least distinctly not like we think about running it usually.

 

Best practice dictates that kubernetes services should generally be stateless. This is a problem because Linux (or Windows. For the record, Windows is still a valid K8s target. The author sat with a colleague for years while the colleague implemented a windows-containers-on-windows-hosted-k8s madness. He was successful. For the purposes of this article, NT is assumed to an abstraction layer that is virtually indistinguishable from Linux.) is generally a stateful system, but let's ignore that for now. Outside of actual application stat (we'll get to that later), there's also the configuration. That's technically a part of your app state. This falls into one of the following.

 

 

 

You can absolutely fall outside of this pretty easily. But your sysops/devops/platform engineer is going to end up hating you. Or kubernetes. Or possibly both. Pursue exceptions at your own risk.

 

Except when it is. Jumping off from the previous point, a timesharing system absolutely can be pressed into the role of a minicomputer. It is not, however, going to like it. Remember the whole "changing your scheduler to make your desktop performance not suck" bit on Linux? Pepperidge Farms the author certainly does.

 

Let's also consider the fact that vertical scaling in K8s can generously be described as "somewhat complicated." Unlike the beowulf clusters of yore, most applications that are slammed into a docker container and then deployed via helm chart are, by no means, multi-node aware. We'll be lucky if they're multithread-aware, but that's another rant for another time. We try to work around this by vaguely waving a complicated architecture diagram at the problem and loudly proclaiming "microservices, conveniently ignoring the complexity that this causes in many scenarios.

 

But before we get there, what of verticality? Well, assigning a kubernetes job to a particular node is doable. But from the ops perspective that means your nice cluster that can run jobs wherever and do all the magic scheduling (mostly) by itself, is now weirdly limited. And for the application developer, this removes part of Kubernetes value prop. Suddenly you do have to care about CPU speeds and RAM latencies. But isn't thjis what we're running away from? Because of all of this, best practice dictates that we must scale processes horizontally. Which leads us neatly into the next section.

 

Remember the meme that I alluded to earlier? "You are not Google?" For a giant enterprise actually doing big data analysis and running a lot of relatively short-lived services, K8s is the best thing since sliced bread. If your data center engineers are forming raiding parties to fight the feral network engineers that have set up their own civilization within the nethers of your datacenter then you may be approaching the scale that Kubernetes starts to make sense at. And damn the abstractions, full speed ahead, you can cram pretty much everything in there, because yes, clustering a bunch of servers makes your platform engineers' lives a heck of a lot easier.

 

But what if you're small? These days, it's not about using or not using K8s, there's plenty of services out there that make the proposition of running a production-scale cluster not suck. AWS EKS comes to mind. Kubernetes, like a decent chunk of cloud and cloud-related-accessories makes sense when you're small enough that you can pick a single model of application and generally stick with it. Where this starts falling apart is when you have to suddenly have a different paradigm slammed into it. Exceptions upon exceptions leading to technical debt. Much like a large enterprise, you can afford to treat everything as the same.

 

But when you're scaling, and the exceptions will inevitably pile up? Well, code optimization... Wait, who am I kidding. Let's be real, nobody really optimizes code anymore. (Yes, yes, if you're actively doing anything hardware-related, there's optimization. But that's a tiny fraction of a fraction of the overall software market.) Minifcation and gzip is not optimization. 

Kubernetes is neat. But much like the rest of the tools we have in this industry, it's prone to wild misuse. For example, it's probably not a great idea to store customer IDs and passwords in the same database that you're storing your production data in. Just a thought, maybe. Likewise, running state-full services that require very dedicated resources on k8s, is probably not a great idea. Maybe.

 

Too many architects and CTOs will proudly proclaim that they are now "all in" on kubernetes. Or insert_tech_here. This sort of rhetoric makes sense if you're a consulting shop making your money in a particular niche. Do what you know how to do well. We, at BTS, see this hyperfocus as an unacceptable risk, but that's not the case everywhere or for everybody.

 

In what we consultants broadly call "the industry," Kubernetes really needs to be not thought of as a tool. Tools come and go. The definition of a platform is somewhat closer. But the closest technological metaphor for this layer of abstraction is "operating system." And because Kubernetes is an operating system, it will follow the cycle that all operating systems go through at one point or another. Development, growth, adoption, and eventual rejection, or simply fading away.

 

When I talk about cloud deployments, a term that has crept into my speech these last few years has been "legacy cloud deployments." In terms of industry adoption, Kubernetes seems to be around the same space where cloud was six or seven years ago. The trickle of "legacy Kubernetes deployments" showing up in the market is still just that, a trickle. This, too, is growing.

 

Back to the hardware sitting on my desk. It's been a few hours. I've spun up a cluster, run some jobs and I've shut it back down again. The nodes are re-imaging their SD cards back to their normal ARMBIAN install. Even though the ARM boards powering it are fairly powerful, I can't get over the layers upon layers of abstraction that are required to get anything done and that the singular home server whirring away in my garage can handle all of my household compute needs without so much as breaking a sweat. What always catches me by surprise about Kubernetes is the initial fragility of the system. But once you make it work, it tends to keep working. It's just that process takes a while. If I was just hosting this on EKS, this exercise would have taken me half the time, but would have I looked at it as an OS? I don't think so.