Brian Grant is Borg and…

Brian Grant, the core developer of Borg and Kubernetes, recently shared the origins of Kubernetes and Borg on Twitter and why Kubernetes was designed to be today’s architecture.

1. Why is there a Pod?

Borg allowed Jobs to be submitted to the cluster at the earliest time, and eventually formed Alloc Instance running entity. Later, in practice, it was found that users would always tie multiple Jobs together to run. The common mode was log and side car of cache. Therefore, Kubernetes design considers multi-container entity Pod as the basic unit of scheduling and management. This design proved to be a very correct decision in subsequent use.

2. The ubiquitous Key-Value

Borg originally only had the concept of key-value attribute on Machine (that is, Node in Kubernetes). Borgmon (similar) exposed metrics had the location, environment, cluster and other information of running services. Job did not have key-value. Borg users place attributes in Job name, resulting in a very long name, and need to customize various character processing to obtain key-value. So at the beginning of Kubernetes’design, all resource objects have scalable key-value, and all resource entities can be refined through label selector without duplication of management.

3. Annotations?

Borg Job has a note field that can be used to store application-specific information, such as DNS TXT records. But a single field is not enough to store more information, so Kubernetes put forward the concept of Annotation. Developers and users can save very complex data in annotation to extend their business logic. Although Label and Annotation are very similar, there are still differences in usage scenarios between them, and it makes sense to separate them.

4. Workload Controllers

What was very different from other container orchestration systems at that time was the declarative API design of Kubernetes and automated controllers. When Kubernetes was initially open source, only one controller was implemented for Replication Controller, and Borg actually supported only one controller: Job (which can be composed of multiple tasks). Job can implement a variety of workloads: Deployment, DaemonSet, StatefulSet and so on. As a result, it is very complex to configure and control logic, so it separates Deployment, DamonSet, StatefulSet and other workloads in Kubernetes. Independent controllers can be very easy to expand, and with community push operators, users can easily customize controllers to implement their own business logic.

& gt; The key is to explicit model state such that it can be externally manipulated.

5. asynchronous controllers

BorgMaster uses synchronous, transactional, edge-triggered state machines, making it difficult to extend, evolve, and scale. For example, some processing logic modifies the resource object beyond the scope of a single transaction. Adding state makes the client unavailable and it is difficult to add new state.

So Kubernetes uses an asynchronous controller and uses the same resource object to store both the desired state of the user and the information about the current state (i.e. spec and status we see). This decision makes apiserver an interaction center, where all objects can be viewed and modified by accessing the apiserver API, and apiserver a source of truth.

& gt; The entire system can now be described as an unbounded number of independent asynchronous control loops reading and writing from/to a schematized resource store as the source of truth. This model has proven to be very resilient, evolvable, and extensible.

6. Schduling scheduling
Scheduling has changed from a single task placement to a very complex resource management field: queue, multi-scheduler, resource quota and resource sharing, automatic scaling, oversell, task placement, rescheduling, preemption, etc.

Because of the flexibility of containers, most users using Kubernetes are more concerned about how to provide the deployment and release efficiency and stability of applications, rather than the resource utilization of clusters. Therefore, QoS, oversell and autoscaling will be so important.

7. Watch mechanism

Google’s internal systems (such as Chubby, Colossus, BigTable) all have watch mechanisms. Later, a more standardized watch definition and interface were abstracted to prevent different systems from making wheels independently.

Kubernetes initially chose etcd as the backend because it was similar to Chubby and provided a watch interface.

Previously, systems in the industry generally preferred to use message queues for notifications. Why did Kubernetes choose the watch mechanism? Because the controller of Kubernetes restarts from scratch, it is necessary to know the initial state (i.e., the current state of the cluster), and correctly handle missing events and outdated events; in addition, the introduction of message queues will increase the complexity of the architecture and operation of the entire system.

For publishing systems, they usually rely on many external infrastructure components: distributed KV to store configurations and selectors, database to store, distributed cache to speed up, message queue to process events (and message persistent storage). Installation and maintenance of so many components will be very complex, but Kubernetes has a unique way, using etcd as an external dependency, as a database and distributed KV, using watch mechanism instead of message queue, and using cache inside apiserver instead of distributed cache, which reduces the complexity of the whole system.

Watch mechanism can be said to be one of the core technologies of Kubernetes. It not only makes controllers possible, which greatly increases the scalability of the whole system, but also reduces the dependence of external message queues. The programming mode and framework based on operator are also very easy to use.

8. summary

Although a complex system will be more complex in the initial design stage, it will continue to evolve with the development of time. As new functions are added, it will gradually become larger and larger, which is different from the original design, and because of many design defects, it will become more and more complex to expand in the future. At this time, there will be some major refactoring of the system, and it may also be rewritten to redesign the whole architecture based on previous experience.