Cloud Native Computing Foundation Kubernetes Community Days Bangalore 2022, 11 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Easy way to test e2e on k8s cluster

Description

The existing Test Framework is intrinsically baked into Kubernetes source code repository with organically grown assumptions over the years that shape how tests are created and executed. This, however has presented some shortcomings including: Difficulty of consuming existing test framework packages from outside of the Kubernetes project, Misaligned framework design can cause dependencies on unnecessary resources, the requirements to use Ginkgo as a way to write tests.
To resolve these difficulties we have come up with a new framework called e2e-framework which is now residing under https://github.com/kubernetes-sigs e2e-framework provides ways that make it easy to express test functions that can programmatically test components running on a cluster. The two overall goals of this framework are to allow developers to quickly and easily assemble end-to-end tests and to provide a collection of support packages to help with interacting with the API-server. It provides a uniform API to work with typed and untyped Kubernetes API objects.

A

Hi, everyone welcome to kcd bangalore, 2022, I'm shweta working at vmware and in this session I am going to talk about new test framework for kubernetes.

A

Testing is a critical part of the software development life cycle, which enables more confident delivery of the product to the end customers. So in recent time we have seen large number of platform. Solutions are built on top of kubernetes, so how these products are getting tested before reaching to end customers, so quality assurance team might be doing manual or automated tests, or maybe both. But when it comes to automated it becomes very tricky because either one can use the existing kubernetes testing framework or write the test from scratch.

A

And if you have a tight deadline, then writing the test from the scratch wouldn't be a good idea.

A

So now the question is: why don't we use the existing kubernetes test framework, uh so there are a couple of challenges with kubernetes test framework. So when I say kubernetes test framework, it resides under github.com kubernetes kubernetes test. So it's a framework which has grown organically with kubernetes and it's harder to reuse.

A

If you are writing a component for kubernetes, and if you want to do testing, then chances are to rebuild the test from scratch and, moreover, if you want to reuse kubernetes test framework on your own components, it's very difficult to manage as it renders all the dependent packages, and I don't think people would prefer to keep all the rendering dependencies as it's very difficult to maintain.

A

Now you might be thinking, don't we have any other frameworks available in the market? Yes, we have several testing frameworks available already as open source projects. Let's see what are those so I have listed down some of the frameworks here.

A

One of them is cubetest2. Cubetest2 is used to orchestrate the steps necessary to run an end-to-end test against a cluster in ci or locally. So it's a localized framework to launch cluster and testers. It does not contain any test or any test solo boy. Sonobo is a diagnostic tool which makes it easier to understand the state of a kubernetes cluster by running a set of plugins. um It includes kubernetes conformance test also, so it's a customizable extendable way to generate informative reports about your customer.

A

Ginkgo ginkgo framework is mainly used for behavioral, driven development and used for unit testing integration, testing, performance, testing, etcetera.

A

um The kubernetes test framework is based on gingko framework, so all these listed frameworks are dedicatedly, focusing on one thing, so either cluster related test or just a framework which describes the test like gingko or to get some reports like cluster state.

A

So these challenges led us to have a new framework which should easy to reuse, which should be a portable one. Avoid direct dependencies on upstream kubernetes packages. Remove the dependencies on non-standard go based test project test packages. Now the question is: are we trying to replace upstream kubernetes stress framework which is already exist? The answer to this is no or never so what we are trying to do is to is to come up with the framework which makes easy to do end-to-end testing so that component developers don't reinvent the wheel again and again.

A

So this brings us here to showcase a more generic end-to-end testing framework called e2e framework. It comes under the kubernetes testing umbrella and it's a great framework to overcome many hurdles. It completely built using go testing and it has ability to integrate and leverage existing test components.

A

Let's see what are the, uh what are the features and what are their in endurance framework? So in e2e framework we have two frameworks, so one is test harness framework and another one is client helper pack framework, so test harness is the one you can use to design the test. So basically, it helps you to design the life cycle of a test. When I say life cycle, it includes from setting up an environment till tearing down the resources.

A

For example, if you want to test some features on cluster, the pre-requisite would be setting up the cluster right and run the test. On top of it, then bringing down the resources created uh during the test, execution process and finally tear down the cluster so test harness mainly helps you to express test as a series of customizable steps, and you can control the entire execution life cycle. Now, let's see why client helper helper framework is for you, people might have noticed the client spelling, so it's not starting with c. It starts with k.

A

So uh it's because we just wanted to make it as a helper framework for kubernetes related stuff, so it is called as a client helper framework. So client is a go package which is intend to provide abstract the complexities of project, kubernetes client code and it provides helper functionalities to make it dead, simple, to interact with clusters and the api servers.

A

uh One good thing about uh end-to-end framework: is it's not necessary to use both the framework so based on your requirement, you can choose either test harness or client helper or both. So if you don't have any dependent dependency on cluster, then you can just use the test harness framework now we'll see what are the features supported in this framework? After that? We'll walk you through the examples, so this framework provides a uniform api to work with typed and untyped kubernetes api objects by having simple client, which includes kubernetes resources, cred operations and control operations.

A

So you can easily create update or delete kubernetes resources. It even provides a type which allows test writers to install third-party tools. So we have integration with help charts. So there is a decoder functionality which provides a utility for creating kubernetes resources or objects from inputs like yaml files or json files. So this is very useful feature which reduces the verbosity and complexity of defining many types and understanding which packages to import may add a burden to test developers. So this feature um manages resources as yaml or json, and it has the benefits in maintainability.

A

uh This framework even provides you, the creation of client cluster for your test, so we have a uh support. Function, supported predefined function to create a kind cluster context propagation. This is one of the important feature in this framework. uh This is very useful to propagate the values, to the next steps or face of a test. So when we see example, I'll explain you how to use the context propagation, the framework also support parallel test execution using the flag, minus minus parallel.

A

This executes the series of test features in parallel, so based on your requirement, you can choose to run test serially or parallelly. Even one can filter the features using predefined um flag sets. So these are the available uh predefined flags. So, for example, if you want to run a particular feature, then just provide minus minus feature equal to the feature name. It will just run the run. Those features. If you want to skip some of the assessment, we can provide minus minuscape assessment equal to assessment name.

A

It will skip those assessments if I'm not wrong in gingko v1. um These filters are not available to use while running your test externally. So mostly they are enhancing. These features on niko version 2.

A

Okay, so let me give you the quick overview of end-to-end framework code base.

A

Let me open the vs code.

A

So this is the folder structure of e2e framework. So if you want to go through the design dock, you can refer this folder.

A

Package folder contains the implementation of test harness and client folder contains the implementation of kubernetes helper functions.

A

Let's go through the package.

A

The environment object is the heart of the framework which allows developers to define and implement features to be tested. This contains all the methods to define the life cycle of a test like setup before each test before each feature after each feature, finish, etc.

A

So, if you see this setup, it registers an environment function that are executed once prior to the environment being ready after each feature it. It registers the step functions which are executed after each feature finish. You can register the functions which are executed at the end of the test order.

A

Let me show you the command line, flag, supported by e3 framework, so these are the command line flags supported by this framework. So if you have your cluster already up and running by using minus minus cube config flag, you can provide the cube, config file path.

A

So client package.

A

To initiate a connection with the api server, you can use one of this function, so if you know already the cube conflict fact, you can use the new function. If you are using in cluster config, you can use new in cluster method.

A

If you want to set the kubernetes context, you can use the new with context name so under creators resources dot go. This is a file where all the cred operations been defined, update, delete, create and list, etc.

A

Now we will see how to use this framework efficiently by examples.

A

Let's see simple test function.

A

So this package shows how the test framework can be used directly in test functions without setting up a test route, see I'll, explain you what is a test suit in a while?

A

So whenever you write the test function, first thing you have to do is define an environment object, so here using new with config and providing empty configuration, this environment object is being defined. This is how the feature has to be defined. So this is the feature name. It has a label type and simple. This is optional and setup. So this is the actual test which will run when we run go test and before running this test cases first, it will do. It will run the setup and using the environment config. You can invoke this feature.

A

Okay, let's see the suits. So what is a test suit? So test foot is a method which is defined in a package level, so one can define setting up the pre-required side or cleaning up the resources in this suits. So in our files main dot, it does main underscore test dot, go uh act as a test suits.

A

So here we define test name. So what is test mean in code testing test main is a function which will become an entry point for all the tests in the package.

A

So we have taken advantage of advantage of this feature of go testing in this project to define a suit.

A

So for this package, the main test will be the test room. So entry point will be here. Let me explain context propagation, also, the same example. So if you see here, uh we have defined a context value with the new, with using new with context.

A

So, let's see how this context value been propagated to other test files. So if you see hello underscore test dot go, this is how the context value being retrieved so context, propagation, helps you to pass on the information or any details of previous tests to the other stages.

A

Let's say filtertest.com, as I said, we have lots of filters available uh for skipping the features or select a select for selecting the features for running.

A

So, as I said, there are lots of filters available for skipping the features and selecting the features.

A

Let's see the example here, so what it does in the code itself, it uh just selects the assessment using the regular expression. So if you see here with assessment, regular expression add hyphen star, so what it does is. It will just run the assessment which has the prefix ad.

A

So basically it will skip this repeat message process like feature set test.go. This uh demonstrate how to uh run multiple features in a test, so you can list down the features list here: f1 f2, f3, whatever it is,.

A

Let's see how to use client package so in the main test, dot go in the setup as a prerequisite. We have created a kind cluster.

A

Even here we have used context propagation to uh store the cluster information and the same a context being used here to retrieve the cluster configuration and using that we are just destroying the question. So once the once this setup is done, then it will run the actual test which is written under catus underscore test dot co.

A

So this feature: what is how it does? Is it just lists down the port center name, space cube system so to access client helper functions. First thing you have to do is create an object of client using configuration dot, new client.

A

So after that you can just access the cred operations or credit methods defined in the client helper functions.

A

Let's see decoder, so if you see here, we have a yaml files and even uh jsn files, so in the decoder test.go, so you can see here using decode any uh you can decode any a single file using decode all files. You can decode all the files under this test, data folder and you can just uh get the objects list and you can loop it and you can convert it to the actual kubernetes object.

A

So if you want to know more about how to use this decoder, you can just go through uh go to this examples. Decoder there is a readme file which explains how to use decoder in different way.

A

Okay, let's see third party integration, as I said, we have helm integration as well. So if you see here, what we are doing is uh here this uh this feature demonstrates how repo based health chat. Workflow works. So here we are um running a regular nginx table and once it is added we are updating this repository.

A

Then it installed the you know, nginx helm, chart and this next step. We are just waiting for this deployment to up and running once it is done. We are tearing down or removing the instant traffic.

A

Okay, wait for resources, so what happens whenever you deploy an kubernetes object? It will take some time to um up and running so if you have to wait or if you want to keep on following the status of that particular product of resources, you can use pay dot for function. So if you see here uh once the deployment has been created here in this test case, we are waiting till this deployment is 50 percentage available.

A

So here the condition with which has been added. So this is done using weight condition, functionality.

A

Parallel features, so if you want to run the features paralleling, you can define feature here so, for example, here in this test case feature one feature: two are defined and we have used test in parallel method to run this feature 1 and feature 2 parallel.

A

Okay, I hope this might have helped you to get some idea how to use e2e frame how to use e2e framework on your projects.

A

Let's go back to slides.

A

Happy to see that already community have started using e2e framework in their projects, git bots, sono boy, creator, service validator, some of the examples.

A

So if you see gitbot, they have used e2e framework for if refactoring integration test to simplify the setup and definition of test cases, even on creator service validator, they have used e2e framework for designing their test.

A

Let's see the roadmap, so these are some of the features in a pipeline which we are planning to add in e2e framework. So, like cube, ctl helper apis, we want to implement cluster ctl api's helper function.

A

uh Even there is already a draft design on kubernetes constructor apis. So this design proposes a functional approach where a developer would use function and method calls to automate the creation of kubernetes object graphs. So when I say kubernetes object graph, for example, if you consider deployment yaml 5 in the deployment, there will be different section right, like metadata spec container section or the prop section. So this would be uh for to define these objects. um Kubernetes constructor apis will be helpful.

A

Special thanks to vladmi vivian for designing such an awesome projects, which makes developer life hassle free thanks. Vladimir, let us join our hands together in moving this forward by contributing reviewing and giving suggestion and feedback thanks for giving your precious time and listening to us patiently feel free to reach us on slack or twitter handler stay tuned for the rest of the sessions. Thank you.