Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 23 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Infrastructure as Software - Paul Jones, Dyson

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Infrastructure as Software - Paul Jones, Dyson

Infrastructure as code has facilitated the automated provisioning of computing resources, however the cognitive overhead of adopting new DSLs, the verbosity of declaratively defining resources in templates, and the lack of basic programming language features can be inhibitors in development time and iteration. Pulumi is a Cloud Native Infrastructure as Code SDK for creating a deploying cloud applications in general purpose languages to AWS, Azure, GCP and Kubernetes. Similarly, the AWS CDK is an open-source framework for defining cloud infrastructure in modern programming languages, by building on the CDK’s library of infrastructure constructs to compose stacks and apps. We will cover both Pulumi and the AWS CDK to explore the options to programmatically define infrastructure, as well as demonstrating their ability to facilitate more testable, extensible and secure infrastructure.

https://sched.co/MPXQ

A

Hello, everyone. Thank you very much for coming it's great to see so many of you here today. I hope you have a good Keep Calm. So far, I'm Paul I'm from bison in the UK I work. As a software engineer there and today, I want to talk about infrastructure, as software.

A

There we go so, let's talk about so it's about evolving, the paradigm for infrastructure provisioning and how we can transition from declaratively defined infrastructure to doso imperatively.

A

Firstly, we will talk about existing solutions for infrastructure, provisioning, both declaratively and imperative. Lee will introduce infrastructure software through programmatic provisioning of infrastructure and then we'll take a look at some case studies, firstly plumie and then the Amazon cdk. So to start with what are the main principles of infrastructure provisioning solutions through our impotency? We want to provision, we want provisioning to be deterministic and continuously yield the same results over multiple deployments. We want to an agreed, centralized, consolidated source of truth for the state of our infrastructure.

A

We want the ability to reason about desired and observed state which facilitates such concepts as the reconciler pattern, a determined state differential and rectify any drift. We want to be able to version the state of our infrastructure, enabling us to apply branching models and apply iterative changes and roll back through tracking changes. We can create an audit trail representing mutations in the instructor's desired State. Lastly, testing our infrastructure ensures that we gain confidence that changes.

A

Don't that changes do have the desired effect and functionality has not been compromised to implement these principles, most existing tools to find infrastructure in text based markup languages, yeah more being the favorite. So how do we currently implement these principles? So canonically sure everyone is aware of both of these will either use terraform or confirmation. Confirmation is first part tool with for desired state configuration management in amazon. Confirmation templates you jamal to describe all the infrastructure resources in AWS.

A

Cloudformation is powerful tool, but it demands quite some learning master it. Moreover, it's specific to AWS, so it would be transferable to other cloud providers as opposed to terraform. This is a open source source financial structure in declarative configuration files, supporting all major cloud providers and even hybrid scenarios. However, both rely on some form of templating to represent infrastructure state, but is this really infrastructure as code? Are we maximizing our development experience and using all the tools at our disposal?

A

So I'll just cover my soapbox for this bit a bit about Yama and ESL's. Yamla DSL restrict our capability to truly maximize our repertoire as software engineers, whilst general is a great assembly language to configure communities and other cloud native tools, it is not expressive in us to capture common requirements and leads to significant boilerplate and copy and pasting. Due to this, there are a few DSL x' like helm and k sonnets that aim to solve this.

A

However, this exacerbates the issue, as it has another level of cognitive overhead to the development experience having to learn the nuances on maintain the behaviors of any other tool sets yama strength is in structured data where it can go wrong is where yama tries to describe behavior variables. Arguments are not structured data. This is programming masquerading as configuration. We've met these concepts before so why are we reinventing imperative programming due to the constraints of templating languages?

A

We have a distinct lack of features like looping constructs, object-oriented principles like inheritance, abstraction, recursion, threading, automated testing, and these are all absent, inhibiting our developer experience.

A

So our software engineers, what we really want is infrastructure as actual code. We want to use the same language that we already know. We want to say in the same editor and use the same frameworks and have the same developer, experience we're going to leverage everyday features like auto-completion, compilation, errors and reuse the developer skills. What we already have we should come up with come up with abstractions to generalize our code and create reusable components. We should leverage the open-source community in our projects and combine our applications with infrastructure code in a consolidated repo.

A

So enter infrastructure of software, the approach of programmatically, defining of structure using familiar programming, languages and Saul says this is borne out of the rapid pace of cloud native developments, combined with the shift to ephemeral infrastructure and the consolidation of application code and infrastructure code, which demands a different view of cloud developments. So knowing JavaScript go means we can start codifying own infrastructure right away. It removes one of the biggest barrier to entry is around infrastructure as code as we know it now, which is another language to learn.

A

Loretha toolset, unlike terraforming CloudFormation, we don't have that problem, avoiding the learning curve associated with learning. Yet another DSL is a violet, sorry learning the learning curve associated with yet another DSL is a huge value. Add. Furthermore, by representing our team structure as actual code, we get to easily our tests using frameworks. We know and the lack of testability in things like terraforming cloud formation. The moment has been a big book bear and obviously, with using programming languages, we can avoid that so adopting this approach.

A

We can firstly leverage software principles in infrastructure, composition, language features like inheritance, abstraction, encapsulation, recursion, looping, constructs error, handling immediately power up our capabilities and productivity. We gave gains fatal controls when it comes to governance of our cloud environments and having our resources specified in an application or library means that we can more granularly check and control what code and the state changes are happening before they are pushed out through our pipelines. We can take advantage of stuffing analysis and linting of code and we can even start doing TDD with our infrastructure.

A

We can have a coexistence between infrastructure and application code where event-driven applications whose deployment and execution are all encapsulated in the same source code.

A

So this ephemerality is often characterized as developers moving down the stack where they're becoming responsible for infrastructure and ops moving up the stack where they're becoming responsible for the code, but the reality of the overall shift is towards increasingly codified provisioning of short-lived infrastructure.

A

To support this shift. Enterprises need the tools that can connect application code and infrastructure code in a way that is logical, expressive and familiar to developers and operations. Teams where application code is increasingly defined, it defines the infrastructure resource requirements of a cloud application.

A

So we've seen this before and I went through all in these, because we need to quickly move on, but, firstly, we've got troposphere, so this is a lot Python library for creating car formation in Python, which can be rendered into the appropriate Yammer or Jason templates. We'll come back to this in a second where we see the cdk, we've also got go formation, which is orientated around bi-directional transformation of go language infrastructure code to cloud formation templates.

A

Also, we've got cloud SDKs, so these facilitate common languages to define, configure and deploy resources across cloud providers. What is lacking with these solutions, though, is the life cycling off the components and the mechanisms the deploy manage the the clarity of infrastructure. It's important to realize that these libraries give direct access to the cloud api's they're, the though they can be used to provision resources. They leave up to the program it to do so reliably, because these are imperative.

A

Libraries attempting to programmatically provision and update infrastructure using them can be error-prone, usually resulting in bespoke homegrown provisioning systems. So, to start with, let's take a look at Pulu me. So paluma is a multi cloud, multi language cloud development platform, which is fully extensible in both its engine and ecosystem of packages. It supports typescript, JavaScript Python and go with SDKs for AWS as your DCP and Cuba Nettie's.

A

The paluma cloud development platform is a combination of tools, libraries, runtime and services that deliver a consistent development, development and operational control, plane for cloud native infrastructure, polemics programming model, extends across infrastructure, managed services, containers, container management and service technologies with full api's for those services exposed which can be targeted by any chosen language. We can build libraries of her packages when implementing standard policies, Network best practices and abstract resource configuration away from downstream services.

A

So these are just some high-level examples. So we see here that we're provisioning an ec2 instance using the AWS library and setting the configuration through the parameters in the constructor. We can export and import values between stacks, enabling portability between environments more on the communities integration to come, but here we're just defining a deployment as all the objects within communities are first-class citizens within polluting.

A

We also can define a venture from applications where infrastructure and business logic coexist and work together to form powerful service applications and paluma also offers a cloud agnostic library to deploy infrastructure to any cloud native environment, abstracting away the underlying cloud implementation and facilitating applications to be cloud neutral, portable and therefore driven by business logic.

A

Instead of stating any cloud specific provisioning pluming supports multiple backends for storing it States the two primary primary kinds are web, which is back-end accessed through CLI and rest calls, and, secondly, the file system, which is the back-end stored locally on file system. By default, the CLI uses the web back-end hosted at a plumie comm. The web back-end stores all checkpoint, States securely and CLI always communicates directly with the cloud providers using clouds, client-side, authentication and all state is encrypted at rets rest and in transit.

A

So, just to compare Ruby to some known solutions, so terraform is the closest comparison to lumi in terraform will declare infrastructure in a DSL and the terraform engine takes care of provisioning and updates and resources very similar to pluming. The major differences, though, are, firstly, the use of languages. So, as I said in terraform, we have to declare our infrastructure in the SLE or HCl.

A

In contrast for Lumiere lets, us use the languages that we already know and we get the added benefit of the features of those languages, so, like I said before things like looping functions, classes abstraction in regards to conformation.

A

So again, we've got the added benefit of using languages, as opposed to recently about infrastructure, as templates, but also paluma is multi-cloud. So we can easily manage resources from different clouds. It clouds easing what would otherwise require manual orchestration and in regards to cloud SDKs. The cloud providers offer SDKs and multiple languages like Botto in so there's that similarity, but it's important to realize that these libraries are different in their aim.

A

These libraries are hitting the cloud api's and provisioning the resources that leap, but leave it up to the programmer to do so, reliably, yeah and, as I said before, it's kind of up to us to ensure that that is done successfully so a little bit on the plumie cloud. So the plumie cloud is a framework within plumie for building container and service applications in a cloud agnostic way.

A

So the polemics our framework, is in preview and provides abstractions that can be will allow us to write a cloud application for many different providers using a common API. This is an abstraction package that defines the API is common to all cloud providers, because they're these abstractions allow us to operate with over different cloud platforms in a consistent manner.

A

Low-Level functionality and capabilities of individual platforms are intentionally not exposed, so just to take a look at example, so the service type in the cloud library expresses a load balance container which can be targeted to run in multiple cloud providers. We also have the API type, which exposes service functions over HTTP.

A

The package also offers simple data abstractions with cloud agnostic resources like tables and buckets so for kubernetes, so pulling me, offers a SDK for kubernetes and allows us try Q&A these applications in the language of our choice and it's designed to fit anywhere that communities is already being used. There is API parity with communities, so users don't have to learn a new API to write communities, applications in alluminium, it's compatible with existing Yamal and helm charts, and it's supposed to be a drop-in replacement.

A

4Q control apply so wherever you have a keep config, you can use plumie. In addition to this, there are multiple benefits from using paluma to orchestrate communities. So, firstly, we have proactive reporting from errors if resources fail to initialize. So if bulimia is able to deduce why something failed to be created in the cluster, it will obviously fail that and therefore no more digging around in the cluster to ascertain what happened.

A

There's also drift detection, so Polly we can offer a diff between the detective stay and anything that's out of sync with the spec in version control, there's a precise Lystra lifecycle tracking. So, like terraform, the plan phase will explicitly tell us which resources will be updated and deleted by an update, there's interim planning. So when we make a change to our cluster, we're told how that change will propagate around our resources.

A

So, for example, if we change a conflict map that it'll tell us which pods will be affected by that update, there's a strong integration with managed kubernetes offerings, so we can provision clusters in eks, aks and gke, and we can also provision resources directly in that provision cluster which will demonstrate shortly and lastly, the communities SDK also supports deploying helm charts. So this is possible through chart repositories or assuming local charts on the file system.

A

So this is just a simple example, so the full API for Kuban it is available just in code, not llamó, so akin to normal communities, resources. We have a deployment specification which can be declared, and then these resources are first-class citizens in our languages.

A

Here's a simple nginx deployment with a service of type load balancer and the load balancers hostname is then exported from the stack and could be used by at the CLI.

A

So if we deploy the stack using the plumie up, we see a preview of the resources that get created as well as their readiness as they become available in the cluster.

A

Cool so taking the community's guestbook application, we can refactor this from being quite a verbose set of resources and take advantage of being in an environment which provides abstraction. We can recognize a common back pattern for a deployment followed by a service which exposes that container optionally behind the load balancer with a stable IP. This can be abstracted into a class which defines all these common components and can be instantiated to create multiple objects. This aid in component, reuse and reducing repetition.

A

The end of this is, we have three deployments and three services with the front end being load balanced and which is the same. However, we've gone from the 147 lines of llamada to approximately 20 lines of code, not not only that, but the code is much simpler to understand and easier to maintain.

A

So the base class of this creates the deployment and service objects exposed to the IP of the service through any, which is obviously the scope of any instantiations of that class.

A

So gonna be preview shows all these resources which will get created in the cluster course so moving on to managed services, so plumies kubernetes integration can be further taken further to by provisioning clusters. Fireman you services in cloud providers, so provisioning and managing and monitoring communities clusters is no easy task. In AWS we can use eks, so you ease a bit of this burden, but it's still difficult to get up and running and to end in code. However, through the polemic eCare support, we can aid this task and get a cluster provisioned using minimal code.

A

The es component takes care of settling setting up all the resources and configuration needed in an eks cluster. So we'll get the eks service role, we'll optionally get a new V PC. We create the control control plane. We get the worker nodes in an ASG which will then automatically join the cluster. We get a config map which allows us that ec2 working hose to join.

A

We create a storage class in the cluster to allow provisioning EBS back to persistent volumes, and we then construct the cube config, which is needed to access the cluster and that can be exported and therefore used on the command line by a cube control.

A

So here we see a preview or V of the ETS stack with all the resources that will get created so yeah. This will end-to-end provision and ETS cluster with worker nodes for us by exporting valleys in the stack we can see references to these values and use them in CLI.

A

This is the instance of taking a programmatically, taking the generator, keep config from ETS stack and using it to access the cluster via cube control. Note that we didn't have to populate the cube control here with any arms or iron credentials, as that's all injected by the eks library during the provisioning time cool.

A

So, on top of cube Nessie's, we also have support for helm charts, so this significantly reduces the amount of boilerplate code required to deploy communities that to deploy to keep Nettie's, deploying from chopped repositories or local charts on the file system till it is not a requirement, although that obviously won't matter as of helper help 3 but Polly me will expand the helm chart and submit the expanded yeah Malta to the cluster. Here we deport in the Jenkins charts from the stable repository and exporting properties from the created resources so running plumie up.

A

We see all the human SS resources that will get deployed via this helmet chart, along with their readiness taking the exported endpoints. We can navigate to the front end via the load. Balancers hostname in AWS.

A

A

Okay, this is where we go slightly overboard, but, given that we have support for home charts in communities, our repertoire is expanding significantly to include CR these operators and entire super sets of functionality like service measures, so through referencing local charts and value jamol's. We can programmatically deploy things like sto, along with a configuration for our cluster and assert on each components, readiness of stability and functionality, so here which is taking a local instance of the sto charts, as well as the values, yama and deploying that home.

A

So if we do a plea, be previous, a that, it creates quite a few resources in our cluster, but what we've done here is programmatically deployed and configured in pi instance. Installation of this do with why cycling to assert that all the resources was created successfully and that we have versioning and rollback of our stacks. Whilst this might be a bit of overkill, it nevertheless demonstrates the possibilities of deploying resources to communities via pluming.

A

So one major advantage of having infrastructure software is that service offerings like AWS lambda as your functions and google cloud functions, can be real language, lambdas severals offerings. Today, divide the developer, experience between infrastructure, which can be or provisioning the lambda runtime itself and writing and maintaining the code that will actually execute in the function itself. One triggered because we're using general-purpose languages, we can create resources and then wire up event handlers. Just like normal event-driven programming. You can create event, sources and handlers in the same program choosing to wire functions to existing resources.

A

The optimum way is to create functions inline. This blurs the line between infrastructure and application logic in a way that makes programs feel like first-class distributed apps.

A

So here we've got a couple of examples of this, so we just create an S us queue, I believe in a bucket, and then we've got some Onix methods which can be used on these resources to register event handlers. So, for the most part, any codes we write here will actually will work, unlocking all sorts of powerful composition or scenarios underneath those onyx methods are many AWS resources. Transitively created, like I, am attachments to facilitate the invocation of event-driven workloads, so just a bit of a segue into testing.

A

So why should we be interested in testing our infrastructure? Firstly, it increases our confidence that any changes introduced to our infrastructure are actually going to be functional. It improves resilience and security and demonstrates a left shift to address code, quality security concerns and environment, environmental consistent deployments through the varying levels or min of inspection. We have a variety of options available to derive assertions of functionality and conformance to SLA x' security criteria and best practices. As we see there's a broad spectrum of infrastructure testing options available.

A

However, we could say that most of all, some of these are retrospective assertions of the state of our infrastructure, and we can use plumie to complement this and expand on some of these mechanisms. So as we're able to use languages like JavaScript, typescript and Python ago, we have the full power of each language available, including access to tools and libraries for that runtime, including testing frameworks, because we obviously might have functions in our code now to say that we want to tag resources or define some event-driven workload.

A

We probably want to test them or do TDD, because this is the same as writing, or during unit tests against any program and our choice. We can, for example, use sort of well-known libraries and here we're creating an ec2 instance with a security group and associated ingress rules. Once we set up our environment in a stack, we can assert against attributes there in here we're just making sure that each tag is not false and not missing an entry for the name key in the map.

A

Also, we can make assertions about the security of our code and assert that we are conforming to security best practices. Here we search through all the ingress rules in The Associated security group and make we don't allow overly permissive ssh, inbound access cool so just to close up on plumie. First in the cloud, agnosticism provides flexibility and portability. It means we avoid vendor lock-in and in some cases we can run our stacks in multiple providers by changing external config and not changing the source code.

A

The kubernetes cluster provisioning, using managed service integrations, reduces the cognitive overhead and cope code. Footprints and management of resources to a minimum programmatic generation facilitates more granular control and inspection of infrastructure resources and, lastly, abstraction aides, standardization across organizations through reuse by downstream services. So next up we have the Amazon cdk. So the cloud development kit is a open source software development framework to define cloud infrastructure, encode and provision through AWS confirmation.

A

A the cdk is currently in Developer Preview and can be implemented via java, javascript typescript net and, as of recently python, the cdks, basically a wrapper around cloud formation, which abstracts CloudFormation definitions of AWS resources into an object orientated library. This allows us to define high level abstractions of resources which can be easily reused. The cdk is a command-line tool for interacting with cdk apps. It allows us to synthesize stacks into classmate's templates and ploy stacks to address environments and defecates deployed stacks to understand the impact of code changes.

A

The construct library includes a module for each AWS Repair Service, with constructs that offer rich api's. That encapsulate the detail of how we this should be used and reduces the complexity and glue logic when integrating various Adria's services.

A

So we can think of for cdk as a cloud infrastructure compiler it provides. A high-level high-level class, provides a set of high-level class libraries called constructs that abstract a diverse cloud resources and encapsulate AWS best practices. Constructs can be integrated together into cut into object, orientated cdk applications. That's precisely define our application infrastructure and take care of the complex of boilerplate logic when we run the CDA application, it's compiled into cloud from a cloud formation template which acts like the assembly language for AWS compare cloud infrastructure.

A

This template is then ready for processing by the cloud formation provisioning engine for deployment into our a2s environments. Again, because cdk is supports, objects and orientated languages, we can utilize language features like looping abstraction inheritance and do some testing. So, as mentioned, the cdk has support for well-known languages so to get started. We can instantiate a new, app or construct library in language of our choice.

A

Here we see a couple of examples of how constructs can be instantiated with resource configuration passed into the Constructors to with the specific granularity required. With both these resources in scope, we can alter their behavior perform operations, create associations and retrieve instance, attributes for particular resources.

A

Here we see the command-line available to interface with our stack, so we have CD case synth, which will synthesize our source code into a CloudFormation template. We've got cdk deploy which will take that synthesize cloud formation template and deploy it into a deeper web iOS environments and we've got cdk diff, which will monitor any state drift between the last deployed state and the current state of our stacks.

A

Here we've just got a simple example of an app so on the Left. We've got the application, which instantiates our stack in the stack. We define a couple of resources and instantiate the construct, and then we have some behavior in the construct in this case, creating a few s3 buckets.

A

If we run CDK diff, we can see the state of the current stack, which in this instance doesn't exist. I'll show you all the resources that will get created so testing as we defined our stack in in this case Python. We can leverage common testing frameworks so in this instance we're using unit tests and just invoking our tests step and and testing whether we've actually got the correct behavior therein.

A

So in terms of service, this is the. What we're able to do is have infrastructure and application code living in together in the same repository, so here we're defining a lambda in our stack and we're having actual implementation read in from just a lambda handler, which is in the same repository. This could be inline, so we have a kind of coexistence between the infrastructure which provisions the lambda and the actual workload that will get invoked in that lambda.

A

The beauty of doing this, though, is that the cdk will also generate any resources facility required to facilitate the deployment and functionality offer desired, stack so deploy in that lambda. In this instance, we have nine role which is created and provided to the lambda resource, providing it with the basic execution capabilities.

A

If we deploy that like most sort of provisioning systems, we get sort of the readiness of all of our resources very similar to how CloudFormation renders it in the console and yeah how successful that deployment was.

A

The cdk also provides an assertion, library containing help helpers for writing unit tests and integration tests for cdk libraries. Using this library. We were able to assert that certain behaviors of your features of our application and stacks are true before we actually deploy that infrastructure.

A

So, firstly, we can inspect stack templates to assert that they contain the expected resources and associated properties, and that synthesising has not compromised the behavior of the stack.

A

Here we're more intact around the stack security, compliance and network configuration in a high compliance environment. We may have specific requirements which has demonstrated here to be checked at the code level before any deployments occur, for safeguarding the integrity of our stacks and environments.

A

So through the concert library, the cdk expose in a bun an abundance of resources and managed services, one such resource is eks, which, from experience requires a bit of orchestration and configuration to provision from code, so moving all that provisioning into the cdk allows for greater control over the specific behavior of the control, plane and work nodes in AWS and also aids, in integrating all the different components together.

A

So the worker nodes, we can do a diff and show us all the resources that will get created. So this case goes ahead and creates the ASG of the worker nodes and exports the are required for them to join the management plane again. Having constructs in code means that we can instantiate them and test against their behavior.

A

This again allows us to have assurances that what we deploy is going to function as desired and also gives us confidence that any changes being made in our code won't compromise from chanela the-- so just to round off on cdk. So again, provisioning ets in code is much much more reliable and we can reason about the actual observed the desired state of our clusters.

A

There's no native community supports that we're still reliant on a bit of glue to actually get the cluster working end to end and, as I said, the lot, what the cdk is still in kind of Developer Preview. So python has come along recently and so there's just a kind of ongoing effort to get SDK parity in the different languages and there's some licks to check out if you're interested so just surround off a bit on cultural readiness.

A

So when we're seeking to shift the paradigm for infrastructure composition into programming languages, we need to first make sure that the surrounding culture is one which which he was ready to for this to be adopted in. If there there are DevOps silos or distinction. Distinctions between infrastructure platform and development, then introducing a principle which essentially blurs those lines needs to be carefully considered.

A

Whilst it might be acknowledged that there are distinct benefits to shifting developers down the stack and ops up, this might actually be detrimental in a culture which is not ready for multi discipline, teams and organizations depending on the cloud native maturity of an organization or culture or technology offering. It may be lesser. There may be certain prerequisites required before leveraging the value of ephemeral environments and codifying cloud resources.

A

There is an evolution through the cloud native maturity paradigm, which must which, where we must have stable foundations and have invested and culturally appreciate these stages of our evolution. Without this, we won't maximize our potential nor deliver the full value of running in a truly cloud native environments, so to wrap up. Hopefully, this is showing everyone, the benefits and justification of moving infrastructure code to a more imperative style and how that change in paradigm can be achieved.

A

We have the tools to evolve the paradigm of infrastructure composition, which facilitates a coexistence of application and infrastructure code, thus resulting in more testable infrastructure. This reduces the cognitive overhead of yama and dsl development and leverages modern languages. Moderate modern language features to programmatically, define cloud resources as application code and supporting infrastructure become more and more blended. It's natural that software developers take the responsibility to define both the next logical step is to use the same set of languages, tooling, practices for both software and infrastructure and as everything. Thank you very much.