Cloud Native Computing Foundation Kubernetes Forum Sydney 2019, 12 Dec 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keynote: Enabling World Class Engineering with Kubernetes at Xero - Jono Cooper, Engineer, Xero

Description

Join us for Kubernetes Forums Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Keynote: Enabling World Class Engineering with Kubernetes at Xero - Jono Cooper, Engineer, Xero

Xero is Australia's largest business platform, helping millions of Australian small businesses thrive. We recently started our journey with Kubernetes, Helm, Open Policy Agent, and more to empower our 1200 developers. This is a major challenge, as Xero has many self-organising teams, each with different tooling and compute platforms.

In this talk, we will discuss how Xero built a new cloud native platform that serves over a million Australian customers. Learn how we optimized the developer experience with Kubernetes and Helm, to improve the velocity of our product teams and enable them to focus on building beautiful products. We will share the quick wins, challenges faced, and how we started a large migration of legacy services without an overwhelming support workload.

A

So yeah I'm really excited to be here today and just talk a little bit about how we use Cuban, et's and cloud native software to empower our developers at zero, but first just a little bit about zero. So we are a business platform that really helps connect small businesses to the accounting tools, apps and data. They need to grow and thrive and I think as our customers needs change. Our purpose kind of evolves with that.

A

So today we're all about trying to make life better for people and small business, the advisors and communities around the world and we're a pretty big company. Xero has over a million subscribers in Australia and New Zealand and over 2 million subscribers worldwide.

A

We also have 2,700 staff, but we still managed to be a developer, lead organization with almost half our staff being developers, and we divide these staff and like two groups, so we've got the product teams that work on 0, the actual product and we've got platform teams that work on infrastructure and tooling for our product teams. So me I'm on one of our platform teams and we're in the cloud specifically aw. Yes, but what does that mean so to help the bookkeeper or counsel small business owner?

A

It means that they can run their business from anytime anywhere on any device and that's really great for them, but twelve product teams. It mostly just means that everyone has their own a diverse account and that's great in many ways like people can release multiple times of day they're, not blocked we're trying to provision new infrastructure, but- and if there's an outage, it's usually contained to a small part of the product.

A

But the problem is: there's now lots to worry about so teams not only have to write great code, understand the intricacies of Australian super, but also to build reliable cloud native infrastructure, and that last part is a little bit of an issue, because what happens is get hundreds of autonomous teams all essentially solving the same problem, but just implemented slightly differently, so one team might be using like Ubuntu virtual machines.

A

Well, another team might be you're anything service on a dress Lander and the deployment pipelines between these teams are just completely different and really, at the end of the day, like infrastructure. Just doesn't add value for our customers, like the small business owner, just wants to run their business. So what can we do about this? How can we like provide the flexibility?

A

The our product seems, needs well-liked, still managing that infrastructure. So communities, that's a thing: let's just create a shared Communities cluster and call it a day, but we haven't really solved anything here now. Product teams just have to learn cuban Eddie's instead of AWS.

A

So what you really need to start doing is think about your platform as a product. So this is all the core idea is about being customer centric and really just working closely with your internal customers to see what they want so like actually just go talk to your product teams and ask them what their problems are and then do your best to try and solve there, and if you do that, well, teams are going to want to move to your platform, so this might come in the form of cluster add-ons.

A

So if managing DNS is a hassle manage that for them, if renewing is sales certificates, is annoying automate that if monitoring and logging is a pain, just just manage it for them like the list goes on, and all of these are like open source very little software through like right ourselves and that's really great but I- think Cuban et's is only the start and think of add-ons as like building blocks to solve real problems for your customers.

A

So this wouldn't be a talk about communities. If I didn't talk about yeah mo the communities, API is great and it's really powerful, but it can be daunting for people that haven't used four, so we use helm to make this a little bit easier for our developers.

A

So how essentially takes your small bespoke amounts of yanil and turns into the fully fledged communities manifests and then applies into the cluster, so we built a generic helm chart that has some opinionated defaults for zeros environments and we give it the developers, so I can spend less time configuring deployment configurations. But how do you actually get people to use helm like at a glance? It's like another tool to learn and it looks kind of complicated.

A

So what we did is we built a deployment wizard so essentially, rather than filling out a value zml or yourself.

A

There's a textbox with a description, and you know all it's doing is passing that what you write into a yellow file, but it's a much better user experience, and so once you've filled everything out, you can go and choose like if you're using teamcity, terraformer Jenkins like and then download a whole pipeline, and this is really basic. There's something took us like a week to make, because it's like all in the browser, client side doesn't talk to any api's. But it's the thing that saved us.

A

The most support people that don't know anything about communities can get up and running faster without deployment wizard than those who've used communities before and as a result, over 95% of our deployments on our clusters, use of generic helm, charts and since the initial success of that deployment, wizard we've expanded our self-service to reduce toil for the platform team. So we identify common tickets that can be automated and we're in a really good position.

A

Now, so product teams can come in and create a namespace and update roles, users, resource limits and then start deploying to our clusters without any interaction with the platform team, but communities can be weird sometimes and things don't necessarily work. The way people expect, for example, like the docker latest tag.

A

If you use it and then you try to a deployment, you might not get the latest version of your software and then protein will come and complain in your sleep channel and say: kubernetes is broken when it's really working as designed so to kind of encourage best practices. We use open policy agent.

A

An open policy agent allows you to author and enforce policy for a variety of cloud native applications, but in communities it can be installed as a mission webhook to prevent issues before they happen and it kind of means we can stop product teams from doing weird or unsecure configurations. So, for example, we used to have this corporate proxy and you had to set the HTTP environment variable on oil deployments, and this was deprecated a long time ago, but the configurations still linger around today.

A

So when we built our Cuba ninis platform at zero, we made the explicit choice to don't even support us, like the network, just wouldn't work, but the network not working is not a great user experience. So with open policy agent, we can put a nice error message in there with a link to the docs on. Why that's the case? Just don't forget to write your Doc's like I, think Doc's a huge part of what makes a great product you know. A diverse and asier have extensive documentation, zero. The product has great docs and I.

A

Don't I think your internal platform should be no exception.

A

Just don't rewrite the official Doc's, because just document your specific implementation and I think if you write great Doc's, it will save you a ton of support and that's really important to us, because we're a really small team, there's only four engineers and hold of zero that build and maintain all the clusters.

A

But it works out really well for us, like product teams, actually want to move their workloads to our platform and going forward. Humanities will be the only compute platform at zero. Just make sure to build your platform like any other products, start small, those like stateless non-critical workloads and make sure your platform works really well. For those first then expand your capabilities when your teams need it and we're not there yet, like our windows, monolith still isn't running, including Ares and Al.

A

We out, we don't have a fancy service niche right now, but I'll know I, know we'll get there through iteration and continuous improvement, and it's already been a journey to get to where we are today. We spent months just experimenting to see what works best with Xero and our customers and what works well for Xero. Might not be the best for your organization, but the beauty of cloud native is that you get the choice we wouldn't have got where we are without thinking about our platform as a product.

A

You have to be customer centric and involve your product teams in every step of the way. I highly encourage you to do the same and your organization, because, if you do so, you will empower your developers. Thank you.