Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 14 Nov 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Building a Cloud Native Culture in an Enterprise - Deep Kapadia & Tony Li

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18)

Building a Cloud Native Culture in an Enterprise - Deep Kapadia & Tony Li, The New York Times Company (Any Skill Level)

The New York Times has been on the Internet since 1996. Long before most Internet or tech giants even existed. The Times is one of the most trafficked news sites on the Internet today with a fairly complex set of services and modules driving our internet presence. We have been using public clouds since 2010 but In 2016 we decided to go all in with the intent to retire our data centers in 2018. As a result one of the biggest projects we undertook was moving our website, CMS and a slew of other internal tools to Amazon and Google's cloud offering. We jumped on the Kubernetes bandwagon and have had a hell of a ride. A lot of our learnings were not technical but cultural. As we went down this path, we found ourselves embracing open standards and tools. We also had learned how to work with product teams who were at various levels of maturity and discipline w.r.t. CI/CD and SRE concepts.

About Tony
I am an engineer on the Delivery and Site Reliability Engineering team at The New York Times, building tools, creating processes, and writing documentation for our many teams of developers. I am currently working on establishing best practices and platforms (Prometheus) to support teams in monitoring and instrumenting their services. Before that I worked on providing the CI/CD platform (Drone) for the organization and developing the launch strategies (Kubernetes) for our datacenter to the cloud migration. The most interesting aspect of my job is organizing and scaling these things to and across an enterprise level organization in production ready states.

About Deep
Deep runs the Delivery Engineering team, responsible for developer tooling and automation, test automation and Site Reliability functions at The New York Times. The New York Times recently underwent a significant transition by moving their entire infrastructure from their data centers to the public cloud. As a part of this migration one of the most significant projects they undertook was to work with the web frameworks and feature teams to move nytimes.com to Kubernetes (GKE specifically). Previous speaking experience: Deep has been a speaker at the MongoDB conference in 2012 in the past as well as appeared recently on the GCP Podcast to talk about the adoption of Kubernetes and other Cloud Native tools at The New York Times.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Hey everyone, hi, 340, I, think we should start hello and thank you for thank you all for coming. My name is deep. Kapadia I work at the New York Times I'm here with my colleague Tony, and we wanted to give you a brief overview of how cloud native computing foundation and some of its projects specifically kubernetes, has helped us sort of build a much more developer, centric culture at the new york times. So without further ado.

A

So this is, this is pretty much what we're gonna talk about. We're gonna talk about what it, what why the New York Times needs, a more developer, centric culture. What is what is our enterprise comprised of? How do we go about transforming and planning to move to the cloud?

A

What are some of some of the culture shifts that we made usually when people think of the New York Times, they think of the newspaper? But it's a lot more than that at this point, but I mean to give you some context like this is this is what our website looked like in 1996?

A

There's a there's, a line below it says: please open your window to the width of the line of this text, which is interesting. The first New York Times website launched in 1996 and was essentially a gif that was put on Triple, D, NY, Times column and then and then since then, we've had a lot of organic growth in 2016 a couple of years ago, we decided that we didn't want to run things in our data center anymore. It didn't make sense for us like we. We build news.

A

Apps we build apps to sort of inform our end users, our readers. We don't need to be managing data centers. We don't need to be in the business of being being super technical about that, so we might created about 300 plus apps to Amazon and Google cloud.

A

We just ended our migration in April of this month, which is last month pretty pretty much so so we were completely shut down. We completely shut down our data centers last last month and in the process we shut down about 140 plus apps, and we migrated 300 apps to Amazon in Google.

A

So this is what the website looks like today. We don't ask you to open the website to the width of the browser it it adjusts there's, but there's a lot going on there like there is. There is like you see some ads. You see like section France, you see different types of articles. There are videos, slideshows, etc, etc. So, like there's a lot of moving parts here, there's also like a search bar somewhere up there yep in the right corner, which powers like a pretty large search function.

A

That goes all the way through our archives, all the way dated back to 1851. So, yes, we digitized their archives at some point as well. So so there's a lot here. What we also do is for specific events. We have, we have specific applications or specific news. Events like the Olympics like we want to get real-time updates about what's going on in the in the Olympics, but at the same time we also want to surface some important stories, collect, create a collection of some stories, etc, etc.

A

So that so we do, we do a lot of this, so so this itself it becomes an app for our for our for one of our teams. We do similar things for elections.

A

The result didn't quite go the way I had intended, but we'll leave it at that, but I mean like the elections, is really really complicated, like we're basically receiving live voter data at like the data is constantly polled and collected and, like you know, the site needs to be updated and we need to be we need.

A

We need to be really really quick about updating the site as well and we get hit with massive amounts of traffic on election nights, so we have a scale the election app very very differently compared to some of the other things that we do so so we have to take these things into consideration.

A

So what I'm trying to say is like the New York Times is evolving becoming more and more complex like like any other news organization, and then we have our new products and ventures. At some point we decided that we were doing a lot of these recipes. We were actually publishing a lot of recipes, but we decided might be useful for people to to collect their recipes. Annotate them share it with their friends, so there's a whole cooking app. Now the New York Times crossword has a life of its own.

A

That's me trying to solve the crossword last night. I was very, very unsuccessful, but there are people who actually like wait for the new crossword to be published and and actually just go at it, the minute they get it. The crossword itself has its own like profit and loss account. It's it's it's it's own thing, but this is something that we do. We build it and then there's all the things that that power.

A

Everything like there's a there's, a content management system where our news reporters and editors or constantly editing stories and and actually sharing them with each other or collaborating on them. There there's an editorial workflow that we need to account for there's I mean we also are a for-profit business. So we we have subscriptions, then there's the archives. As I mentioned, there's advertising, there's, there's, there's mobile apps, the mobile apps are pretty different, like they are powered by by by API.

A

Is that that need to be scaled when, like people get a breaking news, alert and start hitting our API is at one point: we we had. We had our mobile apps be the number one cause for for our ddossing. Our servers, that's not the case anymore, thankfully, so so there's a so there's a lot do sort of process and I'm, not even going really deep, but there are.

A

There are a lot of issues with the way we built apps in the past.

A

One of the things was our internal IT infrastructure team, which was responsible for managing people's desktops and computers, eventually morphed into an ops team that was managing our website. Our our DNS, that's a very different skill like managing internal IT versus managing website that needs to scale on demand, and you know help developers move really fast, so also, there's like changing in consumption, habits, etc. That required us to change the way we we did things so like we tried. We tried moving to the public cloud at one point.

A

We sort of had this notion that, like we won't be able to scale, come come 2015 2016 like we need to move much faster. We started dabbling in AWS, but but but it was just an experiment. At the time we didn't we, we still had a lot of expertise in running our own data centers, so we never tried going too far with the obvious. We ran things in it obvious, as we would run things in our data center, which means access was closely guarded. People would have to actually provision machines before they could do anything.

A

There was not a whole lot of automation built into our provisioning process. There is not a whole lot of automation in our build process. We did go through a.

A

Org wide CD effort, maybe we said, Oh continuous delivery. That sounds like a good idea. Everyone do continuous delivery and you know we we brought in some consultants. They helped us quite a bit. They helped us understand the basic concepts, but then we left it to the teams to figure out what continuous delivery meant for them, which means our our deployment stacks completely fragmented. We had at one point five five to six different ways to deploy applications to AWS. That was not desirable because you know different teams were moving at different paces.

A

Different teams have different levels of maturity. We couldn't we couldn't. We couldn't move fast enough, so we decided to revisit the decision in 2016 and say: where are we at there's like? What do we want to do? And there was a lot of increasing frustration amongst people who were more familiar with new DevOps concepts, people who are interested in running things and containers, people who were reading about kubernetes or me Saussure, and things like that and we're like? Oh, we should be doing this. Why are we not? Why are we not getting on this bandwagon?

A

There's there's a lot of desire to go use things like terraform use infrastructure as code managed secrets in a much better way, rather than checking them into your gate repositories. That does that there was a lot of things that we wanted to change. There were some teams that dabbled in things like nomads there were, there were a lot of things: teams that try to dabble into me, sauce marathon, nomads etc. But those were runoff efforts and were not very cohesive.

A

So we, we formed a team of people who had been doing these efforts in small silos and we decided to actually work with the individual teams to help them improve their level of maturity using common, tooling, common ways of doing things, and we formed what we now call the delivery engineering team, which of which Tony and I are a part of. So what we are is very well depicted by this picture, and this is from Greg, whose last name I cannot pronounce or.

A

His blog, he actually wrote a really good blog about what is delivering engineering around the same time when we were thinking about actually doing something very similar and the idea is, we don't manage people's production code, we don't we don't actually take someone's code and say here we're going to run it on the server or in this kubernetes cluster or anything of that sort. We will create a pathway for them for them to get their code from their laptops all the way to production, and that includes common, tooling common deployment mechanisms.

A

You know a common way to manage secrets, etc a common way to provision your infrastructure. A lot of this is just exactly depicted in the picture where, like these guys, actually create a path for the person throwing the path or whatever it's called I'm I, don't know much about curling, but that picture is really good, so I mean we basically standardized and some some ways of doing things. We basically said we want to have a gate driven workflow.

A

We we all use github at the New York Times, so we we said and github provides us at all kinds of bells and whistles for us to be able to manage our workflow. Let's use that, let's not push buttons and Jenkins to deploy manually.

A

Secondly, we decided to move away from Jenkins. Everyone was managing Jenkins at the New York Times and more time was being spent, managing Jenkins, then actual product building, and we said we don't want to do that. We at one point we had about a hundred plus Jenkins slaves running around or running under people's desks, or things like that which we didn't want. Does that sound familiar?

A

A

Instead of clicking buttons in AWS, we said we don't want to do that. We want to make sure that we are able to provision our infrastructure much quicker much easily. So we decided to use tariffs for that and then actually manage secrets like this. This was actually one of the biggest overhauled we did is like get people away from their esoteric ways of managing secrets, passing secrets around and PGP encrypted files, or sometimes not even just like people would share secrets in slack or people would like checked secrets into their git repos.

A

Every time I get repo got exposed for some reason it was like it was us going through the drill of yanking out the secrets, cycling them out, it was. It was really really bad and at the same time we realized a lot of these tools.

A

That I mentioned up up here are all written and go, and that allowed us to hire for a certain skill set as well so so having those standards was really good, because we we could gravitate towards one single set of tooling and then Tony can talk a little bit about how we do this. How are we able to sort of get all of this synched up and and how does it look like for a developer team today, all.

B

Right thanks Steve, so I'm Tony I work with deep as a sree at the New York Times. So, like you've, said in 2016 as we're doing this evaluation, we had a team that was rewriting our web platform Ashley, so the New York Times com, when you see on your phone and laptop today, so because they were doing this rewrite. We came up with this idea that we kind of are familiar with the Heroku style idea and we put together POC where you take your code, you just commit it.

B

You push it and then voila it's on. It's live. You don't need to do anything else, so we want to do this because we want to keep the local development experience simple and straightforward, and let automation do the heavy lifting if we can prove it with this team. That means we can stamp out this process as a pattern across the organization. So I'll do a live demo. If it goes wrong, then I'll just play a backup video. So the live demo. So here I have my it's not married.

B

Okay, so I'm, look in the thing so here I have my terminal. I have an app that's just saying hello, showing you some variables that I passed to it and and some environment, variables and stuff like that running on gke I have a dev endpoint and a prod endpoint I'll refresh this.

B

So you know that they're live and up and running, and then here I have some code so like I'm, a everyday developer, I want to update some stuff, so I'm, just gonna open my text editor and instead of hello, world I'm, gonna, say hello from gke, cube, con Europe and then save that and then let's say, like deep-set, we took secret management seriously, so we wrote this thing that would fetch values out of vault and then inject it into the container.

B

So for this case our page, the background color is actually sourced from a default white environment. Variable, as you can see, you in production, is yellow and then involved. We're gonna set it to green. So what I'm going to do is I'm going to uncomment this sidecar container, that fetches values from vault and then so I made the changes and then I'm gonna push these to production, but before I do that here, I have a instance of vault running locally and then I'm going to proxy.

C

This vault so that oh okay, let's see.

C

Okay, here's a live demo God's striking on me, I mean I, am connected to the Internet. Just serve you down all.

B

Right, let's do.

C

What's the other port tunneling thing and Iran there we go all right, so we're gonna use that grok instead.

C

This is a live and prompt provides a ssin.

C

All right so, instead of Serbia we're gonna use ball and rock.

C

All right, so we have vault, poured it through and then so I'm going to do. Vault right, we're gonna, set the color to green.

C

All right demo gods come on.

C

Export okay, so it's gonna pass to my local right green. If we read it out, it's that's green all right! So, let's hope this works so I'm going to add my changes as any developer would get commit. We're.

B

Gonna get the color from vault and then.

C

We're gonna push it into github, alright,.

B

So now that it's pushed into github, what our system actually does is it will trigger a build in our CI CD pipeline.

C

So, what's there working there, we go alright.

B

So it's gonna turn our CI CD pipeline. This is drone.

B

What it does is it's connected to github via the web hooks and it's gonna clone the repository down, run our tests on it build a binary and we can notice that we're actually building the binary and passing in the get Shaw of this commit, so that this binary is immutable and tagged with this commit and then what we do is we package the binder into a docker container like you would any other we're also going to tag this docker container with the get saw of this commit as well?

B

They just copy that show there we go and then, after that, what we do is we template out our deployment manifest with the get sauce as an image that we want to deploy. It actually prints out the template manifest so that we can verify that sure we're running our side, code container and anything else, and then it actually does some dry run validation, and that applies it to the clusters. So as we wait for it to roll out after it rolls out successfully it's going to post in slack the status.

B

So there we go and then it's gonna have the status posted. So now, if I actually refresh dev, we can see that it turned green. So let's say we want to promote this now to production right. Everybody loves the color green. Let's change it so that it's not yellow anymore. What we would do is we would actually just go into github and then in our code, will draft a new release and say 2.0, because 2.0 is always better than.

C

1.0 and then we just publish.

B

This release, when we publish this release, what it does is, it is the same thing as when we when we, what happened. We push the get button. Instead, the triggers are slightly different because we build the binary and the container already and the container is in a registry. All it does.

B

Is it it templates out the manifest for kubernetes with the same get shot, because we just tagged on the same get shot, and then it templates out the manifest but applies it to the production cluster instead and then, once it is deployed we'll see, it will notify us in slack again, and then we can refresh the page in here. So you can see. This is a get that I, just I just created right, 3a, something whatever. So this is the old one.

B

So once this is complete, if I refresh production, it turns green and then a rollback as simple as going into our demo or going to our CD pipeline, selecting the previous deployment and then just pressing, restart and then so once this build restarts, it creates a new run of our job. Does the same thing except this time with this get Shaw right, the previous one that we were looking at and then once this finish, deploy which it just did before you go back to production and then there's there, it is so.

B

This is something that we prove that could work, and so, as a result, we decided to roll this out to all the teams that are looking to deploy to kubernetes and then as well as because drone is a plug-in container, plug-in based C sed system. We can, we also wrote plug-ins for Google, App, Engine and terraform, so we'll get into that a bit later. So what we wanted to do is the cloud is a different landscape compared to traditional data centers.

B

This means changes in the culture and the methodology of development must occur if we want to set ourselves up for a good position in the future. The multi-tenant nature of cloud means networking and security has to be reimagined. So how could we take advantage of various isolation levels in project organization structure, to provide developer freedom, and how can we do less to get more out of the cloud? So, with this in mind, we use the cloud migration as a opportunity to improve how we develop at the new york times.

B

It was easier to lean in and shift into this cloud native mentality and embracing this meaning meant improve, developer experience, velocity and productivity. It was a smooth sailing all the way, but here are main takeaways, so we shifted the process of how we do things.

B

Shifting service ownership meant each team moved on their own schedule during this migration, providing them with their own GCP projects, allowed them to experiment with freedom and feel ownership for the operation and reliability of their application.

B

Creating self-service tools, as the SRA team does, for common tasks eliminated the back and forth communication of the ticket based model that a lot of traditional infrastructure teams have documentation was critical to the success of these tools, because if no one knows how to use them, you're going to go back into a lot of back-and-forth communication.

B

Educating developers on how to build modern apps allowed us to make the migration process both easier and kept the door open for leveraging new solution in the future. So if you don't have a 12 factor, application you're not going to be able to connect to that fancy new service service, mesh proxy or configure your application to read from vault, for example, working in the open gave visibility to other developers who are likely doing other similar things. This meant dependencies had clear expectations and could plan based on upcoming changes.

B

So we wrote our C who ever had an idea would write an RFC and all developers in the organization were open and welcome to comment on it and morph how things are being made.

B

A lot of people are going to be interacting with GCP every day, so in-person training sessions we're probably the most efficient way to onboard a lot of people at once to a technology. This also helps satisfy many developers, personal or professional training goals. Developers are you know, people who like to learn and people who like to try things out and having a professional guide them through a quick start and to answer any questions is a great way to boost their confidence and get them in into it right away.

B

So we also shifted our development experience with velocity and developer control in mind, so we preferred consuming cloud-based solutions to managing or hosting tools and enterprise services, from things like databases and queues to get mail servers and HR software. Each slice of mindshare from managing a dependency could be used to work on the product itself. We preferred using open-source tool in like kubernetes terraform open api as tim, maintaining in-house built tools. This way people can change teams internally without much onboarding or friction, because they already know how to do things.

B

The new york times way, which is the way that everybody else in the world is doing it. So we began to ship faster and more frequently, which are which the drone CI CD system allowed us to do. It allowed us to align on common, easily configurable pipelines, they're just like they're, just yeah Mille every step. Look the same.

B

You say which image you want to use, pass some arguments or run some commands in that image, and this was great because it closely reflected what a lot of paid services like Travis and Circle CI were doing, but because this is an open-source project and we ran it ourselves, we had the opportunity to be more flexible change, some things in the installation of it in our in the platform and for the newest version, because we wanted to make sure the system was stable and in production State for many teams that were going to consume it.

B

We have on board a two hundred posit or E's so far annex we are expected to exceed four hundred positives once everybody converts their pipelines to this newest version. More than 50,000 jobs have come through this version since a year ago. This excludes the thousands of bills that were in the first version, first iteration and then in the last four weeks, as you can see, we've had 12,000 jobs, so not every job is a deployment, but it's some sort of build or some sort of tag or some sort of operation and I.

B

Think when I made this was a week ago, so it's probably increased even more, we decided to treat secrets like secrets. We in vault we have 75 namespaces of Secrets, so far, they're used by 21 teams and 54 repositories. Each of these teams or repositories the namespace means they have their own place involved, providing them again with ownership and isolation.

B

So, for example, if your team actually accidentally MUX up the secrets in that namespace, it's not gonna affect anyone else, and then our fault, our team and the people who are on call could go back and fix and recover your secrets without affecting anyone else.

B

The New York Times is, you know like a lot of people visit. The homepage visit article pages all that stuff and before this used to be a giant varnish, VCL configuration file with a lot of logic and rules in it. In this shift we decentralized the CDN logic using the fast using fastly.

B

This meant individual teams can manage their CDN using standardized tooling, that our team has paved the way for and testing methods. This broke up this one or like four giant vcl files into 40 repositories, each with their def stage and prod release cycle, and this puts the control back to the teams that are actually the backends for those various routing rules.

B

We started budgeting in the open with budget configuration as code, we wrote a little tool that hopefully will open-source soon, but basically it's a Yama file. Everything that Yama file these days that you configure to say this is all the juice view, projects that I want under this budget, and this is how much I'm allowed to spend it each month and every time you hit one of those levels, it's going to post and slock like that or some something like that. That tells you you've hit your budget.

B

So if you hit your 50% budget, it's only a week into the month, you know something's wrong and you should take a look at if there's runaway costs and if there's a stale resource or something like that, and because it's in slack and that's where all developers are. This gives a constant, concrete number and a reminder to developers to say we give you the freedom of this GC projects, but with it you have to be aware of your budget and how you're consuming the services in your project.

B

So we also changed how we communicate the develop delivery.

B

Engineering team shifted the approach of support for other developers in order to scale we worked with early adopters, a few teams at a time to make sure that the path that we're paving is in the right direction and with the right tools by sharing responsibilities and goals, we could prove the concept and pave this path and spread the knowledge to developers who would then spread it to the developers that they interact with we rent office hours, which allowed us to answer questions with the chance to continue working on project.

B

If no one had any questions that week, this also meant that developers can learn off each other when they're present in the office hours. Since then many Cross organization teams have also adopted office hours.

B

We must the delivery team. We must always be learning because the technology is constantly changing as you're aware since you're, here at this conference, by the time the developers of products gain grasp of the tools and the documentation, we need to be thinking about solving their next needs and anticipating what else they need, because if you wait until they're experts, it's too late they've, you can't beat them to to it anymore.

B

Turning things off relates to the use it mentality. So, as Dee mentioned, we shut down like a hundred something services. Now what this means is this reduces the burden in my chair again from developers keeping teams focused on their primary products to actually generate value and revenue for the company.

B

Our CTO did a great job, because every time we was shut, something down, he would give a email blasts out and that goddess excited and our team reviewed survey results to figure out what was working and what wasn't this effort took over and what years? And once we got more and more teams onto our tooling into the documentation, we had to figure out if what we're writing is actually solving their problems, anticipate what kind of future problems they'll have.

B

So this is what Google Forms and having consistent measurable like qualitative based questions and that Google Forms allowed us allowed our team to see if we're improving, quarter-over-quarter and our tooling and support.

B

So that was all the stuff that we've have done and, of course, there's still some stuff that we still have to do.

B

One large focus for our team is to continue our education in site, reliability, engineering practices and sharing them with both product and development. This includes, during a shared view of responsibility for the applications, reliability we demo and lead sessions to define service level objectives and application assessments, so that we can provide feedback to teams who are keen to learn and understand what these best practices are and track them over time.

B

We're working on in terms of reliability, service, level, objectives themselves are useless without indicators. You can set an objective, but you have to be able to measure it so defining a path for monitoring like how we did with ownership and deployment is what we're working on. Next. We want to provide a platform for teams to small to manage their own Prometheus instance or their own data collection, and so that we just take some config that they provide to us, and we will fetch the data, the metrics for them and create these dashboards for them.

B

So now that the migration is complete, we're exploring new features and updating non-ideal processes, while product developers return and focus to focus on feature work. A big topic is how to scale on-call responsibilities, to reduces burden on smaller teams. If you have a team of three people, it's gonna be really difficult for you to be on-call. Compare a team of 20 people, because you're gonna get that anxiety about the pager every like at least once a month.

B

Whereas, if you're on a bigger team, you can relax and and focus on, feature work or responding to requests a lot easier. So we're trying to figure out how to efficiently handle this.

B

We can't just the sree team, can't just onboard everybody's applicate, 300 applications and say you're off the hook now, because that's not scaling we're also looking to enhance our support of Google, App Engine and diversify supported paths and platforms such as Amazon's, eks and faregates.

B

We are ramping up our test, automation, team, looking to integrate thing even more closely what development release a monitor monitoring cycle, we're looking at ways to reduce manual testing while improving developer confidence. One of the big tools that we can use for a lot of this is blaze meter. It's does performance testing for applications and we're looking to integrate that into our CI CD pipeline that we demoed and, of course we are looking to integrate even more cloud native technologies.

B

This doesn't mean taking the latest and greatest and then running alphas in production. We've learned that it's okay to wait and a to adopt cloud native technologies. Often they get better and easier to use over time, but sometimes you're put in a position where you need to move fast or you have a time deadline that you need to meet. So you will have to experiment, contribute back to the community to get there. It's do as a tool that seems to cut across many of the concerns that we have.

B

So that's something that we are looking at along with a lot of the other things you I'm sure you've been hearing today and will tomorrow so the C and C F. We are a end-user member of the CNC F and we're very grateful for the CNC F community, which is all of you and the foundation. It's an advocate and supporter of standard and open tooling, something that we really like of best practices and, of course, of vendor support.

B

This gives us confidence to make those technological choices, knowing that there's a community who will face the same struggles and reap the same successes as us, so working on the new york times is you kind of are an agent of change if you're keen on what we have talked about and want to help us build on these tools, help us build trust confidence in our systems, we're hiring. So you know visit this.

B

We also try to document a lot of what we do in our blog blog posts, and a big shout out of course goes to the rest of the deliberation team who's not here today, who still doing all the good stuff on the product and development people at the new york times for learning and executing this cloud migration and the leadership who put the resources towards actually doing this.

B

So there's our contact information feel free to hit us up on Twitter or email us we'll be outside after the session. If you have any questions and yeah I think that's it.