VMware TGI Kubernetes, 6 Sep 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: TGI Kubernetes 089: Tekton Pipelines

Description

Notes archived at https://github.com/heptio/tgik/blob/master/episodes/089/README.md

Content around Tekton starts about 20 minutes in.

Come hang out with Joe Beda as he does a bit of hands on hacking of Kubernetes and related topics. Some of this will be Joe talking about the things he knows. Some of this will be Joe exploring something new with the audience. Come join the fun, ask questions, comment, and participate in the live chat!

This week we will be looking at the Tekton CD project. This is an offshoot of KNative and one of the inaugural projects for the Continuous Delivery Foundation (CDF).

A

All right welcome, welcome, welcome everybody, happy Friday I am Joe Beda and welcome to t GI kubernetes for those not in the know. This is a weekly issue live stream that we do here at VMware. We talk about all things. Kubernetes do some live coding, learn some stuff together, pick a topic and we've been doing it for quite a while.

A

Now we're almost on episode, 90, where this is 89, we have to do something big when we actually get to to 100 so yeah, so just for those who haven't joined us before the way that I generally, this works is say hi to everybody in the comments here, I really want to be as interactive as possible, with the comments make sure that we cover sort of what's new in the kubernetes world and then dig into a specific topic.

A

This week it's going to be Tecton pipelines, something that I haven't, don't have any real experience with, but we're a learn together and.

B

A

I forget to say: I'm Joe, Beda I'm, a principal engineer here at vmworld VMware and came from hefty, oh and actually helped start the kubernetes project when I was a was at Google. So let me first say: oh man! Thank you. Everybody for saying, hi and joining in like Suresh was here early he's joining us from Hamburg. Jason is in the Google office in New. York thanks for staying late, Jason and Jason is actually a tech town contributor.

A

So when we go off the rails, when we do something stupid, Jason's gonna actually help get us back on track. So thanks for joining us, I really appreciate that. Let's see we have a Malik from India joining from the VMware office in Bangalore, you really are staying up late, I'm actually gonna be in India next week. It's the first time I'll be out there in, like five years, I'm gonna be hitting visiting some customers in Mumbai and Delhi and then and then visiting the the Bangalore office.

A

A really quick trip we'll see if I, if I stay awake, let's see Ola from Copenhagen good to see you yatin from Virginia Rory from Scotland. That's a very Scottish name. Rory welcome! Welcome, Aled from from Brisbane Bristol England Shahar from atlanta l'm adi good to see you Sandeep Christy MA from Strasbourg, oh whoa.

A

It just jumped on me here, Nuno from Portugal Thorsten from Germany Nuremberg Harry from rotten, dumb Sean from Birmingham hicham from Paris aren guv 90 we're almost an ID yeah Alexis from or lean or Orleans, which I assume is France Carlton from Austin Duffy from San Francisco how's it goin Duffy, Oh, Aaron, Go, Go Go Gutierrez is from Colombia Joe from Atlanta vidiian Wow Wales good to see you Steve is out in the East Coast, where you're at in Pittsburgh right, Steve, sloka James from Minneapolis Lewis from Cardiff Ishwara from India, oh and then Christie Wilson is also a Tecton contributor.

A

So welcome welcome Christy thanks for joining us squall Bogdan from Romania born Christian from Germany Karim Munoz.

A

You know this is the thing that always catches me by surprise is that folks are really coming in from everywhere, with a lolicon from Nigeria as just super crazy. So thank you. Thank you. Everybody for joining us, okay. So the first order of business is I, moved in here to our offices and VMware in Bellevue, and we are not allowed to keep beer in the fridge.

A

So I got my own fridge and then I went to Costco and I got a flat of like monster, because you know it's afternoon: it's a Friday, so I'm going to decide what flavor I think I'm gonna go with purple here, because monster doesn't come in flavors, it just comes in colors, and so that's what I'm gonna do I'm gonna do purple yeah boggans, saying that saying hi to everybody doesn't really scale so cheers everyone.

A

So let's go ahead and get started so if you go to and I want to switch to my screen here, if you go to TGI Kyo notes, it'll, take you to this hack MD here and I'll. Go ahead and I'll put this in in here, and this is our notes for the episode we can crowdsource these things, and so, if you log in you know, please be nice, don't graffiti, but you can add to the notes if there's a link or if you want to add some time codes, you know we can do this.

A

I, try and clean these things up check them into our github repo. After the fact, and so everybody should feel free to get involved. There. I have a lot of stuff that I want to talk about before we get started so I'm gonna, try and move through this stuff really fast and I'm, not gonna click on every link.

A

So, okay, so last week we didn't have an episode, because I and pretty much everybody was down in was down in in San Francisco for VMworld, and we made a lot of kubernetes announcements down there and the first thing that we talked about was being worked on zoo and I'm wearing the t-shirt, it's kind of our Ironman t-shirt. We got that on the back and on the front logo on the back, um so Thanh Zhu is our overarching sort of portfolio of products related to kubernetes and modern applications.

A

It's just a way for us to give a name so that when we talk to customers, when we talk to folks, we can talk about the entire experience that we're providing not just specific products, and so that's something that we're super excited about. I'm, not going to click through all the links here, but we have something that introduces Thanh Zhu in general, but then we really break this down around. What we're talking about as as build run, manikins build is how do you build and run applications? How do you manage and deploy those applications manages?

A

How do you manage this from IT point of view across multiple clusters?

A

Specifically, how do you actually start meeting that sort of IT policy meets application team and really create the right interface there so that those teams can can it can coordinate and cooperate and then finally and I'll explain what Tom Zhu means in a second here and and then finally, we have rod, and so this, how do we provide a way to run kubernetes in a whole bunch of different places, whether that be on the cloud, whether that be on on something like vSphere, on Prem or even down to bare metal and so, and so we had big announcements in each of these things in the build category we announced that you know- and this was actually happened during the earnings call the week before, but VMware announced the intent to acquire pivotal, which is very exciting.

A

It's gonna be a lot of work on everybody. You know to figure out how to bring these teams together, but it just makes so much sense. Super excited to welcome the pivotal folks on board that manager side. We introduced this thing called ton.

A

Zoo Mission Control, which is a commercial service for being able to being able to manage and set policy across a lot of clusters and then and then finally, there's project Pacific, which we announced, which is really building kubernetes into these fear and are in a pretty fundamental way, which is really exciting in terms of being able to bring new capabilities to vSphere built on top of kubernetes.

A

So it's really not about just running kubernetes on v's here it's really about how can we use kubernetes to make vSphere better for all these fear, users, which is really exciting?

A

Tansu itself, is a name. There's tendu with a Z is actually Swahili. I understand it means sort of like branch and so I think you know the connotation of like hey branching out in a new direction is really really fun and then tansu with an S is a Japanese word for a traditional Japanese cabinet that was actually built to go on the back of horse carts or on ships or whatever, so it's flexible and durable, and so it's like a portable container, which is actually kind of cool.

A

So that's where the town zoo name actually comes from there and then and I see that folks are already asking questions about actually deploying and installing a Tecton which is awesome, definitely go for it and then there's one thing that I did want to call out here. Is we introduced this thing called kubernetes Academy and we're really excited about this and we even got a cool domain kubernetes dot Academy, and this is a bunch of learning resources to start getting into kubernetes, and you know so.

A

We have here like things like a containers, 101 really just getting folks in and some of our some of the the VMware folks have contributed here, whether these be sort of our trainers or some of our architects like Duffy and Scott, that go out into the field and work with customers or just you know, other engineers or folks in nvm VMware, so super super fun and interesting there and so really excited about this.

A

The reception here has been great and a lot of folks have reached out, saying: hey you know: can we actually help build this up and actually get involved with this, and so we're definitely looking to make this to be a more community. Curated resource really excited about it, and- and we want this to be vendor-neutral so, like you know, I'll talk about towns who I'll talk about Pacific, but I think you know from our point of view. The more people can understand and be effective in kubernetes like the better.

A

It is for everyone, and so we're really viewing this as a that's something that stretches across all that and I think also like we're. You know we we have some of the TGI K stuff going on here. Also so so you know you can find the link to that. So we're trying to sort of make that part of the the larger footprint. So that's super exciting, ok, other stuff we got to go on so 1.16 is coming up and there's gonna be some api's removed, some deprecated API s.

A

We've talked about this before a valley wrote a great blog article about it, if you're not ready for this and if you're just like on cruise control, this may catch you by surprise, and so it's something that you should be aware of. The cube con cloud native con schedule is up really exciting. If you follow sort of kubernetes, twitter you'll see that there's a lot of people go and yeah and a lot of people very disappointed that through talk didn't get it accepted it's it's always hard to do these things.

A

Bryan Liles, uh one of our colleagues here at VMware, was on the the program committee and I think it was something like like you know. Only 10% of the submissions were accepted. So, but it's looking like an incredibly strong set of talks. It's going to be a really exciting conference. Tickets are still available and I think if you get in early there's still a way to save money, there's still an early early ticket price. This is gonna, be down in San Diego.

A

If you did not have a talk accepted or if you can travel at San Diego early, there is the rejects conference, and this was been put on by the kinfolk folks, but I know I, think they're there, the main sort of sponsors and coordinator there, but this is really a fun event. So this is really just folks that are like hey. We, you know, I had an awesome talk, it didn't get accepted.

A

This is essentially an is like an extended cube con where a lot of these talks that that that didn't get accepted can gather- and this is really community- draw around grassroots. It's really fun to actually see this come together. So huge fan of the rejects conference and what they're doing there I don't know. I'm gonna have to see if I can actually get in early and be able to do that.

A

I, don't think I've bought all my tickets yet and then, if you're a contributor or would like to be a contributor, then we have the contributor summit, and so this is this is for folks not just using kubernetes but actually helping to make kubernetes better and there's really two tracks to this and george, who actually know. George hasn't been super involved. This time Paris and the contributor experience.

A

Sig has been driving a lot of this, but there's one track, which is current contributors in terms of like organizing making decisions being able to sort of you know, have teaching sessions for how things work and then new contributors there's a whole effort around sort of like how do we make it easier for folks to figure out how to be effective, how to get involved in kubernetes?

A

So if you're gonna be a cube con and you want to learn, you know about how to become a contributor, that's a great place to go, and so please check that out.

A

Let's see so at CD 3.4 is come in announced, which is super exciting, so really great. To see that this is you know, development and Etsy D is continuing to pick up and I think what we've seen is an effective handoff from Etsy D from being a core OS RedHat thing to something that is really being community driven for me. It's really great to see that you know folks can often move around the industry between different companies, but they can also continue to essentially invest in project like Etsy, D and I.

A

Think you know, and looking at this from the point of view of the CNC F scene, project sort of transcend that sort of vendor focus is really really good and so and there's a lot of improvements coming in at CD 3.4. That I was really going to help with kubernetes scalability and efficiency. So there we go, and so here's a here's, a big summary snapshot, a blog post on that cluster API. We're big fans of cluster API here at VMware, putting a lot of work into that view.

A

One alpha 2 is out or is about to be out. Is it out out? Yes?

A

Yes, a lot of improvements, a lot of lessons learned there and then duffy says that at CD 3.4 will likely merge in with kubernetes 1.17 and so something to look forward to there and so cluster API stuff is making progress really great to see that moving forward the end point slice API is coming in 1.16. This is not something that I've dug into so I have not there's a cap on this. That goes into this. My assumption here, I, haven't I, need to learn about this.

A

My assumption is the goal here is to actually be able to get updates to endpoints in a much more efficient way. So this is an efficient way to get that, because these are one of the most heavily trafficked resources in kubernetes. So it's probably a big efficiency thing.

A

Yeah, but this is not something I've looked at, so this is actually exciting to see. Endpoints continue to move forward. One of the things we don't have here, but I'm really excited that's landing in 1.16, is dual stack. Support. I was I was down at a thing in San Francisco yesterday, talking with Tim Hawken, who leads sig networked super.

A

You know long-term like pretty much. You know oh gee, kubernetes and- and we were talking about the the dual stack support and it's actually more than that, their work was done and in and call was driving it and I'm.

A

Trying to remember who else was was really driving this from from Microsoft, but working with Tim in the community to be able to take like we made a mistake early on with kubernetes with things like pod definitions, where we only assumed that there was going to be one IP address one interface and essentially pluralizing all the networking resources across all of the resources and figuring out. How do we gracefully actually move from the old and the new different ways of representing these resources?

A

Lots of tricky API issues there to make that happen, but that work is finally landing. It's been an enormous amount of work, and so that's actually really cool to see that happening. I'm such you know, networking related here, let's see an overview of cron jobs from I know. I don't want to pay for for a medium, so this looks really interesting using kittens. So it's a hadn't seen this.

A

Yet this is super cool and I think to be interesting to look at sort of cron jobs versus some of the pipeline stuff and how these relate as we look at Tecton great primer of kubernetes on the edge talking about some of the some of the things to think about here. Edge infrastructure versus edge devices versus sensor is different, like ways to deploy.

A

Kubernetes I haven't looked at this too closely I'm, hoping that folks aren't suggesting that you connect nodes over way on links because I think that's a bad idea, but given any security on it, let's see, and so yet so, Tim, I and I believe hall will be giving a keynote at coop con about the dual stack support. So so that's super excited exciting to see that stuff coming.

A

Let's see the security on, it could burn any secure yacht security audit, some really good data. Here, it's a good takeaways.

A

You know we're learned, we're learning we're improving! You know: I was at this um yeah enterprise TechCrunch event. Yesterday, you know, like I, said, with Tim and Brendan was there and and.

A

And you know, and one of the questions that came up was like hey: what's up with kubernetes security, and you know and and I think trying to remember who it was I think it was was, was Tim who's like well, you know, security is not a binary thing, it's really a journey, and so you know I'm really excited to see us that were continued to invest in kubernetes security and improving things and then for those who are part of the community. The kubernetes election is underway.

A

This is for the kubernetes steering committee, which essentially helps to guide a lot of the governance structure of kubernetes. So there's a post here in our this is a Google Groups thing that George sent out he's helping to run the run this about the schedule and how this stuff works. So, if you're a contributor to kubernetes. This is something you should be aware of, and you know it's a way for you to to make sure that you know you have your influence in terms of the direction of where kubernetes is going.

A

So I'm on the steering committee now, but after this election I won't be so it's really exciting to see you know the project be be handed off to to to folks who've, really, you know put time and effort and passion in the community. Oh it's.

B

A

I'm doing purple, monster, Brian and then and then and then Duffy just added this thing talking about how does Cube control exec work in detail which looks really exciting, because this is one of the more complicated flows with things like like I'm sure, like you know, we're doing like WebSockets and stuff like that and speedy and stuff like that. So that's looks like a really cool, in-depth analysis there, all right huh all right.

A

So that's me getting through the notes really fast trying to get through this as much as possible so that we can dig in detect on a lot of news to catch up on both between cube, con and and the stuff coming in 1.16 and the stuff that we're excited from the from the VMware point of view. So hopefully, if you want to dig into any more of that, the links are in the notes.

A

Okay, let's start with Tecton, so Tecton is I'll. Give you a little bit of project history and you know, and and folks who know better. You know please correct me if I got any of this stuff wrong and again, we have you know.

A

Jason is is on the line here in the comments helping us out and and Christy also both tectonic contributors, and so they can correct me when, when I get this stuff wrong, so Tecton is kind of a spin-off of the key native project and soak a native originally was really essentially, let's break down the the serverless experience into into a bunch of component pieces and, and one of those pieces was how do we build and deploy another woman was. How do we do like zero to one auto scaling?

A

How do we do like things like build packs things like eventing? These are all things that sort of make up sort of serverless and instead of viewing this as sort of one monolithic project. Candide have always saw these things, that's largely separable and the build stuff. Some of the build stuff was so separable that it actually got split out into a separate project, or at least you know, inspired a separate project but turned into Tecton and Tecton itself is not part of the CN CF.

A

It's part of this other foundation called the CDF, the continuous delivery foundation. There has been some hand-wringing and and drama around sort of CDF versus CN CF, but at the end of the day, I think it's a great sign to see. Google continue continue to contribute at least some open source projects into a foundation so that we can make sure that there's a continued community ownership of these things over time. So it's really cool to see that happen.

A

Does that being said, Tecton still a relatively young project and you can tell because you go to the official web page and it's like go go to github, so we can go to github and and and then get hubs like okay go to the to the official page.

A

No, but like I think you know it's still one of those things where the pretty page isn't up yet and all all the real nitty-gritty is happening in the github in the github repo, let's see, and so there's a there's, an org and look oh look like there's like a CLI and stuff and triggers and dashboard, but the main thing I think is the pipeline repo. It looks like so yeah I haven't seen all this yet so I'm still figuring all this out, so I am relatively new to this.

A

I just had a chance to glance over stuff. I haven't had a chance to play with Tecton in detail, so I'm gonna be learning with the rest of all y'all. So there we go.

A

Okay and then oh man, the purple monster, is very purple holy moly, and so it's called what is it called? It's called ultra violet yeah, let's see so um so. Traditionally, when I've done, TG I can I think each of us who does it differently.

A

You know we'll use sort of a different kubernetes to work against I've used a a cluster running on AWS, but this time I'm actually going to be using kind, and so this is my first time using kind for this type of use case and so I may I may hit my head up against issues with kind in addition to dealing with with Tecton.

A

So you know to two new things together, but but we'll learn and then set saying just got this running with proud triggers a few weeks ago, Jason was awesome and helping me to wire it in go tech on awesome. So that's that's. What I love to see is that these things actually become part of solution versus being like.

A

Here's like the whole thing, and so I love to see that hey these, these things can wire in um okay, so I have a kind cluster up and running, so I can do kind yet, and one of the things that I was trying to dig into is that Tecton is a pipeline for being able to do CI, CD types of stuff. One of the things that you do a lot of times with with with pipelines, is that you'll have one stage of your pipeline: produce an artifact and you'll.

A

Have another stage of the pipeline consume that artifact and to do that you need storage, and so you know and we'll get into the install instructions. But it looks like in terms of artifact storage, Tecton supports either persistent volumes or being ated upload to a google cloud storage bucket and and since, like I'm Ellie kind and volumes is something that I'm still learning about I think it works well as long as you're using a single node.

A

When you go multi node things get a little bit more complicated, so we're gonna be doing a single node kind. Cluster is what I have here. So I can do kind get clusters and then, like I, can do you that know how do I get more details? I.

A

Guess, oh well, so we have a cluster I can do docker PS! You can see that it's a single cluster and then I can do Cube control get nodes. There we go. This has been running for about four hours. Oh my new kind get nodes. Okay, so it just gives you the name. Alright. Well, anyways. We got a cluster up and running we'll see if we hit our head up against anything.

A

Hopefully this will work well and, and James was saying that he's uh he's got kind working with a with with Tecton, so we should be good there, alright, so there we go so now, like I'm gonna, be like hey here: let's install it so I'm, just we're gonna go through this together. The first thing you need is root or cluster admin on your cluster.

A

If you're running on gke, this can be a little bit complicated, with the way that, like gke authentication like intersects with our vac, and so you may have to actually do some of this type of stuff, but we already have a cluster. So we don't need to do that and then the first thing that we do is we can go ahead and actually apply it and we're going to be doing a cube.

A

Control apply against some random y ml and, let's see so, you know what am I okay, so I'm gonna do sorry, I'm gonna do a curl and let's look at this Oh.

A

What what yes go ahead to blow away my homebrew setup and I think my like my code command line.

A

A

Sorry give me a second here: I want to install so with vias code can I do? How do you install the like the CLI helpers.

A

Do you all know?

A

No, it's in my then code will work. No I mean it. You have to there's something that you have to do to actually have it do that to actually make it. Oh here.

B

A

Install shell integration.

A

No, that's not it all right, Oh muck with this later command shift, P.

B

A

Install code commanded, a path there we go does that do a restart to update now all right, I'm out I, have old, old stuff installed. Okay,.

A

There we go okay, so now we have code release TM, all, let's see what do we have going on here? A lot of stuff.

A

Actually, here's what I'm gonna do. Let's look at the proof, so we have oh wow, we're even doing like a pod security policy on this stuff, a cluster role for Tecton pipelines, admin and this thing can essentially get lists, create updates secrets, so it pretty much is cluster root.

A

All right and then we have a service account in the Tecton pipelines and a role binding of that to the admins okay, so we're essentially creating a cluster admin role account in the Tecton pipelines. So this is something that I always have a hard time with. Is that, like I, would love to see a way for these things to be installed in a sort of more localized way? It always scares me when I see like oh hey. We have a we're installing a service account that gets routed across everything.

A

It makes me feel, like you know like we're, creating another tiller now, not that we are but, like you know it's it's something that always gives me a little bit of pause, and then we have a bunch of resource definitions, I like to see the resource definitions being installed via the ammo versus actually being installed on first run, because I think that means that you can take those permissions away later and then, let's see, and then we have another couple cluster roles. We have a service with a pipelines controller.

A

Well, we have a web hook service and the webhook.

A

Okay, then we have a config map, artifact buck bucket and artifact PVC. Do we want to install both of those things or do we want to actually install one they're, both null and then Tecton default with a default timeout and then a config map for Tecton pipeline? So we have a bunch of like config II stuff going on here.

A

Okay Jason says that there's an operator here to help install this. Yes, I was just talking to somebody about this today. So let's take a look at at that operator.

A

Do you think, do you think Jason should I use that or so I, just like slam, the the yeah maligning and move on with my life.

A

Is it worth? Is it worth doing that it slam the ammo? Okay, we'll do that? Alright, so we won't. The operator is really interesting because I think we're starting to see.

A

Is that, like things like sto or Tecton, or you know, these things can be complicated enough to config themselves that you end up with sort of a bootstrap, install or operator that gets install that then configures the larger system, which is really interesting, and then that start to overlap with well, you use hammer to use an operator to use helm, install to get operator and then whatever so Flamel, yes, lambo, that's our new term when you just slam the amal in so a lot going on here. Alright, I'm gonna! Do we do cap?

A

Let's try it see if we can use cap to apply this.

A

Just it would be cap to do what would the thing be, deploy.

A

A

Okay, deploy we have to. I.

A

Like captain, it shows us the preview. So, let's see if we can actually make this work now I'm like losing, you have to give it a nap and then a.

A

A

Does not slamm all shirts so.

B

I'm mixing I'm.

A

Mixing up projects here, but I have any like cap, but this this is oh.

A

F, that was the flag guys looking for alright and so here yeah. So this gives us a summary of all the stuff that we're gonna be deploying Oh. Mr. Windsor how's it going so Michael here is actually the product manager, at least one of the product manager over had people for Tecton and Michael and I worked together back in the day on Internet Explorer, so I started my career here.

A

Working with Michael, and so he has a long and industrious and interesting career alright. So this shows us what we're doing so: we're creating a bunch of C RDS we're creating some cluster roles, a name space, pod security policy, a lot of really interesting stuff and then a bunch of like configs going on here. So let's just go ahead and we're going to slam this in.

A

I'll drink my purple drink ermahgerd, that's right! The whole team is there, so always Kim part of the part of the team. Also, alright. So there we go cube, control, get namespaces and we now have Tecton pipelines and we're off to the races here, and so now you see why I like cap, isn't that cool that's much better than cube control apply. It gives you all this stuff and then actually, when you do a further deploy, it actually tells you.

A

It also tells you what the diffs are, so that you actually have a much better idea of what's going on alright, so we're not going to do the operator we'll skip that for now, so we got that going on and then we talked about resources and artifacts. How do you share what artifacts between things, and so so we have to configure a persistent volume, and so when looking through the amel here and I and so to be clear, like I, think the install instructions could be clear here.

A

I think I know what I'm doing, but I think what I can do here is I can find there's a config map here for the artifact, so it says create one called config, artifact PVC here and then the data right. Now, it's no, but it's saying that I you want to create size, five gigabytes in the storage class name, so I think I can do size.

A

Five gigabytes will do that by default and then Cube control gets storage, class and I think we have one called standard, and this is the stuff built in two.

A

Built into kind as a standard is what we have here. So if I do that, then I think we're good to go and now, if I, do they like the cap deploy again now, it's like hey here's, what's gonna change and it says: okay, we're changing the artifact and then oh. This is interesting. That thing it thinks it's going to do an update to the controller in the webhook and I wonder what those updates actually look like. We do like a deep diff I.

A

A

A

Diff changes, yeah, okay,.

A

Oh, this might be a bug with cap because we're redeploying this and it's actually all the defaulting- is actually confusing things all right, interesting all right! Well,.

A

It's cap becoming a standard cubic tool. I, don't think so. I mean like cap is really I. Would love to see more folks start to use cap because I think it does provide a better experience, but you know I'm not sure how much interaction has been between that team and that and the six CLI cute control.

A

Folks I think this stuff will also get interesting with server-side apply, which is this whole other probably too deep to go to right now about sort of like how do we actually deal with merging changes from different places into it into a single resource, all right cool, so we got that up and running so did I. Do the right thing there in terms of being able to you know the Tecton team and in terms of being able to configure that I guess we'll see as we go, but hopefully that was the right thing to do.

A

Alright, so now I think we got things set up, but what did we set up? I think that's the big question. So we're gonna go through the tutorial here. Actually so before we go to tutorial I. Just looked at the docs, and this is the thing that I think or let's read this okay, so Tecton pipelines are cloud native.

A

They run on kubernetes kubernetes cluster as a first-class type and then use containers this building blocks so that all sounds great I think not just cloud native but kubernetes native in terms of working with things like our back and namespaces, and all that instead of working against it pipelines are decoupled. So you can use one pipeline to can deploy to any kubernetes cluster.

A

So you can have your test deploy thing and that can deploy to your production cluster, so you're not necessarily running a bunch of your sort of build deploy infrastructure on the same cluster that you're actually running your workloads on I. Think one of the things that I think from the point of view of sort of like original kubernetes folks. That caught us by surprise is that you know inside of Google.

A

Board clusters are huge and they're centrally managed by by Borgess re and they're, very much sort of multi, tenant or team, at least across all the users, and so you'll have search running next. To Gmail running next to MapReduce running next to machine learning, all those things will will be running on the same cluster and creating a new board. Cluster was like essentially buying a building, and so it was a big thing. But as we move to cloud as we build kubernetes, the original assumption with kubernetes is again.

A

We were going to do these big bang clusters, but once gke went out there and once we started automating a lot of this setup and teardown all of a sudden folks started seeing clusters. Is these mutable things also?

A

And so we moved from a world inside of Google, where you have these, like big-ass clusters to a world where clusters tend to be more single-purpose and and so or at least they can be, and so people definitely view clusters is more immutable and so I think we're starting to see that the tools are adapting to deal with this Multi cluster world and I. Think the fact that the Tecton works cross cluster is a great example of that. So Jeffrey says a cap like providing lots of insights like terraform, yeah yeah. So that's why I?

A

Like cap- and we did I- did an episode on cap a little while ago and ytt, which is a templating solution. That is, is complimentary, so that's cool, okay um and then Tecton pipelines are type. So the concept of type resources mean that any that for a resource such as image, implementations can easily swapped out I'm interested. Is this using some of the sort of duck typing stuff that is sort of more endemic of Kay native I'd love to hear and see more about that all right? And then as we go?

A

And if we just look at the the docs I, look at Doc's at head looks like the main objects whenever I'm looking a system like this I'm like what are the nouns. So the nouns we have here if we have pipelines and pipeline runs so a pipeline is just a thing that can be triggered and it runs to completion.

A

So it's kind of a little bit like a task, but it's more complicated than that and Jason's saying that this does use some of the duck typing or Christy is also saying that using some of the key native packaged stuff and shared lives and in, but they want to do more in the future. So yeah, the duck typing stuff is super. Interesting I think it's really really cool and.

A

Might be worth doing an episode on on that, okay, or at least you know, I did have pointed out like I, think, veal, a and and and I don't know. Who else did a talk on duck typing at the last cubic on and then there's tasks, which are the individual things that are part of a pipeline, cohesive and loosely coupled and then you have tasks runs tasks can depend on artifacts and parameters created by other tasks.

A

So it seems, like tasks, are kind of like a kubernetes job, but with sort of more interface, information about inputs and outputs, and then pipelines are a thing that string together a bunch of tasks. Then tasks can be invoked via tasks runs and then pipeline resources are those artifacts that get pushed around between them and then there's like how to create a new pipeline and well, let's go through the tutorial, because I think this is the sort of much more detailed but I.

A

Think for this type of thing, going through the tutorial makes a ton of sense. Toriel walk you through creating simple tasks pipeline and running them by creating tasks writing test pipeline. We're gonna create a hello world task and then a hello world pipeline.

A

Okay, so a task defines the work that needs to be executed. For example, the following is a simple test that will echo hello world. Alright, so we'll do that.

A

Tacked on tutorial and we're going to call this hello task.

A

And and let me close this window and if all works well, I can do code and magic happens. Okay,.

B

Cool and we'll.

A

Just put this on here, so what do we have? So this is pretty simple.

A

A

Metadata spec have a set of steps, name, image command and arc, so this is like a simplified pod template. So one of the questions that I have here is, is you know we did things with pods that you could have multiple containers working together. As we look at things like this, do we have a way to actually sort of implement, sidecars or initializers, or that type of thing with this and so like?

A

If you look at something like Sto, a steel will run a web hook for being able to inject sidecars, and so you can definitely have like a an admission controller mutating. The mission controller that can inject a sidecar for you in a transparent fashion, but I think there's times when you want to sort of manually inject the sidecar as part of sort of your buildconfig pipeline, also, instead of actually having something doing that at run time and so Jason saying, okay, sidecar support just LAN, you can specify sidecars siblings, two steps. Okay, so that's interesting!

A

So then, there's a recognition that you have sort of your your main container and then you have a bunch of sidecars, whereas I think kubernetes in general. When you look at a pod, there is no idea of the main container. All containers are kind of equal in that in that level, I think. Maybe it would be interesting to think about like what would it look like if we did define the idea of a main container in kubernetes pods? What would that actually help us with, but.

B

I could see.

A

For something like tacked-on, where you want to actually be able to judge it's much easier to judge success and failure and stuff like that, startup shot down like the sidecar kind of runs forever. But you know hey when the main container shuts down, then you probably want to pull down and actually declare that the task is done so yeah there's some sticky issues with expect to sidecars and things like jobs for sure, okay, so Jason's saying that we can do some of that. So alright, so this is just gonna.

A

Do an echo hello world yeah, so inside cars are killed when the steps complete sidecar, adaptor ambassadors exactly.

A

Okay, so if I have multiple steps, do that do they run sequentially inside of a inside of a task? Do they share, like you know so named.

A

So I assume I can do something like that.

A

Kind of like curious about the breakdown of steps inside of a task and then tasks inside of a pipeline. Okay they're run sequentially in the same pod: okay, sharing an empty during volume at workspace, okay. So the idea here is that it's kind of like a pod but with sequentiality built into it.

A

So that's actually cool alright, where you assume that you're using the same footprint sharing stuff between these things is really easy, whereas sharing stuff between different tasks will be much more heavyweight and you'll see a lot more composability between tasks than within a task I'm assuming okay cool. Let's see so there we go. We got that and then a task run runs the test. You defined here's a simple example.

A

A

So this is interesting.

A

Okay, just like quick feedback on the tutorial you you know- and this is like you write an O'reilly book and they drill this type of stuff in in to you is that you know if you're going to give a code sample, give it a file name and then and then, when you do the apply, make sure you reference that file name, I, know exactly what's going on here, but like it's, you know it's gonna, be a little bit trickier for others to figure this out, because here.

A

Hello task right: okay, so we'll do a new file, low task run demo and we'll go ahead and we'll do that and task graph is echo hello world. That's! So this is a single run of these things.

B

A

Okay pipelines are how you do multiple parallel task runs. Okay, so that's how you start doing sort of the graph on this stuff. Well, look at that Michael's like delegating already okay, yeah I mean how you actually do. The sequential stuff within a pot is gonna, be interesting. I'm definitely gonna be looking at that. Okay, so uh so now I can do cap or I'll just do Q control apply a chef below task.

B

A

So the hello task is sort of the schema. That's kind of like the class, whereas, like the run, is the instance. So now, if I do Cube control get tasks, do I. Do that I get that and then I could do. Oh here's.

B

What I want to try do I have I.

A

Did blow away okay, brew, install okay.

A

We can use this to explore what we got going on.

A

Wait so octant and and I'm Brian did a Brian did a TGI K on this is essentially a dashboard each type of thing. It's sort of a visual cube control to some degree, and so we should see.

A

Custom resources here we have yeah here we have echo hello world and we can see we can see okay, what gets added there and what gets updated. Okay and the relationship between these things. Okay, cool all right. The Tecton dashboard is a good way.

A

So here's okay, so here's my problem with dashboards that run on the cluster- and this is one of our one of other things that we thought about with with octant- is that when you run a dashboard that dashboard runs with credentials and then you have to scope the credential there becomes this credentialing problem where either have to pass your credentials through which becomes a pain in the butt or you have to try and match up or whatever, and so, if you are dealing with our back and scoping things down, dashboards always get really really complicated.

A

And so that's why you know I think you know octant is is from our point of view really interesting, because it actually reuses your local credentials. Actually- and let me start this in the in the background here.

A

If I do this, oh that's I think oh man I had a session. How do I create a new there? We go there's my profile.

A

Kind, get cube and fig path and do.

A

All right so we'll just leave this running in the background here and we'll see what's happening. um So one of the things, if you all are up to it, I'd love to see, there's a plug-in model for for octant I'd love to see us having like things like a Tecton specific set of plugins for octant, which would be really really cool, Oh Jason, saying, hey started looking at that already graph is support out of the box and all that yeah. So that's super cool, okay, but um we're that I have this done.

A

We can look at the. If we have time we can look at the Tecton dashboard. Okay, but now I can do a cube. Control applied a chef, hello tasks, Ron yeah mo so this created it and- and we can look, they task runs all right, so status, okay and we got there's the tasks run, which is actually we have some pods. Those pods are orange, so something weird is going on. We're probably downloading a boot to image. Well bottom of the terminal is cut off. Okay. Let me go back sorry about that.

A

Oh you know that happens when I. When I created the new tab, so so I use this Mac utility called moom and I have a I have a quick key for essentially shrinking the window just so it works just right for for kubernetes I referred TGI, K, okay, so we have this okay, it's succeeded and we have an event that goes with it.

A

This is actually really cool and then, if we want to look there's the resource viewer in the relationship- and what we can see here is that the yamo says has a condition called succeeded. That's set to true, and so this actually is really interesting, and this is something I want to call attention to. This is some of the duck typing type of stuff.

A

That is, is one of the patterns being driven by K native, which is the idea that we have conditions and we have a standard set of sort of like schema for conditions, and the idea is that they have three states unknown, true and false, true as good false as bad unknown as unknown, and then there's things like succeeded or or or you know, ready that our standard things for like hey this thing is actually sort of hit some sort of terminal point, so that's actually pretty cool, and so we can see here.

A

Okay, here's the steps out of each of these things that are completed when they actually completed and and I. Think. If we go back to to the tutorial there, we have the output. Now one of the things that I know you all spend a lot of time on is how do you actually get logs of these things, but I think maybe pipelines?

A

How do I know if it said hello, world logging, okay, logs, can remain in memory only as posts endorse receiver, such a stack driver, see, docks and getting logs from runs. How do we do that.

A

Get grep pod name, okay, so then we just go through and we actually sort of look at the logs and you get this specific container. So, let's see if we can actually see that from octant, so we can go through you. Look at the resource. Viewer find the we don't hyper link to this Brian, so we have a pod there.

A

Okay, this is another thing Brian that we probably want to do. I, don't know if you're listening man is like being able to see. Essentially you know historic, pods or okay.

B

A

Yeah I'd love to like understand this CLI, the the Tecton CLI logs are available via the underlying pod there's a CLI which will stream logs to you from the pod, but it's a pod gone now. At this point,.

A

Oh, no, it's in completed state okay, so it looks like this may be a thing with octant where we actually don't show completed pods.

A

B

Probably something.

A

All right, so, let's um let's go through and.

A

And see if we can, let's see so okay, so let's, let's do it two ways? Okay, so we have the pod here so in cube, control logs.

B

A

This and it's gonna tell me which one do you want. We want step echo and it's gonna say there is a hello world and hello again and I think we don't need to do the dash P because we don't have a new run. It's like the latest run is what it actually shows so.

B

It does run on top.

A

Of kubernetes jobs, okay,.

A

But the jobs are gone, but the pods are still there.

A

Alright cool alright. So so this went through and it ran those things sequentially inside the same pod, which I think is interesting. I'm gonna do so. Let's look at this. Oh, I can't look at the pot in octant. Okay, cube control, get pods cube control, get oh yeah mall.

A

What do we have going on here? Let's look beneath the covers and see the magic okay. Okay. So we have a one container here.

A

Which isn't okay, so we have have a container the args. Our wait. File post file wait file entry, okay, hello, world boom boom boom with a bunch of of this stuff and then okay. So that is that's the boom to image and then eventually echo hello, world. Okay, so.

A

Builder tools, entry point: okay,.

A

Okay, I get what's going on here. This is fascinating, alright. So what we have is that there's a there's a lot of stuff there's a lot of magic going on here, but basically what happens is each of the containers they they inject some binaries into this via volume mount and then they essentially use utilities from that volume to be able to coordinate sort of startup of things across different containers, and so we see well I, don't know. So. Where does builder tools come from?

A

That's the name of that volume as tools, it's an empty derp and that probably gets populated by in an it container. It looks like so that's a creds, init type of thing and there's ko in there somewhere too. Okay.

B

A

That's that's a little bit. That's a little bit of magic going on there really looking I really look to see how the sausage is made. Then Christy is going to be given a cube. Con talk to actually digging a little bit deeper. So this is why I'm guessing y'all are using the abouttwo container instead of something like the Alpine container, because you probably want to assume that you have some stuff in the in the image, which is something I was wondering: okay, so, okay. So the thing that we're seeing here is out.

A

If you look at the pod, there's some voodoo going on here to be able to bootstraps secrets to to be able to inject the right types of binaries and to be able to make sure that it can do the sequencing and coordination between runs of steps within a single pod, but doing that in a way where you can still bring your own container to some degree.

A

Okay, Jason is saying that Alpine works to everything we need is populated by the ethnic container, so yeah. So that's really clever. I think it speaks to how flexible kubernetes is and I don't know if that's a good thing or a bad thing, but that's pretty cool okay. So so we got logs we're good and we can continue on with the tutorial here.

A

Okay tasks, inputs and outputs, Alpine edge, okay, yeah and the most common scenarios. The test needs to needs multiple steps with the input and output resources properties.

A

For example, tasks could fetch it source code from github repository and build a docker image from it. Pipeline resources are used to define the artifacts can be you pass in and out of a task. There are a few system to find resources. Types ready to use the git resource represents a git repository with a specific version.

A

Alright, so here the scaffold we're just using that as an example, get repo we're not doing anything super specific with scaffold.

A

Let's see, but if we have a UI functional test, CITV pom and we need a kind virtual machine with VNC. Ok, so here's something interesting, we're looking at doing something like cube vert as as actually being a resource that you can orchestrate for VMs and so yeah I think launching and managing VMs is PI to the part of these things is not totally crazy, either.

A

Ok, so we're using scaffold just as an example as a git repo and then we're also taking in an image which is a GC our image Leeroy. This is one I think this part of the scaffold tutorial and then we can talk about build and push and stuff like that.

A

Alright, so let's, let's play around with this okay, so we have this thing here again: we're gonna do we'll do a new file. Will Gault call this artifacts yamo and we have.

A

So we have that and then and then we have another one I'll use your project.

A

Okay. So let me gonna be interesting.

A

Credentials get interesting here: um alright, let's let's, let's play around with this, we'll figure out how this works.

A

Let's use GCR just to be safe. Let me I'm gonna switch to my face here, while I actually see cuz, I didn't wasn't planning on using Google, so I have to do Google, ect cons, I'm tempted to say: hey, let's use Amazon but I'll, be nice. I mean login and make sure I'm at the right one. Here.

A

What project are we talking about here this and then, if I do.

A

Where is your I'll switch back, whereas GCR container.

B

Registry, okay,.

A

And I have all sorts of random leftover stuff I have a hypercube from 2017 yeah. You can tell I haven't used this in a while and so okay cool.

A

We have settings and stuff okay, so this is fine, so we can use. This is JB de hefty Oh, as the is the thing there. So we can go back here and my project will be be viewed a hefty. Oh.

A

Now the question: ok, so we'll do this good enough, but credentials end up being a really tricky thing when it comes to these things. So let's see how that works, cube, control, cube control, apply chef artifacts, so those things are up and running.

A

Let's go back to the tutorial and then here we have a task called billdocker image from git source and we're gonna.

A

Touch build and it yamo.

A

Build image yamo we're gonna, go ahead and do this. Okay, so.

B

What do we got here.

A

Can you all see this? Is this big enough for y'all? Let me know if this is too small to read, um so we have a task and that task has inputs. Okay, so one input here is the thing called docker source of type yet, and so I want to look at the artifact again, so the type is so so. The CRD is just pipeline resource, but there's a type there, so we don't actually have, and then these things end up being just key value, pairs and stuff. Okay, so this is.

A

This is like a subtype I mean you could also go down the path where you create a CR, deeper type of resource, but I think that becomes problematic when you want people to be able, define new things, and you really need sort of cluster admin to define new CR DS. So so this is a duck typing type of thing.

A

Perhaps yeah I think this is a place where you can use duck typing or you could either have this generic pipeline resource or you could have more strongly type things that follow patterns that you can use all right. So so we have the build image task. Actually, let's rename this can I rename it to build image task.

A

A

Okay and then we have a path to the dockerfile, and so this is workspace. Docker source docker file.

A

Okay I'm wondering, and then we have path to context well, so these are the inputs. Okay and then we have path to context, which is the build context used by Kanak, oh with docker source, okay, and so so anytime. You do a build of a docker file.

A

It takes things from your local directory and actually you know local environment and uses that when you do a docker build behind-the-scenes is essentially captures all the data it needs tarz that up and then sends that to the docker daemon, so that it can actually run the build process Conoco, which is this sort of I.

A

Don't we did an episode of Conoco essentially goes through and and makes it more explicit what that context is, and so this is the context that you're using I assume that this will actually sort of tar this thing up here and then it outputs an image. And then the steps here on the tasks are build and push where we are.

A

The image we're running is the Conoco project, executors specify docker config is required to allow Conoco to detect docker credentials. So this is the thing that I'm confused about.

A

Where is it gonna actually get this, and how do I actually modify and start mounting secrets and stuff like that into my tasks.

A

Maybe that's in the run, I, don't know and then command we're gonna just execute conte and then inputs params, okay, so this sort of interpolation syntax. This is something that cube control, I believe does where we're actually pulling the params through or no. This has to be. This has to be a task specific thing: I, don't think. Kubernetes knows how to interpolate that yeah. So Jason saying that the tutorial is missing, setting up the creds Jason.

A

If you could help me out here and just sort of like you know, put some stuff in the in the in the chat like that'll get me going, because I could kind of fumble my way through it, but.

A

I'm happy to actually quickly go through and actually create a service account and get that into a secret, and do that when you all aren't looking alright. So these are the the inputs and the outputs, so you can essentially use those as arguments in your steps, which is cool. This is very it's interesting. This is very Google, asked to actually add more and more stuff as command line flags, whereas I think a lot of folks outside of Google would actually do this stuff using just there's.

B

A

Of stuff outside of Google that that would use environment variables, okay, so let's go through, and then we have a task run here and we'll see what we're missing out of the tasks run and how we actually bind all this stuff together. So a new file build image task, Ron, gamal, so tasks run the resource. Here's where we actually bind in scaffold get- and we say: okay, the path to the docker file.

A

It's just gonna, be docker file and I assume that that's relative to the docker source path to context its workspace, docker source examples, microsurgery, Leroy, Leroy web, that's cool.

A

Then the output is going to be the resource that we had here that we already defined. So this is our output resource yeah. It's so understanding like the the secrets flow through this actually gets really really complicated. Okay, so, basically create a service account, give it a secret annotate this as a docker cred and then run then run the task run as the service account. Okay, so let's go through and will fumble our way through this cube control create sa task run.

A

So we have that service account. I have to create a secret. So let me go through and GCR dockers kubernetes secret and see if I can find the way using GCR with some random container solutions. Well, look. We have a hefty Oh thing here. We wrote this awhile ago. This may be out of date, so create an image pool secret container.

A

Alright, so so what I need to do is I need to start with Nabel on container registry api and did all that.

A

Jason Kyokai have to create a service account key and give it permissions and okay.

A

Let me go ahead and I'm gonna. Take you all.

A

Let's see I'm gonna, take you all off here and you're gonna you're gonna watch me be surprised. I'll talk through what I'm doing okay, so we got so you go to API and services credentials.

A

Create a credential, a service account key I, think it won't flash this stuff on the screen. I think we can do this without flashing on the screen, we'll see, I'll delete it afterwards.

A

So the service account oh I, had one that was read-only before well. We had to create a new service account, so let's create a new service account. Where would that be a.

A

Managed service accounts there we go and I can do a create. A service account I'm, going to call this one GCR read/write.

A

Create select a role, I have no idea. Let's see this is probably cloud I.

A

Let me talk about how what I think about GCR I am I mean you know nobody's in love with AWS I am but, like my god at least like you know: ok, filter registry, nope, GE, CR, nope that I have to be okay, so now GC r is actually I. Think so now, I'm like okay, GCS role for GC our project editor nuclear option. I know right like configuring, access control for GTR and I.

A

Think okay, you just need essentially storage admin for this stuff, and this comes back to the history of the way that GC r is actually backed by cloud storage, storage, admin which is sucks because I actually only want to do it for a particular bucket, but with EWS I am. I can actually restrict this as with a policy document that at least I know how to write. But ok, I'm sorry youtube is not sending my messages that have links in them. I'm not seeing them. I'm sorry Christy!

A

If you, if you need to send links you can, you can add them to the to the to the notes and I have been approving the URLs as I've been seeing them come up for sure. Okay, so storage admin continue with GCR and create key JSON key.

A

Private key stored to my computer show and finder, maybe okay. So this is my downloads. Okay, so now I can do copy.

A

Alright, so now that's my key I'm, not gonna cat, that to the screen, because that's bad news.

A

So now I go back to this thing here. I got that and I can do this.

A

Chasing key and then I need to do.

A

A

Think we're gonna do GCR. That I owe.

A

This username thing is like left over or the emails left over from, like the dawn of time,.

A

This is someone nasty because this will actually show up in my history with the full key on it, but you know Yolo.

A

All right, so we have that now. The next thing we need to do is add.

A

That secret to the service account.

A

And that is, we just do secrets and then we're image pull secrets there. We go like that. Okay, so a cube control edit si. What do we? What did we call it? You control gift.

A

So we can do that image all see, secrets and.

A

Gcr Jason key right.

A

Roy saying that.

A

Skips history, if you start a command with the blank space before the command? Oh, that's a good tip I like that. That's cool! Alright! So there we go now. I have a service account? Okay! Now the question is: how do I actually go through and actually say, I want to use that service account in a build image tasks run. Is it some tasks run? Is there a way to actually say.

A

Tacked on service.

A

Well, you do I, don't think. Conoco knows how to understand service accounts. Oh that won't work, then shoot or map the secrets in pipeline off. Let's see exposing credentials, get config, SSH basic, forget docker. Ok, so we have a secret user pass or okay. So this should secret.

A

B

This is a different.

A

A

Okay, Christie put the link to the docks where I should be looking in the notes here. Oh man, we have a slammin t-shirt already holy moly.

A

Okay, I am looking at that: okay cool there. We go I'm, looking at it, sorry, okay, off with Conoco, okay. So here we are down the rabbit hole, so we create a secret, a generic secret with Conoco and then.

A

And then we can just go rename it Conoco secret dot, JSON, okay,.

A

So I can do that we'll go through.

A

All right, download a Jason key for the service account rename the key to Bob Loblaw.

A

All right, so we have that.

A

Read from that bucket and this, so this is running through like different ways of doing this. Okay, so now so now we're doing Conoco.

A

So we have the Conoco secret, but we have to actually map that as a volume and get that in there, and so that's the thing that I'm wondering about so here we have this.

A

So we have a secret name type. It's not basic auth kubernetes.

A

Well, this is a docker config JSON.

A

I'm sorry folks, I'm struggling here, put it at the bottom. Okay, just replace HT to the year with something else like AG connect over the local docker registry. Also.

A

Okay, so this is the thing okay, so this is my. So this is one of the things I'm struggling with right here is that I don't understand the schema for whether this s, this secret should show up in tasks or tasks. Ron I'm, assuming it's tasks run, but it could be tasks also, but I'm, assuming it's tasks run and, in any case we're actually looking at docker config here in builder home got docker I'm wondering how do we actually?

A

What is the right format there and how do I inject that, using using tech time if you annotate the figure with the docker annotation use essay in your tasks, run Tecton will prepend steps in jet creds as the docker config JSON to your workspace. So tasks run okay, so each step is a container spec. Okay, so I have the essay set up I? Have it set up with the docker with the with the the registry credentials? I just need to use the essay to configure the tasks run.

A

So what is the schema for tasks run for tasks runs for the service account? Okay, it's just service account. Okay, so if I go through so theoretically, if I go through here- and this is not inputs or outputs, this is actually.

A

Optional inputs, outputs tab, because this is a top level thing here and we can actually do count like this and then what was my name of cube control. Get si was called tasks run so that theoretically should work.

A

And so now, if I do cubed control applied a chef with a build image, tasks run hilarity ensues. Let's see what we got going on here, octant still running in the background here and we should be able to look at tasks run tas run resolution failed. So what did we get wrong here?.

A

Err when listening tasks retire and build doctor image from good source, that's run billdocker image from get source not found. Did I not install the task? Cube control, applied a low, build image test. I, know I. Think I forgot to do that. Okay, so now, will this thing actually is this thing like hey I'm done or succeeded false? So this is a terminal condition. It won't try and resolve so cube. Troll get runs so cute control, delete tasks, run.

A

We delete that and now we can apply it again.

A

A

A

Containers with on right, not all steps that s it finished executing, so we have the steps, build and push completed.

A

All right so looks like did everything work.

A

Spec status, not all steps have finished executed, so it's still so succeeded is unknown. We're not there yet we're running. I should be a cute control, get pods my machine starting to sound like a leaf blower, so something's happening. You.

A

Brew install kale.

A

Where's, my utility, how do I install this we're racing? It.

A

A

Oh yeah, you know, octant does automatically refresh document. Does logs I'm an idiot okay so where it failed here.

A

You don't have the needed permissions to perform this operation. You may have invalid credentials, so, okay, so something wacky happened, failed to push destination, no token and bearer response. So I don't know so. Apparently the creds didn't work.

A

I'm, sorry Christy about the links.

A

All right, so, let's see if we can figure out what happened here.

A

Step credential there's no logs there: okay, oh and there's like specialized importers and exporters based on types, and so this thing knows how to actually.

A

Touch github commit ID and all that okay, so this thing knows how to like: okay, so we're grabbing scaffold, we were able to create the dur for the built image. There's a lot of convention going on here, I think one of the things I'd love to see is like talk about sort of like what are the different objects and sort of. What is the convention mean around this because there's like magic happening like oh, when I specify a git source, then that means that XYZ gets injected? How do we extend that? Is that extensible?

A

Or is it just that those types are built in that type of thing? Basically, it does not make the link between the SA and the secrets. You must reference. The secrets in the SA I did I thought did I forget to do that.

A

Let's see if we can do that, so if I look at.

A

Custom are backroll bindings where a service account service accounts, tasks, run image, pull secret is GCR Jason key, so I do actually have that. You must so add. Okay,.

A

You must also add to the secret is the fact that is for dr. with an annotation saying it is for doctor yeah, so I used that so I'm gonna flash this on the screen and y'all can like push stuff to my registry. But if I do, can it go secret here? You'll see that this is a service account here? Oh, that's that one okay, cute control get sa no get secret.

A

You'll, see that this is a docker config, JSON GCR key cube control, get that oh yeah moe.

A

And here you can see that this thing is a docker config JSON, that's been basic, ste 4, encoded again copy it off. My screen and type is docker config JSON. So it should know how to do that. So I have okay. So what I have here is I. Have a image, pull secret, that's in the right format and marked as such. I have that set as an image pool secret in the service account and then I. Have that service account hooked up through Tech time? Do I maybe have to let me try cube control.

A

Let's do it. So let's do this because, right now it's an image pool secret. Maybe let's just make it a plain old Cicotte see for a GC our base and key I. Think that'll. Do it? Ok, so now I can do delete apply.

A

Do that again, we're gonna look at pods here this thing's doing its thing. We can look at logs.

A

Credential initializer apparently, but like we can do step, building push okay, downloading the base image. So now, are you all thinking about like hey? Can we opportunistically have a bunch of artifacts for doing things like caching base images and stuff thinking about like the idea of having sealed artifacts, where it's like hey? This is something that is read-only after it's actually created for things like I could see like a whole bunch of like optimizing around artifacts and making sure that you're, not downloading or up downloading stuff as much. What would be would be good so.

A

Bill I did do the create secret similar to that. So we had that.

A

And Kristi's saying I, don't think Conoco knows how to consume the secret. Unless you mounted point can the kind of go path with an environment variable alright, so so we're building it again.

A

And we'll see what happens here and then we're running out of time here so well. If this doesn't work, then I'll just say: hey, you know configuring secrets and connections and image. So this is the type of stuff. When everybody deals with this, they just bang their head against the wall, it's all fun and games until you have to store to get all this configuration set up and so I think you know you know definitely.

A

You know making that be as straightforward as possible. That's definitely gonna be something that I think folks are going to struggle with and then being able to did it work pushed blob, woohoo, alright. So now, if we go through here and if I can find it wherever the heck, this is in the like Google.

A

None of these are any good by the way, just to be clear. I think just just like finding stuff in any of these consoles is really freaking hard, so compute where's gonna, do registry.

A

Gc our container.

B

A

There we go alright, so if things went as planned, Leroy that it worked, we have an image, six point: six megabytes we built it. We push it awesome, alright. So that's super exciting. That's very, very cool, alright! So sorry that that ended up being so hard because I feel like we haven't even gotten to pipelines yet but I think understanding tasks, inputs, outputs and like how you configure those I think is super interesting.

A

We did the tasks run here. Okay, now a pipeline a pipeline defines elicit X to X you in order what else indicating, if any inputs out which we used blah blah blah in the from field. So this is how you can sort of pipe things back and forth.

A

This needs a visualization, obviously now I haven't looked at like this starts to get very similar to things like Argo and I would love to like, compare and contrast what I haven't looked at our going forever, so I'd love to see some of the decisions made with Tecton versus Argo and actually see what's happening here, but so what we have here is I just want to understand what's going on here, so this is a pipeline. So the first thing we're gonna do is build, build and push the image.

A

Okay, and then we have another thing which is deploy web to web, which essentially is you have, let's say, a deployment, yeah Mille and then and then essentially the the thing that we want to modify and slam in there. So the parameter- and the name of the image of this is a task. We have to find this task, so this task actually takes deploy using cube control. We actually have a git source for the yeah memo. We have the name of the image and we have the image that we're using the.

A

So this isn't something that's interesting. So we have the image here. This is modeled as a resource, but also like is that it's a read-only resource, that's essentially a string. So in some ways this is a pram also, so I think you know I think it's also interesting like like, like params versus resources, there's a subtlety there when they're small enough that they're essentially just strings, and so that's something that's interesting. So, okay, so now we're actually going through.

A

We have like you, we're doing a little bit of like yeah mo mangling, and then we have run cube control. That's actually going to apply this stuff and then obviously, if you're, using something like customizer cap or whatever, that's like a templating and apply thing, you could actually do it, then also these things are actually reusing. Let's see so we have the Y ammo path to image, so the the inputs, params path path, is the path to the manifest to apply.

A

This thing is essentially editing that thing in place, but I assume- and this is again like there's an execution context for things in a task where they can actually share data between them. What that path, like there's a bunch of path names in here that feel very magic to me, understanding the environment. That's actually going in with this stuff would be super interesting to me and then just and then we do the run and we're saying. Okay, scaffold get so we're reloading. That gets we end up.

A

We don't have to resync, since this is a persistent volume we actually don't have to resync I, think the the the git repo and then and then that's the the image, the path to the manifest to apply, though I would expect to see that in to run the pipe Oh with the pipeline run.

A

Okay and then we have deploy a workspace source examples mike rizzo yeah, so like where different things get mounted in is something that like like, where does workspace source examples like what is workspace and how does that relate to the git repo right, because I think this is coming from the git repo and what is the rewrite model for that stuff? And what, if I, have multiple git repos coming in? Do they actually be source one and source 2? Ok, so the input name food gets mounted into food okay.

A

So this is source repo here, ok and then I see so that's the resource, oh and then in the tasks the input. So these are actually kind of volumes really and then here here we have an input. So the fact these are called the inputs and not volumes, but they get mounted in a specific place. That's a little bit confusing ok, but this source matches up with this source.

A

So if we had multiple get, we could go ahead and do that and then we have image here, which is a resource called web image and that's from build scaffold web, which is this thing, which is the output of this.

A

Built image resource web image: ok, so this is a resource I wonder what the from does here does the from like. Why do I need the from because they're, both referring to the same resource.

A

You have some resources, don't make sense to me volume on it, but then do they actually show up on disk. I. Think is the question.

A

Ok, but I think I understand sort of this gets this gets mounted in and it's a resource and then the resources that don't get volume mounted those things feel like they're kind of config parameters really, and so that's one of the things like there's a subtlety between artifact and and config here, but now we're going through and we're actually doing a and we have Lakis kubernetes, cube control thing and we're actually doing an apply there. And then this is a summing. This is assuming that you're just reusing your default service account.

A

So there's a permission issue here that you're not addressing to be able to make this stuff work.

A

I, don't see any surface account going on here: okay, we're clear out of time, but I think I start seeing how pipelines can actually, because this is interesting because you're, essentially we have the the build in push of the image, is one thing and then and I actually think it's interesting that like this is because Conoco builds and pushes, but we could have conical produce at our file and then get of another thing: that's like uses some of the OCI tools that takes that tar file and actually does the push so the build and push could actually be separated out into multiple tasks, and so I think there is an open question around sort of like how do you model tasks versus pipelines versus what have you?

A

So if you build an image, a task that gets the image from that image should be able to access the exact image that was built by digest. Oh I see so one of the things that goes into the artifact is the actual digest that got built so that you can actually make sure that you don't get any skew when you have a bunch of these things. Working in parallel right.

B

Because I could see like hey if.

A

You miss configure multiple pipelines and they're all building the same image and then your reference in that image based on a label versus based on a digest. They could actually talk past each other and you could get not hermetic results, okay, so that makes sense and then okay and then that from says I'm actually like I, want to make sure that I don't get version skew when I'm doing stuff, like that, I.

A

Mean another way to look at this is that you could do some sort of resource log right if for Reece, if you know hey, if I'm using a resource in a read, write mode, I can make sure that only one pipeline is using that at a time. But then you have to start thinking about critical sections through your pipeline around actually taking a lock on a particular resource across a set of steps. So that would be another way that I would actually be thinking about doing this.

A

All right pipeline run boom boom boom I, don't think we gonna be able to get through all this, but I think that gives us a good overrun of this very cool and then there's so much more to explore because, like we go here and we're like I didn't get a plane chance to play with the the command line. We didn't talk about the dashboard event triggering what these things ends up being really interesting, because this is where this starts actually overlapping with serverless stuff.

A

When you want to actually hook this stuff up to a to a trigger, you know, can one of the things that I'd love to see is that like do pipelines themselves, actually have inputs and outputs? What is the scheme of a pipeline? So can we actually create some sort of duck typing between pipelines and tasks, where I can actually run a task for a pipeline, but they both have inputs and outputs?

A

How do I actually specify that when I have a trigger, if I hook this up to an H, 2 DP trigger or whatever, how do I pipeline? How do I actually do the the inputs and outputs into the pipeline? So that's some of the stuff, that's interesting, a standard catalog of tasks and pipelines so that it's easier to find these things. It looks like there's a lot here, so you're not actually going through and recreating this stuff from scratch, and then you know there's a lot of llamo wrangling and references and stuff like that.

A

Can we actually create a sort of graphical tool for constructing these things, such that we can guide people to the right problems, because I think if I was writing that stuff by hand it'd be super super easy for me to fat-finger one of these names and have things not line up, and then that becomes a total pain in the butt to debug, so yeah, so really cool stuff.

A

All right so overarching thoughts. Is that I really like the project? I? Think there's a lot to like here. I would love to dig in and understand, sort of, compare and contrast with our ago and I did definitely see in the in the let's see where we go in the there's. The the operator for installing and managing the thing, but in the catalog I definitely did see that there's like an Argo CD thing so like maybe you can do some stuff and actually have these things work with each other, which I think is really interesting.

A

Having this line up with the eventing models that are being built up, whether it be cloud events or or you know, the other stuff happening in inkay, native or other eventing models, I think is really really interesting. I'd love to see this stuff actually be more less Google centric because like when I was looking at the the set up here, you know like I, look at in the install instructions. It's like you can either do Peavey's or GCS, like you know what about s3? What about like?

A

Mini Oh like and I, think you know looking at some of the stuff that we did with Valero in terms of actually making this work across a whole bunch of different storage. Caddy categories is really interesting.

A

You know the concurrency model getting more more clear about sort of like how concurrency works when you actually have multiple things operating on the same cluster is actually really interesting. Also, so very very cool I think I'm totally running down the clock here. Thank you, everybody for joining up. Thank you so much to the to the Tecton team for actually staying up late, because I know a lot of y'all are on the East Coast, or at least.

A

Hopefully you got a beer in your hand, and- and this is always fun I always enjoy doing this things that I want to do in future. Episodes is I had a series where I was building a minecraft controller operator. I think that'd be a lot of fun. I want to continue doing that.

A

You know, I, don't know if Duffy and others have ideas that they want to look at I'm, definitely interested in looking at queue, which is a config language, similar in some ways to JSON it a lot of interesting stuff going on there. If you got you know, Duffy's doing this series where he just digs deep into kubernetes concepts and really explores them. If you all have ideas on what you would like to see, four episodes reach out on Twitter or you can go into the the TGI K repo.

A

It's hep do /t GI k on github we're going to be moving that over probably I think we have to figure out when and how and the mechanics of it we're gonna be moving that over to a tendu repo pretty soon so a lot of fun stuff. There's a lot more to play with here. I feel, like I, only scratched the surface with Tecton, but thank you everybody for joining me and I hope you have a great weekend. I'll see you all later.

A