Istio Community, 20 Sep 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Istio Community Meeting 9.20.18

Description

- Mixer Deep Dive
- community roles update

A

So thanks everybody for joining and do I put the link to the working doc and then just add in any agenda items you want to otherwise we'll take it over to Lynne.

B

Okay, great hi everybody, um so let me send out this link. I guess I can share. My square might be easier. Let me think: how do you do I.

A

Can just the link in the chat if you want to share your screen? Yes,.

B

So I think I got my screen shared. Can you guys see the Israel community rose page, okay, awesome, so the technical steering can access a technical oversight. Committee has recently met to discuss what becomes a member. What makes it a user become a member of the SEO repository and the reason we did it is because we've got a lot of random requests among our contributors.

B

People wants to become a member. They were not sure what's the process, so we feel like it's a really good time for us to huddle together. So this is the new process. I want you to talk to you, everybody sooo. So, within this Rose page you can see to request to be a member of the Istrian organization. You can open an issue which will get there next and then fill out the required template within the issue.

B

So if you're happen to click on the link to open the issue, you'll see it's basically a github issue in the Israel community, repo with the title prevail for you, and it also has a preview template. We think a template.

B

We basically ask you, you know: what's your github ID, which company are you with, or if your individual contributor, that's welcoming to you and the a bunch of requirements you become a member, so we require you to such as the you know, read or documentation on membership guideline I contribute a guideline make sure you get help. Id is secure, make sure you subscribe to the issue there mainly in this and also importantly, to document your contribution to the project.

B

So we will tribution such as altering or review chaos on the github, be able to commenting and providing information on the issue be able to contribute in to work group and also the community discussion like this and and then the other requirements also have people are to sponsor for you to be able to willing to sponsor your membership, and then we also have a little bit of sponsor requirement that the sponsor can't just be randomly anybody who is an existing member. We would like it to be who has close interaction with the the member.

B

It's good the to be member, the user, that who's proposing himself to be a member and be able to interact with that user either through PR, either review that person's P I'll prove that person's PR interact on the design dog and then the sponsor has to be at least in one of the owners file in any of the ischial repository.

B

In fact, we decided discussion yesterday, I think we may tight that a little bit up not only to be any of the sto repository, but not counting the repository of our documentation reports, because our criteria for becoming owner for that repository is slightly different than any of the other repository and they also be able to demonstrate sponsor, are actually form at least different companies, so that this is demonstrate.

B

This is a community interest that you have you as a member, and then you basically list your sponsors and indicate whether you have spoken to your sponsor and also document to your contribution. It's in detail and also, if you as part of the privilege for being a member, you also get to join the developer slack. So you are welcome to you providing your email address so that we can invite you to join a developer only slack. So that's it chillie the process. I, don't know if I miss anything April.

A

No I think that covers it all. Does anybody have any questions or feedback on this? It should be. It's I believe it's based on what kubernetes uses, so it should be pretty familiar to to most. You may have already gone through this with kubernetes.

B

Yeah and that's a new thing we are rolling out, so we certainly welcome you guys feedback on this.

A

All right sounds like there's no questions. Thank you. So much Lynn for covering that and yeah I'm by all means do check that out. If you see something that you think should be clarified or you want to get a little more understanding of, as you create an issue in the repo, let us know how we can make that clearer for you, but this should help streamline things and make sure that those of you who are needing to be active and the org get set up to do so.

A

So all right, so we've been having a lot of questions about mixer on a lot of the recent calls. So I asked me and our if he could join and kind of walk folks through some of the the basics and stuff around mixer, so Mandar.

C

Alright, thank thanks. April I will share my screen.

C

C

You guys see it yep.

C

Okay, so so what one of the one of the things with the deep time is that it's always unclear how deep right so I have calibrated this to a certain depth. However, if, during the call people on the call thing that it's either too shallow or too deep, I can always adjust it. So Mandar.

A

Kudos for using the nautical based puns already I, like it I like it a lot.

A

C

Does it not start okay, I can't see? Oh there, you go okay, now, I, don't know if the zoom is correctly doing this, but I see my presentation on my screen, so I guess works for me. Okay, so we first well I'll just go briefly through kind of why? What and how I? Actually what and how is the meat of the this talk? But why is like what why we need such a thing right and and before that?

C

Just let's go over the basic history architecture right, so there is in the resolve, oi, which is going to intercept all the requests and send make two calls to mixer one on the check path, which is one before the request is sent to its final destination and one after the request is sent back to the requesting source.

C

So the first one is check. The second one is report, and this is just to show where, in the architecture, mixer fits right.

C

This is this a bit backwards, but but I think it's useful to put it here. So why do we need such a thing right now from a very high-level perspective or service mesh needs, some precondition checks. We need to decide if a request should proceed or not.

C

We also need rate limits and quota kind of checks so to decide if a request is otherwise fine, but there are too many of these, so something should be rate limited and then we need observability. We need to collect telemetry now with all this. Another thing that that's important for for a service mesh is that it be extensible and easily extensible.

C

So we have to have the ability to add these above functions easily, and then it's also an integration point. So there are many many existing systems which either perform these three functions or they so, for example, for matrix there are matrix collection back ends right, so we want to be able to have a system that integrates easily with these existing systems. So that's that's kind of kind of the purpose of why why we do this. So what is the mixer abstraction right now we're not yet getting into the implementation. But what is this abstraction?

C

What are we presenting to the user so they're presenting we're, defining a uniform configuration surface and I'm using metrics as an example here, but in the previous slide, I showed that it's checks, metrics and quota. So all these apply. So all the further discussion applies to checks as well. What matrix is just easier to talk through, so the uniform configuration surface talks about when? Should you collect a particular metric?

C

It also talks about how you should collect that metric and where to send the metric to right. So this is the this. Is the uniform configuration surface which is quite agnostic to specific implementations and I'm? Stressing this point because some of it is some of it is going to be important later and then one level deeper mixer also defines a couple of api's right. So the first one is the mixer API itself, which is a JRPG based API and in in some ways it's a it's actually an internal detail between the proxy and mixer.

C

You can go read about details or that API or if someone must've known now, I can talk a little bit more about it. But at this stage it's a front-end API and it's between the proxy and mixer mixer also defines a a bunch of meta api's which are defined by the templates and at this stage since I have not gone through template details. Let's just leave it at that, but it is. It is a meta API and every template essentially defines its own EPS and crucially, that API is between mixer and the adapters.

C

So, in terms of the configuration surface again when to collect is the rule, how to collect is the instance, so you you create, you define the mappings there and handler is where this collected instance is is sent to. So here is an example I'm, starting from from an example and I've divided the Prometheus handler here, because a well first of all, it will come later, and it's not that detailed anyway.

C

So in this case, what we want to achieve is that if you have a certain request head all right, I'm, just making some header up called X collect. It goes to true. If that header is present on the request, then you want to collect a metric, which is request.

C

Count by service, and that is defined on the right hand, side which says the two dimensions are source and target service and sources, source workload and target started service so with when we, when you push this configuration into the system, what we expect is, if you go to Prometheus, these metrics should show up there right that the result there is a little bit so again, disregarding any implementation. What we expect from this configuration is that if this header is present, we expect these metrics to show up in Prometheus.

C

So now we're going to go through the mixer stack to find out how this actually comes into being right, how these things are materialized.

C

So first, let let me just set up like a simple life of request right, because the request starts from some client it's trying to go to a service intercepted by envoy. Now this is where the mixer API comes in.

C

Right is the first step, so envoy is going to translate or package up all the information that is available to it in this attribute based mixer API and then on vocals check, and then it goes report and now now we're talking about report right so again now we're at the top of the stack mixer API is internal between proxy and mixer. It uses some compression techniques to reduce the amount of data, that's transferred back and forth, or there is also batching, which means fewer number of calls are made and for check. Specifically.

C

There is also a very efficient cache implementation, which makes sure that calls are not made on a quick request, or there is a very good cache hit rate.

C

So the next step, once the request is received by mixer, it receives all these attributes. Now some of these attributes have to be augmented. One example- and this is kind of mainstream- it's always used- is the coupon that is in adapter, so Envoy sends a source, UUID right or destination UID to mixer, which is the pod identifier of the part. That's talking to mixer. At that point, mixer is able to ask the kubernetes environment adapter to resolve the pod and get all the labels get all the metal it does sit, which is labels and annotations.

C

Similarly, you can have Drive authenticated principles and these kinds of adapters you can imagine there could be many more right. There could be Cooper that is console any other service registry right that that ID is known to the service registry and the the idea here is that you can go back to the service registry and ask for more information again. This is this is all cached once that step is done. Can.

B

We ask a question on there sure.

C

B

This is once end up in puzzle with me, so I remember so we recently added support for wronging minimum is still well. You wrong is still without mixer, so you can still do basic networking traffic management function without mix you basically just run pilot right. One thing I did find out. The other day is once I add mixer in even before I had any mix of policy or authentication policy, the traffic from distribute tracing that goes to mixer. So I think this. This is the Kuban area adapter you were talking about.

B

So how is this adapter being used even without user? Using any mixer policy? Is that true, a science mixer is in the picture, because if the mix is not in a picture, obviously in the distribute tracing, my god.

C

Yeah, so so sorry I can answer that. So the way when you put when you put mixer in the mix so to speak, we generate like pilot, generates configuration that will ask envoy right that will insert the mixer filter into the filter chain at envoy. Now we don't have the optimization yet anyway right- and this is this definitely on the roadmap. We don't have the optimization.

C

Yet that says that if there are no policies applicable for a particular service or no policies at all, then just alight the call so I think that's what you're referring to. So we don't have that yet, however, so what ends up happening isn't actually that bad? So what ends up happening is on what still calls mixer mixer says that well, the call is a lot to go through because there are no policies that are going to affect you and that information is cached in Envoy is a one-time.

B

C

So one-time call to mixer I mean the cache has a TTL, so one particular expires. It's going to go back, and that is the reason why you see that call going to mix it. Even though you don't have any policies, but that's a good point. It is on our. It is definitely on. Our roadmap has been on a roadmap to say, if you can just divide it completely, if you don't have any policies. Okay,.

B

That makes sense and a side question where you were just talking you mentioned about the mixer is in the picture. You would have this envoy future injected in this voice, ID card. So that's the me if I'm, a user I move from Network only profile to a profile that included mixer I would need to redeploy my control plane to pick up the new envoy with the future. You.

C

Don't have to redeploy your control plane right so.

B

C

You don't even have to redeploy your data plane so as long as you're using sto proxy right as long as you're, using that binary. That binary does have mixer client compiled into it in the El Mundo in a no mixer configuration simply, the configuration does not ask on board to load the mixer filter and when you change, the configuration and pilot starts generating config mixer filter configuration envoy will okay.

B

So it's a dynamic I, don't need to even restart the proxy so.

D

C

Don't have thanks so getting back to our flow, so after the first step of knowing extra information about the request, write metadata but where it came from and where it's going, we move on to this adapter resolution phase, so adapter resolution and decides which of the adapters are going to be invoked for this particular request in which of the which of the adapter supply.

C

So in the rule right, okay, so here is the here is the rule right now match condition here is true, which is like always true in the previous example, actually had a real match condition. Whichever way. The first part of resolution is to decide if a particular condition matches in this case it will match, and then we collect all the actions that are associated with that rule. In this case, the action is send request count by service instance to the Prometheus handler.

C

Now you can imagine that there can be many kinds of routes right, so in our original example, we conditionally dispatched this instance to a handler. In this example. It's doing so unconditionally. Just saying always collect this metric, regardless of what the request sees.

C

So adapter resolution now at the end of this phase, we know that, for this particular request, we need to produce certain instances and send them to certain handlers right so that that's the that's the end of adapter resolution phase.

C

Now the next phase is.

C

Attributes to instance, mapping phase, so that is when we actually produce the instances to send to the adapt. Now this is the instance configuration it is based on the template proto, which defines how a metric template looks like and metric template. This is fairly simple: there is a value which is a scalar and there are dimensions. Dimensions are normally just string to string, but they can be a little bit more of what string. Interesting is the simplest to think about. So in this case, we wanted dimension on source workload and target service right.

C

The source, workload and target service both are attributes that were either sent directly by envoy or that were derived by the kubernetes inverter on the way to come here. So there is a so during this mapping phase it takes as input the attribute bag, which is the attribute that of attributes that have come from Envoy, and it takes this template proto and what it then produces is, and this is kind of going into some sort of the go code, but what it essentially has to produce is.

C

It has to produce a struct of this type right. So the value is in 64 and dimensions are map string to interface, and this is the this is what actually is is produced right now. This is a materialized view of that instance, which is value 1, because that was already a constant and then source workload and destination service are resolved to their actual current values. So now this is the information in the struct form that we are now going to send off to the next phase, which is to the handler using the template.

C

Api now handler is the code that receives this information and does with it what it wishes so, basically, so Prometheus handler would take it aggregate or massage. The data depending on Prometheus, is on configuration and then expose it to Prometheus, expose it to be scraped by the product, but the Prometheus server.

C

The handler interface has the handle metric method right that handle metric method is dictated by which template it supports, and and that's why we call it the template, template API. So if so, the handler has to support whichever template, API.

C

Was defined earlier and it takes it takes an array of these instances which are again defined on top. So now now the last part which we had just kind of so the last part is the handler right like. Where does this go? So here we just define the interface. Now we actually want to see the previous handler implement this interface, so Prometheus handler is defined as such right, like with the adapter and some configuration parameters. That parameters actually needs, and this is an example of the actual Prometheus handler implementation.

C

So it is implementing the interface that was defined a few slides ago, and this is the real implementation of committees of the Prometheus adapter.

C

So, as you can see right, so we with at the top we entered through the attribute API, which Alma had sent, then again we augmented it by environment adapters like kubernetes and adapter.

C

We did attribute resolution adapter resolution to decide which adapters apply and what instances apply then we produce the instances, and then we call those connectors and in order to get a new adapter, you implement one of those EP eyes and and that's well that's compiled into mixer, so that gets into the next part, which is one of the big things that we were hearing or beyond was any time you needed to create a new adapter.

C

You had to compile it into mixer, so what we did was, as you can see, right, the template, API is already an API and templates themselves are defined using a cheer, pc-based tenifer metal language right I mean it. It is a proto, but it also is a DSL that that defines the configuration format and the payload format both and which is why it it's a DSL.

C

So then, what we, what we decided was it's actually better to have this picture instead of this right. At that point, the template API produces these instances and rather than handing them over to a handler, that's compiled into mixer, it sends them. It sends them over GRDC to the adapter that's running outside.

C

So how does that work right? So now, here you see that the message template is kind of defined and that that has always been there and it's in yellow it's in the top left corner in case someone has some color issues, so that defines the template itself right. So what the template is saying is and and you'll see that the template is both talking about how its configured and what is the payload. Now, on the right hand, side we generate new proto's based on this original proto.

C

One of them is instance, map, instance message and it looks. It looks very, very similar to the template method itself. The reason is, I have actually lighted some fields in it, for example, name is the field that we do stuff in two instances. So that's why it is slightly different, but in terms of basing information, it's actually identical to the to the original proto in order to create a JRPG service out of it kind of following the standard conventions.

C

We also generate a message called handle request, matrix, which just has a repeated of this instance message in it and then, on the left hand, side left bottom of you see that there is the actual GRDC service which is defined and it accepts a metric request and it it responds with a standard report result.

C

So now what that does is functionally the G RPC adapter, that's compiled in all the T RPC adapter, that's running outside the interface is actually very, very similar. It is still receiving that instance. There is a and it's receiving the same information, but it is now decoupled from mixer, proper, so extensibility was was one of the was the most important thing that we achieved by by doing this right. So now, templates are no longer compiled into mixer they're, not pure configuration. So an adapter author.

C

If the adapter author wishes to define the new template, they can define the template using the proto right, which is this. Then we have some code generation tools which would produce these other other code to artifacts.

C

They can write their own adapter and then package the proto descriptor. So there is a way to do a deep to proto descriptors in it as a transitive closure. So it includes all the dependencies of the proto. So, for example, if you're using like google erp, see status or something some message like that inside your proto, it would package all that up in a transitive closure, and now that is, data, that's no longer code.

C

That descriptor is packaged as another CR D, which is the template, er D and then mixer loads. All that all that data and mixer is now dynamically able to dispatch using template, API, essentially synthesized directly on the wire these portals and call the services of directly. So that has been a really well I guess: I'll!

C

Let the community decide if it hasn't been it, how useful it has been, but but I think it's exciting that adapters don't need to be compiled in, and some of the use cases were that you don't want to open sourced your app adapter and you don't want to compile it in you. You still want to use stock mixer right. So this gives you all those things. I will pause here for a second. Does anyone have any questions, yeah.

D

Hi hi, so I was just curious about the handle metric part for GR PC adapters. So currently it also resent the adapter configuration. Are there any plans to you know separate out the configuration time calls with the metric time calls? Yes, yes,.

C

There are so that is, that is on the roadmap, so we have zero PC adapters. Have this two holes, one is session based and one is stateless or non session based the first one that we implemented is the Mount session-based. So it did it actually estate listed packages up even the configuration and sends it out on on every call, but we will implement the session based adapter or session based API as well, which will have a life cycle so in in the life cycle.

C

It will first it will force a initialize with some configuration and then will no longer send the actual configuration on every call. So that is a that's a performance improvement and it is definitely on the on the roadmap of you. Are you observing any issues with it? Is that for your boy, boy, you're, asking or just architectural II know I just.

D

Curious because of the architecture here, yeah cool this.

E

Anyone, yes, thank you question. How does this affect adapters that were written prior to this.

C

Okay, yeah, that's a that's a that's a great question, so well, actually, two parts to this answer. One is: we are actually coming out with the bit of formal document which will put on a stereo which will answer these specific questions right as to which adapters belong in misty Ohio repository in the main st repository, which belong in.

C

Contributed by vendors only right so that that that's kind of first question now in terms of grandfathering in what is already there.

C

We had accepted some vendor specific adapters early on when we didn't have any of this, so we are gonna have to kind of deal with them on a case-by-case basis, but if they can be moved out into their vendor repositories, then that's definitely per for, so that we don't need to have that code inside pasty repository, and there is so we were. We were planning to produce a tool and actually I heard that someone already did produce a tool.

C

So we should definitely don't go in that that takes the existing adapters and wraps them inside your PC, Eric adapters or or does some Cochin to create jera PC adapters from existing in memory rather in process implementations.

C

So so it should be easy to move, and there would always be certain like simple or very heavily used adapters that are compiled in at least for some time right. So, for example, Prometheus adapter itself, there is a there is a very good chance that it will continue to be where it is, even though we do have an out of proc kind. Of example, implementation of the same adapter actually dug. That can correct me if I'm wrong I mean do we. Do you see us moving to TR PC adapters for Prometheus.

C

No I guess: okay, Doug isn't here but but yeah, so so yeah. So to answer your question, there is going to be a very small set of adapters that would continue to be compiled in and then we or the community will provide tools to take the existing adapters and move them into well out of the repository.

C

Does that answer your question.

E

C

So where can we go next right after this? So now, if you look at this picture, right envoy is calling mixer and mixer is generating calls to the gr PC adapters, but oops sorry.

C

The API between mixer and gr, PC depresses, well-defined right, that is the part of the mixer abstraction. So in theory, although I should be able to make that call directly.

C

So that is, that is a possibility, and we are we're. Definitely thinking about that. Now that architectural e, both these things are possible, but the good part is the abstraction does not change the the mixer abstraction, which is why I said earlier is to decide when something happens.

C

How does it happen and where it is it sent to in the end, so maintaining the same configuration surface? Both these things are possible and they have their pros pros and cons, but both these things are possible. So that's that's what you can expect to see further, which is another reason why I've I have been encouraging folks to preferred G RPC adapters over compiled in adapters right it. It kind of future proofs you from any deployment or architectural changes.

C

That's it that's, that's all I had and like I said before. If someone wants to go into detail of any part, I'm glad to do so, yeah.

B

I'm- and ah this is really interesting- what's next I'm curious on this picture- would make sort of our boy goes directly to G RPC adapter? Would we eliminate the lead of being centralized mixer? Does it makes a compunction allottee being moved down to this icon yeah.

C

Yes, a yeah exactly so so that so mixer you can think of has like two parts to it. Right. One is what are sometimes called the mixer front-end, which received thing sees things and decides which allow press apply and how to call them and all that, and then there is the part that actually calls out to the adapter, but but to to to large extent, this functionality moves down essentially to mixer filter a lot of avenues for more optimization.

C

So, for example, the problem that you alluded to earlier, which is if there is no policy, that's the point, and then there is nothing to do well. If that code lives inside mixer client, it actually knows- and it just goes straight through it- doesn't need to call out to mixer to purge NPC after in that case. So there are many many advantages.

C

B

Definitely do you have motivation, it's interesting. You didn't bring a break up mixer to tonometry and policy, because now, though it's your one Oh has break up mixer into two components: I, always conceptually picture mixer as two components now. So on this, what snacks I'm curious on? Do you see any motivation to have mixer tonometry drowning the cycle because I see motivation for yes,.

C

So, okay, that's a that's a great question. Yes, so for for policy like latency considerations are sometimes very important and that that means doing it closer to to the proxy helps yes, so telemetry is not it related in that regard yeah, but in that case it becomes a deployment choice right.

C

If you, if we have it, set up in this way, then actually it's remains can can afford deployment, time decision or order later on on decision, but but but generally it would be helpful right for, for similar reasons to also not have the centralized geo telemetry, but let the envoys let the mixer client in envoy, send it to wherever it actually needs to go.

C

There are certain conditions, so, for example, if you want to restrict like firewalls right how many firewalls have access sorry, how many endpoints have access to a particular IP, and then things like that and you can still you can still manage that as a deployment choice, so zero PC adapters themselves are going to be services and they can be scaled just like just like in anything else, so so yeah. So in that view there is no reason to to kind of hold on to to telemetry as a separate deployable component, though yeah, because.

B

I was thinking, I mean, there's no obvious I mean I, think there's a strong needs for having policy running a sidecar for tonometry I think the needs is reduced because you.

D

B

It's not on the request path. I guess well advantage being having centralized tonometry is in managing sto environment. You will be able to read. A cow provider will be easily to manage the centralized component for the user like in charge of its lifecycle or upgrade install. So that would be the advantage. I would see so.

C

So that's a that's a good! That's a good good point. However, if you, if you look at how how dare these adapters are working even now right, what you really upgrade is the G RPC adapters themselves, so you upgrade the implementation and if you, if you actually make a change in the template, that is just a configuration change. So with with that view, you don't actually need to upgrade mixer and similarly, you don't need to upgrade mixer client, which is running an envoy.

E

Everything is porn right.

C

B

I see your point and then in this picture you would consider the adapter to be part of the control plane with mixer. If that's.

C

B

Then the data plane would still be the proxy correct.

C

That is, that is correct, and, and we have, there are already several people who have written their PC, a deaf person I'm very really glad that that's happening yeah.

B

Ibm wrote one for New Relic. In fact, we want to put in a contributor repo, but um Martin hasn't normalized.

C

Of course, yes, yeah.

B

C

That will that will come within the next few weeks.

F

You can't read repo that we could toss it in as well.

C

From that yeah we should. We should consider that that as well like.

A

You know you might know someone that has some perms on that repo right.

C

Yeah indeed, yes, yeah.

B

C

B

C

B

No god, no there's not.

C

Yeah there is, there are plenty of things that are not ready on one one egg. We don't want to add to that. Yeah.

A

C

You Mandar.

A

For doing this, this was great.

A

Here too- and we did record this so I'll get the recording on our YouTube channel today and we'll share the deck as well. We do have one question in the chat: does the mixer pose a networking bottleneck in the event of scaling up network intensive containers like databases, messaging systems like Kafka, so.

C

The short answer is no, because none of datapath traffic actually goes to Lexus right, so mixer is only dealing with headers and metadata about the requests. Not so the volume of the data doesn't really affect. Mixer, oh and and lastly, I- have liberally used in this. This presentation about the really used stuff from Zack and Doug. So thank you to both of the merriment they're on the call or not, but but that it was awesome for me to be able to draw from some existing material and then create some more stuff. So thank you. Yeah.

A

I see Doug's here, thank you Doug for being here and answering some questions and the chat as they came up and I don't see Zach, but we think zach is awesome. So it's great that we're able to collaborate on all of the the many presentations and discussions people have had around mixer before so always good and someone else can take the deck and fork it and make it their own to going forward. So, oh, look, you got a nice photo there very nice, not a sailboat.

A

Oh, that's right. Doug says you sail under so yeah. It's.

A

Awesome, thank you again Mandar. If anybody else has any questions, feel free to reach out on the mailing list or join the working group. Of course, a couple of other things on the radar for misty Onis Lynne, you were just a container camp right. How did that go? Yes,.

B

It was two weeks ago, Ian Linda. It was a very well attended conference. It's a small conference, that's my first time being there too. It's about 300, maybe to 400 technical folks, very technical topics, and so I was able to give it's still talk. You know give what is this? Your white is still and update over sto 2 1, oh so it went pretty well, and this has good diversity of other container related topic, and somebody interests me even tweet about there's no container conference without its real content now. So that was interesting.

B

Well, that's cool yeah! It's a really good conference in general day, wrong I think they run two or three times per year in in Australia and UK, and they are also looking at New York in the u.s.. Maybe next for next year, yeah I think it's a good conference. Definitely good to get some sto highlights.

A

In there, too, great yeah, so um I just wanted to put the word out if anybody's interested in presenting about sto anywhere I'm starting to see more conferences- and you know meetups and things that are looking for speakers on sto. So please do reach out again. You know we have decks that we're happy to share and you could edit as needed.

A

You know. Lynne was just at the container camp. There's a container conf that I just saw on Twitter. That's a covering you know a lot of micro services that so it also has an SEO session that looks to be happening in Germany in November. So there's a lot of interesting I feel like we're we're out of summer. So the conference you know wave is is picking up again and it will culminate in coop con and then will be quiet for a little bit.

A

But if you're interested in doing any sort of presentations, not only can we help with content, but I'm also happy to help with you know diagrams and things like that. We have a design service that I'm happy to set folks up with that. They can use to. You know how things look pretty we're actually also working on a SCO branded template. So that would also be another nice pretty thing for for folks to use when you're doing presentations but definitely do reach out.

A

And if anyone is interested in leading a workshop in Berlin in October ping me because we do have someone who is looking for someone to kind of co-lead. So that's an opportunity if you're interested in speaking other.

C

Than that I.

A

Would assume so I mean.

A

The local food and drink so, but yeah definitely keep an eye out for some of the great conference community stuff it'll be on our community calendar and again, if you want to add some stuff, let me know I know, there's a meet-up coming up in San Francisco and we are hoping to get our regular meetups going as well in the Bay Area. There are also some SEO meetup groups in London. Of course, I know.

A

London is meeting again soon and if you're interested in starting one in your area, please reach out happy to help you and get you set up yeah. So that looks to be all we have for today. We will chat again in two weeks and of course, if you have anything else, you need help with I want to talk about, don't hesitate to reach out. Thank you. Everyone for joining us today.

A

D

Bye. Thank you great presentation.