OpenStack Crowbar, 24 Sep 2014

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: OpenCrowbar Community 2014 09 24

Description

Notes: http://pad.opencrowbar.org/p/community-planning-meeting-2014-09-24

Planning Call for Broom.2 sprint.
Work includes hardware support, Centos 7, Salt & Chef Metal

A

Hello, everybody and well the timber 24th open, crowbar planning meeting, as is normal. This is our it's a bite. We have a weekly meeting with a biweekly cycle, so every other week we talk about sprint, planning objectives and then the other weeks, the midweeks. We talk, design, topics and review review work in progress so that we actually can exchange ideas and the just work.

A

So our focus in this is github issues. Oh and the pause for a thing. Usually, we work agenda a little bit before we start recording. But if it's anybody have anything to add to the agenda.

A

Cool a pretty typical fit for our planning cycles,.

A

Can actually do this so you'll notice that we actually went back and group things into different release. Sprints and releases I have time in here where I do a about release planning, because I believe we're coming to a point where I'd like to close the broom release, what we're going to need to review what the what the deliveries are for that.

A

If I just look at broke, one bye.

A

Alright, this this bug, I, did not close this bug so I'm going to move this to prove to Y.

A

Can do it like that yep Victor, you said this was a hold you're waiting for the poll to go. Go with the nic-10 gay patterns.

B

I, don't remember what I was living on that.

A

A

Quested by victor.

B

Oh yeah, yeah I just haven't done it yet all right. I'm gonna, move on I out some great stuff understand.

A

Move it to broom to or camshaft.

B

B

Just lean to the room, one I can do it it's incredibly to reveal well.

A

I'm gonna, if it this I'm, trying to close broom one. So it was not done we're.

B

Moving a bit of rheumatism.

A

Trying to trying to get them to put in this all right.

A

Rpm installation fails I thought this was closed. Randy. Are you on the phone I see him on the list.

A

You I am I, understand his RPM. Install installation is working but I'm not ready to close the bug without somebody doing it. I.

B

Have attested in a long time, yeah.

A

I saw randy was actually pinging questions about this, so I know he's testing it. I just don't know where it stands. Potentially of that tumor to.

A

I guess we'll do it.

A

Create missing repo dependency, missing, I think this was done you and this is actually patched.

A

And closed closed issue, alright.

A

You already saved this.

A

I believe you merge this patch.

C

A

Should do this absurd, I need to work with with, as I can forget, that would fix quick fix, alright so that that patch got murdered. So we now have the maze for.

A

We now have the menu items or this so I'm, going to close this any any concerns about doing that.

B

Started to emerge.

A

Document using containers for workload, development I'm going to go ahead and move this all the way to cam shaft, because I think that it's really a.

C

Camshaft 0 item.

A

A

Up streams from readystate, this is clearly going to be camshaft work. Some move that dick. Why did that I accidentally moved? This is not the best you I bad, you I, but this the best you condec I can show a skier Scott's teeth grinding and we're in the distance.

A

Let's give up mine.

A

This milestone Kim chef.

A

Bootstrap you I not updated for this, so this is. This is actually done. I was just waiting for somebody to validate it and hold on. Let me demo, it.

C

A

My apartment, I thought I think I have my environment running.

B

A

About six, ah my typing is degrading before before your eyes,.

A

Look at that then actually slide in like Wow. Alright I can actually do two nodes with this.

B

A

State wizard right there off the menu hot dog. Look at that there's, no there's no pm's in it right now, but and then so. The readystate wizard under utilities, bootstrap are sorry. The bootstrap employed and I will show you what how this works, so we can close it. So what what we, when we change the attributes the screen had to get refactored, and so the way it works is if there's no admin network you can gives you a button to create it. In this case it exists. It gives you a link.

A

You have to assign an admin. Node gives you a button. If there isn't, we have an admin note, so it's all good. These are the roles that are bound to the admin. Note and I can add other ones if I wanted, although you can't see them pledge.

B

I would not do that actually,.

A

I'm happy to remove that capability. It's useful! If you don't run to your. If you don't run the production bootstrapping script, then.

A

You need to be able to bind roles to it, but I Victor, if you, if we want to add a bug and they'll I'll yank, that it's trivial thing or have funding.

B

The thing is, if you don't find crowbar admin note, it ain't, gonna, work ever well.

A

You can't remove them, so these are already here. This is just added two extra ones. Oh, you can't see the list so if I wanted to add OS installed or make it a docker container no door. Oh well,.

B

Cute, the thing is, you can't do those on that than those.

A

Which, what can you add on admin, is nothing.

B

A

I will right.

B

Now you can add anything: that's in the RT by all of it.

A

Thank you get put a bug in for broom one and it will be gone, I'm still gonna. I still want to accept this story, but.

C

A

B

Pretty much the only role you should find to the admin mode until you know, we explicitly add something otherwise is over admin noticed all all.

A

Right so here's here's the deal, I'm going to remove that section or change it's a downed roles, so you can see them, and I can add logic that says if it doesn't have curve our admin node flag it, but I I have to work through I haven't worked through the walked through the workflows of the zombies sort of the zombie mode install right now. Maybe this is just the way we do it. If you, we assume you run production on that server. It turns that server into an admin node.

A

This workflow is designed for if you're, not in that case and I, am going to remove the other options because well.

B

Tested the elbow you should also remember that production SH is a hat its assets there, slowly so its axis there just for development purposes so that we don't have to manually do all the stuff that's doing automatically for us and.

A

Until we have somebody who doesn't use it, I I've mixed feelings about showing it here so I. If please open, a bug asked me to remove the adding 44 room, 2 and I will I will do that? Call it a bug, then we'll get it fixed, and then this is this is the other, so there's a set of attributes that are editable in this, and so the only one is DNS forwarders here, and so you could actually edit it and save it.

A

And it's this this review key attributes was the key was the the fundamental, the important change, and then you can go to the anneal and watch all this stuff going. If, if the admin, if you've created the admin role, it's not committed, you actually have to commit the deployment there's a button. For that too.

A

This is actually more interesting. You can run production SH with the dash dash zombie flag, which doesn't mark the admin note alive, which is the thing that gates the work, the things for being set on it.

B

A

Way down deep and out so the way crowbar just put it on record the way this works is we do a bunch of pre configuration in production SH and against. So the cricket have been server using the API. We do a whole bunch of configuration in production SH which could be done manually if you didn't use production of SH and then the last thing that that script does is it marks.

A

The admin note is alive at which point the annealer goes in and and acts on it and so there's a flag that you can set in production on SH. That does all the work except marking the note alive, which I humorously called zombie mode, because your servers not alive and then you get to mark you mark it alive, and then it finishes the annealing okay.

A

Hopefully the helpful description but I'm going to go ahead and take this story as closed. That's all right then, hang in there for quite a while and I stopped at ok.

B

C

B

A

hmm What's a shiny issue, oh you got my shiny issue. So if I look at my room to work for broom.

B

I do I didn't found a milestone. Let me do that. Oh.

A

Would you I got it easy enough, so you did it they're out.

B

Of the mouth then.

A

Did you sign it to me.

B

A

A

Alright assign it to me please it is thank you. Oh let's go whew all issues all right, so.

B

A

A

Alright initial system configuration line- okay, this is when you've decided sent us seven support. I think that's a reasonable request for room. Actually, you know what this is probably a camshaft request, because I wouldn't I, wouldn't block.

A

I wouldn't block for bio, so on 10 to seven inject publicly on a target. Let's talk about that.

C

Actually it is, it should be a broom thing Rob if.

A

You want to make a broken three go ahead.

C

B

Yeah, I'm sorry.

C

I was, I was lurking, worries.

B

You know it would also be a good thing for, if greg almost done with the stack stuff he's been working on, be a good thing for him to add. Okay,.

A

I can kind.

B

Of help him out that way, he can familiarize and stuff with what the provision a looks like these days. Okay,.

A

I'm hoping I would like to see us close broom.

A

3 be the last in the release, but.

A

I mean, let me mark it. October thirtieth lets up then that.

C

Would take a lot of dates.

A

The dates are put up.

A

But we need, we need to figure out the dates but I'm. My goal for this is that we're close we're closing broom one today. So the calendar is weird to this day, so the 24th- oh that's, yeah and.

A

Due date, okay well very symmetrical.

A

Alright, and so this one, the state is clearly, we need to fix, I'd, say I, guess we're going to consider broom to a two week or that perspective I. Guess if we're doing this biweekly, we should let's consider it two weeks.

B

C

So you're considering those milestones Sprint's like.

A

That's what I'm exact exactly I was thinking. Okay,.

C

A

I remember this.

C

Guy there's a while ago so makes sense.

A

And then yeah, then camshaft I'm, hoping that we can see broom 3 as the end of the broom sprint the start of the camshafts print, but that comes back to us. Having the community discussion, thus milestones discussion, a few issues.

A

You you move back, they took public key convert documentation to mark down I, don't see Mike on the call. So I don't know if we can commit this one. But I'd like to see this done before they're released and then we need to pull all the UI code generation. Stuff out is included in that convert you our pages to refresh I'll work with messiest on that might be able to spend them now add interface, UI right! That's a bug! I need to fix that.

A

Nick anything look in here. That's this one I can fix this one I I'll, look at I can fix it. I'll fix it in that sprint.

A

Victor I think you're actually doing this now I. Don't think this is blocked anymore. Is it.

A

So the item and discussion is hardware workload.

B

The rate part of that is done, yeah and I'm working on me, I'm working on parts of the fios part.

A

Okay, so I think the only thing that we're doing is we're not making claims about specific eager that.

B

Does not the jig's I actually actually wrote a rage jig to handle doing great configuration.

A

So this this looks like it's now is not working.

A

A

Any systems with.

A

Which tool set sorry? Are we enabling the mega SE li.

B

Legacy lies the first one. Next on the list would be the Sapphire Singh you based stuff. If I ever get ahold, look if I ever get all of the controllers, thatthat support and then next on the list be whatever other raid controllers. People are interested in okay,.

A

B

Maybe we only we.

A

Only need one right now what's and what about for bios.

B

um Hi us will probably still just be del r series dear via WM, so.

C

B

An exempt are of the other stuff, and it turns out that the super micro there's actually additional license, crap that you have to purchase for certain systems in order to do remote. The BIOS configuration.

A

B

Least, based on the documentation that I've been able to find it, anyone has anything other difference. You know that'd be good to know so.

A

Even like those little those mini ITX ports with need additional licenses, yeah.

B

A

See that in the well.

B

I mean for the mini ITX cords, I didn't even see, I didn't see anything about being able to configure them by any method other than driving the bios setup or going in and doing the plat setup screen.

A

Okay, so we have added be in control, but we don't have a configuration Lestrange.

B

That's that's! That's what I p.m. I gets. It doesn't really get you out of n configuration other than for certain narrowly defined things.

A

Good to know all right, so I means we we need on the community, they make the list that.

A

Micro bios config.

B

ah We're not saying anything whatsoever about three micropiles big right now: sirs, don't want that sport right.

A

No no we're this. Is this a note about offices and observation right? You.

B

B

C

Okay, that's like.

A

They're, what we use Pacific row boxes for we don't need to figure the by oats.

B

I mean we mainly uses that we have a chief Park or target to poke around with IBM I. Wasn't the cell yep.

A

Oh no I was I, agree and I was hoping that there's some supermicro universality towards those machines, but.

B

Well, I mean I, haven't done a big doctor review of the docs those little bored. Yes, okay,.

A

An atom, Singh documentation areas all right so is a that's missing that there people are planning to one- that's not included in this list, or that, where is work that we don't think can be completed in the next two weeks.

B

I think I've got all the work that I can do the handle in there I think.

A

So there's we're not showing a chef metal in this list and we're not showing the Salt Works I mean somewhere. I didn't where's all milestones.

A

And filter you.

B

C

C

B

A

I thought we I thought I thought I wrote of socks to me.

B

Well, there's a salsa pull request has been merged.

A

Add salt stack.

A

A

Is effectively done so we're not we're not doing additional work with salt.

C

B

Yeah so I was saying: if Greg doesn't anything else to do letting him a decent. Some support be a good good tote. Exploratory exercise, I.

A

Think that makes sense and the other thing that we were doing it from an exploratory looking at the pack and packed stack integration tap, but I, don't I! Don't I want to create that as a work item unless it was exploited created an exploratory.

A

Ok, dot! What's what's your thought here you.

A

A

Intessa sorry.

C

Had to grab some eat button, it was like 7 layers, deep of my phones um in particular to what it as far as hitting the CentOS 7 support, yeah.

A

Well, I just push that back to to room 2 yeah.

C

I think that's cool, ok,.

A

And I we didn't, we didn't put a story in for chef, metal, I thought know me: we did I thought I had one you.

A

That's frustrating all right. Let me response to demo the chef's map stuff, so we need a story for it be nice to have a event.

A

So let me let me throw this in as a newish new bat for class. Sorry, looking around the wrong button.

A

You know what I wrote a lot a lot about this, and then it didn't make it in as a story. I don't know if I want to put everybody through watching me copy this over, but so what I'm going to do? It I'll sign it to me: kala broom, to enhancement.

A

A

And then there's actually it's a specific thing that I want to talk through as a.

A

Room 2 item, which is this SH key, the public s HK injection.

A

So we actually need to talk about this little bit. Victor.

A

Make sure people understand where it is, although the people who need to want to consume this or not on the not on the call.

A

C

C

I understand the future grab. That would make sense right.

A

So what what Victor was what Victor said? Well, the admin node has this information in in it in one of these key sets. You know which one.

B

I'm looking at it, ok.

A

Network admin that were coming we're.

C

B

Minutes, Oh access.

A

B

Yep crowbar permission or server access keys, yeah. If you.

C

B

You just add a public key to.

A

B

Catalyst yeah, if you add.

A

B

To that hash then.

B

Ash has should hike will be set up with root access on all the managed nodes right.

A

And so there's there's a separate, so that's already mob.

B

Nd the caveat right now is you have to do that before you bring up the note nodes that are kind of already been brought up? Won't won't get an updated list of the access keys.

A

Ok, there's the API call to give added, so pretty all right all right. So it's a no it's a behavior issue that I think Jason is pretty.

B

Gonna, like just to change that just uh you know at it, give you a new key, a unique name and added into that. The hash, that's basically yes and any new know they're discovered, will have that key injected into them. At all points in the lifespan where we agreed with them were set up. Ssh.

A

A

So this story is really about. Let me refine the story, because with this this is say, I want to inject into the nodes authorized file to end existing.

A

A

Mention ode discovery.

A

Ok, ah let me change this to.

A

And then so from that perspective, I, don't I think we could move this out of this sprint.

A

Since we have a workaround for it,.

A

That that's unreasonable from a work planning perspective.

B

B

You and the longer term fix for this adjusted it refactor. What I'm doing the provision is be its own and a standalone role.

B

Right now, part of the provisioner, because that's where we initially needed it right.

A

I the note, sir.

A

A

We don't, we don't want.

C

A

A

I and then, from that perspective, I'm going to move it I think moving it to broom. 3 is probably appropriate, not all the way two camshafts, because I think it might be a needed for needed for the camshaft before the camshaft release that it's this. This to me is a fact. You know what I'm going to move it to camshaft and will consider it a nice to have feature if it.

C

A

Broom God is that reasonable to you.

C

A

C

B

C

If there's a workaround.

B

Yeah I mean for the cases that we're going to use it for as long as you put it in there before. You discover the notes you're going to want a fitting X Factor, whatever on and right people, so to see affected into the workflow automatically.

A

So that perspective, what I would I think we should do? Is this document poof been typing doctor too much ah SSH key injection work around.

A

This jut needs this I will assign it to Jed.

A

um I like where I could pick it up to what was the issue number anybody.

A

A

A

C

C

A

Alright, so we've planned for broom tip.

A

It's all good I've, ad hoc stuff that we can get to chef metal, we're going to have to demo next Wednesday tall stack integration, demo next Wednesday, so we had some dip. I was coming during the review meeting the design meeting open stuck Scott. Do you want to talk about camshaft and thoughts? For that? Are you.

C

A

I can I can.

C

Briefly, go over or what I'm thinking about it I think it might also make sense just to you know, as part of the design meeting, just go through a little bit of roadmap work. um So perhaps you can do that next week, but the the basic thing I was thinking of up at at this point is that you know broom is effectively the base hardware provisioning release and it brings you up through through the the bios and the raid functions, and you know, basics network setup in those sort of components.

C

um When we start looking at came shafts, that's really where we're starting to look at integration points they were that were effectively as open, crowbars moving.

A

C

Handing off to these other these other infrastructures, such as you know, salt, the salt integration back stack, those sort of functions so I think that's! uh You know, that's what I'm proposing that we target for for the camshaft release, but it's something we should discuss in the desired section.

A

Right, so I think this lines up to what our original roadmap was we want to get so at anvil was the orchestration in VI l.

A

Right broom broom was about adding hardware provision to that cap. Shaft is about integration points, so pack stack is this is going to allow OpenStack or do.

A

Right and actually I think we need to probably go back to chef.

A

Community cooked community to be the OpenStack upstream.

A

I think it's I think we're not it's not as clear on the chef chef, metal integration is really interesting and I think we definitely want to show it, but we need to see more community interest on this.

A

Saltstack is next up, is community interest to.

A

Mvp, so when it seems like we're, leaving we're exiting so Scott, it seems like we have MVPs for salt and chef metal.

A

To set up camshaft discussions right.

C

A

C

That's pretty much what we're looking at is. We have some work ahead items, but primarily what we should be focused on is getting the you know. The base rate and hardware provisioning functions um into the system is part of room right. The the camshaft release is really targeted towards those integrations. If we have some proof concepts and some rarely work or some MVPs that are happening, you know that are happening now is we're getting people to speed or whatnot on the next slide.

A

Yeah, I think that the MVPs are helpful to get funding for camshaft right part of part of what we're looking to do is have community of projects funded projects.

A

C

But it also is a you know. A good place for I mean some of the underlying components that we're doing really only makes sense when you you move it up through the through the stack and it. You know, it makes sense to work up through the through the stack and at least some limited passion for some of these components.

A

Right I'd love to see what the what the app is for saltstack on this. What about things like alternate or additional operating systems?

A

Is there a list on that? Do you thinking staff camshaft? These are released.

C

Well, no, those are really broom functions, um I mean you know these things are you know they don't muddy up right. I mean we're, obviously not going to implement every operating system. We think people could possibly want without a you know, without a driving need to do so. um So this time goes on right, though, if they'll get dropped in, but um from a really see type of Fame thing based off of a few know. What I'm just said right and it's in again, we haven't all gone over agreed with it.

C

um You know it would be the base. You know function laydowns, which would tend to be what we would do in room, which is why I was suggesting that we do the the sent seven stuff in room. Three.

A

Victims, I was I'm thinking more like core OS as a it's, because almost a platform.

C

Yeah that might even buried its own release cycle solutely Korra.

B

I mean coral us, you know I I like it from a theoretical standpoint, but it's not something that really wants to block provide as supported operating system, / platform class whatever and sold. Someone so actually asked for it, because it's.

A

Matt importance.

B

Of town school.

B

B

Oh, can we lose audio no.

A

B

A

B

You I mean, ideally, we have one of the things that needs to be on our to-do list for sending up technical debt is going back in the refactoring how the provision of works to you know kind of brainstorm about. If you still need our current provisioner, we could know just drive something like razor or you know, modularize our current provision, your layer or something wrong with one.

A

B

Interpreter is kind of held together by spit and baling wire.

B

Functionality, but it's not the easiest thing: the world 2 extender to tie a noose or drive programmatically that I'm.

A

Not sure that any any of the other ones are much better yeah.

B

I know that if I like what do I, you know put a little bit of time flowing over one of them and carry the difficulty using them. Broken stuff. Refactoring, the herd, rooster yeah.

A

I was actually arrested. Embedding some of the hatchy Corp tools. I can't the console integrations I think are interesting and the I'm.

B

Very interested in those too I just don't want to invent the whole. We can't not use them cause. It's like the odds of getting something like compo, honest, which is tricky.

A

Which would that be required? I.

B

Don't want to I, don't want to inadvertently make it required on our end. Oh ok.

A

I think we need a, we need a design discussion, some point of it. What you think we would use console to do.

B

Well, what I want to start off doing with it is just monitoring the roles that we deplore and start off just doing that, okay,.

A

Be that requires consolation.

B

Yeah well I mean anything you do with council requires having possible on whatever is being monitored.

B

For me that that's the same as that's kind of the same thing is like with any other monikers, so I must.

B

Right imma get yeah, that's nothing that rust check this.

A

Comes back to the fact that we incur an open crowbar, we didn't put Nagi us back in the system so consoles the.

C

A

For Nadia's, from that perspective, there was there was another Hoshi Corp tool for image. Building that you were looking at and I was assuming. That would be on the technical debt list too, but.

B

A

The thing is packed that pack are.

B

A

Like that is here, br.

B

A

C

B

And I've been looking at the there's kind of a way to distribute a virtual machine version of our axis notice.

A

Interesting: okay,.

A

Yes, some of the weather allows you to do the image image building, which is a little more flexible, restaged amis.

A

And so if you don't let your passion, corpus r, is the vagrant people.

A

Their site looks like it is favorite there you go.

A

Yes, I know you make vagrant.

A

um And so a lot of their ties to feel.

B

A

Like Packer has to feel for a vagrant component for vagrant.

B

A

I feel like we've covered planning. I, don't see a need to extend the time if we've covered all the topics feasible. What we're holding off road map I think we covered in some detail, but not as much scott as I'm hoping we'll go to in the future and openstack options. I can talk about a little bit at hawk.

A

So I'm hearing people in the community and trying to work through some of the interest on how we're going to pull OpenStack as a workload back into crowbar, and we want to do it without being- is entangled as it was in crowbar one.

A

So it seems like there's a lot of interest for people who want centos to use packs back in our do, which makes sense to me, and so we're sort of talking to people who are going through that path and looking for an RDO pack stack on CentOS 7 is one of one of the paths and then I'm still still optimistic. That susa team is interested in taking their cookbooks and bringing them over to open crowbar.

A

They might I, don't know how that you know they want to. They want to be part of the OpenStack chef community, although I don't know that at this point is really feasible to reconcile the work that Zeus is done with. What's in the upstream community and watching the chef upstream community and and trying to get their stuff through a gate.

A

I'm also scratching my head a little bit about what a gate would look like for Shaffer puppet deployment cookbooks so.

A

I'm I'm interested I'm happy to have one-on-one conversations I'm out I'll, stop talking in a second and open up the floor. I really think we need to have some type of gate says an auto NCI system around these workloads to make them effective, but I think it's going to have to be a multi-node deployment different than potentially different than the OpenStack gate system.

A

And that's that's why I pause for discussion.

A

A

All right, it's still it's still preliminary I'm just trying to get it out there, because I I haven't heard anybody very happy with how the physical side of the deployments going, and it seems to me that we could. We could actually play a really helpful role in without being supporting these communities without having to be embedded in the communities and so I'm. Looking for people who can do be that bridge very simply,.

C

A

At the only way, the community forms is when people talk about it.

A

Alright, if there's no other comments, I'm happy to wrap up the call if people back to seven or eight minutes.

B

Okay, mom yeah.

A

B

You gonna have time to get to press the red bar clamp thanks for coming to the week. I.

A

Will make a point to get it done today? Okay, I was trying to spin up my environment. I actually might have to ping you about trying to get my environment working because it's hanging on that docker did the team solve the docker hang issue.

B

What dr. hang issue it.

A

Sounds like it's unique to me. Let me purge all my docker images and see if I'm just out of disk space, that's probably the best will likely cause I'll bang on it right now. If I could get through and validate.

A

How do I know it worked Victor if it ran successfully, do I consider that successor can I inspected some other way.

B

You can inspect it by looking well when you, the rate, configure roll run if it captures the list of raid volume that it created. Okay,.

A

B

And the list of the raid volumes are there, and so at the by default. Right now you should, it should just show up with it should show all of your physical disk, and it should show a single jaypaw great volume, and that raid volume you should be able to go on and go ahead and install an operating.

A

Ok, so if I can install an operating system that I'm happy, which I have.

B

Died means Edwards to workflow, the workflow will look like you. Are you let the note discover and then you add the crowbar installed node hold to it. Yep.

A

B

Commit- and it should a long way it should the rate control should run, and it should blow away any pre-existing raid arrays and create the crazy minimum month of it requires.

B

A

Thanks a lot of sense, so basically is parity with VM. At this point,.

B

Hi Nev, except p.m. don't, usually have raid controllers a good thing.

A

I understand that but I'm my workflow has parody to the DM before I.

B

A

Smith system was spin and it wouldn't install no s, of course, yeah.

B

I I try to make it a point that I don't want there to be any difference in in the workflows unless you want to change it between messing around with the bm's and messing around with physical hardware.

C

C

A

Alright, that's good. I will crank that through alright and I got a document of my OpenStack options and I think we're done cool.

C

Thanks everybody.

A