.Net Foundation On .NET, 27 Aug 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Docker Desktop Integration with WSL

Description

Sign up for the beta at http://aka.ms/docker-wsl2

Buildkit https://github.com/moby/buildkit
Docker docs https://docs.docker.com/
Docker + WSL v2 signup https://aka.ms/docker-wsl2
Host ASP.NET Core in Docker images
https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/docker
Docker images for ASP.NET Core
https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/docker/building-net-docker-images

In this episode we have Simon and Ben from Docker joining Rich to talk about some of the enhancements their team has made to Docker Desktop on Windows for running containers on WSL v2.

[01:10] - What versions of Windows will the Docker / WSL v2 support be enabled for?
[02:09] - How does Docker work with WLS v1
[04:22] - What improvements will developers see with the next Docker for Desktop?
[05:40] - Demo - Develop a .NET Core on a Docker container in WSL v2
[12:20] - Demo - Building the Docker deamon from source on Windows through WSL v2
[17:23] - Demo - Performance improvements for Docker Desktop
[22:00] - Demo - Utilizing host resources
[31:12] - Getting the Docker Desktop w/ the WSL v2 components

#docker #dotnetcore #dotnetdeveloper

A

Today, on the on.net show, we're going to be talking with the docker folks about how they've made docker work in wsl 2 to make your container experience. Better, tune-in.

A

Okay. welcome to another episode of on.net show.

A

So today we're lucky to have two employees from docker,.

A

And so we're going to be talking about how docker is targeting wsl 2 in future versions of windows.

A

So, ben and simone,, how about you introduce yourselves and then we'll get into the topic.

B

I'm ben., I'm the product manager for docker desktop.

B

I'm simone, I'm a software engineer for docker desktop as well.

A

Okay, awesome., I use that product literally every day,, so thank you.

A

Okay,, so what are we talking? About?.

B

Today, we're talking about a new version of docker desktop,.

B

Which we're looking to release in the next week or so., say at the end of july,, which will be docker desktop, consuming wsl, 2 or using.

B

Wsl 2 to run the docker daemon on windows.

A

Okay. now, just remind me,, I think, wsl 2, isn't shipping until next year with 20h1.

A

So if you install this new version on say,.

A

Windows 1903,, I assume magic isn't going to happen.

A

It only lights up when you install it on 20h1 is that right?.

B

Yeah,, that's correct. okay.

A

Yeah. makes sense.

B

So for us,, it's a tech preview as well., so we're releasing it as a limited preview for people to get playing with it, for using it inside of build, to get some early feedback on how it works and to start getting some of that benefit from wsl 2.

A

Right., so will you go to a special link to get this or will it be on the normal windows download page?.

B

So there's a beta sign up page at the moment or a preview sign up page.

B

then, during the end of,.

B

Probably october time frame, we'll be releasing this, or september time. Frame, we'll be releasing this to all docker desktop edge users as well,.

B

So then everyone will get it at that. Point.

B

makes sense.. So we're talking about wsl 2.

A

But there's a wsl 1, that's obviously already in the market that people can use..

A

So why isn't that microsoft and docker didn't decide to do this earlier?, what's magic about now?, so wsl 1 was an impressive effort.

B

To emulate the linux kernel multiple windows,, but there are quite foundational differences between the way the linux kernel works and the way windows, works.

B

Docker relies quite heavily on the linux kernel, behavior at quite a low level,, and that made it quite hard for us to actually do the implementation of docker on wsl 1..

B

We were allowing things, for example, ip tables,, the stuff are networking, and that part wasn't there in wsl 1., so instead, the way docker desktop works. Today,.

B

Is we ship on lightweight linux distro,, which just makes use of hyper-v to run that vm on the desktop,.

B

And apply that native development experience.

B

right., so my understanding is with the new wsl 2,.

A

There's going to be a different vm., so is the idea that,?

A

I was just in hyper-v doing some other stuff just earlier today and as I've seen before, there was this moby thing: that's there, which is basically where all of my linux containers are running.

A

So once you move to wsl 2,, will that moby vm not be there anymore?.

C

No,, it won't be there anymore. actually,, it's a vm,, but the wsl 2. It doesn't appear in hyper-v.

C

It's very low-level constricting into your os.

A

Right., so I guess one of the value propositions of this is,.

A

Docker then just uses the provided in.

A

The box real linux, vm that you can use for everything,.

A

You can use it for docker as well.

A

docker, as a software provider doesn't have.

A

To go out of their way to deploy a second linux.

A

You just get to use the one, that's there, and it's fully featured enough to do everything you need.

C

Exactly and as it's the same linux os.

C

That brings all your wsl and distros,.

C

We, although can enable a very good degree of.

C

Integration between your linux tools.

C

And the docker daemon as well.

A

Cool., so do you think,.

A

For the most part,, the folks.

A

That I think about are obviously.net developers,.

A

There are other developers on the planet as well,.

A

Will they perceive much as of a difference by using docker through wsl 2, or it will mostly be the same thing just delivered a better way or both?.

C

There are two categories of improvements.

C

so for current developers using docker, desktop today,.

C

Which have windows-based build scripts and things like that,.

C

It will just be an incremental improvement., but we also introduced a new feature which we call "linux workspaces",, which enables you to use the daemon.

C

From wsl terminal or from vs code to remote to wsl.

C

That also enables many performance improvements,.

C

Mainly related to I/o performance,.

C

And we'll show that in a moment.

A

Okay. awesome. anything else you guys want to talk about before we get into the demos?.

B

No., I think, we're going to run through some examples and some of the advantages you get from the wsl 2 version.. So we would like to show them.

B

okay, awesome.. Let's take a look.

C

Yeah. so when you install docker, desktop wsl 2 on a windows inside our build you'll, get.

C

This wsl 2 take preview in [inaudible] menu.

C

And it will let you start or stop the daemon running on wsl 2.

C

As you can see, compared to the tens of.

C

Seconds that it takes to run the current docker desktop,.

C

It's pretty fast itself.

A

I'm sorry., but is that a step.

A

That you would have to do regularly or that's,.

A

Once you configure your machine to use docker and wsl 2,.

A

And I assume you won't have to do that. Much.

C

For the technical preview,, we are not completely sure about the quality of the build.

C

So what we decided to do is to keep the original docker desktop logic and the vm that comes with it,.

C

And we actually run the wsl 2 daemon side-by-side with that.

C

So if I go to my console here,.

C

I can see that it has populated a few docker contexts.

C

So there's still my default context, which is docker desktop classic engine.

C

That's the moby thing. yes, and with wsl context,, it's actually running on top of wsl 2..

C

So I can easily switch back and forth between the two.

A

Right., so the one that has the star currently is the one that's in use?.

C

Yes, exactly,, I can do something like docker context, use default, and get back to my classic daemon.

C

So I think I should get back to wsl.

C

One first thing that I want to highlight is.

C

Leveraging as a linux, workspace, stuff.

C

So here I am in wsl 2 in.

C

Bash and I can just spawn visual studio code,.

C

This will run code in a remote mode.

C

As you can see,, it's happening remote to wsl.

C

So with what just happened, there,.

A

You were in bash? yeah. was that bash hosted on windows, or was that bash from the vm?.

C

No. it's bash in the vm.

A

Yeah., so how did that happen?.

A

So that's something-. It looked like magic to me. yeah.. That's pure magic.,.

C

It's actually wsl 2 that.

C

Chooses a feature in the linux kernel called informats,.

C

Which allows you to extend the kernel so.

C

That it can execute other types of executables.

C

What the wsl 2 team has done is that, when they.

C

Recognize our win32 executable,.

C

They automatically spawn the process on the windows side,.

C

And just plan the studio thing.

C

Between the linux side and the windows, side.

C

Okay. so magic., so now I'm running visual studio, code, it's running in remote mode,, so my terminal here is actually linux. Terminal,.

C

And this demo is based on asp.net core.

C

Right. so just pause for a minute.

A

So right, now,, all this source code is back in linux. Land?.

B

A

The terminal is in the linux land,. So really the only thing that's in windows right now is vs code?.

C

Yeah,, the ui. actually,, the vs code infrastructure is based on two processes.

C

One, client processes,, which is a ui running on windows,,.

C

And the server itself that runs the various extensions and things like that is running on linux.

C

Right. makes sense.

C

so for this demo,.

C

What I want to do is actually build and run my application in a container,.

C

And be able to continue developing into why it's running.

C

So what I did there is, I have this.

C

Little start scripts that actually run.net sdk as a container,.

C

And I bind once my current directory.

C

Inside the five system within the container.

C

Usually, when you did that before wsl 2,.

C

You had to use fire polling instead of.

C

Inotify events, because we used.

C

A file share for actually implementing the bindment between windows and linux.

C

But as we are actually hosting my source files in linux,.

C

We can actually disable that and have .net watch work with it.

C

So if I do stopped.sh,, let it start.

A

Yeah, .net watch run's one of my favorite scenarios running in a container.

C

Yeah. so I can browse my application,.

C

I go to see a page and let me edit something there,.

C

Vs02 is some,, I save,.

C

And, as you can see, the change has been picked up: by.net watch,.

C

It's restarting my application, and without having to restart my container,.

C

And without having to rely on eyeballing, it just works.

C

So that's something interesting., so we have inotify support now, with linux, workspace,.

C

Which is a very interesting new feature.

A

Wow. we're going to have to rethink., so I'm actually the one who that you probably saw that that environment variable is set in our images.

C

I was actually the one who put that there.

A

So we'll have to rethink that going. Forward.

A

Yeah., I guess so.

C

So our next demo is actually taking.

C

This whole context of concept of linux workspace and a bid farewell..

C

So here I want to actually develop the docker daemon within wsl 2.

C

That's a good stress, test.

C

yeah.. So what I did is that I've cloned the docker, daemon code,.

C

And as I did, before,.

C

I will just code that,.

C

And the way we develop, the daemon usually on linux is actually to.

C

Use a development container that contains the whole tool chain for building,.

C

Go around codes as well of a c code, and things like that.

C

This development container has the additional benefit.

C

To get exact same version of the tool chain as what we run on the ci,, so that's really handy.

C

So to start my container,.

C

I will just do little make a code,.

C

So make bind here, code.shell,.

C

And this will actually bind once my source directory in my development container.

C

So now this terminal is running my container and interactive mode,.

C

And what I can do is use.

C

My linux scripts from my id running on windows.

C

So you said your linux? linux, scripts. yes.

A

Linux, scripts. okay. linux build scripts.. So if we have a bunch.

C

Of build scripts for building the binaries, running integration, tests,.

C

And so on,, and so I will just use them like I would do on linux,.

C

But from my id running on windows,.

C

And we might even running on the wsl 2.

C

So I made the binary, I can install the daemon,.

C

And I can actually run my new look lg that I just built.

C

Within my development container within wsl 2 from my windows, ad.

C

Right. so for folks that didn't quite get all the pieces there,.

A

Basically, what you do is you built the docker service,.

A

And now you're running it on linux, and then I think you're going to connect to it or no?.

C

No. I didn't. we did not going that far.

A

But actually, we could do that.

C

So what we are actually doing is that we are running a new instance of the docker daemon weaving a container.

C

But we can invent the flow that we use.

C

To develop the daemon without impacting developer, machine.

A

Right., actually I didn't quite get that step.

A

So when you ran docker d that was running in another container?.

C

My daemon here is actually running a container.

C

Oh, yes., I see now., I'm weaving a container and when I do docker here,, it's actually running locally within this container.

A

Yes., so if you did the docker context, ls,.

A

That won't show up because it won't have visibility on that.

A

Although I guess, if you had that correct ip address,.

A

You could somehow connect to that docker daemon.

A

If you expose that port out of the container that's running in?,.

C

It's actually running on a unix sockets.

C

okay.. It's not visible from.

C

We could actually run docker d within this container.

C

Explicitly through gcp and connect to.

A

It from the windows cli., it will just work.

A

Right., but obviously the main thing you're trying to prove here absolutely is these details that I'm asking you about for curiosity is that as a docker developer,.

A

Using the golgi toolchain for presumably quite large code, base,.

A

You were fully successful, like there was nothing that you had to.

A

Do to make everything work, it just worked?.

A

It just worked. okay, awesome.

C

That's fantastic and the association between wsl2 as a fact that we can run the daemon on top of.

C

That and it's accessible from linux and.

C

Yes got daemon wsl, which is mind-blowing in terms of user experience.,.

C

Right. one of the things we talked about sometimes internally.

A

Is windows compelling as.

A

Deb box for arbitrary developer, platforms.

A

I think what we're seeing here is,, if your developer experience, works on.

A

Ubuntu, for example, if that's the developer, you chose,.

A

It should totally work on wsl2.

A

B

A

Yeah,, it's amazing.

C

So next demo is another aspect of this linux workspace, being.

C

So now we're in windows. yeah.. So this one is actually a performance comparison when.

C

Using io for long-range is in runtimes.

C

So for folks, at home, io in this case is either disk,.

C

Disk usage or network.

C

It's a main problem, but we have is.

C

For some, these frameworks, like php symfony.,.

C

The usual workflow is to bind.

C

Out your source code into a container that runs.

C

A php symfony framework and tools.

C

And execute it this way. During development.-.

C

The problem with this is that php symfony rely very heavily.

C

On io performance and for large project,.

C

It's just not usable this way., so you have to actually run symfony directly on your machine.

C

And it can be a nightmare for maintaining your development environment.

C

So, to give you an example,, what I have here is.

C

Just a symfony startup project.

C

And it is dockerized in this way and it's using docker compose.

C

So I'm doing docker-compose up.

C

Oh,, I just forgot: one thing is to use.

C

Right., because this is going to be our baseline comparison.

A

Awesome. yes,, you did screw up.

C

So I'm doing my docker-compose up.

C

And we're going to go to this guy.

C

We'll just very well to say,.

C

Sorry, my english is not there.

A

This is like the poor, man's.

A

yes,. It's my poor man, benchmark.

A

So we can see that roughly it's.

C

400 milliseconds or 500 milliseconds.

C

To run this really simple, page.

C

That's all because of the include prof reference in php symfony.

C

Now,, if I go to my wasl environment, I have the exact same in demo.

C

The only thing that change is that compose file.

C

The bottom, which is 8082.

C

It make sense., so we don't conflict..

C

So if I do a docker-compose up on this one,, let it started.

C

Now,, if we go to this one,.

C

First, one there is, I'll proceed immediately at the same time, but after that,.

C

It's just an order of magnitude, quicker.

A

Yeah, looks like from here like 22 milliseconds.

C

Twenty-Two milliseconds versus 400 or something like that.

C

That's massive.

B

That's massive. actually, one.

C

Of the top issue that we have today,, we have docker desktop and we are really happy to to see that we've wsl2.

C

We are basically solving the issue.

A

Right docker desktop for windows on this particular case.

B

C

Yeah, totally., so what else?.

C

The last thing that I want to highlight is that the wsl2 is also quite clever in term of resource allocation, and it gives access to the daemon to my wool machine.

C

So if I do docker info, you'll see that I have access to.

C

All my cpus, and basically all my memory.

C

I have actually often notice that which.

A

Is by default you only get access to two cpus.

A

C

One windows today.: it might be useful to explain why that is.

A

Today and why you get access to eight going forward,.

A

Or it's not eight,, it's the full set.

A

C

Actually. that issue,.

C

Will undergo this up, we have struggled to enable dynamic resource allocation in a way, but.

C

We really are happy for the stability and things like that.

C

So we never switched on and end the war that microsoft did on wsl2,.

C

Ensuring that it just works,, it's makes it available for us to leverage.

A

Right,, so you don't have to do anything special.

C

No,, it's wsl2 and doing this., so the very good thing is that it has a very positive impact on everything that is either cpu or memory. Intensive.

C

As an example,, so it.

C

Doesn't impact on every docker build or things like that.

C

As an example,, what I want to do is.

C

To use a pretty heavy workload to highlight that.

C

One thing that I thought of was during the development of docker desktop with wsl2,.

C

I had to build a linux kernel and build.

C

Custom wsl2, run it in its kernel.

C

At the beginning, I used to have to set up my machine to lets, say.

C

Proprio, gcc and things like that, and it was a bit troublesome.

C

So what I did is I created a container image with all the tools that I need, and now I can build my kernel using a simple landscapes.

C

So I'm using this built in container script, which is.

C

Just doing a docker run by noting my kernel sources with.

C

My current configuration and just using.

C

This kernel builder image that I take with.

C

So if I run this script.

A

So running on wsl2 in windows, there's no and it's loaded.

A

This is running in your container?.

C

Yes, it's running in a container,, so it's building in a container and as you can see.

C

So you've picked your cpu.

A

Yes,, it picks my cpu,.

C

It's going to allocate quite a lot of memory.

C

Looks like over a gigabyte. yeah.

A

Oh, sorry, three gigabytes.

C

At the moment. it might go higher.

C

But the main point is that it can.

C

Yeah,, that's the point.

A

It can use the whole machine in terms of cpu and.

C

Memory and when it's done using that,.

C

It will just free up resources., so it's not completely instance..

C

So if I interrupt ability, you'll see that my cpu has reduced, but this process is still using free, gigs.

C

But actually it's just a caching,, some caching mechanism.

C

If I trigger something on the windows side, now that.

C

Is resource heavy in.

C

Terms of memory like building the new windows, terminal,.

C

We'll see that wls2 when.

C

I reach some io bonds of memory, consumption.

C

We'll see that wsl2 will begin to free up memory.

C

Right., so mostly it just works. Out.

A

Yeah., so the problem.

C

Here is that my machine has quite a lot of memory., so if we have to wait for it to consume that.

A

So, while waiting for that, one scenario I was thinking about is we've been using buildkit a little bit.

A

So that's for folks at home., that's basically the new version of docker build.

A

I don't think it's the default yet though., it's not the default yet.

C

Yeah,, I'd love it to be., but I.

A

Think one of the interesting things about buildkit is that if you use stages in your docker file,.

A

It has the ability to detect which stages can be run in parallel.

A

Obviously,, if you have a lot of stages, they're, now be interesting from the cpu standpoint.

A

So if it is eight cpus and.

A

Say like 20 stages, that would obviously be a lot,.

A

But you probably wouldn't see a huge time. Savings.

A

Yes, totally. that parallelization would really help..

A

Yeah, have folks been testing with that?.

A

C

Yeah., because I have many interesting lookup build scripts that I use daily.

C

Things like composing kubernetes and over projects that we.

C

Run in the developer solution group.-.

C

Yeah,, it's interest of free bonus, going.

C

C

there is nothing to do and it just come use all the cpus that are available.

A

Nice. yeah, I mean for us. I could.

A

Imagine we have a docker file that builds your app,.

A

And then in parallel publishes.

A

The final version and does testing at the same time.

A

If the publishing part doesn't rely on output of test and just relies on output of build,.

A

Then that should work out. awesome.

C

Yeah., so minus demo is not that conclusive because.

C

Didn't use enough memory? yeah. yeah. actually,. I did prepare.

C

This demo of some time ago and some build possess.

C

The terminal used to use three or four gigs of memory right from the startup.

C

It's not doing that now?, it seems not,, but yeah,. We can see that. a small reduction.. It's a small reduction because yeah,.

C

Based on our optimized.

A

Okay. well,, I I'm sure this scenario is going to be good.

A

Because the thing that you're talking about which is.

A

The linux vm side releasing resources for the window size to use,.

A

That would seem to be an important scenario.

A

yes.. So I'm sure that team will sort that one out.

A

Is there any other points that you guys want to make.

C

No,, I think we are good.

C

B

Okay., I think, other than that we're really excited to actually keep.

B

Working with wslt team to get this out,.

B

People to start playing with. yes.. So what has the partnership been like??

B

So we've been working with wslt team pretty much day on day for the last,.

B

Probably two or three months?.

C

B

They initially shipped this very early, build that wslt.

B

By handing over a laptop to us in their inbox.

C

I've heard about that. yeah. because it was the fastest way to get a build access,.

B

And we've really wanted to start working on it to actually get this as soon as possible.

B

sounds like james bond and delivering something in a briefcase.

A

Yes,, we call that a very modern software, [inaudible] shiny.

C

A

Well, there's this old joke, which is never underestimate. The bandwidth of the 747.

B

Exactly., it was and it worked, and it's really helped us get going starting work on this.

B

So we've been feeding back into the team, some ideas we thought about how to make some changes that would allow us to do even more wsl2 in desktop, and they've been feeding back to us as well changes. I think we should be making them.

B

how we should be approaching some of the further design decisions if we still automate.

A

Right., so the first release should be awesome, and then there's lots of opportunity after that.

C

Absolutely., it has been like the startup time,, which is really, really fast, between them., probably enable us to start with things more easily,.

C

And so being more battery saving.

C

A

So if I'm someone at home watching this video and I'm just super stoked about this,, obviously you guys have to ship your tech, preview,.

A

But let's pretend that's already happened., then. Is there enough functionality in the.

A

Latest 20h1 windows, insider.

A

Builds that the regular folks have access to,.

A

That they could try this out?.

C

Yes,, actually, this laptop is running even not the latest build.

C

Okay,, the public ones., it's a biblicon. yeah.

A

Okay. so basically the only thing left for people to.

A

Try this at home is you guys publishing this tech preview.

A

Totally., so how do folks get the tech preview? yeah.

B

So as long as they've got the latest windows insider build,, they can go to aka.ms/docker-wsl2.

B

To sign up to download the latest version or the preview version docker desktop,, which includes the wsl2 components.

B

okay, and then so once you go beyond the beta phase,.

A

Then that link will just go to the official docker desktop for windows,.

A

And that'll because you'll just need to grab the regular one.

A

B

A

Okay. well,: this is not a ton of value.

A

I think people are going to be super excited by this.

A

Just builds on the wsl2 thing., the combination of docker desktop for wsl2,.

A

Wsl2 itself, and then the vs code, remote features.,.

A

Just seems like developers are just going to be super happy.

A

B

Yeah., that's the goal, is trying to make it as easy for developers to get start to do what developers own on windows as possible.

A

Yeah., I think the thing I like about this as a really takes advantage of using the desktop piece of windows for you,.

A

Say, visual studio code, like you showed, or visual studio,.

A

But then also the best of linux.

A

yeah. yeah. okay. well,, thanks for coming all this way on the show.

A

I think you guys may be the first european folks.

A

On the show that have traveled all this way., so I really appreciate it.. I know my team has worked with docker for many years,, so we appreciate the partnership.

A

yeah, thanks for being on the show.. Thank you so much for having us.

B

Awesome., it's been another episode of.

A

On.Net show today about docker and wsl2,.

A

And all the cool stuff- that's coming. thanks.