Dagster Community, 14 Sep 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Dagster Community Meeting - Featuring Earnest Research and Dagster Updates | September 14, 2021

Description

In our ninth Dagster Community Meeting, we heard from Alessandro Marrella at Earnest Research on using Dagster to power their machine learning pipelines. Owen and Max from the Elementl team presented our Dagster logging improvements and on our Dagster Cloud Private Beta.

👨‍🏫 Today's Agenda 👩‍🏫

Introduction: 0:00
Alessandro Marella at Earnest Research: 7:26
Dagster Logging Improvements: 28:50
Dagster Cloud Private Beta: 47:07
Q&A: 49:22

🌟 Socials 🌟

Checkout our Github ➡️ https://github.com/dagster-io/dagster
Check out our Documentation ➡️ https://docs.dagster.io/
Join our Slack ➡️ http://dagster-slackin.herokuapp.com
Visit our Website ➡️ https://dagster.io/
Follow us on Twitter ➡️ https://twitter.com/dagsterio

A

I am thinking it's this.

A

Shapes like puzzle, pieces.

A

But it's thoughts like this.

A

That's perfect.

A

I tried my best.

A

A

But everything else.

A

A

A

A

B

Welcome everybody we're gonna, give people a couple more minutes.

A

A

A

It's a new emotion, I swear. There's diamonds in your eyes. Come hold me like you mean it.

B

I don't wanna keep this secret call 1203.

B

A

A

A

B

All right, let's get started.

B

um Hello, everybody, my name is sandy and I am an engineer at elemental. Working on dexter um I'll be your host. For today we have two presentations coming out that I'm excited about um the first one is from alessandro morella at earnest research and he's going to be talking about how he uses dagstr um and then uh owen from elemental is going to talk about some new changes to improvements to dexter's logging and then last of all, we're going to have a short talk from max, also elemental, about texture cloud.

B

um Alexander. Are you ready to share your screen.

C

Yes, I am hi, I'm gonna share it now.

C

Okay, can you see my screen.

B

C

Can okay, great okay, hi everyone, uh I'm alessandro, and today I would like to present you the work that ernest has been doing and how the company leverages taxer to achieve it.

C

So first, let's give you a bit of background, so ernest research was founded in 2012. It's a company that sells data analytics products to understand the customer economy.

C

It was funded by kevin carson and it has a globally distributed team headquarter in new york with a growing dublin office and remote people such as me that I'm calling you from italy it has more than 110 employee and, as I said, that is focused on consumer and market research.

C

So what does ernest do in practice? It sells products that let their customers understand uh how the economy is moving, how where consumers are spending their money, and so with that information, how merchants are performing in the market.

C

Ernest is a very data-driven company data is at the core so from the origination of the data, so getting data from several data sources to classification, which means attributing entities to the data normalization so grouping this data, for example in panels and then interpretation so which is providing analytics and uh information about the data.

C

It has a great track record. uh In the past five years, more than 500 potential supply visa misses were predicted against the consensus.

C

Consensus is what the markets think a company, how the markets think a company will do, and so ernest was able to predict better results than the expectations that were then validated at the end of each quarter.

C

Having said that, as you can see, data and analytics are a very important function in earnest and for this reason, the data science enablement team was created with a mission to grow ernest product, with data science and machine learning, capabilities to support initiatives across earnest engineering and analyst teams. So the team wants to enable both engineers and analysts, which are a larger part of the company, to run data science and machine learning on the data to be able to do that.

C

The team built a machine learning platform whose infrastructure consists in jupiter notebooks, which are heavily used for exploratory analysis and experimentation, as we will see later uh pipelines, which uh the team leverager leverages duxter for uh and groups uh and creates uh reusable standardized components.

C

It's integrated with the other teams uh using daxter, sensors and operations are farmed out to manage services such as ai platform, dataflow and bigquery. To connect all this infrastructure, the team created a library called the data science, development kit or, in short, the sdk, which I will talk about soon.

C

So let's talk a little bit about the gap between experiments and production pipelines. So we saw that experiments in the company are created in jupiter notebooks and it's usually python code. The company historically runs production pipelines in airflow using kubernetes pod operator, but to go from the first step to the second step.

C

These there are actually a lot of things that go in between uh which are, for example, writing. The coding python, initially like in your experiment, run tests locally if you're good and then refactor these chain transformations into cli apps, to wrap them individually in kubernetes, pod operator tasks and then either run airflow locally or push it to a remote distance to test. If everything works great, but it never works on the first case. So go back to two and do it again and again and again, and especially the wrapping the cli up.

C

I found it uh very very hard to iterate on uh so like we call it uh cli hell, because you need to keep passing arguments to cli and hope. That is what it wants.

C

So why did we choose daxter? We first looked at airflow because it was uh what was being used in the company and we saw that well. First of all, there are some missing features that are just not there so directly executing bugs or input and output type checking, which would have been nice configuration. Yes, there is, but it's a json blob, that's not really validated, but the most important thing is that the programming model is very task. Centric, you do a then you do b.

C

Then you do c without really much information being passed between tasks beyond like passing, maybe a string via xcom.

C

If you decide to use python operator, your code becomes a little bit more ergonomic, but you would have to deploy a separate airflow instance for each of the of your pipelines, which is not realistic or in the latest version of airflow, uh deploy all the virtual environments that you want to use in the airflow instance.

C

uh If you use kubernetes pod operator, as I mentioned, there is cli health so again for especially for data scientists. Iterating on this was very painful, so we looked around and the established solutions, especially in google cloud, were q fluent effects both are nicer to work on than airflow, but share some of the issues that airflow has. First of all, data input output is still a side effect. You don't pass data between tasks.

C

You pass information about data if you're, lucky and configuration is validated, but it's not as good, and we there is a tfx, also ticks the boxes of directly executing and testing dax and input and output type checking, but is very, very tied to tensorflow, which is not the only library we want to use in the company.

C

So, as you can see from the last column daxter instead, besides ticking all the boxes that we wanted. It also has a great api and, in my opinion, also great programming model by putting data in the edges.

C

So, just to reiterate, why do we like daxter it's easy to run the same code locally, execute pipelines in notebooks, as we will see soon write tests around kubernetes type checking is everywhere, despite it being python, it has a data, centric approach which leads to well-designed dags.

C

I think it's very important to put data as inputs and outputs and not start reading data randomly in a dag and then writing a solid. It's just a matter of decorating a python function and dependency. L is nicely avoided. Thanks to the repositories and grpc servers.

C

So we have this infrastructure and we wanted to build some tooling for our data scientists to interface with using infrastructure. So we built the data science development kit, the sdk.

C

We want also to provide a standard way of developing machine learning products and a way of going from experiments to products in a seamless way, which was the problem really that we're trying to solve. So the the sdk provides several base components and then some components that build on top of them.

C

You can see, for example, data source, just produces data training, get some data output, some model inference, gets a model output, some data, etc.

C

We needed to integrate with different data sources and things, especially in google cloud. Actually, when we started this exercise, we were in nws, so we moved to google cloud uh with the entire company and daxter helped with that too.

C

The execution happens in different layers and the sdk provides integration with all of them and it supports different file formats and utilities that you can use day-to-day in the notebooks.

C

So, having seen these components here, you can see, then these components can be chained because the types match so you have, for example, a transform component that splits training and test data gives the training data to training which outputs a model inference also receives some data and then produces the final results.

C

Each of these components also translates to a duxter solid via a two-solid method. So, and daxter is really at the core of everything that happens in the sdk.

C

We still wanted to have this layer to be able to expand with other uh execution layers and other stuff.

C

We also have a custom I o manager and some custom types. So first talking about the types we have location type, which is essentially a pointer to data that may be computed elsewhere, so this could be bigquery or gcs or local file system and data frame, which instead is data. That's actually computed in the python code that the solid is running or is needed in the python code that the solid is running.

C

You can see here a weird thing, which is that the types mismatch or seem to mismatch actually io manager, the io manager that we built takes care of converting between the two. So we can change solids without having to compromise on performance, for example, a component that runs a sql query. It would be weird to load the data frame just to yield it to the next component.

C

So you can see here that I put some icons about the types that you of location and data frame, that you can import and export. So this is how it looks like in target. You can just swap bigquery with gcs, with google sheet with local file system, without changing anything in the code. So just by changing there it just you can go from a unit test to what you do in production. Essentially, so enough talk, let's show you some code, and so now I'm switching to a notebook interface here.

C

So here, while we are suppressing warnings, because it's a demo, I'm importing the sdk, which is the library I just talked about importing some libraries that I want to use in the demo- and this is the workflow that uh data scientist alternate usually does so they start experimenting with some data, so they load the data. For example, here we are using the rf data set, obviously like in real world data science.

C

This notebook would be much messier, there would be like a retries and the cell numbers would be different, but here it's all presented to you nicely, but the important thing is that data scientists will work with the data trying to implement the classes or the components that I presented before.

C

So we have a transform component which, as I said, gets some data and yields some data training which gets some data and builds a model which you can also type the model and an inference component, which gets some data some model and some data and again a transform component.

C

If data scientists are able to code such that everything fits in these classes or further classes that we're building, then they get a lot of stuff for free and it's very easy actually to implement these these classes because uh they use standard python types, and you just need to forget about where you get the data from and where the data goes. It's just python functions really so by it just being python functions, you can run them and experiment with them locally.

C

You can run them on a row. You can create a confusion. Matrix and keep iterating until you're happy with your results, then once you're happy, let's make it a pipeline.

C

So, as I said, the classes can be transformed to solid, via this two solid method that where you need to specify essentially the solid name and the types that it gets and it yields and these classes automatically become solids.

C

This is boilerplate that could be further obstructed, but it's still good because it's not a lot of work to do. You can then build your daxter pipeline, as you can see, we're not hiding that we're using daxter daxter again is at the very core of what we do so with the solids we built above.

C

We are uh creating here, a pipeline uh with the usual nice syntax, where, like the results, are just results and it looks like python and then the pipeline can be executed in the notebook thanks to the daxter execute pipeline function, so we pass in the pipeline and then we pass some configuration here, some standard, bio manager, config and then some solid, specific configuration. Obviously this pipeline runs.

C

You can see now all the output in the in the jupyter notebook look at all the logs and then you can also inspect the output and look if the results are what you want. Maybe the results are not what you want, because here there is too much mismatch with these classes. So you want to experiment again, since here we are using sqlearn and the interface for inference stays the same, we're just swapping training in this case.

C

So the data scientist implements a new training class with the same interface and it can test again locally, just to sense, check everything, check the confusion matrix so to make it a pipeline. We just need to transform the new class into solid and define the pipeline again, and we can run it again and again we can get the confusion matrix, you get it and okay, so here we got data locally. So, as you can see here, the configuration location, local, I put the url and the data format.

C

uh If I want to get the data from bigquery, which is what happens in production, I just need to swap this, so I just need to say: hey get this data from the bigquery public data.

C

So again you can run your pipeline locally and test if it works instead of a table here, you could specify a query as well, which might be helpful uh to to just like. If you have a like a big table, you don't want to run it in a notebook, so you can do select and limit and whatever um okay, so you get your results.

C

And yeah this was a wrong output. We can also swap executors. So, for example, we if we want to run inference on apache beam because we want to highly parallelize the inference we can first of all run inference in in the notebook. The class has also a two beam function, so you don't need to implement anything.

C

It runs on beam, and here we enforce that the locations are uh specified externally using the sdk types, and once you define your pipeline, you can then uh instantiate a beam runner in this case a local runner uh and specify the output schema because beam wants the output schema for bigquery and run the the beam pipeline in the notebook once this will take a little bit so once you're happy with your beam pipeline to transform a beam pipeline into a duxter solid, we have a function that transforms the pipeline into solid.

C

The the key here is that the the transformation doesn't read or write any data, but we append and attach at the beginning the uh reading, input and output and the I o manager, and the inference component in this case will take care of transforming the data. You can again define a pipeline then and execute it.

C

The beam runner is a daxter resource, so here you can configure it and again. Here we are specifying. We want to use a local runner, so it's it's running beam locally, but running it on data flow would just be matter of using the data floor and earlier so, as you can see, the dangster resource model also helps by making execution swappable, which makes me think that even executor inducts there could be a resource, but this is a topic for another meeting.

C

Maybe um so, anyway, as you can see, you can run your pipelines in the notebook. It almost looks like production. So what do you need to do to deploy to production? Well, we usually have a docker file which contains the sdk a workspace file which gives a dasher hint on what to load and then a python file where we literally copy paste the classes and the imports from the notebook we copy paste the two solid function.

C

We copy paste the pipeline and we we just need to define these three lines of code and this is ready for production.

C

So this was the notebook and the translation to production. I just want uh to reiterate that uh baxter really helps us going from experiments to production.

C

It significantly reduced the friction, especially compared to what we had with airflow before the time system lets us separate business logic from computation and data, serialization acceleration and the resources, and I o manager, lets us integrate very easily with our cloud stack in the future. We want to make more components in the sdk. We want to start using baxtermine to send notebooks as artifacts, and we want to start using the asset api because we are not using it yet and we want to migrate to the new syntax. Obviously, that's it thanks.

C

If you have any questions, I'm happy to take them.

B

Yeah, let's take a few minutes for questions, feel free to um put them in the chat or to unmute yourself and ask directly.

B

Oh wait 30 more seconds and then we can also have space for more questions. At the end.

B

All right, well, alexander, those were very cool, um very good presentation, um very excited to see what we were doing creatively solved a bunch of interesting problems.

B

I'm going to turn it over to owen to talk about logging.

D

All right can people see my screen.

D

Yes, cool, awesome, uh yeah uh again, thank you uh that was that was really cool to see um yeah. So I'm owen, I'm an engineer at elemental and I just wanted.

A

D

A little bit of time to give a quick update on some of the changes we've made around python logging. Recently, some of this stuff is still experimental. We're working on interfaces, so feedback is always appreciated, but this is one of the things that we've gotten a lot of um requests for um to sort of improve this stuff. So hopefully some people find this interesting.

D

um So to start, um I wanted to sort of like disambiguate uh this concept of like dexter, logs and python logs, um because dexter log is somewhat of a ill-defined concept, but um I'm just gonna use a sort of placeholder here uh to sort of explain why things were a bit complicated in the past. So as I'm defining it uh extra logos are where sort of the things that you create when you're doing a context.log call or things that are created automatically by the director framework.

D

um So that would be like all these structured events like step started. Asset materials asset materializations um pipeline failed, like all those sorts of events are also extra logs.

D

All of these things are all these extra logs are sort of inherently linked to some sort of dagster context, so they're linked to the solid that they were run in or the op and future apis like a run id all these sorts of, like runtime information that is very like integral to dagster.

D

um They have some default syncs uh where they are output too, so one of them is standard out. If you're running daggett on your local machine uh or in a notebook like we saw uh you'll, see a bunch of um log messages come in like the standard out stream uh and then you'll also see uh them in the dagster event. Log, which is what daggett is using to to surface things.

D

So in daggett you're, looking at a view of the dijkstra event log, um so dijkstra logs go to those two places by default uh and if there are new places or additional places that you want to have these logs go to uh you do that with uh at loggers and if you're not familiar with those. That's that's totally fine, um I'm not gonna, go into too much detail on that um and then, in contrast, python logs. uh These events just come from custom code and libraries. This is all through the python logging module um they're.

D

They have a default sync of standard out sort of nuances. Aren't super important here um and if you want to have these python logs go to new places or different places, you do that by attaching handlers to a particular logger, so that would look like creating a python logger through the module and then attaching a handler that says: hey every log that this logger produces, send it to a file, um and you can add any number of handlers to a given log, etc.

D

So here these two like different concepts um and just to illustrate that a bit more, um here's like a very simple sample pipeline. So it has two solids. The first one just generates some parameters, um and then it passes them along to this function, which creates a log message that says: hey I'm going to run something with these parameters, then it does something implementation left undefined for now and then once it finishes, it gives an info message saying: hey. I finished trying this thing very, very simple stuff.

D

um So if we look and dag it, we can see that um all of these- and we look at the logs that are created for this one run things step.

D

We can see that uh you know we have all these sorts of automatically generated things, so we have like a step start event. A handled output event step success event, but we also have these like context. Dialogue calls as well and they're all linked to the solid that they ran in, so they all know that they came from run thing.

D

um Similarly, if we look at the standard out, because I ran this locally, we can find those log messages here once again. They're linked to this run thing step, um so they they know a lot about where they came from right.

D

uh So, in contrast, python logs, if we actually go back and look at this uh run thing- and now we say: okay, let's, let's uh imagine that this do stuff uh actually has some implementation, and this is what it looks like. Maybe it comes from a different file, and this is a pretty standard like python logging pattern, so you import the python login module. You create a logger for your file. This argument is just the name. uh Sometimes you'll do underscore, like double underscore name, sometimes you'll, just name it uh like foo.

D

This is just that the name of the logger to differentiate it from other python loggers you might have uh and then within a function you might make calls to that logger. So you might say logger.info here's some information, I've processed this many things so far um and you'll notice. Obviously this has nothing to do with dexter right. This is all like just python module stuff, there's no solids or anything involved.

D

um So if we run this and we look at the standard out um you'll see once again, uh we have all this sort of uh metadata about the things that were logged with context.log. So we know it was run thing et cetera, but all of these log messages have no specific information about baxter right because they were produced by a python logging, module and dagster.

D

Like was not capturing them and then finally finished running thing, and for similar reasons, these also don't show up in the event log right. So none of these log messages processed and things actually show up here right because they were just python like default things.

D

um So this is sort of like the difference between dagstr, logs and python, logs um and sure. Okay, like they're different concepts, um but this can be sort of like difficult right. It's often pretty useful to consider daggett um as a pretty like universal view into the logging information.

D

That's happening within your pipelines uh and not having all this sort of information like surfaced at the same level or at the same um or considered in the same way as these like the extra logs um can can be, make it more difficult to debug uh just be confusing in general, etc, and it's also sort of just tedious to convert like okay sure you could just take that do stuff function and and make everything uh instead of logger.info. You could do context.log that info, but it's not just changing all these call sites right.

D

You also need to change or like feed in this context, parameter in order to do that and um if it's just one function sure that's not the end of the world, uh but you can imagine this getting more and more complicated as you sort of chain uh these or have to like thread this context, parameter through like seven different function. Calls uh even worse! Is it's not always the case that you have like full control over this code right, you could have um situations where you're just importing this do stuff function.

D

That happens to have some python logging going on in it. It might be from some external library it might just be in your company, there's some team making some utility tools uh that are producing log messages, and you don't really like have the ability to change that without messing other stuff up um on sort of like uh in a similar vein, but um like in a bit of a different flavor.

D

uh There's also, this idea that um what I was alluding to earlier, where it's not always just the event, log and standard out that you might want logs to go to so my pipeline, is producing a bunch of log messages. But you might want these to go to some external service like logstash or datadog, to do some more, like in-depth monitoring for your production pipelines, which is a use case. We see a lot and just for simplicity.

D

um These uh generally those services, you actually just write to a log and then they'll have a separate uh like daemon running, to monitor that log file and then upload from there. So you're, not writing directly to log stash or datadog. So we'll just imagine we're trying to write to a file so every event or every log message.

D

We want it to end up in a file as well as the dexter event, um so we're producing gags for logs for sure and those go to something called the dijkstra log manager which will further delegate those log messages, but we're also producing those python logs from the do stuff function and those from the perspective dexter sort of just vanish into a black hole.

D

At this point- and you know, okay, dexter log manager, delegates we have some built-in stuff so that any log message that goes here ends up in the taxer event log and you could create some at logger to take your um log messages and add them to this file.

D

But again, the the problem here is that these python logs are sort of not included in this picture and those might be really valuable to monitor as you're um like as you're doing things and and looking for errors and etc in your production pipelines.

D

uh Another uh sort of tangential annoyance is that um all these at logger definitions need to be need to be defined um in the context of a particular pipeline. Meaning say you have 20 pipelines in a repository generally, um if you're doing something like you know, log stash monitoring you would want this like.

D

You would want, logs from every single one of your pipelines, to end up in logstash or datadog, and uh in order to accomplish that without loggers, you would need to specifically um add each one of these logger definitions to every single one of your pipelines. So there's no like global way to say: hey for every log for every pipeline do this. You would have to like individually for every single pipeline, uh define that and and in a lot of cases, you would really just prefer some sort of like global level, uniform logging policy.

D

Okay. So what do we change? How do we fix these issues? um So the main thing here is that we want to make it easier um to turn python logs into dijkstra logs, where you don't actually have to like change your code and thread this context parameter through um and the interface for that right now um and this fair warning this is experimental and we're also considering other places that we might want to uh add tap points to this.

D

uh But in the diagster.yaml there's now a new set of where a new top level key called python logs, and this is where you can configure um every basically everything you want to configure about how dagster handles python logs. So the first thing is just uh info, uh like the the python log level is info, meaning that that is the level of logs that we care about, but essentially we're just filtering out, debug debug level logs. um The the important thing here is mostly this.

D

This managed python loggers, though what this is is a list of logger names that dagster will automatically capture all the logs from so root. Here just means every single one of them, because python loggers are in a hierarchy, but if you wanted to just capture logs from a particular python module, um this is how you would do it, uh but here I've defined that hey. I want to actually capture every single python log um and then here this dexter handler config. This is also completely optional.

D

This is for, if you actually want a different sync other than like, have it show up and dag it and in standard out, but in this case I've defined a file handler, and this applies.

D

This file handler will apply to every single log message that is processed by dagster, so everything built in so all of the context to log stuff, all of the dagstar event stuff and also all of the logs captured by these managed python loggers and what it'll do is it'll just this is just a built-in class and it'll add them to some log file called mydexterlogs.log.

D

um So what that looks like is now, if I run the same exact pipeline code that I showed before, but with this dexter.yaml instead of like the the default blank, one you'll see that all of these processed end things messages get all of this good extra context right. So all of these, even those these were just uh python log messages by default. They know what solid they were originated from. They know the run id. They know the pipeline name. All this stuff, they're treated essentially exactly the same as any other.

D

Like context.log message would have been similarly for the same reasons, they also show up in daget because they are now part of the event log, so you get a full sort of picture of all of the logs python or otherwise, and then, finally, because I defined that handler, I have this my dijkstra logs.log file and if I cap that out, you can see that all of this information got into that log file, um so sort of just wrapping up this has this sort of, like simplifies the mental model here a bit because we have my pipeline, it produces dagster logs.

D

They just go to the dijkstra log manager as before, but we also have python logs and they're optionally, captured optionally captured uh by something called the dagster capture handler and they're, just typed right back into the dijkstra log manager. So this is where you can define like hey. I want this this and this python log to be captured and turned into like an extra log and then from there all the same sort of processing happens.

D

uh So you have the built-in, like dijkstra event, log stuff, and you also have the file handler telling um or sending all those log messages to a file, and so now we have a full picture in all of these places. We have both the python logs and the dagger logs make their way uh to the desired sinks.

D

um So yeah, like I said, uh we're also considering other places to allow you to configure this so things that, like the repo level or even the pipeline level, um definitely interested in how people imagine themselves using this. But the main thing is we really wanted to unblock those use cases where uh people felt like they had to thread this context parameter through in order to get things to show up in the event block um but yeah if people have questions have a good answer. Thank you.

B

D

um So it is in the newest version of dagster. um I forget what uh it's oh, like 1210 is. Is that what we're on something like that? That's right.

E

Okay, thank you.

B

Wait: a tiny bit more for questions.

B

Patrick says, looking forward to removing context from a bunch of functions me too. Okay, ken asks. Can we write a custom logging sync, so the logs get streamed to stackdriver or such.

D

Yes, so how you would do that is you would create a custom logging handler that just inherits from logging.handler, and you would add that in your dexter.yaml.

B

Cool all right, and now we have a coda from max to talk about dexter cloud.

F

Hi everyone uh sandy: are you sharing slides, or should I be sharing them? If you wouldn't mind, sharing that slide, I we can talk through it.

B

Oh okay, um let me pull it up.

F

Anyway, hi everybody, I'm max, I'm I'm working on the platform team here at elemental, and I you know know that some of you may be interested in a hosted version of tagster. I know that we've we've heard feedback about that and I'm happy to uh announce that there is in fact a hosted version of dexter available. We are um in a private beta with a few users and uh would like to invite you if you are interested to get in touch, and um we can talk about whether it might be a good fit for you.

B

I don't have permission on the slide. Are you able to share it? I can give you. I will.

F

If you will give me permission to share, I will get the slide up.

B

Are you unable to share.

F

Yeah give me a second.

B

Apologies for the technical difficulties, everybody.

F

So yeah, I want to um talk a little bit about uh what kind of distinguishes dexter cloud from the open source offering um because we're hosting this we can be a little more opinionated about how we host it. Obviously, in open source there's a wide range of deployment options that are supported and people are, people have deployed dexter in some very interesting environments, um the hosted version. We can be a little bit more prescriptive, especially about how the host cloud works.

F

I think, what's really most unique about dexter cloud is a what we're calling an agent model so essentially we're running. As you can see in this diagram sort of the orchestration features of dexter. That is, we run the schedules. We fire the sensors. We host the database that stores all of the events produced by pipeline runs.

F

We host dagit, but user code and data remain in the user environment. So there's an agent that you run in your environment, which picks up work off of the queue and then, as you know, for instance, pipelines run or when queries are answered, streams that data back to us. So we can see stuff like pipeline names. We can see stuff like solid names. We can see, of course, the metadata and the logs, but not the data that you're actually processing, not the code.

F

That does the processing and that's obviously interesting for people who have you know, there's many ways in which data and code can be sensitive, and so we think this is pretty cool. um There are also a couple of you know: sort of unique features in dexter cloud. We've implemented uh our back um with single sign-on, through sort of like uh google, auth uh sort of generic saml setups and active directory, which is pretty cool, so sort of uh checks. The box, probably on your enterprise, auth needs and then there's a really nice zero time.

F

Zero, downtime, continuous integration story uh for people, for whom that's interesting, I think, in open source. You maybe need to do a couple of things to make that a reality and in cloud it's it's uh it's much easier to get that up and running. So if any of this sounds interesting to you or if you'd like to learn more, um please just uh take a look at dexter, dot, io cloud, and um we can chat more about it in person. Thanks andy.

B

Thank you max any questions for max.

F

And then feel free to reach out in dm on slack or any other way. If you have any questions.

B

Cool okay! Thank you. Everybody for attending um our next community meeting will be in roughly two months from now I'll post the date soon, uh if you're interested in coming and talking about how you use dagster we'd love for you to do that. um Please reach out to me on slack I'm sandy rizzo or reach out to me via email, sandia elemental.com thanks so much.