Sci Cloj Meetings, 31 Oct 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Malli - plain data schemas in Clojure - Tommi Reiman - Scicloj meeting 9

Description

Tommi Reiman told about Malli, a Clojure/Clojurescript library for plain-data schemas.

github project:
https://github.com/metosin/malli

online playground:
https://malli.io/

repl session notes:
https://twitter.com/ikitommi/status/1188917624287223808

text chat:
https://tinyurl.com/yywaeqoq

previous and next meetings:
https://scicloj.github.io/pages/web_meetings/

A

Welcome to the cyclo sure web meeting number nine subculture is a community initiative to promote scientific programming with closure, and we are doing these web meetings as a means of achieving that goal.

A

For those of us who use the tools that others create when we can join and learn something, and for those that create the tools we hope to give them good feedback and an understanding of user challenges.

A

My name is: Thao, hey Graham, I'm, a closer programmer and I've found programming useful in civil engineering. Today we will hear from Tommy diamond, you might know Tommy from right. It's a routing framework as a metal sin co-founder medicine recently sponsored the closure trade companies in Finland or from its work at making closure go fast.

A

Today he will be speaking about Molly, but before we get started, I have a bit more information to give I want to inform everyone that this meeting will be recorded and the recording will be released on YouTube publicly available. So you know that if you ask any questions, if you're able to I would like to ask anyone to turn on the camera, that makes it a lot simpler to present and somebody will be able to see you if you ask questions or see if you really like something or if you don't understand something.

A

Presentation then feel free to interrupt and ask them. If you don't want to interrupt, then you can either ask me to interrupt in the chats, I hope everyone can see. The chats I just wrote hello in the chats, so you should be able to see that, and if you don't want to speak, then you can just ask you a question and I can read it to Tommy when that is appropriate. To do.

A

I would also like to ask everyone to mute their microphone when they're, not speaking that way, we avoid background noise and it's easier to hear, what's being said and there's a button for that on the lower left corner of your assumed window.

A

Today's agenda is that Tommy will show us Molly and then we will open for discussion and.

A

The meeting will take approximately two hours and the first part will be more with Thomas rhythm and when Tommy has come through the things that he wants to go through then we'll open for brother, broader questions, I think so that's everything that I am going to say for now.

A

Tommy, please take the stage.

B

So thanks for the introduction and thanks for having me it's like honor, to be here, presenting the modeling so I'm from Finland and it started to snow today. So we have winter here which is awesome but like and about the background. So I think we have been doing closure for seven years.

C

B

That I was doing a lot of Scala and Java development and I remember. In the Java days we had the same problems, so we were doing these dynamic forms and dynamic models and those days it was XML and I recall writing at least two libraries. They were not open-source libraries but trying to solve the same thing. So some kind of dynamic data models with dynamic rules. There was groovy sandbox he's running these like source code from company forums, but when I started to do closure and the closure like promise of everything, is data.

B

It was like oh I, opener that the things I was doing in Java bit those Enterprise technologies. They are like really simple in closure, but before today, I haven't seen like there are so many libraries but, and they have really good features, but for some reason being been doing this reinventing these data-driven form libraries or schema libraries, and we have like several of those within the project and we decided.

B

Let's do something for that and let's try to write one librarians would be good enough for those projects we have been doing in enclosure, but also based on those requirements which we have like in the previous life, but I'll start sharing. If that works, and if you have any questions, I'm happy to answer them immediately or or afterwards, let's see, if we can, can you see my screen?

B

Yes, awesome so usually I have slide set deck, which is like 100 slides, but today I thought that I will do no slides, I have only Ripple with cursive an and all the stuff is in here. So let's start on this namespace called Molly dot sky sky. How do you pronounce it's quite close, but and for today's presentation, so the library I'm going to talk about is modeling and modeling means a model in English.

B

So it's in G top, it's quite new. It started like four months ago and the value proposition is this: that everything is data. Basically, so there are no macros in the library you can use functions, but basically there's this hiccup syntax for defining everything, and because of that, you can persist. Schema definitions, put them on wire, put into the database and like written in the closure script.

B

But it's not like just a test that because we need to rate it the routing library which had similar like design principles so that let's try to take these hiccups in ducts and do a modeling library on top of that and just like feature by feature doing stuff and see what comes up. It's still pre alpha I'm surprised that, like it, it was posted in I think Devon some months ago and it was like picked up and there are many users. Many contributors I'm really happy for that.

B

But it's like not finished yet, and we want to like not freeze that there are many open design decisions and we haven't wanted to like lock them yet because there's so many things we have to. First, do and see that after they are done, we can optimize the whole and see what kind of package will come out, but hopefully, within this year.

B

Had the motivation: let's go there, actually it's in the readme and.

B

There are already really good libraries in the closest space. The schema was I, think it was two thousand and 14 it came out. The prismatic schema nowadays plummeting schema and I'm still using that in the project, so the project this is about to end actually tomorrow it has schema better because it was the most robust solution we could pick up for that and and when the project started, close respect was not finished, it's still in alpha so and we are leaving from the project.

B

So we didn't want to leave anything like alpha there because aspect to is coming and it might change everything and we are not there to fix it afterwards. So schema is great, but it's not meant for serial as I like serializing data, so it has records and there are few efforts of making these transit schemas, but they are not actually really working. And another thing is that, like when you have this or or or on tracks and schemas, you can.

B

You are not adding any backtracking of those, but it's still good and whenever there's a Bachynski, my is usually fixed in within few days. So it's still like maintain it's. There isn't any active like development, but still any bugs are fixed really fast. Then there spec.

B

We also use that in many of the projects and like, but it's still alpha and there's some few things that, like I, haven't been that fond of, and it seems that spec 2 is fixing most of the things so alex is doing great work, they're, putting the like schemas aspects as data and this transformation between the currency in tax and like data syntax.

B

But one thing which is missing from there is the runtime transformations it's equation. Basically and like it's not in the scope, it will never come and we did this library called spec tools to try to fix that, but it's full of hacks and it will be it- have to be rewritten partially when spec 2 comes out and there's no skins of respect to you. So just we wanted to do something and start playing with ideas, and so we spin tout this new library called MA I'd like a little online playground.

B

Let's look at in the end, but let's start evaluating things so in Manly. There's this core namespace malik or let's require that and then what we can do is we can validate data and the validate it takes schema and a value. So this is a schema, actually schema AST. Then we have the value and then we could have the options it can. They can be empty, but all these top-level functions they either take valid, schema object or they take the schema ast and convert them into schema.

B

So if you use schema St, it's bit slower, but it's still work, it's kind of nice, but that's the core idea. You have these data structures, which are the schema definitions and- and you do something with that, and all the library is organized it's the it has the main namespace called monitored core I. It standalone there only dependencies, the sei and it will be made optional dependency and then there's these sub, not namespaces, which you don't have to use and I.

B

Think the core is something like less than 10 kilobytes achieved in in the JavaScript side. So it's pretty small the transformations they are in in one namespace. It has like, like from climatic scheme, a string and J's and transformation- it's quite small, also the serialization, using the work dudes a domain. It's also small. It's just like, writes the scheme us to eat it and reach them back. If you reader tax human errors, we wanted to test that too. So it's like one mono repo at the moment and all the like concerns are different namespaces.

B

It might end up being that this, like multi module library, but now it's like just basically one as a separate namespaces, and if you only use the monocore, you don't get any extra dependencies except sei. If you use Eden, you get the ADA meme. If you use generator, you get that test check and this chalk, and so on Molly provider.

B

It's this f-sharp style type provider kind of functionality same as in speck, has the speck provider, and these are the lines of code, so they, the extensors, are really small so because the core, because we wanted to optimize the hole, so we pushed enough hooks to the core so that it's easy to build this kind of extra applications and chase of schema transformations of form only schemas to JSON schema to Schmucker, also really small.

B

There will be few extra namespaces for visualization, etc, but idea small core enough extensions and we are not locking the library because we are not sure with extensions extension points are still needed: some public API s and one dependency.

B

So what do the scheme must? Look like the simplest schema is an like fun core function when you, when we evaluate that, if it evaluates itself, so it's just a function value all the core functions have like two presentations: one is the like valley, format and another one is the unqualified symbol and both are registered and it allows us to go from one form to another. So in the Eden format we see the like this form and when we read it from Eden, we convert it into this format.

B

So there's a mapping table so from this. This function maps to this symbol. If we want to do something more, so we use the hiccup syntax, so it is vector. So we have vector- and the first element is the name of the element and then there are arguments- and here we have a tuple. This is double double and again it's just data, so it evaluates to itself. They can have properties like hiccup. So here we have an property with title location mm.

B

We can compose things, we can have this and we can say that something is an integer and it's greater than 80. We could have, or we say that it's an integer or it isn't string spec has these named branches and modely doesn't have any of that. So I think that spec is awesome for doing like this development time, like tooling and and in in runtime. Molly is optimized for like runtime schemas, so we have the branching information but they're like derived.

B

So this would be the branch zero and this would be actually this would be punch one. This would be Prince two, so they automatically generate this an enumeration.

B

Maybe so this maybe not I, think with spec but like it seems to be that in many real life systems like a nillable is a fact. For example, there's many likes JSON schema, it can have Neil and Jason can have Neil. So we just wanted to support that too. You can have read excerpts this. This way of the schema. Expander has this type page so paste shortcut, so the reg expert can be given in the normal form.

B

Then you can have a custom function, so anything that is a function can be used, but if you use function so spec is good for this, because it has these. These macro suites captures that both the source code and the like, based on that taken I used value, but in Molly there are no macro. So if you would evaluate this, we can see that there's some some like function and we have no idea what is the source.

B

What good thing is that, if we want to make portable function, we just quote that and what we capture is the source code and sei the library we can evaluate that at runtime and that also works in closure script. So if you have a schema which has the function and you put it over wired closed script with eval, you can like run it over there too. There are collections, vectors list sets, and then there are maps and map.

B

Has this bit different, syntax, all all the elements, the parser first delegates it to this like map schema definition, and it knows how to parse its each toils and in maps there's like vectors of this child. So this is a map of X&Y and both are teens, and this is like- and this is like bit larger schema, so we have- and we can have some metadata title first element of the end is a map it has X&Y and an address which is a sub map. It has street with a string city which is optional key.

B

Also. The optionality is like cap supported and the city is Helsinki tamper and the location to topple and then another element for the and is a function schema. It has this error methods, so one of the goals was to get this like error message: I wrote, error message is right both for developers and for end-users. So here we have defined this error message and then a function schema that it takes, gets the value of the first element in the end and checks that X is greater than Y.

B

But again this is just data and like now there are the functions used in here and we can see that, like you cannot put this over wire, but if we would say that let's write the scheme out of that, we will see the form. So actually this is what goes over the wire. So all the like functions there's this registry, which maps from function to the same simple symbols. So you can put this over wire yeah and like how it's implemented. There are two protocols.

B

This is into schema and schema and let's go to see the definitions in skee mice, something that takes the ASD and and creates the schema object out of that, and the schema object looks bit like speck object, actually a lot, and we can ask the name of that. We can create validator out of that explainer out of that explaining also coming from speck big thanks to rich and everybody else who have been doing that we have a transformer.

B

This is the thing that I would have to have in spec, so the value transformation and then schema walking this generic extension point most of the features build. On top of that this you can ask properties and you can ask the form of that.

B

Those are the two schemas and when, if you check like an example, this may be implementation there, like quite simple and bit like the spec implementation in enclosure spec, so maybe schema we're if I this into schema, which takes that like ast. So this is this was like, given we check that it must have only one child, otherwise it fails. We create in like check the first schema. We create a validator out of that.

B

We create the form for that and then we return this reified schema and its name is maybe it has a validator, explain or transform or accept properties and form, and that's about it any questions. So far, let's see I.

A

Guess no I actually have a question if you don't so it's small e based on any of the other schema elaborates or is it built from the ground? No.

B

Actually, it's built on everything else, so here's the list of thanks and I think that it has some own ideas, but most of the things have been reinvented earlier. So big thanks to schema the coercion stuff is basically, we have been using schema coercion for for, like five years close respect. It has lots of brilliant ideas explaining the errors, as data and court I'd, just like a source of inner innovation.

B

In many ways the union types typescript the thing that I know, but it had like been looking for that structure, a JSON schema spec provider and an F sharp type provider. So I would say that it's like a fusion of the best features in the libraries and adding something on top of them and, of course, the like wave. We will trade it the performance and like structuring of things. So it's a combination of those but I think that end result is something that I haven't seen earlier.

B

So it's like JSON schema, but idiomatic closure as TD and so I think it's cool but good question.

A

Just a small small follow up: does that mean inside of malaya there there's no dependency on schema or spec or no.

B

A

B

Can show the dependency so dips eaten, so what it has is it uses closure I started to use depth just some time ago. I'm not sure do I have to depend on that directly. It uses the SC I. It's the only like core dependency.

B

If you want to use the serialization, you used a Des Moines and then like for the generators it uses just test check and this chalk and again, if you know, don't need the generation, you don't have to have them that probably this will be able split into few dependencies that if we just take the Malik or you just get the SCA okay. Thank you. Okay,.

B

So the public API- let's walk that through quickly, so we have a schema ast. It is an ant which has some properties, its integer, which should be greater than 18 and and when we evaluate that, as we saw like the raw form, is just what you gave it. So the quest int question mark is still in the value form. You cannot I see realize that.

B

But when you run a schema for that at the scheme, eyes constructed and the form does the lookup and for all the registered predicates, it finds the symbol and the form is basically the same, so actually the print out. What comes out for the from the printer? It is the M form. So if you ask tight for this, it's actually a schema reified protocol, but there's the print method over written. So it looks like this. You can ask the properties that simple you can validate that. So one is not for correct, for the schema.

B

20 is because it's over 18 and then performance wise in the schema. There was this validator, which is basically building optimized, validate validation, function for the given use case and when we call validate the validate actually creates its validator for the schema and calls it once and throw that away. So it's like convenient, but not that performant. If you want to have performance version, you create this validator and after that you have this pure function, which does the one thing and it's as light as possible.

B

Explain. Thank you. Closure, spec, so I think the programmatic errors in spec are awesome and in schema in Mali there are the similar. So what we get out of this error, we see that the schema which fails was this and the value we gave was 1 and the error was is in part, 3 and the myths means that it's the index, all 0 1 2 3.

B

So this one failed and the value is the top form, and this is the like failing schema, and this is the value, so lots of information, all the information that is available at that given time and when we explain 20, it says that no errors, basically and same here optimized explained you get. You can take the explained or run, for example, in rate it.

B

When we do coercion, it will do get a transformer ahead of time and in request time to as little as possible, and then that's the one thing that I brought issue to spec some years ago. It was too generic. Walker I'm still would be happy to contribute that in in here, this is how it works. So all the spec there's this accept method function in the protocol, and you can just say that. Take this ast and walk that and for its element, there is like return it as vector, so this was like it like.

B

Closure, walk basically top-level thing like then the next things, and then that, like last elements, so that's nice, an order like a JSON schema things built on top of this registry. So what are available, what kind of like predicates and like schemas out-of-the-box, here's the list of all the things that are possible: the default registry, which is used by all the like top-level functions. It's a merge of this predicate registry class comparator and base, and in the predicate we have listed that, like things from closure, call the core predicates, which seemed useful.

B

It's about the same list as enclosure spec. There might be few small differences, but we have this helper that we register both the like symbol form and the function form in class. There's this reg X. So if you are giving something which is of class reg X that is used for comparator comparators test is greater than greater and equals not equals.

B

So on and the base thing is the list of extra things and or map map of vector lists at any may be tuple records and function, but they are separately here, because if you want to make your own registry, you just like say that my registry, if you don't like these, for example, you can say that drop those, but only take this class comparator and base register.

B

And when you have the registry, you can pass it into the top level function and say that these are my specs I know schemas, and if you want to have like a.

B

So repo jumping so there's in the readme, the readme is at the moment source of all the things so custom registry. Here's an example. If you want to have, for example, you don't want to use those predicates. You want to have this like keyword, schema, so input the function scheme of int and its symbol would func function would be a pretty good for paint. You can add your own stuff at your domain stuff and after that they are available and because you didn't put up like normal predicate racket which it's not available. There are guides.

B

Also spec has this mutable registry, which is like easy there's, no such thing in Mali, but you can have that. So you can have an item of default registry and this function, which adds something that and then you can like say that register bang.

B

Let's have this idea, which is intended to have this name, which is string and after that they work and you, if you just like D ref, that when you need it and that's also, if you want really mutable this, is it I, don't know the alternate route, I think it's don't ever use this kind of thing, but evil register you don't have to even pass anything. We just like mutate, the default register, but I think that this would be valid.

B

So if you like this spec tile way of registering things wherever you need them, you can do that, but the registry is explicit in the sense that whatever you is passed to the function is used and by default, that's not like. You cannot new data, a transforming value.

B

So if you have used a prismatic schema, you should I know how this works and actually inspect tools. We added these two wave transformation, so you have different pipeline for encoding and decoding and there's a pull request for adding that to Mali to so we will copy paste from our own library to. But this means that when you have a valium and you have a schema- and you have this transformation context- let's say that Jason you will get this transformation, the functionality that this schema.

B

What was the schema by the way scheme? Oh, it was the aunt thing. So if we give this string one and say that it is, it came from Jason, it doesn't do anything. But if we say that it came from the domain of strings, it will transform that because, if it's, for example, a properties files in property files, you cannot present numbers they're all strings in Jason. You can do numbers, so we are not converting those a same.

B

For example, if you want to do keyword that this like schema, would be keyword and we would have something like Keep Calm like it will be transformed in both of the JSON and string, because you cannot have keyword in JSON, so we have to apply the transformation. But if there's nothing to do it does nothing and the change will be.

B

Let's see that also I did yesterday so that you can add the like decoding and encoding hints to the schemas themselves, and this is from spec tools and the people were wanting to do this and you can easily create these new things, for example, Jodha daytime or whatever, and say that this daytime come thing and I will describe it schema properties how it flows through different domains.

B

Another thing: another content, these humanised errors. This is something that I was studying a lot, because I'm wasn't sure that there is like really good solution anywhere. I was looking for the JavaScript libraries, the closure libraries there's phrase for closure, spec I, think that's good, but like in the JavaScript side, there were all these conventions of how to do localization. So these, like a mixture of everything, Somali.

C

B

And if we do a basic explain for this kind of data structure, so we have a map X&Y and we explain something that X is a string and there's no y, so what it keeps his two errors. Basically, it says that in past one one which is like one one, this one and it's a schema pointer, there's something wrong and the schema is int and in the value it's X, so meaning this one and the value was one.

B

So that's not correct and another one is that, but why was missing key, but this is like good for programs. If you want good for humans, we will run this with error messages and what comes out it attaches to those errors, humanized string of what is the cause, so missing required key and should be int, and if you want like this something that we could, for example, send over wire to the user. This is what spectres, so we can say that human eyes it takes the original data structure and in place of false values.

B

It returns this error and we can just call the message out of that. So now we see that X should be an int and X Y is a missing required key, so that's kind of nice and also because we wanted to do this. It was easy to extend the core so that this comes like trivial, and this is larger example. I have to make it smaller extent like a boss. So this is all the things you can configure. So we have a map, it has an idea, int, it's a size pieces.

B

We have overwritten the error message and we have localize it also. So in English it should be, should be one of the data and in finish it's a PITA all as anti L and H is a function schema, and here we use the static error message.

B

This error message, which is one of the special case error, F and which is another special key and there you can give a function which takes the whole error and like top-level options and based on that they can generate this string that, like Bonnie, it was this and should be greater than 80, and you can localize that also, and then we explain something which is totally wrong and be humanized that and in the human eyes we can also like overwrite the default.

B

So there are default like error texts for in English, but here we want to have finis, so we have to. We can on the fly explicitly say: what's what are the defaults and what is returned? It says the ID is a missing key ID. So that was a special thing it's in here, so the default for missing key is missing key and it's just like. What's the key missing. So it is something like that age.

B

It gets the number what it was wrong and should be greater than 80, and now we can localize that to finish big, because that's nice, who do I need so I spit, I, say hola, cetera, age, 10, PETA alignment, box and post. So like and I think this has. Now all the features which we have been doing in project to project is like internal more so, and everything is explicit again, and this does to look like bit messy because there's lots of stuff, but it's only data.

B

We can say that like define this small, medium or large, it's something like that and we can say define age, which is something like this. So we get nice reuse and we say that this is small, medium or large, and this is actually age and we get the Wow I didn't ripple was dirty yeah, the same thing, but this test, so they might get hairy. If you put all these definitions over there, you can have these encoding definitions and generators and whatnot, but like app, you can make them reusable because it's such data.

A

Any questions for this there is a question in in the chats I'll try to read it. It says, want allowing multiple registries result in parochial, T and or name collisions. Do you see the chats Tommy.

B

I think I have to stop sharing now. I know, I see the chat.

B

Sorry, how do I read it so? The last one just.

A

Scroll up to 7 35.

B

35 it's 8:35 here: yeah support, allows multiple registration and or name collisions. Definitely so you have to have like, but I think that if you have like one registry and and you with those, you redefine this, like the top level, things like validate and explain and transform and like create these helpers, which only use this one register, it always uses the same like.

B

It always uses the same names and like because in spec, fest is like anonymous like T's and I. Think, like everything in Molly uses those so there's no special definition for names based keys at the moment. So if you have users last ID, you can register that into the registry, but then it has always the same meaning. If you have a another like domain, for example, enclosure script, you have defines users last ID is totally different and you start to like change schemas over wire. They might collide, but I think like little like serialization protocol.

B

There has to be in any way like this, both the definitions and then the like values. So if you only use this like on animals or like one-time keys, which we always in the map define that it has a key ID, which is an integer, it has always like local meaning only and if you want to like define that users last ID means something you like communicate that over wire to, and they should probably be some kind of registry conflict. But you are talking about user ID with integer and I'm thinking.

B

It's like it should be string, so they are not compatible, but but I think it's good to that. We can like fine figure it out before any one sauce validating. So when we are like creating the schema object, we can compare the registers. If that was answering the question so.

A

I think the answer they Oscar got I got a good explanation. I say that is. Thank you. When the chats, okay job I will start.

B

Next, generating values so Molly generator it paid based on this check and test Chuck there are the dependencies so looks like because we are using the key function, functions like Specter, so it looks really much like that. Look like inspect, so we can generate random keywords. That's nice! We can use seed, that's exposed to the top level API, so we can like fix that value.

B

So we always get the same results and then we can use soy, sweet sister term in test check that like what is that, like value size and and drag x work to so this is the wire reason why we use test Chuck. So it has this reg X awesome like reverse parsing, so we can generate values from Ray gasps. It doesn't work. Enclosure script at the moment is an issue out of that and I think that was good reasoning.

B

Why it's not implement it would be nice to get that working to and then you can overwrite think so in any of the schemas. You can have these chain elements with this list of the possible values or we can say that it is a string, but if you over always generate one of these and again it's data, you can put it over wire and then there's at the moment this F map, which is directly from test check.

B

So it has this F map function and we can say that whatever gets generated like map that afterwards with this function and can get this like.

B

Actually there see that stick to see the way, that's bad, because it's always the same, but you get something with always starts with kick and I. Think we discussed about exposing some of the generator functions always also to sei. So you could used like this gin slash each of these same ducts. So this also works. But now it's like function, value a front, a function value so like we cannot put it over wire. Now we can put the wire and the same works in closed scripts.

B

Proceedings, persistent schemas I think this is the like main reason why we started doing this library, so we wanted something that we can put into the database. We have built like system for like startups for many years and and it's multi tenant systems, whether at some point we realized that actually the data model is themselves become the first class things that the users should be doing. And if you build this nice editor, you can define your data structures.

B

How do you persist them and at the moment the current libraries didn't have like good solutions for this, so so this is what we did so well. We take the Molly here in name space. We have some kind of schema and we say write to string and what happens is we get a string of string, representation, representation of that and it's basically the same in Eden? If you want to read that back with a red string and what we have is the exactly the same schema now we can do whatever.

B

So we can say that validate and let's say that X is 1 and Y is 2. It's false. Let's explain that.

B

Explain it says that X should be greater than Y, and if it's, if it is, it works- and this is all you have to do- this works in Grutter enclosure script and in growth, vm and big thanks to port dude for making SEI an ada, meme- and I think, is- and with this now doesn't work. It I mean you say stuff. There's like I have no idea how to put that value of wire.

B

Now we can do its bit slower to parse that it's interpreting the function, but it's still the performance is still okay, and if you are running local validation, you define the rules in the server. You can basically eval them if you trust them. So it's as fast as possible- and you can use this like for me to put it to the client and if you have a browser, it doesn't matter if you lose like 100 nanoseconds for validating something locally, because you have say to save the round-trip to the server to run it over there.

B

Yep next separate thing is the provider so spec provider, F sharp? Has these type providers and foreclosures back the spec provider, which is really awesome and I, was checking how that works and is, if there's something that we would like I use in modeling and it's the 66 lines of code and given that we have some kind of values, for example, here here's a vector of two values- and we asked like: can you figure out what would be the schema for that and what it keeps out is this? So it's not perfect.

B

It knows they are both maps, its string set of keywords, it's kind of write address, is kind of right, but long line is a vector of double. It should have guessed that it's a couple of double double. It doesn't do that at the moment. There's a issue out of that there's like so much we can do better here and, like I, think this is really exciting in the sense now.

B

My colleague kima was just using this feature today in the project and he was taking the large configuration files of the client and run through the modeling provider and said that ow it kissed. Basically everything right. So that's also and like then we can provide samples out of that and, of course, they're valid. So we take the samples, create a schema and then validate everything and they all like right.

B

There's the one issue: let's see that there's a list of things that doesn't work at the moment could be better, so maybe tupple annum or mean marks, map off and like this, like, oh I, am actually answered that, but definitely like spec provider has most of many of the things like already done. So, let's it's like use that, but in F sharp you can dis only from example data, but they would be and should be like from JSON schema extracting the male schemas and protocol buffers whatnot and just like I.

B

Think of the whole example the JSON schema, so that given a schema, you get the model II and after that it should be easy for anyone to write this. Like providers.

B

Full example, this is from the readme just like how that like working with Molly work, so we have an address. It's like a map of maps. We can explain that if we get errors, we get this nice structure which we have schema value and the errors and all the pass, which is the actual pointer you can say get in and you will get exactly the day. Schema definition which failed and then, like the value, pointer, humanized errors should be keyword, missing, quiet key should be double and then like real life, Jason transformation.

B

If you have something coming over the wire and in Jason, you cannot have a set of key words, so you will get the set of key words as a list of strings, but when we use the transformation in Jason context, they will be transformed into this set of key words, automatically transformers, there's Jason and like string at the moment. There's also the strip extra keys, transformer and I think there was some collection transformer you can compose them. This is from spec tools. Just took the same code for me here there and clean that up.

B

So here we say that Jason but drop all the extra keys. This is something that is coming over the wire there's extra keys, evil and dark.

B

Then there's this limitation that you cannot have these set of keywords and when we use that transformer it drops the extra Keys automatically and then converts the tags and that's automatic, and this is like that'd- be a lot of discussion like should this be in the scope of spec and I. Think that it should be it's like so important to be able to like easily drop out the keys and do these transformation, and we have been using that for schema for a long time.

B

So, but now it's in Mali there's the schema merge like because it's data, it's quite simple, to take multiple schemas and put them together. This idea of making also this uni difference and intersection over set theory, things and I- think that that's also nice, but the code for this is actually really simple. We just walk the thing and basically merge stuff sample values for that schema and then like provide schema from the sample values. So this is the best guess. If you would generate something that this is the thing that comes out, I.

A

Just want to quickly interrupts and obviously there are any questions.

A

I really feel that I'm learning a lot now, that's the tempo is we have to pay attention so are there anyone who want to offer something that they don't, but they haven't got the chance to ask it's.

B

A

Guess not so I guess we can take it. Thank you. Yeah.

B

Just a few extra features, so the walking over ski must. This is important, so you can write this visitor and say that take the address and walk it over and collect stuff- and this is the like easy way of like making the schema to something else: transformation and, for example, the chaser scheme transformation. You could say that I have an address, I want chase scheme out of that and that's the changing schema, basically because the walking is built in into the schema.

B

So it's like the code, for this is quite simple and the swagger thing this was the 21 lines of code. It's based on the JSON schema, but with few exceptions, so straggler schema has this limitation, so it cannot have these tuples.

B

So it has to have this X items which system like it's the real type performance, one of the things I was building spec tools and I, try to make as as fast as possible, but it was really hard and and formally just like, took the performance as like, first principle, that it has to be really fast, as this validation explained explanation and like transformation there that, like in the hot path, so they have to be as fast as possible. Let's take a test: let's have criterium.

B

This is something that our api consumes from the internet, JSON pizza. So we have the toppings, which is in the thing format that Jason can understand. Then we have the schema for that order. Then we make this.

B

Now we don't use transform function because it creates the transformer and doesn't reduce that we create this transformer for this order in the JSON context, and what we get back is some kind of function which takes the Chasen data and and returns it in the order scheme, and when we run it, we see that the toppings have been converted into right forward.

B

Let's see how fast that is at the moment,.

B

I'm not comparing to anything but like the transformer recursively checks, everything and like if there's nothing to do it returns nil from those branches, its recursive. So if there's nothing to do the whole thing returns basically Neil in here. It should see that this needs to be transformed. It has to do these few operations of like adding this associate these toppings and converting the values. It takes some time to run the quick benchmark, but it will be that long. Okay, it was 650 nanoseconds.

B

Let's make this small change, that our business decides, that we don't need toppings and what happens now is that the Chasen pizza and the order by pure luck? Actually, this order describes something that can be represented in JSON, so we have created this JSON to order again, we can see that it works, but it doesn't do anything actually well, it doesn't do much because it's in the same format as it was so, let's run the benchmark for that.

B

That was like a bit one change that was or one transformation it was 660 nanoseconds with the small change, is 3 nanoseconds. How can it be? Let's see Jace until order its identity, so actually the transformations like the recursion realized that there's nothing to do and in the top-level Neil was returned and the top-level transformer function doesn't want to return nil. It returns identity, but that's basically like there's nothing to do so. You cannot just beat that and that's automatic.

B

So whatever you do, you just define the schemas, you define the transformation context and then you just like say that just work and you get out of this like nice, fast functions but they're still much room for like improvement. There's the boolean simplifier that Myka my colleague started to do it's a nice thing that if you have this and int in course int. So this is kind of nice, but actually the current validation runs all these three and actually this could be reduced into force.

B

Int positive int only so that's the like, and because we are using these separate states for compiling things. So transformer does all the work for you ahead of time. So we can run this this just like. We will I just hope that, can you simplify this and it basically by default, returns identity? If you provide this external simplifier, you can do whatever and another thing is the data flow optimization.

B

So, for example, if you have int and then you have something like there, it's almost a say, but you say that it's greater than one, so these should be merged. That is also should be positive, int and when you generate values at the moment it the generation takes this as a generator term, so it generates any kind of number and checks that they are always 16. But if we could this, like term merge into this in generation, we could say that generate numbers which are always greater than 16.

B

The generation would be much faster, but plan is to just make these like hooks that if you know what you are doing, you can provide like good implementation for optimization, but that, like transformation, optimization was the thing I wanted. So last things, programming with schemas and I will show that like prismatic schema version, because we haven't done this yet, but this is really important and in prismatic schema plumb attic scheme might have been there for like four years, at least so. Let's take a schema core schema tools.

B

Core is library from us and have this like address schema and in schema you define in prismatic schema you define this optional and required. You have this like special markers for the keys, but and because of that they are not normal data. If you say that the from this map get this city thing, you have say that get this optional key city thing, so you cannot like walk it easily, but in schema tools. There are.

B

These helpers, for example, select keys, which knows how to handle this optional required keys, and you will get out this like correct answer, and you can say that remove this ID and country and the country is required key its abrupt, but it knows how to handle those and this. This is what I would like to see in Mali. So you take take any kind of schema. You can say that, like in prismatic, schema everything's closed by default in Mali, we are not sure it might be open by default.

B

It is that common like it is voting in one of the issues of what is a good thing. I think that whatever we do, we would like to hear what people think, but in in prismatic schema they're closed, but we can say that opens open. Everything then make the top-level optional then update something under the country and make them optional, and what you get out is this like map, which says that all the top-level keys are optional?

B

There's this openness definition in the top-level and the country is optional and under that this name, and it's also open so with few like this core, like function kind of helpers, we can. We could make this totally different kind of version of the other schema, and this is something that like definitely, we will be adding this. This one pull request request with started to do this, but there's a lot of more work to do, but we want to do this.

B

Visualizing schema not yet there. If someone is really interesting. Happy to see that we did this for a prismatic scheme as am years ago. So, given this kind of like schema definitions, you will get some kind of liking, UML dish diagram out, and this was big thing when we had the enterprise customer and they wanted to see that, like what are you doing and just like create this?

B

This is maybe 100 lines of code, but just this tool which prints out the craft with thing and every was happy so definitely want to have something like that. Involve the set operations, as I said, Union intersection difference, and this is a really good feature.

B

We have used similar feature in one of the projects and it's really nice, because if you define, for example, human birth, that it has to have this like ten fields and these nested data structures- and you can generate a form out of that definition and then, if you have like partially filled data coming from some other source, you can just like. Take this like difference or intersection of the form and see.

B

What do you have to still like give to me, and you can generate form out of that, and it has like many nice use cases and I think it would be really simple. You know, just like you have to figure out how to handle the optional key. So if you have like optional key in one schema and required key another one, you take intersection, so is it optional or not to just like think bit more, but after that I think it's a nice feature next step. So we hope that Molly will be out this year.

B

There's like all the things we want to do are in issues at the moment, so there's 30 issues most written by us, so more robust errors and like JSON, schema to male and not everything is required for 1.0, but we want to like this is like breadth-first.

B

We want to see everything that will come up and the design will be emerge from that after we have done all the things that are needed, so the core might need some few extension points and after we have, you know that the namespaces are could tell me to publish doc. I.

B

Think last thing I would like to say this: like bit of the differences people have been asking first of all, why but like what is the difference between male and, for example, spec, and here I collected some of the things that like Molly does but spec is not doing and vice versa and then are they coming and I think most of them will be coming so that or is your a reason that we Goodin persist that specs it will be coming? So that's expected in its scope and looking forward to that.

B

There's disclosure verse discussing about like what needs to be happening when you are starting to persist or share schemas over the wire and the question which was asked like what about the registry collation. So there's like one answer to that schema metadata spec doesn't have that at the moment, but there's the open issue for that. This, like error function as data I, don't think that spec will have those spec value transformations. This makes me a bit sad because I would like to see.

B

I would like to throw half of most of the code from spec tools away and, if just like spec core would have these ways of doing those. But at the moment, if you do transformation, use spec and spec tools and when spec 2 comes out, if it doesn't have value transformation, we will force the spec tools to support spec to spec working this issue. For that, and then this JSON schema transformation. You can you expect to swagger to spec tools in for inspect schema values in speculate, expect provider, which is awesome and for human errors.

B

This phrase, so you can do most of the things with spec and I. Think like spec, is much better in many ways for like development time things, because it has the Reg expects and the function specs and it's in the core, so I'm not expecting that this will be like anyhow competing.

B

So if you need dynamic, fully dynamic systems and like API things, you might want to use Mali if you want to have like something development, tooling and like stuff like that, you will probably use Peck, it's better for that things that are inspected or not in Mali, and should they be there? This is asked so in spec. You can have these key sets in Mali.

B

You don't have that, so you have to define the piece all over again, but it could be made so that map instead of having these only these like key value pairs, they could have these registry entries might be a good idea, I'm, not sure or different element, all together, like key set, which would have this a wreck expects like I, think this is like the brilliant thing in closure.

B

Spec at the moment, this is spec to syntax, and it does index in Molly would look like this, so not likely to be adding that, but if that is something that someone would like to do, I think it could be done and pull request welcome if that is needed by somebody multi specs. If we have time we could implement the multi specs, because I was doing that it doesn't take much time to make simple version of multi specs which are like closed, but there's your issue for that, and that's definitely that something is needed.

B

It's like really great feature of closest back and Molly at the moment. Doesn't support that conform on form they are not Molly doesn't have those, and someone asked that he might need those if someone needs them just like right, an issue and I think it could be done. There are no named branches in in Molly at the moment, but there's the explicit indexing because its vectors function specs. This is something that will come. I will make this I'm. Looking forward to this schema.

B

Had this test different definitions which allow the return, value and I think there was one library, foreclosure spec with using to say exactly the same format. So you can have this special dinner with where you can tag that the return type put the topple int and for the arguments you can say that these are the expected types. This doesn't work. Actually, this works because it's not validating but like the syntax would be like this and I talked with the prismatic koi some years ago.

B

That like how should we do that, and they say that definitely just take the code and like app make a new libraries or just copy copy, the code into your own library. So there's one spec version: it is kind of syntax, so I will probably like find who was doing that was it in Orchestra or where and see that if he could say something but cursive understand this. So if you have something which is not used, it's like showing that some syntax definitions but I think that's about my presentation.

B

What is modeling I will put this all the material. This rebel obsession into the gist, but why is Molly I? We thought that this, like this, something missing, that a fully dynamic schema system with everything is data, what is mildly, experimental library there's a Molly playground. I will show that just cook that last week so demonstrating the stuff. So this running closure script, there's no server for that. It's tough, it's empty, like scheming value, and you, if you put something there, a schema and value it.

B

On-The-Fly validates those there's a few example, for example, X&Y map, this kind of schema and like a bit this value, it says these kind of errors, example for user and address and like if you start fixing these, for example. This would be probably three so it's like quite quite fast and there's a small D bounce for like creating the sample values. Creating the JSON schema swagger schema in for schema sample values from the inferred school. There will be others.

B

The visualization thing will be here at some point, but like this wait you can say that topple string, and then you can say that and that's valid and now, if you want to share this with the debounce, it puts the query parameter so I can make it take a new browser and just copy place the thing and it's over there. So you can that's really simple tool but to like there's also already like few bug reports from this tool. So it has given already big value.

B

But, like recap, schemas and properties as data validation and explained the basic functions, a schema, serialization deserialization without evaluate body transformations are generating values, inferring values from but the sources. Now it's only examples later: JSON schema, etc. There will be tools for programming schemas. There is no global state. There's actually explicit. Everything and error messages are first-class because for both for developers and users and highly configurable, everything that is hasn't has an opinion.

B

It's probably configurable, so you can have your own opinion and idea that it's probably like library for making schema libraries instead of a schema library and there's lots of open opinions and they're in the issues. If you have opinion of those like go and write your comments, so we haven't looked basically anything we're targeting JVM note browser and crawl VM thanks to like port dudes, SEI and ADA mean it works on all platforms and looking forward to like community feedback. And yes, it is really fast.

B

You and the links I showed earlier so like there's some new ideas, but mostly like best ideas from other libraries we have collected into one package. That was my also thank you for listening and happy to answer any questions.

A

So, thank you Tommy. So much I really enjoyed the presentation and I felt like I was continually learning, so this is exciting.

A

So are there questions I, don't want to occupy all of the screen time now. So please take the microphone. If you want to.

A

Then I think I'm going to start off with my own question. Yep I was really curious about the playground and I was wondering if you can talk some more about your experience using that in projects, because I can I kind of admission not having to explain people like the data looks like this, but have you tried using it? How do you use it and if so, what are the experiences.

B

The playground I haven't used this like this came out like last week, so I haven't had any like really used for this and I'm, not using Moloch personally in the projects and I think none of the metazine guys are so Kim was like poking. That could it be used, but I think that this is basically, if you have checked the CheY CheY LJ condo web, so I've been chatting a lot fit for good and dead. There's somewhere over there where's the web, but like the playground idea is originally from here, don't sit here.

B

So if you look at the graphics, it looks pretty much the same. The buttons, the bootstrap, everything and I think these are really great tools, for example like running the linting or refined it of their multiple tools. I think sei has at the moment a playground and I. Think like this is great because in closure you can do this, you can see I. Do we have that no back to safety, so I would think that this was basically the tools we have been doing the project. They look bit like this.

B

So if we are given like an schema editor for business users, to do it's either like graphical or then you can expose the like hidden thing, and this is just post like a test for how to make it in today's library. So this code mirror and I like Eden and the Molly libraries and just like Putin together and small debounce.

B

Oh and it's it's really easy to do so if you need some kind of and because you can run basically, if you look at here, so we can have the like things that look like source code actually the source code. So you don't even have to quote these.

B

So basically you can add these business rules here and you or you could bill the rule system where the like rules are close. Your code and there's different schemas I, just put them together with these small.

A

Libraries, so I just want to say that worked. It actually posted the link in in the chat which is convenient when the library authors are watching.

B

B

That's one and the other one was I.

B

Don't have the other one put it whatever like. Let's fix that, one? Why? Why is that should be another link like? Oh, it might have broken that yeah. The chat seems to be killing the the last thing is broken, but like my references and it should be in the chat, the ones thanks, Karen.

A

Are there any other questions.

D

Yeah, so maybe if I can ask I'm Daniel I wanted to ask if you have thought about the opportunities with to link that could use playing data schemas. So, for example, you could imagine that the lintel could know about the type of data that it expects and then it would shout.

D

Oh, you could imagine that the autocomplete of your IDE or editor would complete beta examples that conform to a schema and I. Think in the F sharp world. They have nice examples of doing that with the type of providers. So I was wondering if you were imagining these kind of opportunities of how people could use Molly yeah.

B

I would definitely hope that, because I think it's it's so simple that, because the couple is just a vector with few things, the syntax is like really easy to like cab right and it's really easy to parse, because it's just data to the library to do that.

B

The Molly is the core is so small that it could be embedded in many places and I was just taking that, as you mentioned the f-sharp provider things, and it has this visual studio integration where, for example, if you point it out to the external JSON schema, you get this out of completion.

B

So what kind of futures dot address, dot city dot, something you can like browse the whole model and I think would be like really like awesome and I think what I have been like wanting is the the multi closed-mouthed aspecting and I was just yesterday. Thinking about that. So would it help the tooling? Because if you know, for example, you have this multi like schema, which has three possible options. So this, for example, you can add map which is like order or then you can have this like custom and created event or three different options.

B

And now, if you want to pattern match those, you would probably like generate the code to match that from the definitions or even sync them. So if you are trying to like read that you have this one pipeline, which gets data, which is one of these three like schemas, so could we check that? Like? Are you checking against the right data structures?

B

Because now in closure, you just might have a function that handle event, and then you read the type out of that and based on that start destructuring, the keys and when the like schema change, your destructuring code might be totally wrong. You had this like reading the wrong keys and I think that this simple data definitions could help there.

B

So let me know that this is the like thing that is coming in, because in statically type languages you Det usually have some Union type and you just say to their ID that, like something is coming in that generate me the code and you have this pattern matching with these other options and they keep in sync and like in closure.

B

I think we could have like 80% of that, if it just like I, don't know how to write the tooling I hope that, like someone, picks up this long idea of having these simple portable data definitions and writes the to looking forward to that. But my question would be the same that like here's Molly. Can you do something like that with this? And if you need something we are happy to add, because it's just like the beginning of the whole story.

B

But I think that one thing I'm going to use that is, and one of the reasons we started to do was in rated there's so much configuration and at the moment we are using expect one to validate, for example, configuration and inspect one there's no way to merge deeply nested specs together. So what I wanted to do is having these closed, spec validation for the whole thing. It starts like that.

B

You have like this top-level configuration, which might be like in the current configuration you define the databases and servers and the servers have options for porch and virtual host and they have some options and and somewhere you have the routing table which has all these in rated there's multiple components which contribute to the specs and you have to merge them. And then you want to say that, if is this valid and just I cannot do it expect one I probably could do it expect to. But I didn't want to wait.

B

So I started to doing male so that now I know that, like I want to deep merge two things which saw madly deep and they just work and if they don't work, I can blame myself so, but then having this like app error, messages, ie didn't show the developer facing error messages, but I talked to Ben upon expound. So would he be interested in contributing to Molly and he was busy, but we were chatting a lot about how to do that.

B

So there's actually a branch of having this expound great error messages for modeling, and that's really nice, if you have this like the whole system, is deforming with model schemas and for newbie experience. You spin up this like big web server and you just put some Eatonville some port, which is like in from place. You will get this like fig wheel, does so that you, this level has this extra key and it should be in this Long Island and if you run it in the background, you get this instant feedback. Now it's like in reading.

B

There's this nice error messages, but if you use top level integrand, it has different mechanism for that. If you use a mutant, it doesn't have any kind of validation for the parameters. If you seeker a database, it doesn't have just having this one I used to put all the configurations and run the validations from the whole stack.

B

That's something like wanna test, and it also might affect how the like knowledge would be model. So.

A

Are there any other questions.

C

Yes, I would have a question, so how would I, how could I use it with write it so, basically right now I'm using it just with closure spec and how could how could I integrate it for the coercion in there.

B

Yep, so for that there has to be this rated coercion, module for model it and I think like. That is something that should be happening next, because that's a good test to see that everything is like working, so I think the feature wise everything is there already? So it's just like we haven't done that and, for example, the spec coercion module is maybe 100 lines of code and schema cursor, maybe 50 lines.

B

So it would be really simple so that I cannot promise it to you too until before tomorrow, but it should be really trivial and and and one thing that needs to happen. We have to like push out a version of Molly because ready-to-use like closure, so we have to have some work in there that it would ever work. It doesn't use tips right. That's that's also like app in in spec. At the moment, there's like two syntaxes one is the data spec.

B

We did the simplified thing and it's too simple, because then people ask that I have these like map, like definitions, I will share the screen just for a second.

B

mmm Let's go to the examples, there's that, like rings, Peck, swagger and then there's this.

B

Here is other in data specs.

B

No sorry needs to be this ring swagger swagger.

B

Yeah yeah so like here are the parameter definition. This is the data spec, it's a layer on top of culture spec and it tried to simplify things, but as soon as you need something extra. So we reused all the like for maps and sets and vectors and thought that if you just need those everything is simple as soon as you need something like X is an or of integer and something else you have to like. You fall in between these two realms and so like closure.

B

Spec one is quite like if you can use that also, but you have to predefined the things if you want to like inline things. Data spec is good, but I think Molly, thus both quite nicely because you can have this. This query would be a map of x and y, and you can into the top-level put these titles and things that like contribute to the swagger specs, which is a common question. So how do I like add a title to this total?

B

You cannot just do it, but if we take this code and make a modified version of that, no, it would look. Probably of like.

B

Where is it like, say that hit some was totally mad, which would have total in and now the question? How do I like jet a JSON schema title here, I can say title is my body. If you want to an example, you can say that, like.

B

Json ski schema example- and you can say that total is, for example, 1, so you can do stuff like this and and like there's enough syntax for making any kind of complex skineri work, but it's still only data, so you didn't have to predefined it. Even in line everything you want, I.

B

Lost my battery over there, that's by output.

B

Could you please tell something say something hello.

A

B

A

Hear everything? Oh no! Well, Tommy! It's fixing, I guess! This is another good time to ask. If you have more questions.

B

B

Hello, yeah I ran a battery from my headset, sorry yeah, but I was saying that this is the way it would work with really done, and it's just like maybe few hours of integration work and few tests and then it should work.

D

Yeah, so maybe maybe if I could ask something more. So in the last few weeks at my workplace at localize, we are experimenting with kind of domain-specific language for doing some data processing, and we wish to now a large group of people who are not developers who may not be knowing closer to to fight to be able to write this kind of domain-specific language, which is some kind of hidden structure.

D

And we feel that it is a general question and that it's maybe an epoch, important question, because we wish to see how people who do not know closure can learn closure gradually.

C

D

At some limited example of Eden of Clausen book and what we are trying to do is indeed to define it as a limited case. Not all kinds of data structures are allowed, but something limited can be constructed and then it is again a tooling question. Oh maybe UI question: how can we create the perfect user interface to allow someone to create it unless they structures that conform to some to some schema, and can we automate this relation between schema and why? So?

D

These are our naive thoughts and we are having naive experiments but I wonder if you have similar challenges, sometimes definitely.

B

So in one of the projects there was after the approach there was at some point we needed to have this simple rule management system. It was like templating, basically rules for how to the like print out templates work, and my first idea was that the I will create is eaten editor and people will start right Aden and it was using the code mirroring and schema validation. There was all this nice, like floating error. Message sees and I thought. Okay.

B

This is something I can cook in a one day and thought that it will never go through. It has to have these like lots of forms, because there are business users and like like government people, so that, like they, they are not like hardcore computer science, users and I presented that the tool to them, and they said ah this seems to be good, so I think we can do that. I'm like wow. This is really nice.

B

So what, after the project, they've started to use that and they had to make this organizational change, because the title for the person using that application have to had some kind of like I, don't recall what was the term, but they didn't have this kind of position in their company.

B

They were like filling out forms and doing this like domain specific things, and suddenly they were like almost programmers, so they had to have this new title and one of the persons his older person like took the title and said that he will be start doing that and he was really proud of that and I'm not sure if they still use that, but like it was so I opener that, like it's really simple anything good tools, for example, put mirror on that.

B

My Molly play playground has to like pour in first, so it automatically puts the brackets at the moment. It doesn't show the errors, but a de meme shows exactly like which lines and which marker is back. If you just add that also there. So you have this limited Sandbach of things. You can do and there's no out a completion at the moment, but it could happen. I think the toolings get better. We don't have to do anything just like some of the JavaScript code. Mirror things have outer completion, I think there's a hook for that.

B

Also- and this is like big things, because in closure we can think like dynamically. So everything is dynamic. You don't have to like fix up things in my last Java project. There was one last thing we had to change and it was like going through all the layers it was like in the database. We change the table and the FiberNet mappings were change and to date, data transfer objects were change and everything was changed and the rest client questions. The server was changed and was huge amount of change. No, this.

B

This can't be this hard, so huge change for small things and now in closure, because things are more dynamic, so you can put any you can have non SQL databases and have these lose keema chicks in the database, and you can just like easily store things and you can push like the boundaries forward, so boost push the power to the users to do stuff and in the Java times, as I mentioned the one way to make this. These rules dynamic.

B

We have this like groovy sandbox and you, the business people were writing groovy small text box and it worked. It was quite kind of Awesome, but in closure. Now that you can just like write closure functions which are safe through the SC I and ADA meme into this Eden thing, and they just works.

B

So that's like that's like brilliant I, don't know what we can do with this, but I definitely think this a huge opportunity for writing, apps for business people and the live preview and everything. So it is just like built in the into the language. Basically.

B

And one thing like in the project I'm currently doing is where we use the prismatic schema was that we have these like state machines which were running the whole system, other like processes, the state machines we're like described as Eden in the back end, and only thing that was not serializable was the like scheme and schema definitions, and the project is about to end. We had this closure script, common closure, common files which had the schemas, but it was like, but because we had to know them in advance.

B

I would have liked that, like this one just this definition, we can put it to the database and this just like peat from the database, and this is the process, and it like runs like this, and these other beauties which are generated. These are the actions which are out in which step and with Molly, it would have been like available one year ago.

B

We would use that and just like extra layer, dynamism, sort of build this like web editor for the processes and just like after one months, we would have been done because so say that we build this engine and you can do anything yourself. I mean I, think in closure. D. There was nice talk about this state machines, workflow engines, enclosure each match. I recall that remember that from the title. But it was also really good that, like adding like live stuff into the like Java based BPM engine burp flow engine via the web like interface.

B

So there was a bill ability to add some like attributes to the processes and they were adding like closure code and running the ripple there. So they would like parameterize the processes through the web UI, and this is something that I think that there would be more of these things coming in the following months and years, because we can do that. The JavaScript people can do that with JSON schema. But it's like really big way and like, and hopefully the mica Juxt guys there, the genes library for doing that.

B

So it will make hazel schema like more closure ease. You can use that from closure, but, like molly is another like way of like taking the closure first, so hiccup under Aden, but please find used cases for that and if you need something we are happy to add it to the library.

A

I'm, just I was really curious, a follow question, because when we were as I'm hearing you everything becomes data driven. So we take all the parts of the application that we can and we make them data-driven. But what does that look like in the end? Do we then exchange our code editor for running something against a database? Or what do you envision that we're doing, or is it more for business people more powerful tools for business people so that we don't have to do anything if.

B

A

Want to comment on that.

B

Definitely I would say the latter one so that what I'm seeing at the moment is they're tools for the business people and someone has to build the engines, but like the business people at the moment, like seen so many project where business people want something and what they have do, they will create a backlog item and the software guys will do something and it will might end up having some some kind of UI which you can see or then it's just behind the cover something happened and like in many cases you can expose the things for business people and an enclosure, because you can validate things and you can like preview things that they don't have to like be afraid of things that they will break stuff and, for example, one of the features in SCO and what molly is using this termination safe.

B

So because you can write any kind of closure core code into these function, things and there's an interpreter. It has these limits that, for example, if you have the infinite sequence- and you say that take the last one of those it will have default like it goes only 100, tops and then says two months. So again you can create these limits and mother we made our actually was added to SC I. Was this termination safe, like mold?

B

So when you enable that the things that might like cost like an on termination, they are not allowed that you cannot use your ECAR recur and few others, and that's kind of nice, because after that you know that you can run simple things if those macros looping whatever, but like it's safe, you can run it wherever and if it's come from the browser, you can still run it on the server, hopefully I'm, currently testing and trying to make it break, but like it seems to be that it's good.

A

So you kind of be comfortable to use that's without having someone be really clever and have your machine spin, Maps 100%, all the time I would.

B

Definitely recheck the like integration to SC I, because at the moment Mali is depending on some sure it's not like in like, depending at the moment on the stable version, because Mali is not stable. I just moved some namespace a few days ago into some other place. So if there are pre alpha user, sorry for that, but try to like freeze most of the sting things as soon as possible and like caching, the libraries that they all like good before having this public release.

B

It would be really bad if there would be some evil somewhere over there and you got just like hack anyone's computer using modeling. We don't want to do that and like now, there's a termination save thing that testing that but I think like I, wouldn't want to program in like web editor. Definitely not, but I would like to build tools that enable people who were using just having these JIRA ticket, like pipelines, monitoring stuff from the developers having the power to Diamond process engines, rule engines like dynamic, schemas and running graphs and stuff, like that.

B

A

I think I actually have a somewhat related full of question. Is that something you you want to ask from the closure ecosystem? If you could ask people working on closure as a whole to to focus down on one problem? Would with that be part of it, or would you kind of point in a different direction.

B

Sorry, could you repeat the question: I was just thinking something yep sure so.

A

I I thought about this question earlier when you were talking in and basically what do you want to ask from the closure ecosystem if you could direct a bunch of us to work on something? What would you consider to be important and since we're talking about this rule-based systems, but at least I think would be powerful, but you require a lot of effort. If that's a direction, you would want to go in.

B

Definitely like a direction but I think like closure community at the moment, has like so many like brilliant minds and it's like overwhelming to see in like the quality of libraries that are popping up and just like, enjoying the ride and seeing all these linters and like whatever she J ljs NPM tools, shadowchild s and stuff, like it's like really nice, to see that what is coming out like, but the rule engines.

B

We have done like what has been needed in the projects and I know that if there would be infinite amount of time, I would probably like doing like all the tooling, because we have like we have seen what is needed and like it would be nice to do but like we don't have like unlimited like time to write all these stuff. So it's all usually, for example, it's my only like because just like enough ideas where in the table- and we decided- let's do something for that and try to push it not.

B

But now this 162 libraries I think we are maintaining so have to do something with that, but just like for community. So keep doing the good stuff and- and one thing that would be nice- is the I think the like developer experience and- and there was just like questions about like what can we do for that? What is expected for closure 1.11?

B

Is it the next one and like my biggest wish, was that, like the compile times, would be like smaller in the client side versus initiative of like having this client-side compiler äôt cache thing, which was all this, but it's something that we cannot do it's the like core team, which is thinking about this and it's hard problem, but like what we can do.

B

We have this library called beer head mythos in library for having these L état like created, like good-looking here or many things, but they have no content, basically so rate. It omits those and I have this branch where or the Molly errors are printed with that, so they look really nice, but the core errors there's enough data to build beautiful errors from the core messages. But we should have this common formatter and happy to put this our media library to GL gekommen.

B

If that is a good start, if it's a bad stuff, looking forward someone to making a better, so it's something that would be nice to have this like coherent experience, so tools that, like start quickly and and you get this configuration which is like unified in somehow an error message. It isn't somehow like like on the same grade and but I, don't know it's that. It's something that things that should happen and I don't know how to make them.

B

So, hopefully, someone applied for error messages for glj, together and I'm, happy to like point out to the what we have been doing so this a in four things and we don't have probably time to finish them all, but but it will be like between grant and Molly. We will have like nice error messages, I.

A

Also just want to say about I think it's right what medicine is doing for the closure community a place thanks. That's that's my impression. Thank.

B

D

Look so I guess tell dole, has some connection problem for a moment, and maybe he was going to ask us if we have any questions- and maybe we will be with us in a moment hmm so can I ask what was difficult?

D

Did you have any kinds of data where you saw that Molly cannot express your assumptions about the data or that you cannot do the influence of the schema from the data because it was too complicated or difficulty in any other way.

B

um I think like what is the best part of closure spec is the Reg expects for, like sequence, like specs, basically, there's no such thing in Molly. At the moment, this plum attic schema had released a plumb attic schema. It is really simple, like sequence, specs and like they good for most of the things, but when you want to like describe, for example, function, parameter these warr, arcs and stuff like that closure. Spec is really awesome for that and like if I would have. Data like that and I I would be using Molly.

B

I would definitely like like see how it's implemented in close spec. If there's something like cap, we could take from them if we just want to use Molly or then decide that okay. This is a better like this problem.

B

Is this like spec III, can't remember like where spec original I took that posited from racket, but there's probably like few sources which describe how to do these record specs and like a crystal grant, has this Express library, which was doing something like that, so it and anyway Bank building Molly was like really interesting in the sense that, like you, have to poke all the corners you have to take the providers and the generators and how they fit together and, like the validation explained and the human errors talk to tens of people, how they think about the world like just trying to get something like good out of that, but for the provider question hmm at the moment, like the list of things that the provider infer cannot do is like long and I.

B

Think it's it's math and you are the math guys. So, for example, if we get this like vector of values which are all integers at what point, do we decide that that's a tuple? Is it like? If you have one sample, and it has like a list of one number, is it couple of one it might be? It might not be if you have two samples and both are of two lengths. Is it's now a tuple so properly? There's like lots of options, configuration like these limits that need to be set.

B

So if you want to infer, we have like Tara theta petabyte of data, and you want to infer the values and, like you have to have these like some kind of limits that couple you need to have this much certainty for it to be double, because if you mark something as topple at the based on the first 100 values, its tupple of two and the one hundred and first value would have three values. Then it doesn't validate against the schema.

B

So you have made two strings so I don't know how to do but like there are smarter people who probably know know how this works. But one thing I was wondering for that provider. Is that I? Don't know whether people have done that before, but they should be like two things. One is the samples and one is that like correction data, so, for example, if you want to have this like, we have 100 samples and it seems to be a couple too, but the infer doesn't know that and we want to fix that.

B

They should be this. Like a small schema definition where we look, we say that on the user under address under location, we mark it as couple and then win Remini infer. We have had this like partial scheme of its fixes, some values and the inferring data there combined. So you would get all always the same results and like the best version of this, like partial schema, would be the full schema.

B

So we know exactly ahead of time, but this is the exactly scheme on which we are looking for, but you could have these parcel things, so we would say that we know that all things that are under a dress look like they are toppled, because we just know that that business guy, just in whisper, tell us that is the old system. He has these tuples strange suppose, but it's like it would be easy to add.

B

But just like have to dig in very deep and see it and some papers and books and see how it goes and because, for example, spec provider already has done bit many of these. So probably just like join forces and see like. Can we do like one two, both or how? Let's go.

B

But I think is like statistics like numbers certainty, isn't.

A

Like probabilistic data schemas is something you wouldn't tear up side of the closure community.

B

What was the term.

A

Probabilistic data schemas- we know for in with a ninety percent certainty that the schema.

B

And I think, like that's, also like a nice food for thought like how do there like, like a union schema Union is like quite simple, but the intersection and difference. How do they really work? So is it just like you just look at those and see the math, how it should go and it's like trivial or or they're like you need to parametric eyes so that how does it go?

B

So if you have two different types is like or or and or hardest causal I, just like haven't thought of that, but probably there's some something to think about, but I think that that would be like nice to be able to do.

A

And now you're still talking about inferring CMS from values.

A

B

And yes, and not so I think like if you have Stu schemas like user and address, you could merge them in the user address, probably just like like putting them together or if you have likes a user, a user and a neighbor, and they look a bit like almost the same event. They're overlapping me. So how would you combine them? Or if you say that, like we need to collect usually a neighbor information, but we already know everything about the user.

B

So we don't have to ask that and we can just like cut out the like app intersection from that. So we see that like remaining neighbor, looks like this, so we just take two schemas and what's the difference, I'm not sure how useful they are, but I have this thought that they might. It might be really useful because at least in one of the project we were doing this, so it was basically process.

B

Tweets was guided to like fill out these like datasets some of the day that was coming from like automatic sources somewhere like set by human, and if you have this like big data, like definition, and it goes through the whole like process and you get data from different sources, you can always ask what is left and events.

B

It's like cut out the big schema with the part you know, so you can take values and schemas and take that intersection of those, and then just you know that this is the remaining schema and and if you get data which conforms to this schema, you are okay and when you have can have a tool which take schema and creates, for example, simple UI out of that with UI component, you can sustain that okay manual guy, we couldn't get all the data you need to fill out.

B

These 17 forms, and it's just like dynamic, so fill out the things. So we need these before this. Can ship.

B

So it has like known use cases, but not just today, not for today, but I think the features to the libraries and then when you need that, and we need that and see that okay, I knew that this will re-emerge this problem and we have solution for that, and that's interesting, like that's I, think like Molly, is the best like quality library. We have done so it's like really tight and small and like now, that's we don't have to.

B

We didn't have to lock anything discussed a lot so how it should go and iterations on like one hour iterations like a few times a week, this more small thinking sessions and implementations destiny. It's just like it works and grows and hasn't been like many bad choices. Yet do is fun.

A

Now that you talk about code, quality I just want to note that I took it out previously, that's I think I learned a little bit about namespaces and just how to organize things and how to make it clear. So from from what you showed today, that was educational, a place for me, but back to what you said because, like you subtract, one schema from another or you can take data and you remove this part of the data and now we've kind of understood this part.

A

And then you can say what you have left was that in a sense what your months.

B

Yeah, if you go through an example, so if you have like a user- and it has like first name and last name and like address if we have like we have to get this email in our amount of information for all customers- and we like start with- we know that this is a requirement schema basically, and we get the address from some address system, and we know that. Okay, what's the difference, what is at them?

B

Oh yeah value, which is the address, and we have the schema, which has that, like common part and the address- and we check what is the like difference of that- and we get that like the remaining schema is just the top-level thing. If we get like the first name from some public system, then we know that the remaining schema is like reduction of these or other things. Every time we have this smaller ski and at some point we can sustain okay, we need this information still and we can ask that from the user I.

A

Just killed some generic user interface I can kind of ask you about these things once you've yeah.

C

A

B

A

B

This I think this, like all the generated you eyes, are usually really bad, but if you have like something like workforce, so you're doing like manual labor, some kind of installation thing, and the last thing is that you have to fill out for things. You don't have to have this like it's not meant for 1 million. You end users, which are like loving your product. It's just some mobile that install a guy has to like fill these three parameters and to generate that UI.

B

We can take a schema and at the moment, Molly doesn't have this from schema to web UI kind of namespace or library, but differently. You'll have at some point there's for JSON schema, there's at least few of those I mean in many other for prismatic schema, there's one library which takes the schema and emits user interface, or this form out of them, and it's just data. So it should be like trivial to do from modeling, just like seeing how other people have been doing that and are there any good practices?

B

And after that, like having the processes state and the schemas data you can like make this like dummy, workflows and like mock, the process is really fast.

B

A

There is this other project that I, don't really remember the name of, but it's it's related to the atomic and I think it tries to do something similar to this is that ring about fear or.

B

Maybe if you see I.

A

Don't remember the name right now: I'm not stupid,.

A

But they kind of they have this web UI, where they work with, they know with the atomic model and they interact with this model, both at the URL every layer and at the backend, and the way they solve a system is, let's say about they're able to build thanks in two days when other people build it in two months.

A

Just that's where my folks or what you were saying left me.

B

B

Probably know the library you are referring, referring to I haven't used that but I guess no one else remembers the name of the.

A

I think that Direction is pretty interesting because it lets us think about the systems and different way. What if collecting data from users was simple. If you have specific Asians and I, don't want to write a credit application now, I want to focus on other things. So let me write my data schema and let me just work with the auto-generated stuff.

A

Meanwhile, and then it might be really simple to change its follow until we know and for.

B

Example in Finland its Finland is the promised land of forms like all the like electronic. All the like paper processes are put into the internet at the moment that it's basically like you have this before you were feeling this huge amount of papers. It's like. We had paper a lot and now they are put into the internet. You huge amount of forms and like so many projects, are doing those, and we have done our share of those too and remember like one of those like form, libraries and data definition. Libraries we did.

B

It is indeed hub. I, don't know it's in Finland, many of the like public, publicly funded projects end up in the github because they are paid with public money. So few of those are there, but the good thing is was that it was like a building permit model and, like rules were really complex. So there were these like the rules that, for example, if you have field that, but information over there and some other information somewhere, it effectively puts a warning into third field somewhere.

B

So it was like guiding how to fill them and actually in the paper, it was like one one paper form which had questions and then two or three forms full of like texts. So how do to fill that and just like putting that debt in the form that machine can understand and guide you that? Okay, you just at this information and that they are like because of these two information.

B

This is the next thing you have to move, so you have the same things but like it's much easier to do, but so many forms in here so and I I think I would pick like three or four form schema libraries. We sat on enclosure and in in Scala in G, tops I, don't want to find them now, but like they, they exist.

A

A

So I guess, if you have.

A

Hours, what do you think.

D

A

I guess that finalizes, the official part what's going to happen now, is that the YouTube recording is its Sheriff and, as a fun part of the video I would like to say a huge time to use Tommy both for writing. Molly and explaining it yeah.

A

So thanks a lot.

B

Thanks for having me, it was fun.