Next Generation Shell Meetups, 4 May 2021

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Next Generation Shell meetup on 2021-05-04

Description

Next Generation Shell is a programming language for DevOps.

This meetup was about:

* Which problem NGS solves
* Brief language basics (principles, concepts, syntax, etc)
* Use cases
* Cool language features
* Q&A

Links:

* https://github.com/ngs-lang/ngs - NGS on GitHub
* https://discord.gg/6VqgcpM - NGS Discord

A

Okay, so the recording started. I want to thank ilya for delivering this meetup uh by the end of the meetup.

A

I will send you, through the meetup platform, the link for the slides and the video, and I will also send you some more stuff free stuff you can enjoy and let's start so yeah. This stage is yours. Please.

B

Okay, so hello, everybody, um I'm glad that we are all here.

B

Oh, I have this background. Okay, hey, okay! um So um let's start- and I guess- okay- we- I will be talking today about next generation, shell, so and the first question uh would be you know which problem it actually solves and uh it solves this absurd situation in which uh devops is actually devs and dopes and dev's got all the languages and ops got bash and.

B

So basically, uh ops, people or what's called devops or whatever systems engineers um they don't have a good programming language. On one side we have bash, which was created in different era for different problems and it doesn't meet. Let's say today's expectations from programming language.

B

On the other hand, we have python ruby, go whatever, which are general purpose.

B

Languages which are being a bit abused to do, uh devops, uh yeah and so bash is also abused, because it's you know if you want, you know structured data which you have all over today, uh you're using jq typically- and um I don't find it to me, so it's uh and that's the point where I um I thought you know I have bash, I have python, and none of this is feeling actually convenient to do what I'm trying to do and what I'm trying to do, what what is actually devops scripting?

B

uh What do we mean by that? So if these operations here are uh frequent in your script, uh that would be considered the dev scripting from my perspective and that's where next generation, shell, sorry, I have uh this here- uh yeah, okay and that's what ngs is about so.

B

So, let's see how the things should be, and things should be like that as that's, uh if you use ngs okay, um why? I think, I'm in the position to help anyone uh I'm doing devos for quite a while um I've seen stuff, because I was freelancer, so I was all over different companies startups, mostly startups.

B

And uh at 2013 I decided okay enough of that. uh There is no good solution, uh which is a programming language for devops. Let's do it um and since ngs is working for us and be me, I hope that it can work for other people too.

B

So I would today I will give a bunch of examples, but I think that a short introduction to the basics of the language is kind of in place to give to get some uh basis for uh the examples.

B

Okay, so language design principles, that's um what's at the base of the language, so I want the language to be relatively small small in number of concepts that it has, if you're doing, devops scripting, that whatever you're doing should be in standard library or directly in the syntax of the language, depending on the frequency of uh how much you use the feature. So, for example, running external program would be a syntax. That's number three here.

B

uh Additional rule is that small scripts do not pay, and that means that if you have like a script, it should be two lines.

B

It will not be a public static, void, main and the class around it, because you know the language is all about objects or whatever, and another aspect of that is error, handling, which is exceptions and not something which is returned, and you need to check it every time so because many of the scripts in devops they it's fine if they just fail- and you don't want them to be like twice as long just because you're handling every error, consistency.

B

Well, that's that's pretty hard one uh because it needs many eyeballs, because you don't always see that something is inconsistent because you're so much in in some particular view of the language or whatever you're doing now. So that would be nice to have help just open issues for if something is inconsistent.

B

So what are the big concepts in the language types? uh We will go over uh all of this in a bit more detail a bit later, but you know overview of the big concepts types methods, multiple dispatch, which is somewhat tailored for the language, exceptions and patterns. What is uh notably missing, I think, is classes.

B

So if you take a language which has classes- and you compare to ngs, if you take types and methods in ngs that will be roughly equal to classes in other languages,.

B

Two syntaxes because uh well I wanted a programming language, but I also wanted to have like concise way to run external programs. So one of the syntaxes, the number one looks like you know, typical real programming language and the second one looks like a pretty much subset of bash. I would say um so types how you declare type you just type and the name. That's that's how you declare type. uh I was surprised by myself how far you could actually get without declaring the fields and well by this.

B

Until this day it just doesn't have fields, declaration on types and it's still somehow fine. So if a type needs to inherit, that's the second second line, that's type child and then parent, which is, I think, quite obvious. How you create an instance. That's c equals child roughly like python, then you can assign to any field because fields are not declared you're free to assign to whatever field.

B

So the multi methods and methods. So on this slide, you see two methods defined and since they have the same name, they form a multi method and when you call you call kind of both of them and ngs decides, which one runs so as opposed to many different languages.

B

This search is not by you know, which type fits the most perfectly, which parameters may fit the most perfectly: the searches bottom up and wherever uh arguments match the parameters, the method is invoked.

B

What's interesting is that the method also can have have guard that's for fine grained control and this guard, if the expression in the guard returns false. That tells the language you know what that was, not the correct method to invoke. Please keep searching up.

B

So if you call wake up on mondays, uh we'll see oof, not on my days we'll see. Okay,.

B

So what what is also interesting? What happened you know? Okay, now, let's say you have uh types and they inherit from each other, how that is that related to the order of methods?

B

Well, usually, if you inherit from a type like here that that means that, at the point that you declare the child type, the parent is already loaded and probably all its methods are already loaded. So the order is fine in many cases, so how you call the methods and multi methods? Well, that's pretty much uh regular stuff that you see in other languages.

B

uh What's interesting is that uniform function call syntax, uh it's not unique to ngs, it's a feature where you can just take the first parameter and take it out and place it before the function and the dot between them and that's how you can move the c out of the parenthesis on the left side and put them in front of the operation. It's just pretty convenient thing.

B

B

Well, it just has exceptions, it doesn't uh like uh well, it also has a result type, but it's something different it. It mostly works with exceptions. um What you see here in the wake up function uh throws error, that's actually uh um how language design manifests so in other languages it would be. If c is sick, then throw error.

B

But since I have seen that this pattern is so common that you have a, if then condition then throws then throw and then something that this became a shorter syntax, so you have throws, you have condition then throws and then whatever you throw, also it works with a return and which becomes returns and breaks and continues so I have shortcuts for this, and the catch is uh pretty much ordinary, as we would expect in.

B

Like many other languages you catch, you can also specify the type of the exceptions that you are trying to catch and, uh if you want uh like really fine grain control catches, also support guards. So that means that I have example later it's less important right now: okay, so let's get to the fun stuff use cases.

B

So what do we do? In ngs? We run external programs, we manipulate data, we do things in parallel. We work with files and also not the least important is ngs should be used as one of the command line tools in your pipe when using other shells, uh for example whatever, and these are kind of the intended use cases for ngs so and um we don't have to actually dive into every one of them, but I I want.

B

I want you to feel that you're you're covered if you're, using ngs and you're doing things like this. You are covered if you're not covered that that's a bug and you can open an issue on github if there's something devops is that you're doing and it's not supported, or it's not convenient to do. Ngs, that's a bug, okay, which is actually a very different from uh let's say uh python.

B

If you go to let's say I just imagined today you go to python community and say you know what running standard programs is not very convenient in python. Maybe what do you think? Maybe we'll have a syntax for that? Probably not. um Maybe we can have maybe uh exit codes handling. We should throw exceptions, and now you just use the library it will be. Second, the second label citizen anyhow.

B

So if you're running external programs, uh ng has got you covered in these aspects, um let's, let's just go just one by one, not very deep but uh quick. Okay, first of all you sometimes you need to check that the program is installed on the computer. That's how you do it if program blah blah.

B

Actually, I can show how that feels and by the way n is alias, which is I'm just frequently using that's ngspi print, inspect, okay and let's say uh program, ls.

B

Okay program with name ls, okay, so in boolean context that will check if it's installed. Okay, this one is installed how about this one, not okay. So that's how the safe works um assert the program. So sometimes you have scripts which require some programs to be installed and they cannot work without this program. So just put assert in front.

C

B

At the top of the script, uh typically yeah, so if you have uh yeah, you probably guys remember that we can also place a cert after this. If you do the dot assert.

B

Nope exception because the asset failed. Okay, that's okay! That's less interesting, but you know sometimes it's useful and just as well facility to um create command line arguments when you are preparing to call external program, because sometimes you have nulls. Sometimes you have what not and uh it's becoming tedious to prepare uh this command line arguments for the next program, uh so there's a facility for that. I'm not diving into that.

B

um Comparing to bosch and just behaves like uh like many alternative shells today, if you have a variable and you use it with dollar bar name or whatever, uh it will not expand to multiple places. This madness is gone.

B

Okay, if you want multiple places that uh that's dollar star- and you know syntactically, that will will expand to uh several arguments, um actually zero or more um okay. This is um not very interesting.

B

Okay signals, mildly interesting, I would say: let's say you run a web server, you do some tests on it and then you need to kill it that that's how it would look like in ngs. uh This is actually very similar to code that we had in vime.

B

We had command line tools and that's one of the tools was actually starting web server, so we need to check whether it was working correctly. That's that's roughly how it looked like um okay, getting the output of the external programs that you're running um notable differences.

B

um The first line looks like roughly like bash when it captures the output bash by default, actually not by default. I think it does every time um strips the new line at the end of the output and that's the information that you lose, that you cannot ever recover. Okay and just does not do that. You will have the new line at the end of the output uh for the utilities, which return something useful as a one as one line of output, you should use the line option oops the line option.

B

uh It will give you the first line without the end of line character.

B

If you want all lines, just do dot lines uh and, uh in the second part of the slide, you'll see that you know if you run a program and you waited for it to finish, you will have the standard output in this field and then that works. Unless you did some redirection or whatnot, which is also supported by syntax, but we are not looking into that um okay, this is a. I think this is a notable feature of ngs uh because it shows like several aspects of the design, and it's also pretty frequent.

B

uh So this double backtick syntax that you see here. It runs external program and parses the output. So let's say: if you have an api which turns json, that's exactly how you would work with it. Let's say you have like monitoring or whatever.

B

Maybe we can, um we can actually run something like that.

B

Actually, uh you know what my ip, I think, my ip I'm looking at standard library of ngs by the way, my ip okay, no, it does fetch for some reason, but it would also work with backticks. Let's see that curl as this is an api which returns json, so, let's first of all print all of it. Okay, it has ip a country and country code.

B

uh Let's say we want the country code just for the kicks of it: okay, string! Okay, uh so that's how it works. It runs the command. uh I think minus s would be this silent right, yeah without this by the way you you get this verbose output, because it's print inspect. If you want uh just the il part, you just use print.

B

If, for some reason, you won't make a json out of it, this will be valid. Json output, okay, anyhow, we'll get there. um So what else is it this as many this syntax? It's actually calling uh several methods while it's doing its job and that's how this uh syntax is customizable and you can parse whatever you want as long as you define how to parse it. So here, for example, you see definition for find find will parse uh the output into array of strings.

B

uh I don't want to dive into this either, but you need just need to know that you can customize and if you have your own commands, which output in your own format, whatever that is, you can customize that okay, another another thing which is, I think, um differentiates ngs from.

B

Oh, oh sorry, the font is not readable in here in the shell.

B

ah Come on that's leggy.

B

Okay, uh because nobody apparently seems to actually care about exit codes uh and just does okay. So let's compare this example uh with, let's say: bash: okay, we have grep. Here um we have zen, and else so what's interesting is that the graph actually has three uh exit codes: zero, four, two one for false at two for an error and if you're using bash or pretty much, uh not anything else, but many other things. They will not get into this details about what is exception. What's not and then yes we'll get into this detail.

B

So this, if goes actually- and it just goes three ways: it's either then else or exception because, for example, if you like misname the file here and grapple, the return two and if you say you're in bash, you'll, see the error, maybe if you're lucky, but generally it will continue the execution into the else close, not so energies.

B

Anyhow, let's say you have your snowflake program and it's actually okay for that program to return 10 exit code 10 and it's not an exception. Well, in ngs, you have a syntax and you can specify what are the non-exception exit codes and that could be just a single integer or array of integers or a range.

B

I think that um the area also supports ranges inside I'm not sure, but it's pretty flexible, and if it's not flexible enough, you can customize uh more by defining additional method which is called finished. Okay, there is finished.

B

Okay, maybe I should rename it, but anyhow there is several of them defined. You can override that. Let's say you have special program which you you need to test. You know the state of the universe. Before you check you can tell what exit code means you can override this method, uh not overwrite, but actually uh well kind of override, it's close to overriding additional method to the multi method.

B

Okay, so here's how you handle exceptions, uh exit codes, two exceptions, uh okay, signals, not very interesting. You also have control over signals how to treat them. Okay, let's get into the more interesting part data manipulation uh which is actually uh under development still in ngs, but I think it's pretty interesting already.

B

So the main topic uh is the universal pattern. Matching so you can think about this, as you take regular expressions and you generalize it generalized to data structures such as areas and hashes and and.

B

You try all of it to be kind of the same mindset that you just have a pattern. You have your data and you check whether the data matches the pattern by the way after that, after I called this universal pattern matching I thought maybe it's not generic enough, because it's universal pattern, something because you actually might be creating new data structure from old ones, so it would be like transformation and not exactly matching, because you will be like using the data that was matched in order to create something new.

B

So, let's, let's take a look at a couple of examples here, let's say uh we want uh to filter a list and this list is a bunch of objects and they have the field n, so here's ngs versus python.

B

Actually, it's it's very interesting because I went to stack overflow, even though I I knew answers because I thought okay, maybe people will figure out something more interesting, more concise. Sometimes they did sometimes they didn't, but it was a very interesting experience and that's why also it take a lot of time to prepare this presentation because, oh and how they do this- I I maybe I know, but maybe maybe there's a better way so and how's it do this in jq. How about you know ruby stuff like that anyhow?

B

So that's why it took so much time to prepare this presentation, so we have filter which will return the subset of the list.

B

B

And any would just check if there is any matching element in the list, so um this slide is not very precise because on the python side, if the field n is not present on that object, that will be an exception and on the left side it will not be. It just means that the pattern will not match.

B

So, if you don't um do discounts in python, that's how things would compare actually. um So I think that uh whatever we have in gs in this case is more readable um and on python, on the other hand, is more verbose. um I personally um actually never connected with this list: comprehension, syntax.

B

Somehow it is not aligned with how I think um I don't know some people say it's completely fine. I think it's maybe people with more mathematical backgrounds.

B

I don't know anyhow, ruby is more close to how ngs works, uh but this select by the way select method has several names in ruby uh anyhow, um the select method which is kind of filter. It takes a piece of code to the right which is trans. It doesn't take a pattern exactly by the way so patterns in ngs are actually super set of predicates.

B

If you pass here in in the blue, if you pass a function, that means this function will check each element, so this function will be called with each element uh and depending on what the function returns, the element stays or goes it's like in ruby or on in many different languages which uh in which filter method takes a predicate okay and what else we have okay, uh assorted examples with different uh different stuff that we can check. So my list matches the pattern.

B

Repeat repeat: that means that, from zero to uh unlimited number of repetitions in in this particular case, you can specify number of repetitions. So this one uh checks that each object in the area will has the field number of chickens and that field is of type integer. uh I I guess it's some somehow reflect realities. Unless you know somebody is sick and then you can count half chicken. I don't know anyhow.

B

The second one checks uh finds the first item, which has state running and throws exception. If it's not found uh this, the next one is almost the same. It finds the first element which matches the pattern, but it could. We call the filter example as some sort of structure in syntax. uh It's not this structure, you can. You cannot for now. The universal pattern match doesn't have the structuring feature in it, so it doesn't work like that. Yet it is planned feature and I'm just not sure how to do it right.

B

Yet I had several thoughts about that, but nothing conclusive. Yet, okay. So let's continue so for the first here with host localhost, that's the default. So if you element with uh this pattern is not found, uh this object is returned to the right. Reject is the reverse of filter.

B

uh So, for example, pending running rebootings are ec2 machines which are like in kind of in good shape, and you typically can't delete the rest.

B

So if you want to find the ones that you want to delete because there's a bunch of states and it's not convenient to filter by the states which are bad- you want to filter by the shorter list, then you need to invest the logic so pending running and rebooting are okay and you you want to throw them away and just to have the machines which you can delete. For example, terminate um the one is interesting method.

B

It looks in the list for one element which should match if there are zero or more matches, that's an exception, because you have many situations when, when you know that there should be exactly one element like that, but if you do this session in the code, we it's it's much more, uh let's say maintainable code and you know don't need to comment. Okay. There should be exactly one element like that.

B

um A couple of other examples um um find whether any any element is as much in the pattern or all male all elements are matching the pattern or how many elements are matching the pattern. But you got the idea. Okay, that's less interesting!

B

um Oh, actually, we are almost out of time. Is that right.

A

Yeah, you still have 15 minutes. That's fine!.

B

Oh 15 minutes: okay, 15 minutes is good okay. So if you need to run things in parallel, which is another aspect where gs I think it's quite convenient- uh you typically use pmap in ngs- tells uh in how many threads you should be running your stuff.

B

So all these urls doesn't matter how many of them there are they'll, be they they will be um processed in five threads. That means that you have parallelism of five. That's how it works.

B

uh On the other hand, if you would do that in python, that would be much more variables and the example on the right which you see from stack overflow. uh It even does not include the parsing of json. So um well, I guess it's just much more verbose um with is order maintained. Yes, the order is maintained.

B

The question was whether, in parallel execution, the order is maintained. Yes and that's by intention. So let's say we have a list, one two three and we do pmap and say two threads and we just map it into the.

B

Sequence: dollar x.

B

You see one two three, uh we could do, uh let's say a.

D

B

So you would see it's actually like more numbers. uh That's one! It's a zero to ten, not including.

B

Ten, so you see it's it's um it's specific, because it's map the order is important. It's still a map right, so you you can do same thing if you remove the number and you remove the parallel from here and that's that should behave the same okay um back.

B

um Who do we got yeah? Okay, uh we are here here, okay, now, the next example. You want to do something in a thread I took example from stack overflow. I translated it to 20s on the left, okay, and you can just see that it's shorter by the way this uh this, if name equals main it just goes neatly into uh you, declaring the main method, and that's exactly for that, because it's so common and you see it in python all over. I thought: well, that's common that gets its own.

B

You know feature in the language. So if your script, your script runs top to bottom. If it happens to be that your script defines the main method, it will be invoked. So that means that you have. If you have small script, you don't need to define main method. You just write your code, you know top to bottom uh and it's just executed anyhow.

B

So what we have here, uh we have thread start here on the right which we don't have in ngs or ruby for that matter, because we thought you know if you creating a thread, you probably want to start it right away.

B

That's not typical use case when you create a thread, and you want to start it later, uh what else not much it's actually pretty similar, except that you need to do a couple of imports in python here and just doesn't have a built-in slip, so it runs external combat for that other than that.

B

It's like quite similar, I would say, um working with files, um not particularly interesting until until it gets into the interesting features which are on the bottom, where you can fetch the well file and parse it in one in the same command fetch for example, or you can store structured data in the file with one command. You don't need two separate steps to like encode json and then store that into a file.

B

How this store knows that it should be json, guess what the file is named.json. So here you go um okay, what else? Maybe less interesting you can get the lines from the file. You can run your own code on each line, which will later have some efficient implementation uh right now it doesn't.

B

um You can write lines into file with one command. What's this that's the regular stuff, uh maybe files? That's that's more interesting! Okay, that's that's a concept!

B

Okay, that you have a file which might or might not exist, and then you don't want all the comments which work with this file to fail, because you know it might not exist you you expect that so, for example, if you read from that file, you'll get null not an exception.

B

If you want lines of that file, that will be an empty array. If you call lines with callback, it just will never be called so everybody knows. Okay, there might be file, there might not be a file which is fine, because you you, because you told everybody that you know that it might not be present.

B

Okay, uh tmp file, temporary file. What's uh let's say neat about this feature: is that whatever, whenever you create temp file, it gets deleted automatically when the script exits? So you don't need to take care of that.

B

um Which is uh actually it it records uh how this is implemented, there's a hook on exit and which, whenever temp file is created, it gets yet additional hook. uh Callback and just delete the file when it's called on exit, um so a bunch of commands which are dealing with files when given no arguments where to read or which file to work with they just read from standard input.

B

So that's that's how you can fetch json here, for example, you run is to describe instances. Then you call in yes, you fetch fetch it's all the json from the pipe and well top label over. There is vpcs and and then you're looking for the one which is the default, and you want it sweet pcid.

B

So that's how you would actually there's something shorter, but I'm not getting into that because and just a brief description in bacteriobactic uh syntax in ngs. When you call aws ec2 commands, they passed a bit more, they not passed into json and like like simple json. They passed one label beyond that, so you would not need to do dot vpcs on that, because uh the json, which is returned, it's a dictionary which has exactly one key and one value.

B

So in that case, if it's aws command, you just get the value and the value is typically an array. So you will not have the vpcs over here, but that's just a step aside um exit codes, another uh another time when we get exit.

B

uh Another time where, where we see that ngsa uh takes care about exit codes, now these are exit codes from ngs.

B

So let's say you use mgs, as you know, part of your greater shell script and let's say we are talking about the same service which returns status, so this thing uh equals equals that will evaluate into boolean, and uh here boolean true, would return. Zero and boolean false would return 1..

B

So this is that's why it's so easily and straightforwardly embedded in brush if, for example, because the exit code, it was taken care of by the way you can you can, uh if you um can't condition, we said to control when it's deleted automatically um on six, only delete uh yeah we'll get into okay. There is a question I'll get into that in a moment. Okay,.

B

Yeah, so if you define your own type, uh if you define the exit code method on your type, that's how you tell in just how to convert your object into an exit code. So, let's say maybe it's not very clear. Let's say we define type t now we have oh switch to hey bro. Okay and now we are returning.

B

um Instance of type t what would be the exit code zero because it doesn't know about this type. So it's okay, the guess was it's okay, but if you define exit code of x, which is of type t and say it's ten, oh that's a problem exit code. Why it's not working ah because we are not returning t anymore.

B

That's why it's not working! It's just left over here yeah, that's that, but you can write your own custom code here, maybe check something inside the object and there tell tell everybody where it's, okay or not, or whatever, okay anyhow, so the question was whether temp file, uh whether you can control if temp file is deleted.

B

Let's say you had an error, and now you don't want the temp file to be deleted. That's interesting! This is currently not supported, but if it's your use case, you are welcome to open an issue, and I think this should be supported.

B

Like maybe dot you know, don't delete or something like that on the on the tmp file. Yeah, that's completely valid use case and it should be in um okay exit.

B

So uh since I noticed that you know if you have exit somewhere in the middle of your program, uh that means something wrong happened, so the default exit code for exit is one. If you want you can override with something else, but uh typically that's that's, because an error occurred.

B

um I want to skip about some slides here, because they are less interesting.

C

B

Are almost out of time? Okay, so this switches is why and just integrated well into the common pipeline, because you can output your stuff in many formats. Many, not many, but you know quite a few print lines will print. You know the items of an area each on its own line, print json. We will print you know, convert you know your expression to json the result of your expression to json print json lines will convert each item of an array into json print that and print inspect is actually for humans.

B

Okay, that's yeah! Oh yeah, another interesting feature um pt. Actually I don't think it was mentioned. It's print stable. So if your expression returns, let's say array of objects, they will be uh printed in a table. This is actually integration with integration, not really integration, but I'm using output from jc gc is interesting project in in which, whatever command you give it. It parses the output and returns json instead of it.

B

So if you had, uh you know file it, it is now parsed into json, so over pipe here, there's json and js just prints it as a table.

B

um Just a couple of cool language features which are not associated with any particular use case.

B

The common line arguments are passed into the parameters into the arguments of main and the only other place where I have seen that happening. That's in raku formerly uh named pearl six um because it's just convenient and if your script is small, you don't want to waste like five or ten lines, just to parse your arc v into something that you need so this this, for example, is optional because it has a value. It's like in python um section. That's interesting one! That's how you just organize your code.

B

It doesn't have any like deep semantic, meaning um it's just easier to organize the code, because in you know, if you have this curly braces, then you can indent your code and then, when you look at the code, it's just more organized also. I found uh you know recently. I found another use case for this. Syntax is that instead of leaving to do you just leave a section with name and it's empty, and that's how you know you need to write this yeah customizing exit codes. We have seen that actually uh guards and exceptions.

B

I mentioned that. That's how you should be feeling now.

B

I hope, uh let's see whether we have some yeah um future command line interface, the shells, the shell is not ready and that's uh because the shell itself should be in ngs, and you know, ngs was the language box. I started working on the shell, but it's uh just starting uh autonomous is not interesting. The pattern matching that should be. I think it should be developed uh very much. It should be like big facility.

B

um um Yeah destruction should go there. uh I think it should be something very big, because I think it's very useful speed could be improved, uh but you know that's um right now: it's not very important um yeah. All the rest is not very interesting.

B

So what guys? I uh propose! You do next. Okay, uh first of all uh start running next generation, shell. uh Some of you already did. If you didn't uh that's a link you can. I think I can be copy link, okay, I'll paste, that yeah uh join the discord.

B

Okay, I'll just paste it into chat for your convenience. Oh, this one didn't copy copy link, sorry for the spam, uh what it doesn't copy, I think uh copy link. Yeah, no I'd say it's not copying the link. How about this one copy link.

B

No, it just the copy, doesn't work anymore. Okay, so no links, sorry guys.

B

Okay, so uh you are invited to join discord uh if you have a use case which fits neatly into the devops definition and it's not covered by ngs. Oh thank you. Julio, open issue like with this tmp file.

B

This is completely valid use case where you had an error, and now you don't want to delete the stamp file you actually want, maybe to log the name instead, uh thanks for the links- um and you are of course welcome to contribute by uh actually you can do many different things to contribute uh feedback help participate in language design. It's a it's, because language design is something that is very easy to get wrong and the more eyeballs and more opinions you have that's.

B

This is much better implementation, of course, and uh yeah talk about ngs and you know tell your friends um of course. uh Oh, we have bonus slides, but we don't have time for them. So I guess we should uh start the. Maybe questions if you have.

A

Yeah uh wow, that was amazing. On my end, I enjoyed. I have lots of questions, but first there are a few questions in the chat and maybe others wanted to ask something.

B

uh The questions that I have seen in the chat I answered if there are, if there are some which are not answered you are like please welcome just.

C

B

Voice or by chat, you know just ask.

A

If anyone who, from those who placed the questions in the chat uh feels that he didn't receive an answer, please ask again: okay: yeah!

A

We shall provide that. You should organize the slides and people will get access to the slides in the video. Of course, yeah.

B

I just didn't have enough time. I wanted to do that. I wanted to provide the link during this this meetup, but I just was not prepared enough. Can you share? Yes,.

A

B

C

Yeah, it's it's.

B

uh Because I want to place it on some, not just share it from here. I want to place it on some good url and- and here then I I think I just can probably on the meetup.com I can just update. You know comment with the link.

A

We can discuss it later. There are lots of okay to choose from. uh uh I was curious to about the the support you have for floods. uh Is there any limit irritation? We should be aware of I'm talking. Do you have anything like the gear, and we know from python that we should be aware of.

B

Nope nope, specifically not because I knew about this issue and I didn't want it to happen so that, but that that means that you're, like you, need to use locks when you, when you need that it's my it's kind of manual well, there will be higher level facilities like this pmap. Then you don't need to care that much uh there will be. um There are already some facilities which are higher levels and you and you don't need to care uh about locks, uh but yeah there is. There are no limitations.

B

uh There is an issue where you can bomb yourself in pmap. If you do not specify the number of threads, it will use the num one thread pair items that you have. So if you have pretty big array, uh maybe you want to specify the number of threads.

B

Maybe maybe I should place you know built in some limit on that uh yeah.

A

So, basically, the operation we count on the operation system to optimize and use the the cores it has any sort of um control.

A

For example, if you have a computer with a nine core 10 cores at like three numbers and and do we have any way to control, to split the world between cores or do we let the oppression system do the optimization? It knows and that's it.

B

um Well, there is no explicit support for like pinning something to a particular core, I'm not sure how to do it, but you know if this c limit the number of virtual cores um limit by number of virtual course. uh So the suggestion is to limit the number of threads to virtual cores.

B

um Well, it depends on what these threads are doing. If they're doing like heavy cpu work, then this is correct, but let's say: if they're reading from somewhere and can be blocked, then you can have much more of them. So it's not very straightforward default is number yeah. I I know yeah so many tools do that, uh maybe as a default it it should be fine, but I don't know it. It needs discussion yeah so by the way guys.

B

So this discussion, for example, that can go into github issues like uh okay, here's the issue, the default limit of a number of threads, here's my opinion: what do you guys think and then we can conclude something and then it can be implemented after that.

A

E

They ask something sure um when it comes to functional facilities, I mean like filter, map, etc within the core language. What's available right now, apart from filtering map that you're already seeing.

B

Okay, uh a bunch is available, uh I'm not sure it's. I have this on the slides, uh so filter map, you know, is the usual suspects. uh You know they want. They all get all the functions that you have seen, which uh are um taking um uh sorry forgot. The word now.

B

They can't take um well, I'm just tired. They can take predicates okay. So that's your support for functional programming. So let's say you have filter, but uh on hash you also have the the regular filter will take a call between two parameters, one for key inverse for value, but I see that in many cases you want to filter only on the value or only on the key and hence there are filter key function and filter v function and in the end you will get a hash filtered by either key or the value.

B

Does that make sense? Is.

E

That, for instance, a reduced function of sorts or something like that.

B

Yeah there is reduce; actually I um it is there, but is the the use, is not that I don't use it that much. I don't know. Maybe it's me there is reduce. uh There are actually two of them. uh One is taking the start element, one which doesn't well, and the only example I have is actually some. So I don't know it's it's. No, it's not that frequent. I don't know uh it is there. You know just for completeness, but.

E

It's common in, for instance, in jq, when you have to merge objects or something like that to use a reduce function. For instance, just okay.

B

Objects: okay, okay, if you're merging objects, it's just a plus okay.

A

B

Ngs, let's say we have a one, you can just merge it with b, two done. Okay, that's that's one passion.

E

Another question about the types of data structures, or rather the serialization format. You talked mostly about json and then you've shown the how to deal with the output of you know now separated lines from find, for instance, should one wish to employ a different format, for instance, one that is not readily defined.

E

What should one do to actually should one actually write a parser for it, or is there some other way to actually define it?.

B

uh Right now, uh you have exactly two ways to do that. One is find c library which does that and add this to the project and another one is to you know to do this kind of manually.

B

uh Well, actually, uh since we are talking about shell, uh it's also appropriate to use external tool, which does the parsing and converse, let's say to jason. Well, it's.

C

B

Anyway, you know.

E

But basically you're telling us that you can actually um take advantage of c libraries to actually.

B

Well, it needs to be coded yeah, but uh yes, the project doesn't see itself. So is a part of the project in is in c and whatever I could that's implemented in the standard library. You can see it's 7k lines, it's even more a bit um because uh I said you know if one day I want to rewrite in go well, probably not go, but let's say rust.

B

uh If one day I want to rewrite this, I want the the c part to be like really small, as small as possible. Actually, um therefore, you have the standard library and you have. uh There are a couple of other files which, like less frequently used, but you know functionality is in there uh yeah. So if you would like to add a c library yeah, you add it to the project.

B

B

Okay, yeah, so that's that's. How json is implemented by the way.

A

What about in in in in interaction with a code written in other programming languages, ex a part of the possible the possibility that you give us to execute a separated program? Is there anything else that your ngs supports.

B

Okay, so uh what is supported now is running external processes. That's how you communicate with these guys. I uh was I'm not sure whether I opened an issue for that, but I was actually thinking uh this node.js guys. They. um They came up with something interesting, which you generate no gs code. You generate code in other language and then these guys communicate over standard input and standard output where the node.js runs kind of a server, and you can call methods.

B

So it's like it is running external process, but it's uh not doing it for every comment that you're doing you just keep it saved open and you can execute um a particular method call like with whichever method call you can do, but it doesn't need to rerun the program each time, so it also can have state on the other side. uh I was thinking about that. I'm not sure about that, because you know it's a.

C

B

And you know it's you know, running external programs is okay for now. If this.

D

B

Issue you're welcome to open yeah go ahead.

D

Yeah we did, we did. We did a very interesting project about a year ago of integrating ngs into juniper in the juniper notebooks and there you actually have a possibility to match it on. You know: juniper variable level, so you could write an ngs expression. Take the result of that expression. You know which will be a variable and consume it in python and that you know initially appeared very powerful, but we didn't do much with it.

D

It's just kind of we did it for one thing and then you know we completed that thing and didn't continue with it, because we don't use juniper nowadays on a regular basis.

C

Actually yeah, I just posted the next.

D

You know that yeah, but that was a very powerful uh thing that you know really seemed to promise a lot of power.

D

E

You know an interesting.

D

Way to combine ngs with, for example, python.

E

I've got another question um what about just saying: I'm recently, I'm working on in integrating in a initial workflow, some some degree of interactivity. So there are quite a number of tools such as universal plumber or fsa. Death, for instance, um does ngs support something on the interactive front. I mean apart.

B

From, unfortunately, sorry not yet uh it will be, I hope, somewhere with the shell itself, uh there's nothing interactive right. Now, it's a programming language, I think, actually it differentiates ngs from alternative shells.

B

I would say that in gs is programming language. First, because alternative shells I can see you know several projects and they all start with the ui. The ui is implemented in. You know, go rust whatever, that is, they start with with the interactive part and they grow the language and the language is typically uh very bash-like with pipes and I went another route, I think that's like more full-on programming language should be there.

B

So that's, I think, the main difference in approaches. Okay,.

B

Yeah, that's that's a long answer of why we don't have interactive.

B

Yet: okay, okay, okay,.

A

um Okay, so I believe we should oh yeah.

C

Can you show us an example earlier about running against the aws small thing.

B

C

Feeling how it looks like uh experiment.

B

I don't know I I I'm afraid to expose some sensitive information.

D

C

Know the names of.

B

Instances or what not so.

C

Oh just count just count the the amount of instances that you have running or something like that.

A

Yeah everything is uppered on a video, even what we do now, so there will be.

B

C

B

Let's not okay, I I can. I can show a script. I work today on uh that's at the dollar job, so uh we are comparing two data structures.

D

And almost jason.

B

Yeah so the almost jason's we made jsons out of that, we said actually said story and and then we are processing this with ngs.

B

So in our case we need to compare some bunch of stuff only by one field, so the lookup is an array we take. Each element, elements are objects, we take name of this and we only compare the names and that's how lookups are equal for.

E

B

Yeah, so it's um like in python that all these uh operators are actually a method calls, but just more straightforward, because I don't rename them equals equals. It's just. You know it's as it's the same name, the same as the operator itself, not like underscore underscore equals or whatever that would be in python same idea. um I think just better implementation question.

E

Does the support for aws, for instance, in uh in ngs map directly, um you know api to aws api to ngs? I don't know function, name et cetera or is there a bit of? I don't know a composition of sort. For instance, there are just coming to mind, for instance, that you might have in aws in deeply, for instance, a something like ac2 described instances, for instance that may refer volumes by id, and then you got a different command, which is described volumes that describes the actual volumes.

E

But of course they are not related to each other directly, and so the question my question would be: is there is a just one-to-one mapping or something like that?.

B

E

Is it no? No, it's more complicated complications.

B

So there are two things in ngs for aws um one is the bacteric parsing, which does a bit over the json. uh The second one is, uh I would say, experimental library in uh which you cannot like say, walk through the object, so dot ami, for example. It will not do that, but uh here's example of um how you can create like a minimal uh installation with an instance.

B

E

So it's semi-descriptive in a way.

B

Yeah, it's uh it's declarative, but.

B

It's running top to bottom, but you know if this: if you have this converge here, it means that if resource is already there, it will not do anything.

B

It's like, if you say, uh take uh something like puppet and just run.

E

B

E

B

Yeah, oh by the way this is. uh This is thing that you cannot do like in say in terraform. If this instance is stopped, okay, it will run it. Okay, you just don't have support for this in terraform. I think in cloud formation either.

E

Yes, you need something like ncbo or some. You know some sort of state machine.

B

Ngs yeah for that matter, yeah, because you can uh here you describe how you find this machine right. Typically, that will but tags right here is by everything right, but you can just say: tags: okay, define the tax tax is just just a hash here and you know: converse state running and you're done.

E

Okay, that makes sense.

B

uh Yeah, so the um the aw library is like kind of experimental. I used it at work, it works uh it's just. uh They made the api so inconsistent.

B

So so I could not like generate or do some generic code. It's just impossible. They are awful. uh You know the there's, almost no, the number of dimensions on which api is different per service. That's amazing! Even working with tags, you know in in ec2 you fetch the text together with the instance and with load balancers. You need to do a separate api call to gauge the text. What the is wrong with you, people yeah. It's a.

E

Sedimentary walk something.

C

E

Been written yeah, I I don't know, I see something in it at my work as well, you got many people writing many different pieces and consistencies goes to hell.

B

Yeah, so, uh on the other hand, you know I understand that the aws teams were able to move quickly forward, but the end result. I I mean aws, probably don't care that much, because you know that's not why you will probably not use the cloud. That's my that's my kid.

C

B

You know the end result is you know, it looks pretty bad.

E

Yeah and let's not talk about obstructing at a multi-cloud level, or something like that. Oh.

B

No, let's not go there.

E

I just seen actually an article when there was a system engineer talking about how this level of automation and also the api. Actually it's the the promise of obstructing complexity away is being a bit defeated because there is so much uh so many differences that you know obstructing things and yeah and even within a certain environment the the capability of this obstructing thing. You actually get a one-to-one mapping into a different language.

E

May that be terraform or something like that. But it's you need to actually know everything about what you're working with anyway yeah. So the the the decreasing complexity. It's not there. Basically, it's actually! I know. Personally, I write ansible playbooks, just that sort of mystic combination of of a data structure within imperative uh functions, et cetera, it's so complex that you're actually wondering why you're not actually writing a in.

C

E

Actual computer programming, language yeah.

C

E

Yeah, it has more time actually trying to work around the syntax than actually doing what you what you wish to do. So it's it's sort of a more than a force multiplier. Of course, I don't know how divisor I don't know what's to how to express that yeah.

B

um About that complexity, yet, therefore, doesn't hide this complexity. Unless you write your model, maybe they have modules for something, but you know, typically, the resources are of this particular server.

E

That's very brittle because you change one thing and it goes yeah.

B

So yeah about the ansible and stuff like that, so uh you you can see many domain specific languages going the same route exactly they start small and they say oh yeah. We only have this problem and they grow over time.

B

Programming, language with the configuration language, syntax and it looks pretty bad and it happened at puppet. It is happening in ansible.

B

I don't know people when.

E

I think last year in the declarative languages, these sort of minimalistic languages that room where there were people working on geeks. The new distribution base basically uses uh a lisb derived language for pretty much everything user, functional uh package management management, etc. So they were commenting about this new. You know newfangled um revolutionary development in terraform, which are loops basically and they sort of were laughing saying. Okay, we've got that in lisbon.

E

We had that, for I don't know, 40 years 50 years, and so what's the point just use a programming language for that, instead of actually you know doing some assaults from by trying to you know fit into a declarative language, something which is not declarative at all and yeah.

B

So yeah, so the original promise was, you know you have like different people on the team. Let's make it, you know, let's make some order, you will use, it will look fine and then you guys, you don't want to know which you know puppet modules. I have seen. You know how they look and you know so. It's just worse.

B

C

B

Chef, you know this cookbooks, you know it can start fine and then, after that you see oh yeah. We need out of this 20 min 5 and now.

B

E

And there is, there is a different approach from our software engineer, point of view from a software developer, point of view, a software from a system engineer and a software engineer. A software engineer, person looks a system and then says: okay, I'm going to translate what I do in a programming language using this nice different language does thing, but they usually don't know what it means to actually manage an infrastructure. So they don't. They expect everything to work, just fine and never to actually touch anything manually which in the real world doesn't exist.

E

Then you've got the system engineers which, most of the time not say every time, but most of the time are not programmers, so they don't actually know much about languages that are lost in this. This is changing these languages, which should not be programming. Languages might actually have sometimes more complicated than next language and they make a mess of it or they just you, know, copy and paste stuff that doesn't quite actually work or scale. So it's it's hello. I don't know really if we are actually solving the complexity problem of your edge.

E

Just you know making it worse in a in a way. Nowadays, perhaps there will be some revolutionary technology which will help us, but I don't know actually, if we actually should tell you know system engineer, just you need to have a developer. You know background to do system engineering nowadays, but.

B

Definitely that's the case.

E

Because I don't know it's, I think the system administrators in the old uh in the old-fashioned, you know definition of the job, especially if you are managing cloud infrastructures or something of that degree of complexity.

E

I don't know the skills that 10 or 15 years ago were there for that sort of job description nowadays and no. You need to actually have some kind, at least some kind of background in programming or in development to actually be able to to deal with the complexity.

B

Yeah also, I I had this uh funny, uh it's not even story, but it's a small thing, uh not even sometimes the system administrators, don't understand. You know the cloud so, for example, and company I said you know, I need the ec2 instance and the guy says wait opens the list of stopped instances and says yeah which size you need I'll I'll start for you uh and I'm uh I'm saying you know, that's not how you're supposed to manage the cloud so yeah yeah but yeah, it's okay! For now it's.

E

B

um Yeah yeah never mind. It was just interesting experience.

E

Yeah, but I guess watching what ngs tries to do apart from you know, trying to go beyond the paradigm of you know the the three standards, the standard inputs and the outputs on there plus you know text passing, which is, I guess, the problem nowadays, because after all, the shell is not put out for at least the shell or the paradigm you know around. The shell is not without they post structure data passing most most of the time.

E

That's why you use jquery, you use all those sort of crutches which uh you, actually you are the no pearl or python or whatever to actually go around it. But- and that's- I guess- is a way to answer to an evolutionary problem because of course,.

B

The problems have changed, you have structured data now and dealing with structured data in bosch is painful because it doesn't have it and whatever you need to do you do it with jq. Typically, and it's uh external program call and the it doesn't look convenient not anywhere near even.

E

Yeah, you have multiple layers for nothing, basically, because that's not what you need.

B

Yeah by the way, it's interesting to see, you know that bash over time got, for example, associative arrays. They were not there in the beginning. I I think maybe the areas I'm not sure where they appeared, but since we have this behavior of splitting a string by spaces into multiple positions, I would assume that the area was not there from the beginning and that's how they were doing array.

E

There is a problem, I guess as well, which is you know, related about the training of people. Recently my company started training. My colleagues on you know when you are dealing with structured data, use format, aware tools, don't use text-based data, you know test-based tools or line based tools to deal with format with data structures because yeah you can do a splitter csv with cut or some you know or oak, or something like that and.

B

E

Yeah, because you never know when it's going to fail, etcetera,.

C

But nowadays, all the time.

E

That's you know, many people don't actually know the tools because they are actually coming up pretty pretty quickly around github or places like that. Proper tools to deal with this sort of data is usually one of those you know untold or un untaught, and I say things about you know the the profession or, for instance, how well.

B

I I I do see this. I do see this issue addressed in. uh If you take alternative shells which are like pretty recent, they.

C

B

Work with structured data, it's all understood that this text parsing thing should end for the better life of everyone involved and they do structured data and some of them over pipes, even powershell power. But.

E

B

Has uh has other issues, I think that the language itself is just a bad.

E

I would say very portable, I guess in a way well.

B

It I think it runs on linux, but uh I I wonder, which parts exactly the run yeah. I think that the uh language, how the language behaves in the syntax and all that they took kind of the bad parts from shell and peril, and they stuck it into powershell for some reason, which is unclear.

B

I did see that powershell is very actually powerful and that is actually deserved uh to call it powershell and you can do like pretty much anything. But it looks just horrible.

E

That's my opinion might be true.

A

Okay, I think we shall conclude the meet. uh We should send you an email, not an email, a message through the meetup platform, with uh everything we promised um and that's it and and uh yeah I enjoyed, I usually talk, and this was an opportunity to leave them and I enjoyed.

B

Thank you, I'm I'm glad to hear that yeah. Thank you guys.

A

Thanks take care thank.

A