Internet Engineering Task Force 94, 1 Nov 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: IETF94-TUTORIAL-CBOR-20151101-1600

Description

CBOR session at IETF94
2015/11/01 1600

A

A

See poor lightning tutorial and cast bowmen those slides at this point here, so it somehow magically turned on you. Don't know why it did me for five minutes, but then it's about it.

A

So in this lighting tutorial, I want to cover three questions. One is what is C ball and when might you want it? Second is how does Brooke and the third one? How do you work with it? So, let's talk about Chiba and I'm, going to steal some slides from David Crockford here who was the inventor of Jason, so.

A

So 30 years ago or 40 years ago, we were all writing our COBOL programs and generating data formats completely at Hogg, then we started integrating around databases and then finally, XML came along and we had a common data format that we used for interchanging information. Excel is originally a document farmer, but it can be used for interchanging data.

A

It's just a little bit remote from what you would want to use and finally, people came came up with interchange formats that are actually based on how programming languages would want to represent data, and this is the thing that Jason finally became the most used in sense of so the JSON format. Finder changing data is the basis for a lot of work right now, both in the ITF and and generally in the application interoperation environment.

A

Now in the ietf, of course, we do our data forms this way.

A

So there is something called TV's type length, value representation and in our RFC's we write this down in box notation box notation, because we have all these boxes with. There is things in them. Okay. This may seem a little bit archaic to some people, but it has served as well so I'm not trying to belittle this. This actually works reasonably well.

A

The main problem is that every single new protocol comes up with a new TV format and you have to write all these particles and these particles have bands and they lead to security problems and so on. So maybe it's time to move on, and the obvious next step is on this slide. So this is how are your typical XML document looks like after a few iterations, and so you want to specify this because this, of course no longer can can be reasonably described.

A

So we have another thing, called the XSL XML schema language, xst and for reasons unknown to mankind. The people who invented this decided to write this down in XML. So it is as unreadable as the the XML. There is something called relax ng that helps a little bit but yeah generally application. Programmers and the to fund of this you can make it work. The Java environment helps you with that.

A

A lot there are tools out there and so on and so on, but we in the last 10 years people have been convert converging on the JavaScript object, notation JSON and the idea for jason was that it is minimal that it is text-based and in order to stop all these discussions, when you define in your data format, then there are two million people who who can do that as well, and they all have an idea how to do this and they want to do it slightly differently from you just to stop these discussions.

A

They were referred. I decided to put this as a subset of JavaScript, so he took the object notation of JavaScript and used that as the interchange form, of course, that helps with acceptance in the JavaScript community, because it's a very natural data format for them and for everybody else. It just means you don't get a turn and say how about changing this comma into a semicolon and and putting it in a space there? No, it's all defined in JavaScript.

A

So this was a pretty quick standard to establish. Now. Javascript has a certain data model and in javascript you have strings x-rays, numbers mullions, something that's called objects which in most other languages, are called dictionaries or tables arrays and the Navajo. So that's the data model and that's the underlying model of behind all JSON data items.

A

So an array in Jason might look like this. Of course you can have nested, arrays and so on, and a table are called object in return. Might look like this is familiar with Jason in this room?

A

Okay, so it's a pretty way known format by now. These things are called objects, because in JavaScript they are called objects. Also for a computer scientist, these are really maps and that's the term I going to use in the next twenty minutes. Now the nice thing about Jason is you don't need schema information for ingesting this information? Now you just point a generic puzzle at the JSON document, and it gives you this structure with arrays Maps strings and so on as a programming language object and you can start working from that right away.

A

The way and most programming languages have something like boolean types, number drive, string types, a special and Oliver you some form of maps and and some form of race.

A

When you define a JSON object, then you often will want to use what is called object from maps in Jason and structures. So you like the example here. You use the names in the map just as indication. What what the thing on the right side of the colon actually means, so you say would and then you give a number that is to be interpreted as with. So this is the typical way in which you do structured information in Jason, but you also can just use arrays and.

A

Define the meaning of a position in the array, something that I would call record usage now, one of the nice things about this is that various structure in this map approach- you can just add new keys, buy new items by defining new keys and generally Jason consuming applications just ignore what they don't understand. So this is a way to do evolution.

A

If you want to put new information into a data format that applications already understand, you just use new key names and then yeah the old applications will still work, but of course, they won't understand the new information that they didn't know work out.

A

Okay, so this is Jason and Jason is well-established works very well, but it does have its limitations, and that is particularly interesting in the constrained space where we do Internet of Things types of applications and the the biggest problem really is that Jason doesn't have any idea of primary data, of course, not because it's a text father, so you you wouldn't have an easy way to put in binary data, but also it's really a format that originally was meant for a programming language, not for data interchange.

A

So, for instance, all numbers are in decimal and it is actually surprisingly hard to convert between decimal and binary representations of numbers, not for integers numbers, that's trivial, but for floating for examples. It's it's really an art that has taken about a quarter century until fully understood it. When you implement something in Jason, then you usually have to do copying because strings use quotes around them and if there is a quarter in a string you have to do escaping and so on and well.

A

If you somehow do want to use binary, then you use something like base64 encoding and that also requires captain Jason was deliberately designed not to be extensible, so Jason is Jason.

A

If you do want to have some additional representation types beyond the seven that I showed string, number and so on. The only thing you can do is use something like the structure, format that I showed and say date, : and then put in the a font. So you cannot add new types at the representation level, you only can add limit the semantics 11.

A

Finally, at the implementation level, we know know about interoperability issues in Jason, there's a whole RFC that tells you how to use Jason in a way that minimizes probability issues and the ugly thing here is that this reduces the accessibility you have when, when using Jason.

A

So for a long time, I have used this slide here. In my talks about data formats, the XO many people actually have at some point. You find something called XE, which is their efficient exam, interchange, binary format for sending around XML data, and we just had no idea how to do this in Jason.

A

So over time, Jason is no more than 20 years old. A lot of people have defined their own binary, Jason's and one one of them is called besom and that's actually the most cited wonder if somebody thinks about binary jason it they are going to say reason just because the abbreviation is so great, not because the format is so great. The format is really not so great, but reason is not the only one of those that there are dozens of them.

A

They have, they all have different objectives, so they know they're not just trying to do a binary Jason, but they have specific ideas in mind. So, for instance, riesen was designed for specific databases, MongoDB and its main design features that you can update data at rest. So when you have a recent representation in your database, you might want to increment a number in the middle of that representation, and that, for instance, means Beeson cannot really use variable length numbers. It has to use fixed names where numbers so there's this updating case works.

A

So you see that different requirements, different areas of communication lead to different binary Jace's and, like 20 of them around now, actually interesting, the most of them actually more complex to implement. Then Jason. You would expect when you go to binary, there's it's much easier to convert between the binary, interchange format and the binary representation within the computer, but that's not the case. This performance out from complex.

A

So when we started looking at data interchange formats in the constrained space where we might have a microcontroller with something like 128 kilobytes of code, we were thinking about a way to get this simpler. Want you to build something simple and focus on small code, size and compact data size, but in this order the implementation was supposed to be simple, so you can actually implement it in a small number of codes and and my C implementation is 822 bytes of code, but kilobytes not megabytes of code on an arm.

A

So this was one of the objectives here to be useful in the IOT space.

A

So the data size was supposed to be come, but not with extensive fiddling, which would have increased the code size again, but just with a very natural mapping between the internal representation types and the interchange types.

A

So three years ago, three years ago, we actually started writing this up and came up with something which is called concise, binary object, representation. How did we get this title? Well, this is Jason, but it's not a notation, it's representation, so it's object little baby and it's not JavaScript and the two most important actually woods here. It's comic-con size. It's meant to be simple and so on and it's binary.

A

So how do you pronounce this c ball and what is the C ball? Maybe something like this? Okay. So after we defined this, we can finally fill in this field in cable.

A

So more specifically, the design codes were being able to encode all the data that we actually use in IOT and generally IETF pottekkatt make a compact implementation possible. That's the concise them being able to pass without a schema description that the same thing is with Jason and that's very different, for instance, from XE, which you usually would I want to use together with a steamer description to get compact representation.

A

This compactness thing is secondary to implementation. Compact, nestled simplicity was more important to us than getting the last bit of efficiency out of it.

A

Our main to target areas were constraint, nodes, those IOT things, light switches and high volume applications, so applications where you don't really want to throw a lot of code complexity at consuming or producing the data. This is not hdf5 or something like that. It's not not optimized for sending around terabyte size data bases, but still you might have high volume applications. We are sending around lots of, for instance, sensor measurements, and you want to do this in an efficient way.

A

We wanted to support all JSON data types with a defined conversion to and from Jason, and finally, we wanted the former to be extensible. So we are taking a different stance here from Jason, which is Liberatore not trying to be extensible, but we think protocol evolution usually requires some extensibility.

A

So pretty much two years ago we finally got in RFC out of that, and people since have actually started implementing that, and there is a website called see what at I/o that actually collects information about. What's out there in the implementation space.

A

Ok, so let's talk a little bit about how it works. Now, if I I was telling you how Jason works, I would probably use this wonderful, racetrack, diagrams, syntax, diagrams and so on. But of course SIBO is binary. So SIBO instance looks like this again. The point was to have a binary interchange performer. This by the way, is a table out of RFC 70 49. That shows this piece of data in various formats.

A

We actually now also have see seven three one in that table, which is interesting very early hour, see that get but most of the things right more more, so that what has come in between anyway. So this is how things look like now. If you look more closely, a data item has an initial bite that tells you what it is, and this initial bite is divided into a three bit field and a 5-bit field.

A

So the three red field tells you which of these eight major tribes you are having and the other five bits tell you something like length information or if it's just an integer, it might be the integer.

A

So the major types are unsigned: zero are positive and negative, integers byte strings. This is the binary data. I was talking about text, strings and text in 2015 means utf8 arrays. Yes,.

B

A

Would have been one way of doing things or we decided we would have all positive and zero numbers in the unsigned, major type and all negative integers in the other. If you know protocol buffers, they have this interesting thing called zigzag encoding, and this is essentially a way to do a zigzag encoding without having to shift around the data. So we put the least significant bit of the zigzag encoding in through the major type and and the rest is in the right of you.

A

So we have arrays, we have maps, you have something called tags I get to this in a minute and the simple types like bullion snow and also the floating point they. So this is actually certified for floating convicts. So what could we do? That worthy is five bits if those five bits are available between zero and 23, it's an immediate value so, for instance, encoding.

A

The number zero gives you a badge with a zero in the major type and a zero in the immediate value, and that's all you need for that, and they use between 24 and 27 mean that a few more bites follow. One. Two four and eight there are some reserved they use, and finally, we have a swinging mechanism where, if you don't know how many data you will have this indicates, there will be terminator at the end of the sequence of that hardest.

A

Yeah, so so this the immediate value or the one two four and eight bytes value that follow, generates an unsigned integer and for major type 0 and 1. That is the integer, either directly or in one's complement for 7.

A

This is numeration type for things like true false null or undefined, or it might actually be the actually certified for representation of a floating-point number 4 times 2 and treats the length four types length in bytes and four times, four and five, it's a count in items or I can have some apps or it might be a tag value.

A

It's come come to that in a minute, so the major type seven is doing these false true now and if they use- and we have a Ino registry, if we ever need more of them so far we have and if we are using the values for two bytes for bytes or eight bytes. Actually, the where you in there is a half a single or a double floating point and finally tags.

A

That's an interesting idea. So how do we make this format? Extensible? We defined one major type which allows you to associate a number with a value and those numbers might where these are registered values and they tell you what what did value actually is so, for instance, number zero means the value is a text string that is structure according to ISO 8601 in the RFC 359 form. 1 means this is a unix time number relative number of seconds relative to 1970 january 1st.

A

There are big numbers: we have decimal fractions, big floats and a few other things that not everybody needs, but that are useful to have predefined. If you need them and anyone can go ahead and register tag, the good numbers, this hot numbers require a little bit higher level of effort, but we have enough tags that we will be able to register a few of them.

A

So the Perl community, for instance, have immediately picked this up and done a few tags for products specific things like Perl reference, you Express to two zero, nine eight in binary. It's our V reference value, which is what people think is exactly, though I think this tag should be so. Tanks can be defined. For specific applications- and we might want to define a few more of these more general purpose tags here so, for instance, ASM on object, identifiers tank arrays, like you have in JavaScript.

A

These are examples for four things we might be defining as generic types.

A

So this means well, you probably have to eat, to digest this slide here and a little bit more time or read the equivalent cable in RFC 7 0 for 9, but 0 is represented as 0 and 1 as 123 years, 16 17 hex in the 24 as 18 18 hex, because whether 70 24 is the first one that doesn't fit into the immediate value in the first byte. So we use a one byte extension and write be 18 in there and 100 is 1864 in a thousand s 19 or 3/8, and the million is this.

A

It's pretty straightforward and it's very cool decode. Now at some point we are leaving the 64 bit space and then we are going to big numbers which actually contain a check. C 2 means C is 6 6 x 2. So it's a tag with a value 2, and this is the ID string that represents that. Are you.

A

These are negative numbers floating points, infinity, nan, minus infinity. These are all defined in our ability, 75, for so it's easy to transport them in in C, ball, there's no way to write them in JSON, false true, and now our with these simple types, and then we have hex strings text, strings and arrays and maps all the interesting thing about maps is you can use any data type as a map peanut not like in JSON, where your own abstracts.

A

So this is how it works again. I could talk hours about that, but I think it should be obvious. There really is not more to it, so you could implement it from those three flat. So how do you work with it? First of all, if you just want to play around with it, there is a seaboard playground called seaboard and E, and this essentially is two fields where you can either put in something that looks like Jason are put in bites and press these conversion arrows and it will convert between one or the other representation.

A

So this is for playing around now. The thing here looks like Jason, but it's not limited to Jason. We have to find a diagnostic notation for seagull, so we have a common way of target to talk about CEO of values in text form, so we can represent binary strings and tags and so on in this diagnostic notation for what you find in section 6 of the RFC.

A

Okay, if you want to do more than playing around with, you probably want an implementation. There are couple more than 25 implementations right now or a couple languages like JavaScript PHP, cool rust, rust community has been particularly emphatic and picking up to see what I have no idea. Why Lua Python movie Erlang and Alexia Haskell c-sharp, Java, C, C++ and D any of your favorite programming language. That's missing!.

A

Ok, are we well, then Java? So if you, if you need it on the Java Virtual Machine, you use the Java implementation and just link it to whatever you need.

A

It's just a one call can't quite see this, but yeah go go to that list. Yeah somebody hasn't released.

A

It's probably going to be one line in this first thing: okay, so this is, if you are using it like JSON, that's really all you need to know because it will just work now when we write text based specifications, we have something called a B and F that allows us to write in a slightly more formal way. What pieces of text I actually meant here and actually, if you look into the history of the RFC's, the first RFC that uses a form of BNF is our c 14 from 1969.

A

I think so that has a very long history if we always have described text based protocols using something like Vienna, so we thought is there maybe something like like a PMF for jason, like data Mart's and of course there is a ton of proposals how to do scheme our languages franchise.

A

You could use one of those email languages and it would also probably do the same thing, but we try to limit this to essentially the function of a B and F and we actually copied the syntax of a B and F that's relatively similar, but the theory behind it is really the theory behind relax ng. So if you know what what relax angeas, but you will feel at home very quickly, yes, another proposal called JSON content words where we stole the rest off of the great ideas.

A

Very little of this is innovation, but I think we have put it together quite nicely, and we now have something called a seaboard data definition, language that has been stable for a year or so we are still adding a few things to it. Where we learn from from the actual specification work going on in city ideas that we want to make a few additions.

A

There also is a tool available.

A

This tool is user friendly, but only to the users that it likes so well at sometimes is a little bit verbose in its around messages. But the main use of this tool is either to generate an example instance. So you have a CTD a specification. It's just going to generate a random instance that is conforming to that. That can be very enlightening.

A

There's something called a B and F gen. If you are writing a B and F, that does the same thing. All you have a way to check an instance against the definition and that's what we use for writing specifications. We want to know whether the examples we have in the document are conforming.

A

So this is an example how a large part of RFC 7071 would have looked inside here. That's actually a JSON spec, but because jason is a subset of seaboard. Of course, CD DHN can describe JC document. So a replication object is a map with two keys application and reference where application is a text. Value and reputa is an array of zero or more refuge on the ref. Neutron is a map that has a rattle. It resonates an assertion that detects a writer and it'll take a rating which well okay.

A

We took a little bit, but you here is a float16. Assuming that you don't need a lot of accuracy on that rating, and then it has a number of optional components: confidence, normal writing, synthesized integer, generator design set inspired expires, is an unsigned integer and then it has some extensibility. So you can go ahead and you find additional entries for that matter, so that previous I don't know five pages out of that RFC.

A

So that is the retroactive. We have looked at 7071 as an example of a document that has a not completely trivial jason representation and it takes days to read the description and it takes five minutes to understand this stuff and for for new development.

A

We have an example here, which is grass grass is the generic autonomic signaling protocol used in autonomic management of systems and the grass people for once have decided not to invent another tier V. For with this, that would, of course, be the natural thing. Do you find another tier V, but they have decided to just use C bar and so for instance, the grass message is an array with three elements: a message type, a session ID and zero or more options. So actually it's not three elements.

A

It's two to any number of elements and the message tribe is 123. The session ideas. Very you up to 16 million and options are defined like this. So waiting time option is one of the options that is actually in array containing the Oh waiting value and the waiting time, which is a number in milliseconds. So it's pretty straightforward. It looks a lot like a B and F, but it's the square brackets. Of course, don't need optional here, but they mean array, as in Jason.

A

Okay last slide: where do you go from here? Well, first of all you go to our c7 t-49.

A

Then maybe you go to see what dot io you might want to subscribe to the Siebel mailing list, which is relatively quiet, but maybe a good way to find people to talk to the CDD a locum. It is not an RFC yet because we still wanted or to learn from specification efforts, but we expect to move it forward to to a working group and then to stannous fact of human next year.

A

And finally, the edge' team has asked me for all of you to go to this place here and tell us how good this, like in tutorial, ask questions.

A