Hyper Cloud hyper, 11 Nov 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: hyper cache with NodeJS

Description

In this workshop, we will demonstrate Hyper Cache and demonstrate the basic API and how to compose data and cache

Github: https://github.com/rakis-me/hyper-connect-playground/tree/cache-example

Create your account on hyper today for free: https://dashboard.hyper.io

A

Imposing the data and cash service with hyper hello, my name is tom wilson and today I want to show you how you might go about using the cash service and the data service for hyper in your application.

A

One of the most powerful features of hyper is the ability to compose your services, because it's one api and it's backed with promises you can compose those services to uh you know make it just very easy in your business logic, to do multiple things.

A

So with that said, what we're going to do is start a node.js project from scratch, we're going to configure hyper connect and then we're going to create a manual kind of composition between the data service and the cache service, so that you can see how leveraging caches can make your application faster and more performant. So first thing we're going to do is sign in.

A

To github and then we're going to just create a new repo I'll just call it. You know hyper connect playground, because that's a really cool name and we'll make it public, so you can review it and then we're going to use get pod. So we have the gitpod extension installed, where you can essentially click a button in github and it will open up a vs studio, virtual machine for you in getpied, which is really cool or you could.

A

You know open your terminal and um you know pull down the clone down the repository and literally start in the same space.

A

Hopefully this won't take too long, but while that's working we can go and if you haven't already created a hyper account, you can go to dashboard.hyper.io or we could go to hyper.io and click dashboard up here and to create a free developer account all you do I'll go ahead and just sign out all you do is click the github and then, if you haven't signed in before it'll, take you through github authorization and then you authorize hyper to validate your github account and actually don't do anything.

A

But just validate that that you are a real user and then you come in here to uh the developer preview and we'll add an application, and in this application we want both data and cache. So we'll we'll create that.

A

Cool got that now, let's go over to our playground. Sometimes this happens. I can just hit refresh and it usually just pops me right into the ide there. We go okay, so we're in visual studio code and we're going to create or set up a node app. So we'll go, you aren't in net and I'll do dash y. So I don't have any comment blocks, but you can.

A

um If you like questions, you can just hit yarn and net and answer the questions as you see fit, and then we're going to add a couple of things. We're going to add yarn, add and let me just bump up the font. So you can see what I'm doing now.

A

It's kind of annoying when you're watching a video, and you can't read the text, and maybe that's just me, because I'm old but um anyway, hopefully you can see that better now and we'll do add, hyper dash, connect and we'll go ahead and add node dash fetch and I think that's it for now and then we'll do yarn, add and we'll do dash d, dot, emv, so dot env will just make it easier to manage our environment variables and we'll go ahead and create a dot emv file and we'll create a register.cjs file and we'll create a index.js file.

A

Okay- and you can do that on the editor as well, um but um touch when you're in the command line is an easy and cool command to use. um So the first thing we want to do is set up our hyper environment variable in the dot emv file, we'll type hyper, and then we'll go to our dashboard and open up our app.

A

And it's going to give us the key secret connection string. We want the connection string, so we're going to copy that and we're going to paste that in our environment, variable like that.

A

Pretty cool, that's cool, and now we need to do something that, hopefully we can make better in the future, but- and I don't want to go into a long story but like if you use the browser fetch is built in if you use dino fetch is built in if you use cloudflare workers fetch is built in there's lots of javascript server runtimes, where fetch is built in no js fetch is not built in, and node.js has been around for a while and they're constantly improving it, but it currently uh supports es modules, which is fairly new, but it has a library called node fetch that we installed that provides fetch compatibility.

A

The only problem is, is you have to import it? Hyperconnect is, is built on the latest technology and not on common js. It's built on um es modules so, and it's also built to run in multiple environments. So it's looking for a global fetch and a global request, object of fetch.

A

So the way we're going to manage that is use this register cjs file, so we're going to um in the register gscs cjs file we're going to create a function called register and in that function we're going to do a dynamic import of node fetch, so we'll do await import, node, fetch and then we're going to set the global.

A

This for fetch, equals node fetch default right and then we're going to set the global this for request.

A

Equals node fetch dot, request and re re request. Is the global kind of class or request objects, and then response is the global class for response objects so we'll we're not going to use it, but we're going to set it because that's how other environments have it response there we go and then we're just going to call register down here and the way that we're going to use this module is we're going to use it on the um on the command line.

A

So we'll run the require um same thing with our dot env module so that it can be loaded into the environment before our app runs. So we need to go to package json and add a script.

A

Scripts, let's call it dev and in this we'll say, node dash r dot, emv config dash r register cjs.

A

And then we'll call our index.js file and what this will do is it will register our.

A

Register the fetch module inside of our application so that fetch will be available globally and I'll. Show you how that works by um opening index.js.

A

I think my dog is wanting to be on video shh. You want to go outside. hmm Are you gonna? Let me finish.

A

A

All right, sorry about that, okay, so what we want to do is show that um fetch is available globally. So we'll just do console.log batch right right there, and if we do yarn dev we should get fetch is defined.

A

uh So one more point is we have to make our module. We have to change our package json to change the module or the the type of project to be module or es6 modules.

A

Now, let's see, we've got our dot. Emv we've got register, um wait, import.

A

um Okay, so I think it does get loaded. It may just not get loaded immediately. Let's, um let's try to get hyper connect to work, that'll be a good point to check so we'll say: import connect from hyper connect and then we'll um instantiate hyper by getting the node the hyper environment variable and then fg parentheses got to do that. That's kind of the the name space or we could do default and just empty parentheses just is default right.

A

So with that, let's, um let's see if we can't add some data, so we'll go away. Hyper.N hyper.data.add.

A

And we're going to.

A

Add a document so id game dash one equals type.

A

Game and name donkey right.

A

Okay and then, let's log, the result.

A

Let's see if everything works, cool everything works, so we loaded that and it registered the um fetch and everything works. Just fine sweet, okay, so we've got everything set up and just to make things a little bit easier, I'm going to create a sync function called main and then we'll just call main there. Even though you can see it supported um uh top top weight right, so you can call weight without having to wrap it in a sink.

A

But what we're going to do is we're just going to create some pure functions up here and then this will be our kind of.

A

You know call stack kind of thing, okay, so what we want to do is let's go ahead and.

A

Remove the document that we just added and then.

A

Hang on lost my place: let's remove the document that we just added. So let's do await hyper.data.remove.

A

And we'll give it an id of game, one.

A

And if we run cool so now, we should have nothing in our data store, nothing in our cache.

A

So what we want to do is every time we add a document. We want to also cache it. Okay, so and then every time we read a document, we want to read from the cache.

A

And then, when we delete a document, we want to also delete it from the cache. So this way we'll basically every time we write a document, we'll write it to our database, but we'll also write it to the cache and then when we read it, we're going to read it from the cache if it exists, if not we'll fall back and read from data and then when we delete it, we'll remove it from the cache and then remove it from the database and the cache.

A

And then this way, what we'll create is um this workflow that will leverage high read throughput on the cache instead of having to hit our database every time for a read? So that's what we're going to do, but let's first maybe get familiar with kind of the cache api.

A

Because it is fairly new and let's see.

A

So the cache api is very simple and you know it's meant to be simple: it's meant to be composable, so it'll look very similar to the data api, but it's actually going to a temporary cache. So if we wanted to cache a document, we would do.

A

Result equals await hyper.cash.ad.

A

And we'll give the document a key, so we'll say name one and then a value and we'll say id game. One type game name donkey kong cool. Now we could also give it what's called a time to let live, live.

A

One day, one hour, etc, and that could be two hours could be five hours five days, five seconds whatever, but we're going to say no ttl, which means to store it until we delete it and then we'll just do a console.log.

A

Result now, if I run that get ok, true, so able to add to the cache now to read from the cache it's as simple as calling a git command, so we'll say, document equals away hyper dot cash dot get, and we just give it our id.

A

And we should be able to say console.log doc.

A

And let's run that.

A

And let's see what did we.

A

Do game one that would be helpful cool and we got game one from our cache sweet. Okay, and if we want to remove the document we just call cache remove, but before we we do that. Let's just do query and query is a pattern matching. So if I wanted to find all of the keys that start with the text, game g-a-m-e, I can just add an asterisk and then it should return me a response or a result that contains a list of documents um that all match that pattern right.

A

So it's kind of a pattern matching query. So, let's give that a whirl cool, so it returned us all. The keys which there's only one that match that, and you can, you know say, ends with one so star dash one and it would return all the um keys that end with dash one or you could do contains.

A

So that would say, starts with game and ends with one. So that's about all you can do with caches. Mainly they're not meant to be like this aggregate query engine. It's really to store uh data to be read very quickly, so get and set, and then that's the last one. Let's do set real quick um set. We give it a key and then we give it a new value and then um that value I'll just copy this.

A

And with that value.

A

You can modify it right so.

A

So we could say, let's just enhance it published 1982 or 81 whatever. So we run that get ok true and then, if we do a get.

A

We should now have published as part of our document cool so add, get set, remove and query. That is the cache api super simple. So here we'll remove. Okay, true and.

A

Now we can get back to our challenge so, like I said what what I said before is because the apis are all promise based and they're very similar. It becomes pretty easy to compose these, and what I like to do in this screencast is compose.

A

uh Show you a way to compose these uh just using plain old promises.

A

So what we have is we need to add um and and when we add it's kind of like a cache ad, so we want to add to our data service and our cash service, and then we need to do a get with a fallback right. So with our get, we want to fall back to data, if not in cache, and then we want a delete where we remove.

A

Data in cache, if it's this and then and then remove data in.

A

Data remove the dock and data if exists, so we basically want to remove them all. um We don't want something to be in the cache when it's not in the database right because then, when we read it, it'll read from the cache like it's still in the database, but it's not okay, so we gotta handle all three of those cases um and we'll just use some pure helpers.

A

That'll help us really quick, we'll add.

A

These nice pure functions log, it's not really a pure function, but it's a shortcut so that'll just tell us what's going on and then always will always return, whatever value, no matter what's called in the value before and then constant is kind of identity. So whatever you pass in is going to pass and that will just help us with our promise management make things a little bit easier and then what we're going to do is uh work on cache ads, so cache ad.

A

We need to add to the database and if that's success, successful we want to cache it. um So we're going to create a little helper function that will take the document and then it's just going to call hyper dot data, dot, add document, and so what we'll return from hyper is if that's successful, it'll return an okay true and if it's not successful, it'll return, an okay, false so or it will throw an error.

A

So we'll do it then, and then we're going to say always return the doc right and maybe what we want to do is get the result and say: result.okay.

A

Return, the document right.

A

Or we might want to promise dot, reject.

A

So that it doesn't uh try to cache anything right, because if it's false then.

A

Then we'll just return the response, promise dot, reject response right, um but the key is is if it's successful, we want to return this document and if it's not successful, then we'll just reject it, so it won't get caught in our promise chain.

A

So that's cool that gives us our ad, and now we want to create a cache function, will take a doc and then it'll call hyper dot. Cash dot add with the doc, and we need to give it a key, so doc, dot, id and value like that.

A

um We want to give it a key, because you could have a different type of key than what your id is. So we want to always give it a explicit key and then, if it's successful or not we're going to just use, always doc right cool.

A

So now, let's create our um cache, add function and and the cache ad is going to take a document and then we're going to use promise.resolve to start our chain.

A

So if you, uh I think, it'll look a little bit better but we're starting a promise pipeline, which means we're going to take the document and we're going to pass it into the add function and whatever the result of that function is we're going to pass it into the cache function and then, whatever the result of that is we'll pass it back to the caller and in this case we'll pass the document that we added so we're going to do dot then and we'll call add, and then that then cache right.

A

So we take the document and we pass it into ad, where once it's successful, it's going to return the document and it's going to pass it to cash, and then cash is going to cache. It return the document and then return it back to the caller.

A

So now, in our main function to add game one, we just go ahead and wrap it in log, but we're going to call it cache, add right and we're just going to give it a doc. So we're going to say id game, one type game and name.

A

Donkey kong, okay and now we should be able to run that and we should get oh promise pending. So I forgot the await right now: let's do an another one: let's do game two and we'll call that super.

A

Mario brothers, okay and now, let's run that with the await.

A

Cool so we got game two so hopefully now we've got two games in our system that are in the database and they're also in the cache. So let's verify that let's comment this out and then let's just do log await.

A

Hyper dot data dot, get game one and then let's do another log away: hyper dot cash dot, get game two right and we'll switch these later. Let's just check that out cool, so game. One is in data in the database and game. Two is in the cache. So let's just flip those and see if game two is in the cache and game. One is in the database.

A

And they are cool now.

A

We we've got our ad so anytime, we want to add a document, we can just call cash add and it's going to add it and cache it at the same time pretty sweet, but now we want to do a get where we actually read the cache and then, if it's not in the cache, we'll we'll read it from data so in in order to do that, let me change this. It's driving crazy!

A

Indeed, you see two there we go um and in order to do that, what we need to do is to test it. Let's create a document in just the data and we'll say id game, three type game and name.

A

Mario paint stuff like that cool, so let's run that so now we've got game one and game two in our cache and data in game three, just in our data. So now we can have some functions to call to test this implementation. So the first thing we want to do is create some helper functions, so we want to create a function.

A

Called is cache it's cached and it's going to take an id and then it's going to call hyper dot cash dot get id and then it's going to just do a check on the result. Because if the document is there, it's going to return the entire document.

A

Otherwise it will return false it'll return, okay fault, so we'll take the result and we'll do a check and say is equal to false, and if it is, then we're going to do a promise.reject id and I'll explain why I'm doing that in a minute. Otherwise we're just going to return the the result, which would be the document right.

A

So that way, if it's cached and we just get the document, but if it's not that doesn't exist, it's going to return okay faults. So we want to reject the promise and that's going to push the id on the right side of the promised and we'll show you how that will work in a minute.

A

But let's create one more helper function, we'll call it read and what read will do is take an id and it's just going to call hyper.data.getid.

A

Which it will either return it or it'll return. Ok, false, okay! So now in our get we're going to um call this just get that's a good name for it. Let's get and it'll take an id and then what we're going to want to do is we'll start our chain with promise resolve with the id and then we're going to say: is it cached and if it is, then we're going to just return the constant we're going to return whatever value it is and then in a promise.

A

This uh own rejected method is where we'll handle our read. So if, for whatever reason it's um not there we're going to get the id and read it and then the last thing we need to do is kind of catch this, because if there's an error, it's going to go in there and it's going to read it and blow up. So what we want to do is like catch and then whatever that error is that's. Okay, we'll return the id okay.

A

So if the cache doesn't exist for whatever reason, then we'll just return the id, and that should be good.

A

So now, with our get, we should be able to have a single get that's going to return from our cache if it exists or a return from data, if it doesn't so we'll do log um await get game, one that should return from our cache log away, get game two again should return from our cache and log get game three. Now this one should fall back and return just from the data. So let's see how that works.

A

Okay, so we got game three, but we got the id. So our read didn't work, so we.

A

Oh now, this catch came in here, so I'll have to remove that and move that down here right. That's what I have to do. Okay,.

A

I think that's right.

A

Cool, so that's the get right so that that will get our data from the cache and then the next thing we need to do is delete so it'll get it from the cache or not so.

A

We can just well well, I can prove to you, okay, game three is coming from data, so let's do time, yarn, dev, okay, it's 1.1, and then let's do game one and it's 0.8. So so you can see the difference in speed. Now, don't freak out or read in the in production won't take a second to read it. um That's just because we're local and we're having to run the entire node.

A

So it's the entire life of everything starting up and starting down. So it's not the exact time, but you can see the difference in in retrieval from the cache in the database, but don't worry about the exact time. That's not! um That's! Not close. um Okay, so delete delete is pretty simple, we'll create a delete, function, it'll take an id and we'll do our um we'll actually just call a function called d cache with that id.

A

So we want to decach it and delete it at the same time. So let's take a look at what that will look like.

A

And it's going to look a little weird, but it works. That's the cache takes the id and we'll call hyper.cache.remove id. So if the id exists, that's fine, so then we'll say always id to pass it down the chain. If it doesn't exist, we want to use always id2 to pass the id down the chain, then we'll say hyper dot data dot remove and then that will remove it from the database.

A

um So in order to call that we will say log await d e l game, one.

A

Log away d e l game two and log away b: e l game three: now the first two is going to remove from the cache and the data and then the third one is going to remove just from the data because it's not in the cache. So if we run that.

A

Cool now we can verify that by doing a get.

A

Get and get okay.

A

I'll get rid of this time thing because it's annoying oops.

A

So they all are gone from the cache and the data so that that is, you know an example of basically composing data and cache, and what we hope to do in the near future is to create these plugins, where you can just say you know, I want to use this pattern and then every time for a given document type, you can just use this pattern and always cache it.

A

And yeah: that's it I'll save this.

A

um Let's do touch dot, get ignore and then echo the modules dot get.

A

A

Cool all right, let's add all these.

A

Let's just check out a branch, we'll call it a cache example.

A

And we'll commit these.

A

To the cache example, branch.

A

Like that, okay, so with that that's the code, you can kind of check out and.

A

Now, you've kind of seen how hyper data and hyper cache work together. Hopefully that was helpful thanks.