Rust Programming Language RustFest Barcelona 2019, 28 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: RustFest Barcelona - Neville Dipale: Powering Crowd-sourcing of Public Transit Delays with Rust

Description

“Your train is arriving in between 20 to 90 minutes”. This is a reality for about 3 million train commuters in South Africa, who not only battle to commute daily, but are faced with a lack of accurate information and high uncertainty about their trains.

This talk explores how we progressively use Rust to power crowd-sourcing of train delays, and are helping reduce the uncertainty of arrival times in our ailing train (and bus) services; all with little computing resources.

We focus on how the Rust language and ecosystem helps us to create tools and services that run at scale and low cost.

https://barcelona.rustfest.eu/sessions/public-transit-delays

A

Neville: good morning,, everyone.

B

This talk is going to be about powering crowdsourcing of public transit delays using rust.

B

Morning, everyone.

B

My name is neville dipale., I'm from south africa. I'm on twitter.. I work as a data engineer,, mostly as a freelancer.

B

I started as a hobby about ten years ago. Now, I run a public transit info service,. I started.

B

Using rust about a year ago and recently became an apache arrow committer, contributing to.

B

The rust implementation. today I am going to be talking about trains in our region and challenges with them. So.

B

I will give an overview of how they work,. I will talk about the crowdsourcing solution,.

B

How we use rust and what the benefits of having used rust, are.

B

So, as a background to the train service,, we have two train services in our region.

B

Out of interest,, how many people have been in south africa, by a raise of hands?.

B

Cool, thanks., so we've got two trains and we've got the fast train and the metro train.

B

The fast train was actually built for the 2010 soccer world cup..

B

We hosted it, and won it..

B

The metro train has been around since the 80s, so it has very aged infrastructure,. They.

B

Haven't replaced some of the trains in quite a long time.

B

The metro train is very cheap.. You can get from one side of the province to the other side of the province for about.

B

2 to €3 and then most trips would be 1 to €2.

B

The fast train is a bit more expensive, so it goes from the airport into the city and.

B

You pay about €10 a time for that train, so the metro train is used mostly by poorer.

B

Communities and students, though, rely on the service.

B

Let's guess which train is which here. we've got this one., which would it be? fast train.? Then you've got this one, being the metro train..

B

So we've had this problem for a very long time: where,, because there aren't enough trains,, they get overcrowded and people get desperate.

B

To get to work and then they hang on to the trains..

B

They are changing that now with new metro trains, but we've had problems over the years.

B

With procurement, government, issues,, etc. we were supposed to have these new trains about four years, ago.

B

I think now, there's only three running and they've promised that we will have all of.

B

The trains being replaced by these ones in the next ten years, so the struggle is still.

B

Going to continue.

B

with metro train delays, they've got severe infrastructure issues, as I've mentioned.

B

It's estimated that 40% of rolling stock, ie, the train coaches,, are out of service,.

B

Are not available,, and this affects peak time services quite significantly.

B

It's only during the day when there's fewer people taking trains, fewer services,, where.

B

They get to catch up and then the same problem then begins again in the evening.

B

Trains can get stuck in the middle of the tracks, cable, theft,, signalling, problems,.

B

Etc. it's not uncommon to be on a train, stop in the middle of nowhere for 15, 20 minutes,.

B

And then the train continue again.. I did an interesting analysis, not here though,, for something else where we were looking.

B

At the average train, travel times, factoring into account the delays, and one of the trains.

B

Took about, I think it travelled at 30km/h effectively and was travelling about 100 kilometres.

B

So you can imagine how long that train trip was.

B

Delays of over 60 minutes in most cases.

B

Part of my pitch,, it was dramatic,. I said, "your train is arriving in 20 to 90 minutes",.

B

Because that's the kind of uncertainty.

B

there are communication issues also from the operator.

B

None of it is digital,, all of it is analogue..

B

We don't have an api that provides information as such and a lot of the times, even for the.

B

Train to say, move themselves,. They rely on phones.

B

The train conductor will stop the train and say:. Can I move on?.

B

Then they have to wait, 10,, 15 minutes or so..

B

In the past 15 years we've had a number of collisions of trains.

B

One big one, 20 people died or so,, so it has been very bad over the years.

B

Here is a graphic that just shows some of the data that we've collected.

B

I've split it into delays,, so a histogram of delays where the first two packets are.

B

Trains that are early or on time and in the ten-minute category, 53% of trains are delayed.

B

By up to ten minutes,, so this is great but also bear in mind that we don't crowdsource.

B

The entire network,, so it's only a snapshot of some of the trains.

B

Then you will see that the delays sort of go down.

B

You still have quite a number of trains that are very late in most cases and each time,.

B

This is normally either in the morning, somebody has to get to work or a meeting at 8.00 am,.

B

They end up getting there at 9.00 and, as a result, a lot of people have lost their jobs.

B

Because you would come late every day in the week and your employer, just says: well, sorry.

B

There are people who wake up at 3.00 am just so that they can get to work at 8.00 am in.

B

The morning which is quite terrible, and then the same again happens at night,, where people.

B

Would leave work at 5.00 pm, if you work 9.00 to 5.00, and then only get home at 10.00 pm.

B

So our solution has been to provide metro commuters with information on train delays.

B

We use twitter and push notifications on an android app,, so we've got a android app that.

B

People use to process this information., we crowdsource delays either by sharing a train trip or spotting trains. So if you would.

B

Have noticed on the first picture,, there's normally 4-digit numbers on each train and.

B

We use those, map them to the schedule, and people provide us with the train number and.

B

We figure out which it is. we calculate delays based on the actual versus scheduled.

B

We use a linear model: if the train was supposed to be here, five minutes ago, and the average.

B

Speed has been, let's say, 20% less than what it should be,. We just do a simple linear progression.

B

Of what the delays would be., then we generate estimated arrivals at stations and then notify commuters about delays.

B

This is what we use rust, for.

B

okay,. So in terms of submitting train delays,, the architecture is quite complex..

B

There's a lot of services that are running around because I have been doing this for.

B

About seven years, now. some of the stuff is in rust. a lot of it is not in rust, but for the talk, I'm just going to talk about the few components,.

B

Specifically, three components that are built in rust.: we've got a location sample service that checks for sample validity and accuracy. So imagine.

B

When you are crowdsourcing, one of the problems that you have is that some information might.

B

Be accurate, some might not be accurate., there's nothing stopping a person from saying:, I'm on train 1235 instead of 1234 and then.

B

They give you incorrect information., one of the things I saw in the early days of deploying this was a train that was moving.

B

In reverse- and it took me a while to figure out that somebody is actually on the other.

B

Train but they are submitting train locations for this one,, so it looks like the train just.

B

Keeps going backwards., then this location sample deals with that..

B

We've got a service that saves samples in the database.

B

I will explain more in detail about that.. Then you've got a vehicle stream service.. If you were in rust, lab in italy. I talked specifically about the vehicle stream service.

B

It calculates delays of trains.

B

The user sent delays was samples and then other users are notified of delays.

B

I will talk about this in detail.. Here is an example of a process.

B

So train a is expected at station 3 at 8.00 am.

B

There are no samples yet so we are just relying on schedules.

B

the users. Different users are waiting for the train at stations, 4, 5,, 6 and 7, and those.

B

Are the estimated arrivals at those stations.

B

once we receive the first sample,, let's say at 8.12 and that's between station 3 and 4,.

B

Already for station 4 there's a delay, so we will just project those and then I've marked.

B

Them in red to show the users that you are notifying..

B

One of the problems is that we don't want to spam users and notify them every other.

B

Minute to say, your train is arriving in four minutes,. Now it's three minutes, etc,, so we.

B

Project those and then at the next sample that you get. We only then calculate the difference.

B

In delays., if the delay is still more or less the same, we use a five-minute range,. We don't notify.

B

The user again., so when you get a sample at 8.16,, the station for the train was already passed,. The next.

B

One is at 8.25, it's within a four-minute difference,, so we are not going on notify.

B

The user. we are only going to notify the user, whose train is now arriving at 8.33.

B

In terms of the rust services themselves,, the location sample is a grpc server that.

B

Uses tower grpc,, it receives vehicle samples and processes, them.

B

It matches samples to trains, and so we take the four-digit train. Number, match it to the.

B

Train and generate a last known location., it uses actix to monitor state of users and trains.

B

I will talk more about it. there,. I've just shown a screenshot.. I know actor is more popular than actix.

B

We have a trip status struct that tracks the last known.

B

Details of each trip. going back to the previous slide there, you see that we've got user trips and trips.

B

A user trip just matches the user to say the user is on this train so when they submit.

B

Another train sample,: we automatically reject it because you can't be on two trains at the.

B

Same time., the trip just checks the overall status of a trip.. So in the trip status you will see, we've got the trip id, route id,, the last distance and.

B

Then location., the last distance is very important because we calculate the distance based on the geometry.

B

Of the trip, so if somebody submits a sample and says that the train is close to station.

B

3,, let's say it's 2 kilometres away from station 3, and another user submits a sample. Five.

B

Minutes later, saying: actually,, it's close to station 2. We are going to reject the.

B

Second sample because the train can't go back in reverse, so that's how we work around that.

B

Issue. then also keep track of congestion and occupancy.. We are hoping that, with a lot of users, we will be able to reliably tell people that.

B

Your train is actually full,, it's better, maybe to wait for the next one, if it's coming.

B

Okay,, so we've got an actix actor that receives a sample and handles it.

B

The benefit is that I don't have to worry about multi-threading, because actix has a.

B

Mailbox so as long as I have a big enough mailbox to handle it, one benefit of rust.

B

Is that it's very fast, so I never have a big backlog at any point in time.

B

The sample produces a location or an error if, for example, the location is invalid.

B

So after the service saves the valid sample onto the database before it returns it to.

B

The client, this sample then gets picked up by the next service that I'm going to talk.

B

About., if you have worked with geospatial data,, it gives vehicle co-ordinates and then some information.

B

About the vehicle., so the next service that I then used is the vehicle streams. So this reads: data from a.

B

Database periodically every 7 to 10 seconds and it projects the delays per route. So this.

B

Is for estimate and live sorry, scheduled services and live services.

B

It calculates the estimates and keeps churning them out throughout the day.

B

It starts running from 2.00 am when the first train moves up until 11.00 pm when the last.

B

Bus is moving,, so it's both for trains and buses.

B

It sends all the results to a service,, it uses nats.io for that,, and then it uses a.

B

Tokio interval and runs it every 7 seconds to generate my sample.

B

How does rust help with vehicle streams?? I first wrote the service four years, ago, initially in javascript, where a small crash.

B

Would collapse the whole thing.? I moved it to kotlin in the jvm, hoping that it will be better,, but then I had to keep.

B

Throwing more ram at it.

B

You will remember, I talked at rust, lab about going from 20 gigs of ram to less than a gig.

B

Of ram at peak time, so it took a few months.

B

This was about 14, 15 months ago, and I'm seeing very impressive performance.

B

I previously talked about how the service just runs for months on end.

B

I never need to restart anything..

B

I use a third of the cpu usage..

B

I talked before about going down in ram, and the main benefit is that I was able to replace.

B

An in-memory cache with a simple hashmap..

B

I will talk more about this and then the service is easier to maintain and expand.

B

Then there's a notification service because I mentioned that users have an android app.

B

So when one user is submitting samples,, other users are relying on that sample.

B

It's a firebase cloud messaging service.. I think it was abandoned, but somebody picked it up and started using it, which is one of.

B

The nice things about open source in general., it notifies users about a range of things.

B

There's effectively a generic cloud messaging app where I can just keep on extending it.

B

By different types,, so it's not really train-delay specific.

B

We personalise information for individual time delays so if you've got five people waiting.

B

For the train at different stations- and you notify one user now,, but the delay doesn't.

B

Affect the other person, when it does affect the other person,, the second user, we only.

B

Notify them., so we keep the state of each individual user and one of the things we try to avoid is spamming.

B

Users, by sending them hundreds of notifications per second, because immediately someone will.

B

Uninstall your app and you will never see them again.

B

The service also uses actix, but instead of using grpc, we use nats.io, so within the whole.

B

Architecture there are messages being passed around quite a lot.

B

I used to use kafka. subscriptions are edited in realtime.

B

A user says I only take the 7.00 am train,, or I only take trains from 7.00 am to 8.00.

B

Am, between station 7 and station 24.

B

So once the train passes station 24, we don't notify about anything because it's no longer.

B

Relevant for you.- this is one of the nice things, because the fast train service that I talked about, they.

B

Will send you a notification about a bus in some far-away place, not running an it's just.

B

Irritating and a poor user experience.

B

all transit data state is loaded at start-up, when the application starts, and I keep it.

B

In memory so specific data like what the train numbers are,, what the stop locations are,.

B

Etc,, as well as the trip data.

B

the benefit here, is that we end up storing everything in memory, because the application.

B

Effectively,, as long as the application doesn't crash,, which is one of the benefits of rust,.

B

It will keep running indefinitely., so if you use actor systems quite a lot, you will know that some actor systems support.

B

Snapshotting, where periodically you'll save the user state onto a database. So if the application.

B

Crashes, you can recover that state.

B

here,, I don't use that.. I just load everything in memory and keep it running.

B

So why rust then?.

B

The main benefits are that I'm really getting stability out of this because being able to.

B

Write a programme where I can prove to myself or to a compiler that it won't crash, I handle.

B

Every state that will cause a panic, helps me to create a cheap in-memory store using.

B

A hashmap for example,, instead of using bootstrap or something else.

B

I don't really use rust for something very novel per se,, but it's really just to give.

B

Me, the performance.

B

actix has been good to avoid concurrency issues.

B

I rarely have problems with borrow check or lifetimes or anything, because everything is.

B

Simple., the cost saving is the important element for me, because this is a hobby and maintaining.

B

This means, if I was doing it as a profession or whatever I would need to wake up at 2.00.

B

Am in the morning to notify users of cancellations of trips and all of those things, and then.

B

Stay up all day until late at night., because it's a hobby,, it's very costly.

B

every hour that I spend on this is a hour that I could be working as a freelancer and.

B

Putting food on the table,, etc, and also the other thing that I save on a lot is server.

B

Cost. initially, about two years ago, we first started using rust,. I had to have a 64 gig server.

B

Because the kotlin service would use about 20, gigs of ram and all the other services.

B

Would cumulatively add up so I used to normally run at around 40 gigs of ram usage, just running.

B

This aspect of this public transit stuff and now the services that I just talked about.

B

In rust,, including others that I haven't mentioned in this talk,, I can run all of them in a 1.

B

Gigabyte container with 2 cpu cores, so that's significant for me.

B

The other thing is scalability in terms of performance..

B

I use two cpu cores, but I use very little.

B

If we were to expand, maybe even 100-fold, I would still be able to use them within the.

B

Same resources. okay,, that's it from me..

B

Are there any questions?.

B

Hello, thanks for that interesting talk.

C

I already saw you at rust: lab., that's amazing. about the scale of the thing,, so how many users does this actually serve concurrently.

C

And what's the frequency of notifications and messages to users, just to get an idea?.

C

Neville: in terms of the number of users, we get about 100 users a day submitting samples.

B

There's a lot more who receive samples, but it's not very frequent and it's only really.

B

Busy in the mornings,, when a lot of people are taking the trains and then again in the.

B

Evening. throughout the day, you will see one sample once in a while, but it's not a lot.

B

Then, in terms of notifications that are sent, there's a hard rule that says we won't send.

B

A notification to someone for the same train within five minutes,. But we send about the.

B

Last time I checked, it was about 400, 500 notifications per hour, at peak times.

B

But it's still quite a young service.

D

Any other questions?.

D

Again,, a very interesting talk.

E

You mentioned you run it from about 3.00 am until late in the evening.

E

Does that mean you can shut down your service and,? If you didn't have that option, would.

E

You have to change anything about your infrastructure?

E

neville:, so in terms of running it,, the running aspect of it at one point is standing on a.

B

Chair,, looking at cancellations, more than delays.

B

The whole delay infrastructure. It's the cancellations where there's a whatsapp group.

B

From the train operator, where they say:, this train has been cancelled.

B

So those ones we can't automate as yet., we have to capture a cancellation and send that out.

B

Everything else is just automated, so at night when it's based on schedule,. So if there's.

B

No scheduled trains that are running the service will just keep calling every 7 seconds, but.

B

Do nothing. once activity picks up,? It starts running.

F

Hi. great talk by the way. I was just wondering,: does the app send regular updates, or does the user have to actually.

F

Send the update themselves? press a button or whatever? neville:, my apologies,. I actually didn't talk about how users submit.

B

There are two ways.

B

First is ... depending on where they are when a train passes by you can spot the train by.

B

Saying I just saw a train passing by and we use that one sample to calculate.

B

But then, once you get on the train,, if you've used waze, it's something similar, where once.

B

You get on the train, you can say I'm on this train and then it will take a sample. I think.

B

Between every 45 and 60 seconds., that's how we get the sample., then, once you get on the train.

B

F

A very nice talk.

A

So you mentioned, you store the state as a hashmap in memory.

A

Have you ever had the issue that the state got? You were not able to forget?.

A

So something you use is etl, so you can forget an entry automatically;. Do you need to handle.

A

That in your application, or is that not an issue?.

B

Neville: so I used both.

B

there's a crate called etl index,. I think,. I cannot remember.

B

I used that for state where I want to keep stuff .... I restart the application once a.

B

Month,, I will just load that information.

B

I do do incremental updates where there are changes that happen and in terms of the application.

B

Getting big, right now with about 7,000, services, or 7,000 different routes,, yes, and 7,000.

B

Vehicles,, I use less than a gig,, so in the vehicle stream service I use about 500 megs.

B

And then for static information I use about 400 megs,. So if I were to increase the service.

B

Tenfold, still say 10 gigs of ram, so that's not a concern.

G

Do you have trainspotters in south africa and could you maybe change your app a little.

G

Bit so that it gamified it for trainspotters?.

B

Neville: well, there's a lot of stuff. I didn't talk about and I see there's a lot of questions.

B

About,, so one of the things you do on the app is we keep track on. We've created a.

B

Points system for trainspotters, where, based on how much the trains, how much you travel.

B

The first person will get more points.

B

at the end of the month,, because people are often students, etc,. We buy them data in small.

B

Prices that they can afford, and then the other one is we keep track of how many kilometres.

B

In total, you've travelled and we also incentivise those people, so we do gamify, because otherwise.

B

People don't see the benefit of spending their data, helping the service,, even though it helps.

A

More question?.

A

I don't see any.

D

Do you know if the train drivers use the app as well, or do the company use it?.

D

Neville: some background,, one of the problems we have in our country, is that the government.

B

Outsources a lot of its sources.

B

The problem with that is that,, even when you are coming with good intentions, trying to.

B

Help out,, they can't just take your services,. They have to go through permanent services.

B

And things like that., we have put in a proposal a number of times where we said instead of putting gpss on trains.

B

Because the trains are old anyway and might get vandalised,, let's create an app for train.

B

Drivers, because all train drivers have smartphones and then those drivers, when they get on the.

B

Train, they can say:. I'm logging in on this train.

B

One of the problems, though, that we have,, because we've tried that with buses also,, is that.

B

Bus drivers are very unionised,, so there's a lot of organised data and there's always.

B

Issues about whether this information will be used to the prejudice of drivers.

B

Sometimes the drivers aren't very honest in the sense that when a bus is supposed to leave.

B

At 9.00 am or it leaves, and every 15 minutes,. Sometimes the drivers will intentionally not.

B

Leave at 9.00 and leave in a bunch together.

B

there's a concept called bunching.. There are quite a few issues that work against us trying to implement that, but we are trying.

B

Because if you do that, then you don't need the crowdsourcing.

D

Anyone else? okay, thanks, neville, for your talk.