►
From YouTube: Knative demo: BigQuery processing pipeline
Description
In this video, Mete Atamel, developer advocate at Google, builds a BigQuery processing pipeline to query some public dataset about COVID-19 on a schedule, create charts out of the data and then notify users about the new charts via SendGrid with Knative Eventing on GKE.
A
And
the
second
pipeline
I'm,
going
to
show
and
I'll
go
quickly
with
this
one
is
that
this
is
a
big
query.
Processing
pipeline
I,
don't
know
about
you,
but
when
I
started
working
from
home-
and
there
were
a
lot
of
cases
like
covet
cases
in
London
and
I
was
obsessively
checking
for
the
news
every
day.
At
some
point,
I
decided,
okay
after
like
a
couple
of
weeks,
I'm
like
I'm,
not
gonna,
check
news
anymore,
because
it's
not
productive.
So
what
I
would
do?
Is
it
like
everyday
around
like
five
o'clock?
A
I
would
just
go
to
this
website,
we'll
get
some
stats
about
UK
and
my
parents
say:
oh,
they
also
live
in
Cyprus
and
I
would
check
this
that's
from
Cyprus
as
well,
but
then
once
I
check
it
I
would
agree
and
start
reading
the
news,
so
I
was
still
not
being
productive.
So
what
I
did
in
this
pipeline
is
kind
of
find
a
way
to
get
the
news
without
having
to
check
it
myself.
A
So
I
built
a
pipeline
that
would
carry
the
cogut
19
data
for
the
countries
that
I
care
about,
and
it
will
send
me
an
email
notification.
Every
day,
around
5:00
p.m.
with
the
data,
so
the
way
this
works
is
that
we
have
cloud
scheduler
that
creates
a
job
to
to
basically
call
a
service.
And
then
this
service
will
is
called
query
Runner
and
it's
a
Canadian
service.
It
will
basically
go
to
bigquery
and
Bitcoin
has
many
public
datasets,
and
one
of
them
is
now
called
Corbett,
19
dataset.
A
So
the
this
service
will
go
to
this
public
dataset
and
it
will
basically
run
a
query
and
extract
the
Corbett
cases
in
the
last
30
days
or
so
for
the
country
that
I
specified
right
in
this
case
at
UK.
It
will
get
the
data
and
then
it
will
save
it
to
a
temporary
bigquery
table.
And
then
once
this
is
saved,
coroner
will
send
a
customer
custom
cloud
event
that
will
be
received
by
a
chart
creator
and
Chuck
crater.
A
It's
a
Python
app
that
will
simply
read
this
table
and
then
use
math
Lib
to
do
a
simple
chart
of
cases
in
the
country
and
then
once
the
charges
generated,
it
will
save
it
to
charts
packet.
It's
a
storage
bucket
on
Google
Cloud,
and
then
it's
not
a
fire
service.
Will
we
listen
for
notifications
from
this
packet
and
when
and
when
the
chart
is
saved
here
you
will
get
a
notification
and
then
it
will
use
SendGrid
to
send
an
email
to
the
end
user
in
this
case
me,
okay.
A
So
this
is
what
I
set
up
I?
Guess
we
won't?
We
don't
have
to
go
into
much
detail,
but
couple
things
to
mention
is
that
I
I
used
class
scheduler
source
again
this
is
another
event
source
on
Kenya
native
GCD
project.
To
do
this
scheduling
job
setup
but
and
all
that
kind
of
stuff,
then
I
use
custom
events
to
send
a
message
from
here
to
here
and
then
check
criteria
was
used.
A
As
my
flip,
even
though
I
don't
know
Python
that
much
it
wasn't
that
difficult,
yeah
and
then
I
use
cloud
storage
source
to
get
notifications
here
and
then
send
do.
It
was
really
easy
to
use.
Actually
I
was
pleasantly
surprised,
so
I
used
some
great
to
send
an
email.
So
all
the
details
are
here
how
to
set
it
up,
but
I
just
want
to
show
you
how
it
looks
like
in
the
end.
So
when
this
works.
A
So
I
had
this
charged
bucket.
You
see
that
once
the
child
is
created,
you
can
see
like
charred
cypress
and
charging
charging
at
the
kingdom.
So
these
are
the
charts
that
I
created.
You
can
see,
it
says
cold
cases,
United
Kingdom,
it
just
gives
you
some
numbers
and
then,
if
everything
is
set
up
with
SendGrid,
you
basically
get.
A
Let
me
see.
I
show
you
one
of
my
things,
so
you
basically
get
like
an
email
like
this.
That
says
a
new
chart
from
beaker
pipeline
and
then
you
get
one
full
site
with
an
one
poetic
Kingdom,
so
I
get
1
4
p.m.
for
Cyprus
1
by
p.m.
for
yet
to
Kingdom,
and
that's
it
that's
all
my
Corbett
19
news
source
nowadays,
which
helps
me
a
lot
in
terms
of
staying
sane
and
all
that
kind
of
stuff,
yeah
yeah.
That's
what
I
want
to
share
today.
Hopefully
this
was
useful
and
yeah.