►
From YouTube: Making Complex R Forecast App... Natalia Costa Araujo, Pedro Szloma Herr Zaterka & Matheus Sesso Gay
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Making Complex R Forecast Applications Into Production Using Argo Workflow - Natalia Costa Araujo, Pedro Szloma Herr Zaterka & Matheus Sesso Gay, 4intelligence
R has tools to help multidisciplinary teams succeed. Usually, the processes developed in R are created for problems restricted to academics researchers. As more companies search for data and scientific methods to guide business decisions, creating a scalable R environment will be a critical step towards success.
A
Hi
everyone,
so
today
we're
going
to
be
talking
about
how
here,
at
foreign
intelligence,
we
are
making
complex
art,
forecast
applications
into
production
using
our
go
workflow.
So
just
to
give
a
little
bit
of
a
background.
We
started
as
an
economics,
consultancy
company
that
focus
mainly
on
time
series
data
and
we
are
now-
and
we
have
already
a
first
version,
developing
what
we
call
forecast
as
a
service.
A
The
idea
is
that
people
with
little
or
no
background
on
statistics
or
economics
can
run
analysis
very
sophisticated
running
from
traditional
statistical
algorithms
up
to
automl
functions,
and
they
can
do
that
with
just
a
few
clicks,
and
we
also
focus
on
having
very
good
front-end
applications
where
the
person
can
easily
access
all
of
the
results
and
have
a
good
understanding
of.
What's
going
on
in
there
as
we
started
to
shift
from
consultancy
focus
to
a
technology
company,
we
had
a
lot
of
challenges.
A
The
first
one
was
that
we
were
used
to
using
r
in
all
of
our
developments.
So
if
you
look
at
the
background
of
our
scientists
data
scientists,
they
are
mostly
from
statistics,
economics,
engineers
and
use
our
on
a
daily
basis,
so
we
needed
to
bring
archer
production.
We
needed
to
use
that
team
know-how
that
we
had,
and
we
also
wanted
to
do
this
while
optimizing
our
cost
in
the
same
line.
A
We
wanted
to
have
simpler
and
faster
deploys,
because
in
the
beginning,
it
would
take
hours,
maybe
on
a
friday
night
friday
evening,
and
we
knew
that
it
was
something
that
we
needed
to
do.
But
one
thing
that
we're
not
willing
to
let
go
is
the
reliability
of
our
results.
We
really
focus
on
having
accurate
forecasts
and
while
bringing
interpretability
to
of
the
results
to
our
clients,
but
how
did
we
get
there?
So
our
data
scientists
working
on
different
projects
would
see
the
need
of
a
different
approach
or
a
different
methodology.
A
A
We
started
doing
super
repos
where
we
put
everything
in
there
and
we
would
have
maybe
version
one
two
three,
but
there
was
no
actual
version
in
control
at
the
point
that
we
started
doing
that
thing
doing
that
things
got
a
lot
better
for
us.
So
another
big
issue
that
we
faced
was
that
we
were
running
the
codes
locally
and
with
all
of
the
time
series
that
we
had.
Maybe
we
were
running
a
thousand
of
them
with
very
exhaustive
cross-validation.
A
It
would
take
a
lot
of
time
and
once
you
find
a
bug,
you
have
to
start
all
over
and
we
saw
the
need
to
go
to
the
cloud.
It
was
something
that
we
couldn't
postpone
anymore
once
we
lift
and
shifted
our
coach,
the
cloud
there
were
a
lot
of
gains
because
we
noticed
that
the
productivity
of
our
data
scientists
grew
a
lot.
We
also
were
a
lot
more
sure
about
the
reliability
of
the
results.
We
knew
that
that
it
went
through
our
entire
pipeline
from
pre-processing
feature
selection,
modeling
and
post
modeling.
A
All
of
the
analysis
that
we
have,
we
were
sure
that
they
were
being
done
properly
and
we
did
all
of
that
using
r.
If
you
look
at
the
way
that
we
used
to
do
things,
we
used
art
for
api,
orchestrating,
outputting
and
communicating
with
google
cloud.
Even
our
front
end
was
developed
in
r.
We
also
had
a
monolithic
structures.
If
I
was
running,
maybe
97,
it's
a
very
classic
case
for
us.
We
were
running
97
time
series
and
I
sent
the
request
via
api.
A
So
as
the
lines
started
to
grow
and
the
jobs
would
take
a
lot
of
time
to
run,
we
started
looking
for
new
solutions.
As
our
focus
started.
Shifting
scalability
became
a
bottleneck,
so
we
needed
to
be
able
to
scale
what
we're
doing
so
we
could
grow
and
what
we
wanted
to
do.
We
wanted
our
code
to
be
generic
and
accurate,
so
we
want
to
run
as
different
time
series
as
possible.
A
A
At
this
point,
we
decided
to
look
at
the
pros
and
cons
of
r
of
keeping
using
r.
The
advantages
were
first
that
we
had
a
legacy.
All
of
our
codes
were
written
in
our.
We
were
very
comfortable
doing
that
the
team
know
how
even
from
developers
to
users,
they
all
knew
how
to
use
r
very
fluently,
and
we
also
have
the
fact
that
r
is
largely
used
in
academics.
A
We
focus
on
having
new
implementations
and
most
of
the
statistical
new
implementations
are
available
in
r
very
fast,
so
we
could
have
new
approaches,
new
methodologies
within
clicks
and
the
drawbacks
are
that
there's
very
limited
native
support
to
cloud
environments,
and
once
we
started
needing
help
from
other
developers
outside
of
this
academic
world,
they
were
not
familiarized
with
r
they
used
more
python
or
other
languages.
So
we
had
to
make
sure
that
we
were
on
the
same
page.
We
got
to
the
point
where
we
map
our
solution
needs
we
needed
scalability.
A
So
we
need
to
be
able
to
run
lots
of
time
series
at
the
same
time
and
also
long
ones
and
within
be
ready
within
a
few
minutes.
We
need
a
resilience.
We
need
to
have
some
retry
strategy.
We
need
a
decoupling,
dockerization
cost
efficiency,
monitoring
and
microservice
friendly.
So
next
we're
going
to
talk
a
little
bit
about
the
solutions
that
we
showed
in
the
previous
slide,
and
we
start
with
the
decoder
step
once
a
job
is
sent
via
api.
It
goes
to
the
decoder
which
divides
the
jobs
into
different
vms.
A
A
So
what
we
did
at
this
point
is
that
we
fit
our
sessions
in
small
images.
We
start
we
took
that
big
images
and
we
divided
it,
so
we
could
be
faster
and
lighter
it's
good.
To
mention
that
we
got
a
lot
of
help
from
the
our
minimum
community.
They
helped
us
build
a
specific
version
that
we
needed,
and
once
we
started
working
with
the
images
we
faced.
A
A
very
non
challenge
for
our
users
is
that
we
had
package
dependencies
that
conflicted,
so
maybe
I
needed
a
specific
version
of
a
package
for
one
package
and
for
another.
I
would
need
a
different
one
and
with
those
conflicts
we
realized
that
maybe
it
was
best
for
us
to
build
our
own
image
and
with
these
images
we
decided
to
use
as
little
external
packages
as
possible
to
avoid
conflict.
A
B
B
So
we
wrapped
all
the
our
pipeline
in
a
in
a
python
api
and
we
plugged
it
after
the
decoder.
What
it
did
for
us
is
that
it
greatly
increased
our
performance,
the
all
the
jobs
and
the
pipeline
was
running
much
faster,
but
we
had
some
issues
with
cost
control.
We
had
some
problems
with
cost
control,
so
we
decided
to
change
the
focus
and
look
for
another
solution.
So
that's
where
we
landed
on
kubernetes,
which
is
a
much
more
modern
approach
to
this
kind
of
pipeline,
and
for
that
we
decided
to
use
the
argo
workflow.
B
What
argo
did
for
us
is
that
it
enabled
all
the
micro
servers
like
structure
with
a
scalable
and
more
cost
controlled
architecture
all
running
in
an
isolated
ecosystem,
so
we
can
run
python
processes
and
r
processes.
All
that
within
argo
and
argo
will
deal
with
all
the
communication
with
the
google
cloud
platform.
B
Regarding
the
costs,
argo
allows
us
to
customize
our
clusters
and
our
pods
our
nodes
to
know
how
much
we
will
spend
in
certain
situations
in
we
can
predefine
all
memory
usages
and
how
much
machines
we're
running
regarding
the
resilience
part.
What
what
it
allowed
us
to
do
is
to
implement
retry
and
failure,
strategies
and
disaster
recover
tools
as
well,
and
also
the
application
became
stateless
and
much
more
fault,
tolerant,
which
all
of
that
combined
formed
a
much
more
bulletproof
architecture.
B
Regarding
the
tracking,
I
think
it
was
one
of
the
our
biggest
improvements
was
that
now
we
have
completely
the
couple
log
files.
So
when
an
error
occurs,
we
know
exactly
where
the
error
is.
So
if
it's
an
infrastructure,
error
or
a
pipeline
error,
and
that
allows
us
to
call
the
right
people
to
solve
the
right
problems.
B
It
provides
us
with
a
lot
of
less
time,
debugging
and
looking
for
what
exactly
is
the
problem
and
what
is
occurring
and
also
we
implemented
real-time
monitoring
with
argo,
workflow
dashboards
and
prometheus
and
grafana
one
of
the
biggest
accomplishments
that
argo
brings
to
us
is
that
it
allows
us
to
move
much
more
towards
a
micro
service
approach
and
it
allows
us
to
plug
more
and
more
application
that
makes
the
all
the
pipeline
much
less
language
dependent.
So,
for
example,
we
started
everything
in
a
pipeline
with
r.
B
Now
we
have
machine
learning
models
in
python.
We
have
deep
learning
models
in
python.
All
that
can
work
well
with
argo
to
build
a
pipeline
that
brings
the
best
of
both
worlds,
so
it
can
have
the
best
r
can
provide
and
the
best
python
can
provide.
We
can
have
the
latest
implementations
at
the
newest
solutions.
Also.
What
what
it
brings
to
us
is
that
it
allows
us
to
have
multiple
processes
with
continuous
deployment
and
continuous
integration,
with
much
more
reliability
and
now
moving
on
to
how
our
architecture
actually
looks.
B
This
is
how
it
works,
the
our
user.
It
can
be
both
an
external
user
or
an
internal
client
such
as
our
data
science
teams.
They
can
send
their
jobs
directly
to
or
forecast
as
a
service,
or
they
can
use
our
created
time
series
that
we
provide
both
types
of
clients
to
better
improve
their
models.
It
will
go
through
an
api
that
will
trigger
a
pub
sub
event.
B
So,
with
the
architecture
defined,
we
saw
the
need
to
set
the
parameters
to
allow
it
to
scale
in
an
efficient
way.
First
of
all,
we
use
aggro
conflict
map
to
limit
the
rays
at
which
pods
are
created.
To
avoid
overloading
the
case
api.
We
also
limited
the
maximum
number
of
incomplete
workflows
and
the
max
total
parallel
workflow
that
can
be
executed
at
the
same
time,
to
avoid
using
excessive
resources
to
mitigate
ip
extrusion.
B
We
are
now
running
all
our
application
in
private
clusters,
with
no
external
access
at
all
and
all
our
processes
run
asynchronous.
So
we
have
as
many
different
series
as
memory
allows
us
in
a
different
virtual
machine,
so
we
are
not
locked
to
a
one
job
per
viral
machine
and
I
think
to
wrap
it
up.
We
also
have
now
running
it
on
a
multi-region
privacy,
key
cluster,
which
also
brings
more
resiliency
so
to
wrap
it
up.
B
We
change
far
from
entirely
built
in
our
application
in
the
cloud
with
third
parties
packages,
doing
the
communication
to
a
completely
cloud-native
solution,
which
is
completely
independent,
from
which
code
we're
using
in
our
process
and
now
we
can
plug
in
different
applications
and
work
without
any
bound
to
which
technology
is
trending
at
the
moment.
Thank
you
very
much
for
watching.