►
From YouTube: The Cargill API Platform as a Building Block in Establishing an Engineering Culture at Cargill
Description
When building and growing an engineering culture, capabilities such as an API gateway sound like an easy win. However, the continued push for self-service and automation is always in contention with the need for governance — visibility often conflicts with security. Join Cargill’s Colin Schaub as he discusses the growth, evolution and foibles of Cargill’s API platform using the Kong API gateway and developer portal while on the journey towards building a foundational engineering culture at Cargill.
A
A
A
A
The
oldest
of
the
three
we
call
the
data
platform,
you
can
imagine
a
traditional
data
lake
with
hadoop
and
lots
of
databases
and
tables,
and
such
kergel
has
so
many
legacy
systems
around
the
world,
with
varying
data
formats
and
languages
and
structures.
Almost
all
of
this
data
gets
loaded
into
the
data
platform
in
some
fashion.
A
The
next
is
the
cloud
platform.
Our
cloud
platform
can
be
considered
our
operational
infrastructure.
It
provides
us
with
the
ability
to
deploy
to
the
cloud
provision,
data
stores,
monitor,
availability,
alerting
and
all
the
other
capabilities
one
might
think
of
when
envisioning
an
I.t
landscape,
it's
opinionated
and
pretty
strongly
on
purpose.
A
Logging
and
monitoring
are
only
provided
if
you
stay
within
the
guard
rails
think
about
source
control
for
a
moment
as
a
global
I.t
organization,
cargill
has
source
control
systems
of
every
variety
having
an
opinion
about
which
source
control
system
to
use
has
tremendous
value.
If
you
use
this
source
control
system,
you
can
leverage
the
cicd
pipeline.
A
A
A
A
A
A
A
A
This
was
the
easy
button
right.
It's
why
we're
all
here
it's,
but
it's
difficult
to
talk
about
kong
issues.
We've
had
because
more
or
less
kong
just
works
and
does
what
it's
told
and
just
keeps
on
keeping
on
the
only
consistent
issues
we
have
with
kong.
Is
that
because
it
is
the
front
door
to
the
many
apis,
we
have
it's
always
the
one
to
blame.
A
A
We
also
thought
we
should
have
an
api
first
design
tool.
Now,
two
to
three
years
ago,
there
weren't
many
options.
Swagger
two
was
out
and
pretty
popular
swagger
3
was
on
the
horizon
and
perhaps
just
released,
and
we
chose
a
product
based
on
api
blueprint
because
it
seems
safe
and
actually
pretty
easy
to
read
and
create.
A
It
wasn't
very
popular
though
one
team
used
it
and
enjoyed
it,
but
for
the
most
part
no
one
was
interested
in
it.
The
tool
was
recommended
by
a
consultant.
We
were
working
with
at
the
time.
The
day
after
we
signed
up
and
had
our
first
team
working
with
the
tool
it
was
purchased
by
one
of
the
largest
db
vendors
on
the
planet
and
well.
A
You
know
where
that
went
that
one
team
used
it
through
the
middle
of
this
year
until
we
eventually
moved
them
off
of
it.
No
one
else
has
ever
wanted
to
use
it
in
the
developer
portal.
We
quickly
found
out
that
coming
to
the
table
with
these
preconceived
notions
of
what
the
community
was
looking
for,
was
well
pretty
ill-conceived.
A
It
was
the
initial
step
of
what
we
thought
comprised
an
api
platform.
What
would
folks
need
a
gateway
check,
design
first,
tool
check
the
dev
portal
check,
as
it
turns
out.
Most
of
the
developers
at
cargill
were
using
a
top-down
code,
first
approach
to
api
development,
which
I
think
is
pretty
common
swagger
libraries
that
used
introspection
to
build
a
swagger,
spec
and,
of
course,
swagger
gooeys
lots
and
lots
of
swagger
guise
pretty
quickly.
We
learned
that
the
level
of
interest
for
a
developer
portal
just
wasn't
there
right
or
wrong.
No
one
really
cared
for
it.
A
Governance,
that's
governance.
With
a
small
g,
we
wanted
teams
to
be
as
free
to
create
what
they
wish,
while
steering
them
towards
well-engineered
apis.
We
want
them
to
be
self-reliant,
but
stay
within
the
guardrails
teams
can
create
their
own
api
policies,
which
are
reviewed,
approved
and
deployed
by
the
cappy
team.
A
We
also
ask
the
team
self-review
via
peer
reviews,
get
another
member
of
their
own
team
to
review
a
pr
before
the
pr
submitted
to
the
cappy
team.
This
fostered
knowledge
sharing
within
the
team
and
added
a
bit
of
a
check,
your
own
work
step,
while
saving
the
cappy
team
from
unnecessary
review
of
easy
gotchas.
A
A
Some
things
that
we
govern
are
paths
since
essentially
we're
working
with
a
multi-tenanted
environment
of
50
teams.
We
needed
a
way
to
govern
uniqueness
of
these
apis
on
the
gateway
as
well
as
path
meaning
across
hundreds
of
apis.
We
help
the
teams
do
this
in
a
nice
and
friendly
way.
Of
course,
security.
We
also
wanted
to
make
sure
each
api
was,
has
the
appropriate
security
and
is
using
the
most
appropriate
identity
management
solution
and,
of
course,
the
cargill.
We
have
multiple
cores.
We
also
govern
cores.
Why
do
we
do
this?
A
A
We
didn't
realize
how
much
so
when
we
started,
if
you
can
imagine
a
deployment
pipeline
that
hides
most
of
the
complexities
behind
kubernetes
and
provisioning
relational
databases
and
distributed
caching
layers,
the
missing
link
so
to
speak
is
how
to
wire
all
that
stuff
together.
Database
connections,
connection
pools
secrets,
health
checks,
sso
ssl,
only
redis
caches.
A
A
Maybe
isn't
all
that
adept
at
modern
development
practices
to
begin
with
we're
learning,
and
thus
the
beer
demo
came
into
existence.
It
was
maybe
not
the
first
reference
implementation
we
built,
but
it
was
definitely
the
one
that
gained
and
maintained
traction
across
the
organization
simply
put
beer
has
nothing
to
do
with
cargill.
You
don't
actually
sell
beer,
although
we
probably
in
some
way
handle
the
malting
greens,
but
the
point
is
not
that
people
can't.
The
point
is
that
people
can't
argue
about
property
names
or
an
entity
relationship.
A
Kick
the
tires
a
bit.
Look
at
the
logs
view.
The
cpu
usage
learn
how
to
play
with
an
application
in
an
existing
infrastructure
which
leads
us
to
community.
This
is
where
we
see
the
most
excitement,
the
most
appetite
and
the
most
actual
need.
The
technical
stuff
is
awesome.
Kong.
Does
all
these
really
amazing
things
it
works
right.
Kubernetes
is
super
awesome.
Service
mesh
is
super
awesome.
A
A
Much
like
we
think
of
kong
as
a
product.
Cappy
is
a
product.
We
open
ourselves
up
to
criticism
and
interact
with
some
of
our
most
consistent
users
and
ask
them.
Are
we
doing
what
you
need
us
to
be
doing
customer
advisory
board
in
a
recent
cab
meeting
we
listed
the
high
level
deliverables
of
our
upcoming
release.
A
We
were
wondering
if
teams
were
interested
in
us
pursuing
this
as
a
platform
capability,
one
team
said
they
discussed
it,
but
decided
to
not
pursue
it
due
to
the
what
they
perceived
as
difficulties
of
managing
security
across
entities
within
the
queries,
interesting,
but
officially,
not
interested
due
to
security
concerns
that
they
didn't
want
to
address.
At
that
point
in
time.
A
At
the
cab
meeting
the
developer
portal
again
with
team
leaders,
we
just
received
a
resounding
nope,
not
interested
basically
came
down
to
their
current
working
models
being
good
enough.
They
had
swagger
documents
with
which
their
apis
generated
and
exposed,
and
they
were
happy
enough
using
that
we
also
talked
to
them
about
canary
releases,
not
very
interested
here
either
again.
The
current
processes
were
working
at
this
point
in
time.
Basic
api
gateway
functionality
was
enough
for
now
and
developers
weren't
ready
for
more
in
a
large
global
it
organization.
A
A
Not
trying
to
force
complexity
where
it
isn't
wanted.
We
set
out
to
help
the
user
community
more.
All
of
our
platforms
use
chat,
ops
as
the
primary
form
of
communication
to
many
of
this
seems
like
a
no-brainer.
Everyone
uses
chat,
ops
nowadays,
but
for
cargill.
This
was
new,
first
and
foremost,
we're
trying
to
grow
a
community
and
move
people
away
from
problem
solving
and
requirements
gathering
via
email.
A
We
also
hold
weekly
office
hours.
Ask
us
anything
events
once
a
week
for
an
hour,
we
host
an
ask
us
anything
session.
Obviously
this
worked
a
bit
better
when
we
could
do
face-to-face
meetings
when
this
was
still
safe,
but
this
is
still
possible
to
do
now
that
we
have
all
the
modern
video
conferencing
tools.
A
A
We
also
hosted
a
few
road
shows.
We
took
cappy
to
the
p.
We
took
cappy
to
the
people
so
to
speak.
It
is
one
thing
to
to
perspect
to
participate
online.
You
can
sometimes
feel
the
excitement
just
by
using
maybe
the
number
of
questions
in
the
api
engineering
lounge
as
a
measure,
but
this
doesn't
always
show
the
underlying
enthusiasm
showing
up
and
holding
local
office
hours
in
the
same
time
zone
as
a
face-to-face.
A
We
didn't
realize
when
we
planned
the
trip,
but
when
we
got
there
we
basically
had
a
line
of
teams
waiting
to
discuss
their
projects.
They
wanted
that
personal
interaction
as
an
anecdote.
We
spent
a
week
in
bangalore
with
our
offshore
dev
teams,
just
getting
to
know
them.
During
one
meeting
we
were
diagramming
the
cloud
infrastructure
and
talking
about
how
to
deploy
an
api,
I
asked
the
group
I
was
talking
to
what
languages
they
were
most
familiar
with.
A
A
We
also
hold
monthly
demos
where
the
topics
might
range
from
the
mundane,
like
cappy
operations,
we're
going
to
upgrade
this
sunday
morning
to
the
exceptional
like
graphql.
This
is
similar
to
an
agile
demo,
but
the
audience
is
often
much
larger.
In
cross-cutting,
we
get
everyone
from
the
developers
building
and
deploying
apis
to
managers
who
want
to
just
check
in
because
they've
heard
about
cappy,
sometimes
they're
boring,
sometimes
they're
exciting.
But
the
point
is
that
they're
very
regular,
every
team
does
them
and
the
community
comes
to
expect
them
over
time.
A
A
Graphql
did,
however,
come
up
from
the
business
service
api
community
organically.
It
was
a
true
facebook
story.
The
team
was
continually
seeing
requests
for
specific
single
purpose,
apis
unable
to
keep
up
with
the
demand
they
built
the
very
first
graphql
api
and
demoed
it
at
our
monthly
demo
to
the
large
api
community.
A
A
A
We
talked
about
how
the
iphone
revolution
is
also
seen
as
a
driver
for
api
growth.
Everyone
there
had
a
cell
phone
and
we
showed
them
how
apis
are
used
and
the
apps
we
all
use
each
and
every
day
this
is
all
pretty
obvious
stuff,
but
to
the
average
smartphone
user.
The
fact
that
apis
are
what
allow
them
to
have
the
cool
technology
that
fits
in
their
pockets
is
often
another
light
bulb
moment
and
finally,
tacos.
A
A
A
A
A
We've
since
transitioned
50
to
60
swagger
specs
to
the
to
the
developer
portal
and
every
time
we
show
it
to
people,
we
are
a
combination
of
wow
and
aside
relief,
psy
relief,
the
developer
that
the
developer
doesn't
the
developer.
Now
has
one
less
thing:
they
need
to
do
in
a
wow
that
it's
actually
being
used
as
if
they
were
late
to
the
game.
Somehow
and
again,
the
beer
demo
specification
loaded
into
the
dev
portal,
consistency
and
repetition.
A
We
can
spend
lots
of
time
thinking
about
better
platforms.
We
can
plan
road
map,
execute
all
of
this
matters
and
it's
very
exciting
stuff,
but
what
really
matters
is
building
the
community
around
the
platform
and
allowing
the
community
to
steer
and
guide
the
platform
in
ways
that
are
most
helpful
to
them.
A
Providing
a
developer
portal
seemed
like
the
right
thing
to
do
in
the
beginning
of
the
journey,
but
in
hindsight
the
user
users
weren't
ready
for
it.
Yet
when
the
community
was
ready,
the
deployment
of
the
developer
portal
felt
more
like
a
community
driven
effort
and
had
more
energy
because
of
that,
the
api
first
design
tool
also
fell
into
this
category,
engaging
with
the
developer
community
in
meaningful
ways
that
are
both
personal
and
helpful.