►
From YouTube: D3L2: Cybersecurity, Data Mesh, and Delta Lake at HSBC
Description
In this session on August 30th, 2022, Ryan Harris, Principal Cybersecurity Engineer at HSBC, follows up on his Data+AI Summit 2022 session Accidentally Building a Petabyte-Scale Cybersecurity Data Mesh in Azure With Delta Lake at HSBC with Denny Lee for this fun ask-us-anything technical session.
We can dive into the infrastructure and architecture employed, ranging from the landing zone concepts, secure access workstations, data lake structure, and isolated data ingestion, to the enterprise integration layer. Ask your questions on how to build a flexible, secure, self-service environment that is unlocking your team’s data capabilities.
Quick links:
https://delta.io/
https://go.delta.io/slack
https://github.com/delta-io/delta/releases/tag/v2.0.0
https://groups.google.com/g/delta-users
A
Or
we
will
do
our
best
to
answer
questions.
This
is
a
much
more
free-flowing
session
so,
in
other
words,
we're
going
to
talk
about
cybersecurity
database
and
delta
like
obviously,
but
by
the
same
token,
we
do
want
lots
of
questions.
So
please
go
ahead
and
chime
in
with
your
questions
and
for
starters,
why
don't
you
go
ahead
and
tell
us
right
now
like
base
where
you're,
based
out
of
in
my
case,
I
am
based
out
of
the
kirkland
washington,
ie
seattle
area
and
my
guest
happens
to
be
also.
A
Yes,
in
fact,
actually
what
was
really
funny
was
that
ryan
and
I
actually
both
I
used
to
be
his
neighbor
and
then
in
newcastle,
and
just
never
do
that
which
is
pretty
brutal
so,
but
it
is
what
it
is
so
we're
all
live
on
youtube,
which
is
great
so
now
we're
just
sitting
for
live
on.
Oh
looks
like
we
are
actually
live
on
linkedin,
which
is
great
as
well
so
perfect.
Let's
go
ahead
and
start
the
show
so
again
to
everybody
that
is
wondering.
B
A
Attending
the
first
of
a
monthly
series
called
d3l2
or
delta
lake
discussions
with
david
lee,
I'm
here
with
my
first
guest,
which
is
ryan
harris
and
you
are
attending
the
session
cyber
security
data
mesh
and
delta
lake
at
hsbc,
I'm
going
to
start
sharing,
because
the
focus
will
just
be
mostly
on
conversations
between
myself
and
ryan.
So
let
me
stop
sharing
number
one
and
number
two
ryan.
Why
don't
you
introduce
yourself
a
little
bit.
B
A
Cool
okay,
and
so
that
always
glad
glad
to
be
chatting
with
you
just
to
provide
a
little
additional
context,
ryan
and
I
have
known
each
other
for
quite
some
time.
In
fact,
we
both
used
to
run
the
spark
cl
sparking
ai
meetup.
One
was
still
called
ciel,
spark
being
a
fun
fact,
so
that
that's
sort
of
aging
us
or
defining
our
definite
our
age
here,
a
little
bit
too
much
so
we'll
be
working
on
spar
for
quite
some
time.
A
Let's
see
one
of
the
things
I
want
to
start,
oh
right,
some
quick
logistics
again,
if
you
have
any
questions
on
what
we
talk
about,
whether
you're
coming
in
from
linkedin
from
youtube
or
from
zoom
the
lynx
foundation
zoom
by
all
means
just
chime
in
and
we'll
do
our
best
to
answer
your
questions,
but
let's
dive
into
some
of
the
details.
First,
so
actually
before
we
even
dive
into
those
details,
ryan.
B
And
I
started
my
career
in
the
air
force
got
kind
of
my
background
there
right
out
of
college
and
after
that,
I'd
been
working
in
kind
of
highly
regulated
government,
environment
and-
and
that
was
one
of
the
places
that
cared
about
cyber
security.
And
when
I
got
out
really.
Banks
were
one
of
the
only
other
people
that
actually
cared
about
cyber,
and
I
got
involved
with
zion's
bank
down
in
utah,
and
this
is
like
before
the
first
fishing
attacks
were
even
a
thing.
B
B
We
started
looking
at
our
log
data
and
were
able
to.
I
realized
that
we
could
identify
customers
that
had
visited
a
phishing
website,
even
if
they
had
it
logged
in
to
banking
in
that
session,
through
cookie
and
stuff,
like
that,
tying
them
to
their
referrer
logs,
but
that
we
could
then
reach
out
to
customers
and
say
hey.
Did
you
go
to
this
kind
of
strange
email
site
that
you
and
oh
yeah
yeah?
We
did?
B
Okay,
you
shouldn't
have
let's,
let's
change
your
password
change,
all
this
kind
of
stuff
and
also
working
in
credit
card
skimming
and
other
sorts
of
activities
like
that
to
identify
compromised
atms
and
things
of
that
nature,
so
got
involved
with
doing
kind
of
data
analytics
and
working
on
log
streams
of
data
at
zions
and
spent
almost
15
years.
Doing
that
and
was
then
kind
of
looking
for
the
next.
B
A
B
Yeah
I
mean
zion's
was
a
you
know,
western
regional
bank,
and
they
had
different
rules
based
on
what
state
they
were
located
in
hsbc
global
bank
different
rules,
depending
on
which
country
you're
operating
in
so
instead
of
dealing
with
you
know:
10
000
people,
employee,
wise
or
whatever
we've
got
a
quarter
of
a
million.
So
the
the
data
sources
that
I
worked
at
at
zions
were
in
the
large
number
of
terabytes,
and
now
I've
got
stuff
that
are
over
petabytes.
B
So
but
that's
what's
nice
about
solving
these
problems
in
a
kind
of
big
data.
Scalable
way
is
once
you
have
a
solution
to
the
problem.
It
doesn't
really
matter
how
big
it
is.
You
just
cost
a
little
more
for
your
compute,
throw
more
nodes
at
it
and,
if
you're
solving
the
problem
efficiently,
it
scales.
A
B
Yeah,
well,
I
would
say
we're
probably
processing
at
this
point
50
to
75
terabytes
of
that,
the
the
that
that
total
was
the
kind
of
total
stream
of
exhaust
data
that
we
assumed
to
be
getting
generated,
and
it
is
our
mission
to
try
to
capture
all
of
it
and
do
analytics
on
it.
But
but
we're
working
towards
that
goal.
A
Got
it
got
it
and
then
I'm
just
curious
what
what
are
some
of
the
data
sources,
you're,
typically
utilizing
or
something
like
this-
I
mean
it
seems
massive
and
I
love
the
idea
and
we'll
definitely
dive
into
this
a
little
shortly,
which
is
like.
I
love
the
idea
that
you're,
like
oh
yeah,
we
just
toss
a
note
into
it
and
you
can
solve
the
problem
but
by
the
same
token,
I'm
just
curious
like
what
type
of
data
source
is
using.
How
dirty
is
that
data?
I'm
just
curious.
B
We
track
all
of
those
individually
track.
What
security
controls
are
applied
to
them
and
get
the
logs
that
would
produce
artifacts
from
any
of
their
activity
and
then
correlate
those
feeds
together.
So
some
of
them
are
really
straightforward:
netflow,
it's
like
10
fields,
source
destination,
number
of
packets,
stuff,
like
that
other
ones
like
the
endpoint
defense
logs.
B
B
In
in
cyber
security
systems,
generate
lots
of
data
and,
if
you
think
about,
if
you're,
just
going
to
browse
a
web
page
you're
hitting
dozens
of
links
in
that
one
web
page,
each
of
those
links
is
generating
an
artifact.
Spinning
up
the
web
browser
is
starting
a
process.
There's
network
traffic,
that's
going
on
any
artifacts
that
are
getting
downloaded
or
getting
scanned
by
av,
and
all
of
these
actions
are
generating
unique
log
events.
A
Gotcha
and
then
I
guess
one
of
the
things
that
that
you
noted
before
that,
in
addition
to
all
the
data
processing
issues,
there
seems
to
be
like
this
lack
of
documentation
and
data
dictionaries,
and
you
typically
had
these
bespoke
vendor
data
silos
that
you're
typically
working
with,
and
I'm
just
curious.
Can
you
describe
that
a
little
bit,
because
I
think
this
will
segue
quite
nicely
into
the
conversations
about
data
mesh.
B
Yeah,
the
individual
vendors
will
have
data,
that's
relative,
relevant
to
their
application
or
system
that
they'll
store
in
some
way
that
that
met
their
goals.
Maybe
they're
it's
a
micro
services
api
to
hit
and
get
the
data
out
of.
Maybe
it's
a
back-end
database.
That's
got
a
relational
structure
to
it.
B
Back
in
like
a
single
line,
json
array,
that's
got
hundreds
of
gigabytes
of
data
in
it
that
doesn't
easily
parse
out
so
well,
so
you
need
to
pre-process
those
sorts
of
things.
The
vendors
didn't
think
about
how
data
from
av
may
relate
to
the
proxy
data
they
do
relate,
but
they
didn't
help
to
map
out
how
you
should
relationally
join
that
data
to
provide
the
context
of
what
you're
actually
looking
for.
A
Right
and
so
then,
okay!
Well,
then,
so,
then,
how
do
you
with
this,
especially
with
all
of
this
data
flowing
in
which
it
seems
like
the
document,
the
schemas
can
change,
they're,
the
the
inferences?
What
they
mean
can
change,
yet
you
need
to
have
a
this
is
standard
to
work
off
of
right.
So
that
way
you
can
do
your
actual
analytics.
How
do
you
actually
get
this
all
these
systems
to
actually
work
together.
B
Mostly
by
following
the
the
bronze
silver
gold
approach
to
delta
table
curation,
where
we
have
the
raw
data,
that's
that's
coming
in
in
its
native
format
and
then
we're
doing
we're.
Our
goal
is
to
create
initially
silver
tables
for
all
of
those
data
sources
that
represent
in
a
delta
table
nicely
structured
by
the
the
relevant
fields.
Column,
statistics
collected
partitioned
appropriately,
but
that
contain
all
of
the
data
in
those
sources.
B
That's
not
necessarily
how
you
want
to
use
it
for
that
use
case,
but
then
building
curated
gold
views
and
and
materialized
tables
from
those
silver
tables
that
can
be
linked
together
to
find
to
either
display
the
metrics
dashboards.
Things
like
that
that
we're
trying
to
display
that
we
know
that
story,
we're
trying
to
tell
or
that
can
help
join
disparate
sources
of
data
together
and
then
sometimes
the
threat,
hunters
or
other
data
analysts
are
just
jumping
in
to
those
silver
tables
and
doing
ad-hoc
analysis
on
them.
A
Okay,
perfect,
so
let's
break
it
down
a
little
bit,
so,
let's
start
actually
with
the
data
processing
in
the
medallion
architecture,
as
you
say
so,
just
to
provide
context
to
everybody.
That's
attending
right
now
that,
if
you
just
haven't
heard
the
umpteenth
time
of
the
various
delta
lake
conversations,
we've
had
right,
the
context
is
basically
the
data
quality
framework.
A
I
can
go
ahead
and
do
those
things
against
those
those
data
sets.
So,
let's
start
with
the
the
broad
section.
First,
though,
which
is
basically
again,
you
had
noted
that
you've
got
all
of
these
different
sources,
which
may
have
different
schemas
may
change
over
time.
From
your
perspective,
when
you're,
when
you're
you're,
taking,
for
example,
that
one
example
where
you
had
where
the
the
rest
api
was
barfing
out
like
a
single
array
right,
something
like
this
is
is
due.
Then,
in
this
case,
and
by
the
way,
there's
no
right
or
wrong
answer
to
anybody.
A
That's
listening
to
this
conversation.
Everybody
has
different
approaches
to
this.
I'm
just
curious.
Do
you
take,
for
example,
that
one
you
do
you
just
write
it
as
is,
or
do
you
try
to
parse
that
data
in
the
bronze
table,
or
do
you
basically
just
allow
the
barf
to
happen
per
se?
It
gets
written
to
the
storage
and
then
basically
have
a
specific
pipeline
that
allows
you
to
process
that.
B
So
our
raw
data
lake
is
kind
of
outside
that
medallion
architecture,
where
we
capture
the
complete
raw
data,
in
whatever
form
we
had
and
and
that
that
means
that
if
we
ever
find
that
we
messed
up
somehow
in
how
we
were
processing
it
or
made
any
mistakes
or
have
questions
about,
why
did
that
processing
fail?
We
can
go
back
to
the
original
source
that
we
captured
and
say
this
is
this
is
what
it
is
and
then,
as
we're
going
into
that
medallion
architecture.
B
Typically,
the
bronze
table
is
more
of
a
view
or
disposable.
We
don't
maintain
that
bronze
table
as
it
gets
loaded
to
silver.
The
exception
would
be
in
those
cases
like
the
one
that
barfs
out
a
single
string
of
json
that
needs
to
be
pre-processed,
that
pre-processing
will
land
in
kind
of
a
a
persistent
bronze
table
that
we
keep
around
just
in
case.
We
need
to
back
load
it
because
it's
so
much
like
storage
is
cheap.
A
Got
it,
and
so,
from
your
perspective,
like
do?
Are
there
typically
different
teams
that
are
accessing
these
different
data
sources
and
or
which
is
basically
where
the
silo
is
coming
from,
or
is
the
style
coming
from
other
aspects,
because
you
know
that's
just
one
example
of
you
taking
that
bro.
You
know.
I
love
the
fact
that
we're
now
officially
calling
the
rest
apis
barfing
things
out,
but
you
know
rest
api
is
barfing.
It's
data
out.
A
B
One
team
is
focused
on
cyber
security,
metrics
dashboarding
reporting
kind
of
the
status
of
all
of
these
security
controls,
and
these
are
dashboards
that
are
viewed
on
a
daily
basis
by
cis,
regional
cisos
and
other
management,
executive
management
just
to
know
the
coverage
or
if
the
system
is
down
and
like
hey,
antivirus,
isn't
being
updated
here.
What's
the
status
of
control
x,
so
that's
kind
of
one
use
case,
there's
other
use
cases
where
threat
hunters
are
involved
in
the
data.
B
Now
the
threat
hunters
care
about
some
of
the
same
data
sets
that
the
cyber
security
metrics
cares
about.
But
metrics
team
is
looking
for
a
much
more
curated
defined
view
of
that
data,
whereas
the
threat
hunters
are
just
want
the
raw
data,
they
want
everything
if
there's
anomalies
in
the
data.
Those
anomalies
might
be
interesting,
so
don't
cleanse
any
of
that
sort
of
stuff
out.
So
some
data
sources,
maybe
initially
one
team-
wants
but
almost
any
data
source
that
we
have
there
are
use
cases
from
other
teams
and
that's
why
we?
B
B
That's
also
looking
to
to
do
this
core
processing
of
all
of
those
raw
sources
into
a
common
data,
lake
environment
that
everybody
can
access
and
now,
if
somebody
else
wants
to
see
the
proxy
log
data,
they
don't
need
to
figure
out
how
to
process
that
syslog
stream
of
gzip
files.
Whatever
all
of
that
data
exists
in
the
both
the
the
curated
view
that
metrics
may
be
using
as
well
as
that
silver
enrich
table
that
has
all
of
the
data
and
then,
if
they
decide
well,
I
need
a
view.
B
That's
more
like
this
for
my
use
case,
we'll
create
another
view,
maybe
materialize
it
depending
on
their
needs
and
have
multiple
views
that
stack
on
that
that
enriched
table.
And
then
we
can
have
a
lineage
that
tracks
from
the
view
going
back
to
the
source
data,
going
all
the
way
back
to
those
original
files
that
we
ingested
and
can
ensure
that
we've
ingested
all
of
the
data
and
if
there
were
any
gaps
or
issues
during
data
ingestion
that
those
are
accounted
for
in
our
data.
Lineage.
A
Right
so
then,
I
guess
there's
an
implication.
So
when
you
build
up
these
data
mesh
environments
right
because,
for
example,
centralized
platforms-
typically,
we
can
say:
hey,
there's
one
group
that
core
team
that
ensures
all
the
data
is
exactly
the
way
it
is.
But
of
course
it's
extremely
slow
right
or
what
you
had
been
fighting,
which
is.
A
Basically,
everybody
was
silent
off
together,
so
they
did
their
own
thing,
but
then
they're
not
doing
some
shared
version
of
a
data
dictionary,
shared
definition
so
also
metric
x
between
the
the
cyber
the
threat
hunters
versus
the
metrics
dashboard
also
mean
different
things
because
they
literally
use
the
same
terminology.
But
terminology
excuse
me,
but
they
actually
use
the
different
definitions
of
said
terminology.
So,
but
then,
if
you
have
a
data
mesh,
does
that
imply
there's
that
aspect
of
a
centralized
system
where
you're
going
to
have
to
have
all
have
the
same
data
dictionary?
A
B
Yeah,
I
mean
part
of
that
I
think
goes
to
how
you
choose
to
implement
a
data
mesh
within
a
given
environment,
and
you
know,
typically
with
a
data
mesh.
The
team
that
is
generating
a
data
source
is
the
one
that's
supposed
to
be
responsible
for
that
documentation
and
lineage
and
maintenance
of
that.
Now
most
teams
don't
really
want
to
do
that
sort
of
thing.
B
So
you'll
kind
of
find
this
kind
of
ad
hoc
evolution
of
data
where
people
will
first
throw
stuff
together
and
we
have
a
collaboration
zone
where
they
can
share
off
their
kind
of
work
product
with
other
teams
within
the
environment.
But
it's
obvious
that
this
is
a
data
source,
that's
being
generated
by
team
x
and
tmx
may
have
put
together
other
data.
You
know,
sources
that
exist
in
the
enriched
lake,
combined
them
joined
them
in
a
certain
way,
filtered
them
technically.
B
If
they
are
going
to
be
the
ones
maintaining
that
authoritative
source
going
forward,
they
should
be
the
ones
that
are
responsible
for
all
that
responsible
for
the
the
maintenance
and
data
quality
of
that.
But
typically
those
teams
don't
want
that
responsibility
and
that
initiates.
The
conversation
of
how
do
we
migrate
what
we've
built
here
into
something
that's
centrally
managed
that
you
all
will
take
care
of
and
we'll
just
come
to
you
when
we
need
to
make
changes
to
it
and
that's
kind
of
the
software
development,
ci
cd
maturity
model.
A
A
You
know
the
definitions,
you're
good
to
go
and
some
teams
will
be
more
like
okay,
we
just
this
is
temporarily
we're
done.
We
can
advocate
for
why
this
is
applicable
for
everybody.
So
that's
why
the
central
tour
team
needs
to
take
it
over
and
then
the
central
core
team
takes
it
over
things
of
that
nature.
I
guess
there's
a
lot
of
horse
trading.
It
gets
in
these
type
of
scenarios.
B
Yeah
and
a
lot
of
data,
I
mean
it
really
depends
on
how
data
mesh
exists
within
an
organization.
Some
teams
may
be
really
a
micro
services
type
architecture
where
they're
managing
a
thing
that
produces
data,
and
in
that
case
it
really
should
be
that
team
that
is
managing
it.
But
in
our
case,
most
of
the
results
of
data.
The
data
products
that
are
produced
are
the
results
of
combining
data
from
upstream
systems
that
typically
aren't
owned
by
any
internal
team.
They
come
from
external
vendor
systems
and
who
really
owns
that.
A
B
A
Right
from
the
starters,
so
all
right-
I'm-
hopefully
I'm
saying
your
name
correctly,
so
I
apologize
if
I'm
not
since
it
has
this
great
question
from
the
linux
foundation.
Zoom
link
here
and
like
I
said
before,
you
we've
got
questions
from
linkedin
and
also
from
youtube
just
for
everybody.
That
you're
more
than
welcome
to
chime
in
the
question
here
is
since
hsbc
is
a
global
bank.
Do
you
have
separate
common
or
raw
data
lakes
per
region
to
to
adhere
to
regional
compliance
requirements.
B
B
It
it's
it
kind
of
depends.
We
we
have
a
roughly
a
team
of
five
to
ten
people
that
are
focused
on
the
data
acquisition,
getting
the
raw
data
from
all
of
these
internal
external
sources,
landing
it
into
the
raw
lake.
We
have
another
core
team
of
five
to
ten
data
engineers
that
are
producing
all
of
that
data
into
these
common
delta
tables,
but
then
there's
additional
data
engineers
and
analysts
that
are
distributed
into
all
of
these
teams,
and
you
know,
as
an
ml
engineer,
a
data
engineer.
A
No,
it
makes
a
lot
of
sense,
cool
and
so
all
right
going
back
to
the
earlier.
You
had
mentioned
that,
basically,
with
all
the
data
that
you're
processing,
all
you
had
to
do
is
just
add
another
node
and
you
can
automatically
solve
that
particular
particular
problem.
I'm
just
curious
in
the
end.
Is
it
just
because
everybody
in
hsbc
or
like
did
you
have
a
problem
convincing
people
to
switch
over
to
spark
in
order
to
be
able
to
do
that
and
apologize
for
folks?
That
might
know.
A
We've
ryan
and
myself
have
had
previous
conversations
about
this.
So
that's
why
I'm
just
jumping
directly
to
the
part
that
says
yeah
we're
using
spark.
So
I'm
just
curious
like
what
were
some
of
the
problems
that
you
faced
and
then
this
invariably
will
lead
to
also
like.
Why
did
you
also
utilize
delta
lake
by
the
way?
But
let's
start
with
the
spark
aspect.
First
here.
B
Yes,
so
like
one
of
our
data
sources,
the
the
endpoint
defense
logs,
that's
about
three
to
four
terabytes
of
compressed
data
a
day
uncompressed,
that's
about
30
to
40
terabytes,
and
it
was
such
a
large
stream
that
nobody
was
getting
it
and
processing
it.
Before
we
started
the
data
is
available
through
a
vendor
portal.
There
was
ways
to
search
that
data,
so
it's
not
like.
We
were
completely
blind
to
it,
but
we
weren't
maintaining
our
own
history.
We
couldn't
go
back
years
into
that
data
and
do
searches
previously.
B
Now
my
background
in
working
with
hadoop
and
other
big
data
processing,
I
I
knew
that
I
could
process
that
at
scale
with
cloud
resources.
I
just
needed
enough
to
do
it
and
spark
was,
was
the
next
step
now
initially,
when
I
was
processing
that
it
was
throw
a
bunch
of
nodes
at
it
and
and
get
it
done.
And
but,
as
I
was
looking
at
that
processing,
I
knew
that
it
was.
B
It
was
too
much
compute
for
really
what
was
needed
and
that
there
were
ways
of
optimizing
it
so
that
entered
the
next
phase
of
like
great.
I
can
throw
more
nodes
at
this
to
solve
the
problem,
but
now
how
do
I
make
this
more
efficient
and-
and
that
was
a
significant
challenge,
but
now
it's
more
efficient
and
we
can
still
throw
more
nodes
at
it.
If
we
need
to,
we
just
have
to
start
with
far
fewer
nodes,
gotcha.
A
Gotcha
and
then
so
so
this
allowed
you
to
basically
handle
the
scale
and
you're
becoming
more
efficient,
so
it
doesn't
cost
you
as
much
money,
which
is
always
a
good
thing,
but
then
there's
a
great
question
actually
from
prejud,
and
I
again
apologize.
If
I'm
not
saying
your
name
correctly
like
what
do
you
do
like
what,
for
example,
okay,
I'm
gonna
ask
it
slightly
differently
than
the
question
that
was
asked,
which
is
like
why?
A
B
So
what
makes
delta
tables
great
for
us
is
that
they
are
a
streaming
source
and
sync
for
these
data
pipelines.
So
when
we
were
initially
building
a
lot
of
our
data
pipelines,
we
were
using
a
batch-based
approach,
it's
very
common,
it's
what
people
understand
and
it
was
easy
to
implement.
A
lot
of
our
data
sources
were
coming
in
an
hourly
daily
basis,
and
even
these
endpoint
defense
logs
that
are
flowing
in
every
five
minutes.
B
We
just
said:
look
we're
going
to
start
by
processing
these
once
a
day
and
then
move
to
kind
of
an
hourly
basis,
but
by
building
things
with
delta
tables
and
migrating
all
of
our
pipelines
to
streaming
that
we
just
run
on
a
daily
basis
or
an
hourly
basis
or
a
continuous
basis
for
the
stuff.
That's
arriving
every
five
minutes.
It
means
we
can
have
one
common
type
of
etl
approach
that
works.
Regardless
of
how
frequently
we
need
to
load
the
data
and
then
we
don't
need
to
come
up
with
well.
B
A
B
Correct
and
I
would
say
that
a
lot
of
the
data
that
we're
doing
doesn't
necessarily
require
the
full
acid
compliance.
We're
not
making
those
sorts
of
updates
in
place
to
tables
and
stuff
like
that.
We're
mostly
pulling
snapshots
of
other
data
and
joining
those
together,
but
delta
also
makes
it
very
convenient
when
you're
landing
a
new
snapshot
of
the
data.
But
you
already
have
an
existing
reporting.
A
B
B
A
Got
it
thanks
a
lot
ryan
and
just
for
everybody's
context
by
the
way,
there's
lots
of
different
aspects
when
it
comes
to
like,
obviously,
in
this
particular
case,
ryan
hbc,
they're
they're
a
database
customer.
But
the
fact
is
that
you
can
definitely
do
these
same
approaches
when
it
comes
to
open
source,
and
so
I
we're
always
advocating
for
that
aspect
of
the
side
of
things
and
so
there,
for
example,
there's
a
project
called
data
hub
that
actually
will
allow
you
to
go
ahead
and
place
the
metadata
into
its
set
of
stores.
A
So
you
can
track
it.
Obviously,
working
with
an
hms
system
allows
you
to
track
all
the
track.
Excuse
me
all
these
different
tables,
and
so
you
put
comments
or
extra
descriptions
to
provide
access
to
that.
So
there
are
definitely
different
approaches.
The
reality
is
most
of
these
systems
are
have
their
pros
and
cons,
and
so
just
exactly
as
ryan
described
in
his
situation,
and
so
I
don't
think
anybody's
figured
out
the
cleanest
way
yet
at
least
yet.
A
I'm
sure,
as
time
progresses,
whether
it's
databricks
with
unity,
catalog
or
azure,
with
azure
purview
or
lake
formation,
with
aws
or
whatever
else
that
you're,
possibly
using
there
and
glue
excuse
me
there'll,
be
things
will
get
better
on
this
front,
but
it's
it's
still.
It's
still
a
bit
of
a
progress
to
put
it
lightly.
A
Yeah,
it's
it's
it's!
It's
that
delicate
balance
of
just
trying
to
figure
out
when
which
things
do
you
want
to
tackle
yourselves
and
which
things
you
want
to
just
let,
let's
we'll
just
let
the
vendor
tackle
it
so
cool.
I
actually
had
one
other
question,
but
actually
that's
more
directed
to
me,
and
so
I'm
just
going
to
answer
that,
and
this
is
just
going
to
give
an
opportunity
if
anybody
else
has
any
questions
by
all
means.
Please
chime
in
there
was
a
question
from
linkedin
or
from
deepan.
John
again,
I
apologize.
A
If
I'm
not
saying
your
name
correctly,
what
are
the
best
ways
or
practices
for
an
api
to
access
delta
lake
and
that's
the
question.
The
context
is
assuming
an
external
application
wants
to
access
the
delta
table.
What
would
some
of
those
options
be
and
so
long
story
short?
It
depends
on
what
type
of
system
you
want
to
get
access
to
it.
So,
for
example,
if
you're
talking
about
a
distributed
system
like
spark
well,
you
can
always
have
spark
or
trino
or
presto
db
or
flink.
A
These
systems
can
actually
access
your
delta
tables
directly
right
and
that's
not
a
big
deal
at
all.
By
the
same
token,
if
you
wanted
more
from
an
ap
rest,
api
style
perspective,
the
most
common
approaches
right
now
is
to
use
the
delta
rust
api.
It
also
has
python
binding,
so
you
can
actually
use
that
or
using
the
delta
standalone
from
the
delta,
connectors
or
delta
standalone
project.
A
So
you're
sort
of
your
call
on
which
one
you
want
to
go
to
and
so
saying
that
we're
actually
almost
near
the
time
up,
so
I'm
going
to
go
ahead
and
go
through
a
couple,
more
questions,
but
before
we
go
through
those
questions,
ryan,
I
just
want
to
ask:
is
there
anything
else?
You
think
you
think
you
want
to
add
from
a
standpoint
of
like
best
practices
around
of
data
mesh
and
data
lakes
per
se
and
delta
lake
per
se,.
B
I
guess
one
thing
I
want
to
mention
real,
quick
on
their
connectors
power,
bi
we're
connecting
directly
to
that
from
the
delta.
I
o
connectors
page,
so
yep.
It
depends
on
what
what
your
needs
are,
but
but
there's
lots
of
good
resources
from
that
that
connectors
page,
but
as
far
as
best
practices
with
with
delta
lake
data
mesh.
B
So,
if
you're
implementing
a
data
mesh
really
think
about
how
your
organization
and
the
way
you
all
work,
and
that
there
is
no,
this
is
the
way
to
do
a
data
mesh,
but
the
data
mesh
should
benefit
your
work
environment
and
the
the
way
your
teams
need
to
interact,
and
I
think
the
way
you
design
that
mesh
and
the
way
you
define
the
interactions
between
teams
really
depends
on
how
much
centralized
control
you
want
over.
That
and
and
just
kind
of
where
you
want
to
place
that
needle.
A
Awesome
awesome,
awesome.
Okay,
actually
tons
of
questions
are
coming
in.
So
let's
try
to
see
how
much
how
many
of
these
we
can
actually
help
answer.
One
question
which
I
I
didn't
want
to
get
in
front
of
something
that
you
you
asked
the
question:
comparing
delta
lake
versus
app
store,
synapse
analytics
one
things
we
don't
do
in
their
sessions.
A
We
don't
do
comparisons
compete
scenarios
mainly
because
it
does
require
us
to
go
dive
much
much
deeper
into
each
situation,
and
I
I
can
honestly
say:
none
of
us
are
probably
asa
experts,
and
so
we
don't.
We
want
to
be
fair
to
any
of
the
potentially
competing
projects
so
understand
you
you're
interested,
but
you
might
want
to
chime
in
on.
For
example,
you
can
always
go
into
the
delta
lake
user
slack
or
which
is
go.delta
dot,
io,
slash
slack.
A
You
can
ask
the
questions
there,
where
people
will
be
provide
a
lot
more
context,
so
apologies
for
not
being
able
to
try
to
answer
that
question
right
away,
and
so
tara
chit
asked
a
cool
question:
what
are
the
potential
risks
of
pursuing
this
delt
data
mesh
architecture?
From
your
perspective,
since
you've
built
this-
and
just
like
you
said,
you
know
all
the
hotness
about
before
they
even
the
terminology
itself,
but
what
were
the
risks
you
know
for
doing
such
things.
B
That
can
all
be
mitigated
through
communication
and
central
controls,
if
needed,
but
with
the
data
mesh,
depending
on
how
independent
and
self-service
you're
doing
it.
Communication
is
definitely
a
key
in
some
places
where
we've
let
people
do
their
own
kind
of
infrastructure
in
a
self-service
basis.
Writing
their
own
terraform.
B
Their
terraform
didn't
necessarily
follow
the
the
core
terraform
that
we
were
writing.
So
then,
when
we
inherited
the
maintenance
of
that
project,
it
meant
we
had
tech
debt
that
we
had
to
refactor
to
kind
of
bring
that
mesh
node
into
compliance
with
how
we
were
doing
things,
but
at
the
same
time
it
allowed
that
node
of
the
mesh
to
move
quickly
and
independent
of
the
the
time
restrictions
and
availability
of
the
central
team.
So
that's
that
balance
of
moving
fast
versus
accruing
tech,
debt.
A
A
Got
time
for
only
two
more
questions,
I'll
answer,
one
of
them
and
then
we'll
have
you
answer
the
other
one?
Okay,
so
the
question
is:
can
we
have
a
millisecond
latency
to
get
our
data
for
ui
purposes
from
delta,
like
the
quick
answer
is
that
delta
lake
is
designed
more
for
olap
purposes,
online
analytical
processing,
so
typically
you're
doing
a
batch
process
so
because
of
that,
typically,
it's
not
millisecond
latency.
Now,
if
you're
talking
about
like
getting
the
data
directly
from
the
from,
for
example,
spark
structure
streaming.
A
Yes,
that's
completely
designed
for
that
type
of
purpose,
but
from
the
standpoint,
especially
with
project
light
speed
coming
in,
but
from
the
standpoint
of
getting
directly
from
the
table.
Typically,
it's
we're
talking
in
the
not
the
millisecond
area,
we're
talking
in
the
second
area
like
in
terms
of
being
able
to
get
an
aggregate
of
that.
Okay,
so
just
wanted
to
give
you
some
context
and
then
the
final
question
again.
I
apologize
for
not
answering
all
the
questions,
but
we
yeah
we,
but
this
is
we
have
time
for
the
last
one
here.
A
B
It's
not
really
difficult
that
difficult
at
all.
In
our
case,
the
the
main
difference
between
a
unified
shared
environment,
with
kind
of
one
data
lake
and
one
databricks
workspace,
is
just
that
each
node
of
the
mesh
ends
up
with
its
own
data
bricks
workspace
and
its
own
azure
data
lake
service
that
they
can
maintain
their
own
kind
of
private
lake
house
within
that
mesh
node.
If
they
want
so,
the
mesh
node
becomes
either
a
subscription
in
azure
or
a
resource
group.
B
It's
just
an
isolated
pocket
of
resources
that
are
controlled
exclusively
by
the
needs
of
that
participant
of
the
mesh.
So
if
you
had
a
centralized
environment,
you
could
kind
of
turn
that
into
a
mesh
just
by
saying.
Okay,
this
one
central
one
is
the
core
platform
and
now
we're
forking
off
another
zone.
That's
that's
using
that,
but
doing
its
own
thing
independently.
A
Perfect
ryan.
Thank
you
very
much.
This
has
been
an
awesome
session.
I
already
chimed
in.
If
you
have
more
questions,
please
join
us
at
go
to
that:
go
dot
delta,
dot,
io
slack!
You
can
ask
us
questions
there.
We
again
we
apologize
for
not
answering
all
of
the
questions,
but
again
you
can
just
chime
us
there
so
ryan
again.
Thank
you
very
much
for
spending
your
time
with
us
today
on
and
answering
providing
context
around
cyber
security
data,
mesh
and
delta.