DASH / Behavioral Model WG

Detailed Notes:
• Chris and Andy discussed a Service Tunnel feature that Andy was trying to simplify down to 64 bits. Chris suggested filing an Issue and creating a PR to get the discussion going offline. Chris also suggested using blame to see who wrote the code and pinging them in the issue.
• Chris and Vincent discussed an issue with compiling Docker images, which may be related to a race condition. Chris suggested filing an Issue and tagging himself and Kamil.
○ Vincent and Chris discussed a compile issue with Docker that Vincent noticed when he pulled up the latest Dash to rebase. Vincent mentioned that Kamil had made some changes to the 'make' files that could be causing the issue.
○ Chris and Marian discussed a bug in the bmv2 code; discussed the possibility of returning more than one object type, but Marian was unsure if it was possible due to the API signature.
• Chris and Saad discussed a Python-based behavior model that Saad has been working on for the past few weekends. The model is a drop-in replacement for the existing P4 behavior model.
○ Saad has been working on a Python based behavior model to replace the existing P4 model. He has completed the functionality and is now working on integration.
○ Chris and Saad discussed a project involving a drop-in replacement for an existing model, which would be written in Python and have the same interfaces as the current P4 runtime approach.
○ Chris and Marian discussed two issues related to the implementation of the model. The first issue was discussed yesterday and Brian agreed to take another look at it. The second issue had code issues and Marian was cited to help.
• Chris and Marian discussed a code path in the P4 test Gen that was found to be dead code. Marian explained that the NVGRE encap function is agnostic to the overlay IP family and that the service tunnel is before the IPV6 translation.
○ Marian and Chris discussed the code path of NVGRE encap. Marian explained that the function is agnostic to the overlay IP family and that it supports both IPv4 and IPv6 headers.
○ Chris, Andy, and Marian discussed the potential of using a code path as a CICD test case for P4 test gen. Marian suggested that the solution should not be to remove the condition from the function.
• Chris talked about Kamil's work on refactoring the SAI Thrift server in the SAI repo to compile the SAI Meta into a Lib Sai Meta.
○ Chris discussed the progress of Camille's refactoring of the side thrift server in the Sai repo. The goal is to compile the side meta into a Lib Sai Meta, which will eliminate the need for it in the vendor libsai.
○ Chris and Agrawal discussed the status of test cases running on hardware. Vincent offered to check the test cases and provide feedback. Chris suggested that other vendors could also try the test cases on their own implementations.

Kamil has made changes to SAI Repo, with (for example) build times cut in 1/2 again
SAI Thrift server and tests
Volunteered to separate SAI Meta from libsai

[sonic-net/DASH] Sai attributes for ACL rules should include list of IP prefixes for the Bmv2 compile (Issue #360)

ACL Tags: Support for tags on the control plane
Kristina to clarify this in the HLD
What is it that defines what we should (at libsai) down, vs NB on the API
State this this doesn't trickle down to libsai, reference Orchestrator

Discussed the Issue from Intel checking out the P4 DPDK and checking for an update from Usha. Also discussed the pipeline and behavioral model, the code work done, and the known issues and gaps.

Paul was introduced to the project.

Emulation of P4 into Python - Saad
To solve language limitations in the reference model; also has 'static analysis' (big string input)
Stateful features not in P4 or bmv2; translated to Python to see if it is useful?
Look at DPDK and bmv2 - perf constraints are not there; it is functionally working
Chris: integrate w/SAI PTF & SAI Challenger :) - offered to work together
Chris: more annotations from Marian?
Marian: orthogonal to Saad's questions. Source for SAI API generation is the .json file generated by the compiler. Today you can find both examples in the generated file. Native field or field from annotations to understand the 'type' information and flags (SAI flags and annotations, for example).

DPDK BE test/experiment
Usha is working on this

P4 DPDK on horizon; BMV2
https://github.com/sonic-net/DASH/issues/408

2 Intel Engineers (fruffy - Fabian) contributing to TestGen in Issue399
Length of time to generate packets (consider this), run the test to see how long it will take
PINS team is using TestGen at different cloud company
Ping MarianP offline for updates

Andy likely waiting for DPDK developers, compiler issues are open, then Andy can resume BE experiment

DASH Behavioral Model Workgroup call

PR365 - Adding support for metering (Outbound)
https://github.com/sonic-net/DASH/pull/365
Mentioned in DASH HLD 2.4.2

There was a need to introduce new annotations. Also, the ACL code in P4 has a compile flag - be careful here for bmv2 to SAI generation.

Continued ACL tag conversation
Do we have a list of tags, or 1 tag? Clarify...

PR327 - Add Private Link Mapping
Service Tunnel is for VMs to access public resources via Internet; the overlay will have a public IP as the address
The service is exposed as public IP, but there might be different scenarios where we have the same service hidden behind NAT, which is where we need Private Link.
The transposition happens on a different table.
We re-use ST encoding

PR358 - Support SAi attr generation from annotattions

Marian provided overview as to why we are doing this:
Need to add more metadata to SAI headers to explain relations between objects.
New method for generating attributes for keys and action parameters.
If you place this directly into P4 code in the form of annotations,

Merge PR246 - Marian to sync to main, then notify Kristina

PR308 - does not compile for DPDK backend (expression is not implemented)
Andy added IF/DEF, does not handle multiplication
Marian will fix for NVGRE
Andy will find link for specific lines of code
Here is the link to some sample code with #ifdef to handle this kind of code: DASH/dash_vxlan.p4 at main · sonic-net/DASH (github.com)

Go overbmv2 for ST/PL
https://github.com/sonic-net/DASH/pull/308

Andy: Compile DASH P4 code checked into DPDK software switch
Facing a compiler bug right now - checking it out
Once complete it will be nice to have instructions published for everyone

PR's Requesting updates:
https://github.com/sonic-net/DASH/pull/336 (Prince) - 2 different changes
https://github.com/sonic-net/DASH/pull/246 - (Marian) not risky to accept this PR, however would like to have test cases - we could merge (Volodymyr). Chris to take out of Draft, Kristina to Merge, however PR has Status Issue.

DASH Community call content request: Inbound Pipeline for ST/PL - Prince Providing Overview
Q: Reshma - please cover the Inbound piece of ST/PL (not related to bmv2 implementation)
PR341 how Inbound flow works
Section 2.3 - Service Tunnel (ST) and Private Link (PL) packet processing pipelines

For now we have only Source-Based ST/PL scenario

Added Reshma as a Reviewer to take a look

Go overbmv2 for ST/PL - push to 3/16/2023
https://github.com/sonic-net/DASH/pull/308
Documentation reviewed and work in progress on behavioral model, which needs changes based on the documentation
s/be complete next week for ST
Kristina to create 30 min call for next week

WIP snapshot of something that compiles with the P4 DPDK compiler Thank you Andy!
https://github.com/sonic-net/DASH/pull/324

Andy: Intel project IPDK - possible minimum viable set of features here? (DPDK software switch, P4 to - DPDK compiler, P4 Runtime API support (allow Chris to re-use code)
Chris: P4 DPDK gives us P4 Runtime (working hypothesis)
Reshma: we could have a real-time software switch. Perhaps Shitanshu can help out w/Andy

PR's Requesting updates:
https://github.com/sonic-net/DASH/pull/336 (Prince)
https://github.com/sonic-net/DASH/pull/246 - (Marian) not risky to accept this PR, however would like to have test cases.

No description provided.

More guidance needed for inbound processing.
- Review Logic on a returning packet that had a src IP change

Capture task to not only update the code (PR) but also creating issue that will follow to update HLD

PR#332 Will be merged, all comments resolved. Soon to be Merged
Add #ifdef's so P4 code can target BMv2 or DPDK back ends by jfingerh · Pull Request #332 · sonic-net/DASH (github.com)
- Not only checking pna architecture as well as dpdk

Do we use the same sha tag for both docker images (dpdk & p4c)?

Consider freezing sha for p4c and floating for dpdk & pna
- Upgrade as needed once confirm things are working as expected

Go overbmv2 for ST/PL - push by 1 more week to 2/16/2023
https://github.com/sonic-net/DASH/pull/308

Discuss: Add/Transition from bmv2 to P4 DPDK target (P4 runtime on DPDK)

Merge https://github.com/sonic-net/DASH/pull/332/files ?
Can merge once out of Draft Status

WIP snapshot of something that compiles with the P4 DPDK compiler
https://github.com/sonic-net/DASH/pull/324
Thank you Andy!

Add Private Link mapping by marian-pritsak · https://github.com/sonic-net/DASH/pull/327

Discuss: Add/Transition from bmv2 to P4 DPDK target

AI: Andy will try to run maybe next week or so to see if it executes (check for gaps)
Should not be a mess of IF/DEFs
Chris created a docker file w/DPDK backend for Andy, can compile this version of code w/every version of CI

Marian: execute on real DPDK BE software

Add-on-miss available in PNA, implemented in DPDK, there is an active development community there. We want to make sure there is no degradation if we switch to different BE (do not break out party hats yet folks)

https://github.com/sonic-net/DASH/pull/331
Correct syntax errors in PNA_CONNTRACK version of code - Andy
These had IF/DEF PNA conntrack around them

Q: range port list and list port list (in bmv2) public?
A: Marian: simplified version s/be on par
unroll the range by looping through it

Saad: open contrail issue in P4; need proven xn tracking implementation
paste code into new work item
Kristina opened: https://github.com/orgs/Azure/projects/129/views/6?pane=issue&itemId=19631774

Push ST/PL bmv2 PR308 to next week Feb 2 2023
https://github.com/sonic-net/DASH/pull/308

Q: Andy: match, key gen ID, will this be in P4, per ENI (Y), CP will decide updates (batch or real-time),
will the key gen entries be replaced?

From former HA notes:
It is the implementation - depends upon your hardware and what is optimal for the technology provider. Packet triggered here…
Config level of the VNIC/ENI, change config/policies on the ENI, config version changes from 0 to 'not zero', cache the version for the ENI that is maintained IN the connection, look at version maintained in the connection for the ENI (from when the last packet triggered the fixup), and tells datapath whether an update to the State in the connection is needed, packet will go through policy lookup and update. (For every connection we will have a cached version). Will not transit the EST flow for forwarding…or do we use the flow entry that is there? We don't have to update the flow (per se). Gen ID should make it's way into the P4 code. Andy: Aggressively delete old or let them be aged out? Not sure if this is defined in P4 model.

Hanif: In P4 community, has bmv2 caught up?
Andy: Open Source PLang projects review: majority of code is delivered via paid developers
BMV has been in maintenance mode
Would a switch from bmv2 to dpdk go smoothly today? Likely not… w/reference model & control plane changes (src & dest address list-match-type, etc…) - no simple datapath implementation at high speed. No path into DPDK.

On the other hand, people are working on P4 DPDK (useful, performant, c/be turned commercial). BMV2 is local…
No idea if we have gaps/feature support.
Andy: does it even compile? Gaps? Run through P4C DPDK Compiler.
Chris: workflow? 1) execution engine P4 DPDK 2) Compiler 3) P4 code in behavioral model
P4 code to SAI header file (autogeneration) not platform dependent
libsai IS platform dependent

Volodymyr: https://github.com/p4lang/p4-dpdk-target and https://github.com/p4lang/p4-dpdk-target/tree/main/examples/pna

Reshma: TDI integrating into sync_d of SONiC as a direct call (not an RPC call)
Want to implement DASH onto DPDK
Connection tracking will be DASH connection tracking
range port list and list port list (in bmv2) upstreamed into P4.org? Andy to check? Power of 2 ranges accepted only?

DPDK can be a replacement for bmv2 - may need to make choices in terms of implementation

Kristina to perform squash & merge for PR321

Cover PR308 next week
https://github.com/sonic-net/DASH/pull/308

Stateful processing in bmv2?
No, working on ST as priority - updated Issue 6

P4C compiler - needs a refresh
https://github.com/sonic-net/DASH/pull/249
Get and build onto ACR Registry. The building thereof requires a lot of finesse to put PR together, must work w/Marian

Go overbmv2 for ST/PL next week

DASH API in SAI Master - Reshma approved this morning
Add DASH API by marian-pritsak · Pull Request #1590 · https://github.com/opencomputeproject/SAI/pull/1590

DASH/pull/310 Add ADDR_FAMILY attr generation (for any table that has IP address, IP prefix, or IP prefix list as a key)
https://github.com/Azure/DASH/pull/310
API used/needed for SONiC CRM (Critical Resource Monitoring)
SAI API to query resources left in the ASIC for sai_object_type
github.com/opencomputeprojct/SAI/blob/master/inc/sai.h

API allows you to look more closely into resources that are left
Can raise alarms in the db

Underlay Routing - in DASH we have complex LPM routing and the 2 default routes w/2 ToRs.
Do we need to implement another LPM table?
No: this is possible to implement w/o an extra LPM. Implementation can perform the hashing on the 5-tuple & algorithm? Can we assume ECMP-in and return packet on the same port w/o a hash? (We have a default hash assigned by the SAI definitions - check opencompute project). What we DON'T want is for an implementation to PREFER a port. We can either 5-tuple hash or send out the same port (add to HLD).

Is there a way to have 2 default routes & decide routing w/o ECMP?

We can clarify in HLD the default route we learn via BGP from the peer.
Expectation is not a lookup and route (take care of both cases w/o a lookup in the DPU)
1. default route
2. ECMP

What if there are multiple hops out of a port? (Not what Azure does, however it is a good question).
Using this tech in another enterprise w/o a CLOS (equal cost) network, they may have to design this differently via the DPU implementation.

Next week come to the call with ST/PL questions if you all have any :)

We have tests marked 'don't run' in BMV2

Big challenge - not all of us are P4 savvy, if we had chosen C there may have been a bigger universe to work on it
Simulator, Compiler, etc...

Open Source DPDK backend for the compiler (Andy)
Here we have not done scripts that take SAI APIs down to 'something' in the P4 lang

P4 DPDK support is happening, in a few months we might be able to use it better from a DASH standpoint w/features in P4 DPDK

Need BMV2 owner; knowledge and expertise is not widespread.

Gerald to check w/Engineering (via Deepak and MSR) for resources.

Vincent:
SAI Build = x86 specificity (SAI Thrift Server?) - if there is a problem we can flag this?
Running SAI Thrift on ARM server?
maybe check opencompute/SAI

For reference from SONiC: https://github.com/sonic-net/SONiC/blob/master/doc/platform_api/new_platform_api.md This is the HLD for platform 2.0 APIs

BMV2 implementation ignores and sends packet out on same interface. Test will still work with default route that is pointing to both interfaces. Gap is underlay MAC address that was pushed needs to be updated. If we can plumb in MAC address using existing SAI API.
We need P4 table updated just enough
Marian – agree MAC address portion is open

Created Issue 292 and assigned to James Lo (Accton)
Basic underlay routing bmv2 to handle MAC address updates/resolution #292

IPv6 & IPv4 scale again

Underlay routing for overlay tests to run on bmv2 or hardware
Volodymyr writing functional tests for overlay, but have expectations on underlay routing, cannot run on bmv2 as written.
Underlay MAC and possibly DMAC programming, perhaps another table. Minimalist SAI API to support it.

Continued the discussion of how much underlay support is required in the bmv2 implementation to ensure consistent test-cases for hardware or bmv2

Default routing out same port. We only need to add underlay MAC support

Filed issue #50 against sonic-dash-hld

PR249 to introduce a new container - mainly about dockers, it builds P4c used to build/compile the bmv2
New image, rules to build, docker file itself; things left to be done are CI updates and docker repo (away from Chris')

PR1630 in opencomputeproject/SAI list of prefixes instead of /32's or /128 IPv6 addresses

SAI annotations is something we would like to have - backlog - use P4 annotations instead of heuristic in the code generator

PINS project: there is no public code w/PINS implementation w/SONiC VS or bmv2 at this moment

Chris Sommers provided overview of structured annotations, P4 runtime spec
Virtualized ReDis SyncD addition to BMV2 GitHub project
Closed PRs 268, 264, 263, 232

Behavioral Model Workflow README.md
https://github.com/Azure/DASH/blob/main/dash-pipeline/README-dash-workflows.md

NVidia is working on SAI API SONiC changes in priority over bmv2, then will go back to P4c changes

https://github.com/Azure/DASH/pull/240 - Make SAI attrs mandatory for drop-on-miss tables
Big 'P4 Annotations' discussion - Chris to experiment

https://github.com/Azure/DASH/pull/246 SAI bulk add/remove implementation - the idea is to support the definition; we have entries and objects w/bulk create and bulk remove

Reshma hosting :)

PR207 Marian will approve right now

PR201 includes more than just renaming APIs to be compatible w/SONiC software; Fixes make file, Autogeneration of utils, etc…
Chris to play w/autogen today

PR202 Marian to try this today now that he has his environment back

Chris Sommers to continue working with ACR and Runners for images

Working toward a stable SONiC API - we are 2 PRs away
XN tracking is also pending this work

8/11/2022 -
Mukesh: PR187 - Default drop support for P4 table miss w/o generating SAI drop action
Correcting lingering issues to restore positive tests
Drops at end of pipeline (for counters)
Hanif: are we saying this is a stop-gap until we have add-on-table-miss for Connection Tracking table?
Mukesh: add-on-miss is different than table-miss-drop general table

Mukesh: PR193 - Outbound pipeline packet test
First time where we are testing the pipeline where it will be quasi-functional
Added Reshma as reviewer

Marian: PR194 - Fixed Inbound VNET Routing
Idea of routing table is to point to a VNET
Need to look at source VNET (VNI values may overlap), need to look at VNI and prefix of source public address
Code generator is failing, need to fix

Underlay PR176 Marian - fixes
Mukesh: PR167 Fix SAI object create error in libsai adapter
Vijay Srinivasan: PR163 add vnet table to the pipeline; bring into line w/gNMI wrt VNET_ID and VNET_Table

PR163 add vnet table to the pipeline; bring into line w/gNMI wrt VNET_ID and VNET_Table
Other PR149 to be approved, published and merged
PR has SAI header changes, separate PR that has not been changed yet, and a VNET_Out test case PR (more objects)

PR 141 needs 1 more look before Marian approves and merges
PR 152 - Marian has a similar PR (stages w/ENI as a key in the ACL)
The problem is changes to the configuration. ACL is a security table needs to be bound and unbound atomically.
Can we split this into BM and SAI API then Test? - Marian will review both today and merge.
PR160 https://github.com/Azure/DASH/pull/160
- IPv4 option parsing
Current code does not take into account IPv4 optional headers

What is priority based routing? Priority based routing lookup - (Marian) can we delay this question until next week?

CA-PA validation has to happen against specific CA address, does it need to come from a specific PA? We need to check whether the PA exists in entire mapping table.

Prince to sync w/Michal what to program and add to HLD




We need to document DSCP bit - Michal

PR Review: 141, 149, 152
P4 bugs/issues needing volunteers

Discussed PR141 - will keep a separate action
P4 fixes needed
#146 to saad-mhzr

TCP state machine discussion and requirements
WB/BB testing discussion
Alignment to P4
Next steps to define APIs

June 23, 2022
Created new work items
Discussed need for testing

June 16, 2022
Upcoming Keysight CI/CD pipeline automation tests
Keysight/NVidia updates next week

June 9, 2022
Discussed 'Hello World' basic test

June 2, 2022

May 26, 2022
Discussed ONF presentation
Review of project work items

Chris/Reshma
https://www.youtube.com/watch?v=mT7-t_aDozM - P4 as a Single Source of Truth for SONiC DASH; use cases on both SoftSwitch and HW

Venkat Puella
https://youtu.be/ymU2d9Oe_dI -
Venkat's presentation at P4 conference. This will become useful to DASH later as it uses P4 modularity and composability.

May 12, 2022
Discussed updates to work items, new CI/CD resource, IPv6 PR approval, test infrastructure, and new content in pipeline ReadMe.md file

May 5, 2022 Behavioral Model WG Call
Demo by Marian

No description provided.

No description provided.

No description provided.

March 1, 2022 Behavioral Model sync call

Initial behavioral model sync call