P2MP connection problem: the intermediate node w/ an end host in the tree doesn't forward traffic between its parent and child nodes.

[congwang09]

Multipoint request on renci testbed that includes 4 sites. Duke is not correctly forwarding traffic to/from NCSU.
In this example, Duke is both endpoint switch and tree node (see the diagram below), the code here in L2MultipointPolicy.py, first checks the endpoint switch, which is skipped in the following tree nodes breakdown rule.

atlanticwave-proto/shared/L2MultipointPolicy.py

Line 216 in 88f5f47

if node in covered_nodes:

L2MultipointEndpointLCRule does generate a set of flood ports to flood the traffic, so need to check the detailed flow entries.

Request:

curl \
-X POST http://atlanticwave-sdx-controller.renci.ben:5000/api/v1/policies/type/L2Multipoint \
-b cookie-mcevik.txt \
-H "Content-Type: application/json" \
--data-binary @- <<EOF
{
    "L2Multipoint":
    {
    "starttime":"2020-09-04T10:00:00",
    "endtime":"2025-10-09T15:59:00",
    "endpoints":
    [{"switch":"uncs1","port":12,"vlan":1423},
    {"switch":"rencis2","port":8,"vlan":1425},
    {"switch":"ncsus1","port":12,"vlan":1424},
    {"switch":"dukes1","port":12,"vlan":1422}],
    "bandwidth":8000
    }
}
EOF

Tree structure:

2020-10-01 17:48:49,657 debug.sdxcontroller.topologymanager: 139622684415744 DEBUG    find_valid_steiner_tree: Successful [('rencis1', 'rencis2'), ('rencis1', 'dukes1'), ('rencis1', 'uncs1'), ('dukes1', 'ncsus1')]
2020-10-01 17:48:49,658 debug.sdxcontroller.topologymanager: 139622684415744 DEBUG    find_vlan_on_tree: ['rencis1', 'rencis2', 'dukes1', 'ncsus1', 'uncs1']
2020-10-01 17:48:49,659 debug.sdxcontroller.topologymanager: 139622684415744 DEBUG    find_vlan_on_tree returning 1

Rules:

2020-10-01 17:48:49,637 sdxcontroller.rest: 139622684415744 INFO       - JSON data {u'L2Multipoint': {u'endpoints': [{u'switch': u'uncs1', u'vlan': 1423, u'port': 12}, {u'switch': u'rencis2', u'vlan': 1425, u'port': 8}, {u'switch': u'ncsus1', u'vlan': 1424, u'port': 12}, {u'switch': u'dukes1', u'vlan': 1422, u'port': 12}], u'bandwidth': 8000, u'endtime': u'2025-10-09T15:59:00', u'starttime': u'2020-09-04T10:00:00'}}
2020-10-01 17:48:49,644 sdxcontroller.rulemanager: 139622684415744 INFO     add_rule: Beging with rule: L2Multipoint(2020-09-04T10:00:00,2025-10-09T15:59:00,[{'switch': 'uncs1', 'vlan': 1423, 'port': 12}, {'switch': 'rencis2', 'vlan': 1425, 'port': 8}, {'switch': 'ncsus1', 'vlan': 1424, 'port': 12}, {'switch': 'dukes1', 'vlan': 1422, 'port': 12}])
2020-10-01 17:48:49,666 sdxcontroller.rulemanager: 139622684415744 DEBUG    _reserve_resources: VLAN Tree: ,1
2020-10-01 17:48:49,667 sdxcontroller.rulemanager: 139622684415744 DEBUG    _reserve_resources: B/W Tree: ,8000
2020-10-01 17:48:49,668 sdxcontroller.rulemanager: 139622684415744 DEBUG    _reserve_resources: VLAN Port: ('uncs1', 12),1423
2020-10-01 17:48:49,669 sdxcontroller.rulemanager: 139622684415744 DEBUG    _reserve_resources: B/W Port: ('uncs1', 12),8000
2020-10-01 17:48:49,671 sdxcontroller.rulemanager: 139622684415744 DEBUG    _reserve_resources: VLAN Port: ('rencis2', 8),1425
2020-10-01 17:48:49,672 sdxcontroller.rulemanager: 139622684415744 DEBUG    _reserve_resources: B/W Port: ('rencis2', 8),8000
2020-10-01 17:48:49,674 sdxcontroller.rulemanager: 139622684415744 DEBUG    _reserve_resources: VLAN Port: ('ncsus1', 12),1424
2020-10-01 17:48:49,676 sdxcontroller.rulemanager: 139622684415744 DEBUG    _reserve_resources: B/W Port: ('ncsus1', 12),8000
2020-10-01 17:48:49,678 sdxcontroller.rulemanager: 139622684415744 DEBUG    _reserve_resources: VLAN Port: ('dukes1', 12),1422
2020-10-01 17:48:49,679 sdxcontroller.rulemanager: 139622684415744 DEBUG    _reserve_resources: B/W Port: ('dukes1', 12),8000
2020-10-01 17:48:49,681 sdxcontroller.rulemanager: 139622684415744 DEBUG    Sending install breakdown: <shared.UserPolicy.UserPolicyBreakdown object at 0x7efc7103b250>
2020-10-01 17:48:49,681 sdxcontroller.rulemanager: 139622684415744 DEBUG        L2MultipointEndpointLCRule: switch 203, 8:([1]), ([(12, 1423)]), 1, 8000
2020-10-01 17:48:49,682 sdxcontroller.rulemanager: 139622684415744 DEBUG    Sending install breakdown: <shared.UserPolicy.UserPolicyBreakdown object at 0x7efc7103b290>
2020-10-01 17:48:49,682 sdxcontroller.rulemanager: 139622684415744 DEBUG        L2MultipointEndpointLCRule: switch 205, 8:([23]), ([(8, 1425)]), 1, 8000
2020-10-01 17:48:49,683 sdxcontroller.rulemanager: 139622684415744 DEBUG    Sending install breakdown: <shared.UserPolicy.UserPolicyBreakdown object at 0x7efc71032b10>
2020-10-01 17:48:49,683 sdxcontroller.rulemanager: 139622684415744 DEBUG        L2MultipointEndpointLCRule: switch 204, 8:([1]), ([(12, 1424)]), 1, 8000
2020-10-01 17:48:49,684 sdxcontroller.rulemanager: 139622684415744 DEBUG    Sending install breakdown: <shared.UserPolicy.UserPolicyBreakdown object at 0x7efc71032f90>
2020-10-01 17:48:49,684 sdxcontroller.rulemanager: 139622684415744 DEBUG        L2MultipointEndpointLCRule: switch 202, 8:([1, 2]), ([(12, 1422)]), 1, 8000
2020-10-01 17:48:49,685 sdxcontroller.rulemanager: 139622684415744 DEBUG    Sending install breakdown: <shared.UserPolicy.UserPolicyBreakdown object at 0x7efc71032a90>
2020-10-01 17:48:49,686 sdxcontroller.rulemanager: 139622684415744 DEBUG        L2MultipointFloodLCRule: switch 201, 8:([23, 2, 1]), 1

P2MP Tree created:

Connectivity between end host pairs:

               renci2     duke    unc    ncsu
renci2.       --           ok        ok       no
duke          ok           --        ok       ok
unc            ok           ok        --      no
ncsu          no           ok        no       --

Flow table of br22 in dukes1:

corsa-1# show openflow flow br22 full 
                                                                                                                                  count : 25
  +-------+-------+---------------------------+---------------------------+--------+---------+--------
  | table | prio  |           match           |          actions          | cookie | packets |  
  +-------+-------+---------------------------+---------------------------+--------+---------+--------
  |     0 |     0 | -                         | goto_table:1              |    0x0 |    2282 |  147182 |         
  +-------+-------+---------------------------+---------------------------+--------+---------+--------
  |     0 |   100 | in_port=1,dl_vlan=1411    | output:2,output:11        |    0x0 |   
  +-------+-------+---------------------------+---------------------------+--------+---------+--------
  |     0 |   100 | in_port=2,dl_vlan=1411    | output:1,output:11        |    0x0 |   
  +-------+-------+---------------------------+---------------------------+--------+---------+--------
  |     0 |   100 | in_port=11,dl_vlan=1411   | output:1,output:2         |    0x0 |   
  +-------+-------+---------------------------+---------------------------+--------+---------+--------
  |     0 |     1 | in_port=12,dl_vlan=1422   | set_field:4097->vlan_vid, |    0x4 |      
  |       |       |                           | goto_table:1              |        |         |         |           |           
 	+-------+-------+---------------------------+---------------------------+--------+---------+--------
  |     1 |     0 | -                         | goto_table:2              |    0x0 |    2306 |  149348 |         
+-------+-------+---------------------------+---------------------------+--------+---------+--------
  |     2 |     0 | -                         | goto_table:3              |    0x0 |    2306 |  149348 |         +-------+-------+---------------------------+---------------------------+--------+---------+--------
  |     3 |     0 | -                         | goto_table:4              |    0x0 |    2282 |  147182 |         
 	+-------+-------+---------------------------+---------------------------+--------+---------+--------
  |     3 |     0 | in_port=11                | CONTROLLER:65509,         |    0x3 |       0 |       
  |       |       |                           | goto_table:4              |        |         |         |           |           
 +-------+-------+---------------------------+---------------------------+--------+---------+-------
  |     3 |     0 | in_port=12                | CONTROLLER:65509,         |    0x2 |       0 |       
  |       |       |                           | goto_table:4              |        |         |         |           |           
+-------+-------+---------------------------+---------------------------+--------+---------+--------
  |     3 |     2 | in_port=12,dl_vlan=1      | CONTROLLER:65509,         |    0x4 |      
  |       |       |                           | goto_table:4              |        |         |         |           |           +-------+-------+---------------------------+---------------------------+--------+---------+--------
  |     4 |     0 | -                         | clear_actions             |    0x0 |       0 |       0 |         
+-------+-------+---------------------------+---------------------------+--------+---------+--------
  |     4 |     0 | in_port=1                 | clear_actions             |    0x0 |       0 |       0 |         
+-------+-------+---------------------------+---------------------------+--------+---------+--------
  |     4 |     1 | in_port=1,                | output:2,output:11,       |    0x1 |       0 |       0 
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  | output:12                 |        |         |         |           |           
+-------+-------+---------------------------+---------------------------+--------+---------+--------
  |     4 |     3 | in_port=1,dl_vlan=1       | output:2,                 |    0x4 |      35 |    
  |       |       |                           | set_field:5518->vlan_vid, |        |         |         |           
  |       |       |                           | output:12                 |        |         |         |           |           
+-------+-------+---------------------------+---------------------------+--------+---------+--------
  |     4 |     4 | in_port=1,dl_vlan=1,      | output:2,                 |    0x4 |    1126 |   
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  | set_field:5518->vlan_vid, |        |         |         |           
  |       |       |                           | output:12                 |        |         |         |           |           
+-------+-------+---------------------------+---------------------------+--------+---------+--------
  |     4 |     0 | in_port=2                 | clear_actions             |    0x0 |       0 |       0 |         
+-------+-------+---------------------------+---------------------------+--------+---------+--------
  |     4 |     1 | in_port=2,                | output:1,output:11,       |    0x1 |       0 |       0 
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  | output:12                 |        |         |         |           |           
+-------+-------+---------------------------+---------------------------+--------+---------+--------
  |     4 |     3 | in_port=2,dl_vlan=1       | output:1,                 |    0x4 |      11 |     
  |       |       |                           | set_field:5518->vlan_vid, |        |         |         |           
  |       |       |                           | output:12                 |        |         |         |           |           
+-------+-------+---------------------------+---------------------------+--------+---------+--------
  |     4 |     4 | in_port=2,dl_vlan=1,      | output:1,                 |    0x4 |    1110 |   
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  | set_field:5518->vlan_vid, |        |         |         |           
  |       |       |                           | output:12                 |        |         |         |           |           
+-------+-------+---------------------------+---------------------------+--------+---------+--------
  |     4 |     0 | in_port=11                | clear_actions             |    0x0 |       0 |       0 |         
+-------+-------+---------------------------+---------------------------+--------+---------+--------
  |     4 |     1 | in_port=11,               | output:1,output:2,        |    0x1 |       0 |       0 
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  | output:12                 |        |         |         |           |           
+-------+-------+---------------------------+---------------------------+--------+---------+--------
  |     4 |     1 | in_port=12,               | output:1,output:2,        |    0x1 |       0 |       0 
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  | output:11                 |        |         |         |           |           
+-------+-------+---------------------------+---------------------------+--------+---------+--------
  |     4 |     3 | in_port=12,dl_vlan=1      | output:1,output:2         |    0x4 |      24 |     +-------+-------+---------------------------+---------------------------+--------+---------+--------
  |     4 |     4 | in_port=12,dl_vlan=1,     | output:1,output:2         |    0x4 |       0 |       
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  |                           |        |         |         |           |           
 +-------+-------+---------------------------+---------------------------+--------+---------+--------

[mcevik0]

Let's save the flow tables on all switches first. Then, we will perform some preliminary tests, and tests with this policy afterwards. However, we also need the CTAG tunnels include VLAN range 1510-1559, so recreating the tunnels and re-deploying is necessary.

Current policy

[root@atlanticwave-sdx-controller script-sdx]# ./curl-0.sh -c cookie-mcevik.txt -o get_policies

============================================================================== 
--- Get POLICY - http://atlanticwave-sdx-controller.renci.ben:5000/api/v1/policies
============================================================================== 
{
  "href": "http://atlanticwave-sdx-controller.renci.ben:5000/api/v1/policies",
  "links": {
    "policy2": {
      "href": "http://atlanticwave-sdx-controller.renci.ben:5000/api/v1/policies/number/2",
      "policynumber": 2,
      "type": "FloodTree",
      "user": "SDXCTLR"
    },
    "policy3": {
      "href": "http://atlanticwave-sdx-controller.renci.ben:5000/api/v1/policies/number/3",
      "policynumber": 3,
      "type": "EdgePort",
      "user": "SDXCTLR"
    },
    "policy4": {
      "href": "http://atlanticwave-sdx-controller.renci.ben:5000/api/v1/policies/number/4",
      "policynumber": 4,
      "type": "EdgePort",
      "user": "SDXCTLR"
    },
    "policy5": {
      "href": "http://atlanticwave-sdx-controller.renci.ben:5000/api/v1/policies/number/5",
      "policynumber": 5,
      "type": "EdgePort",
      "user": "SDXCTLR"
    },
    "policy6": {
      "href": "http://atlanticwave-sdx-controller.renci.ben:5000/api/v1/policies/number/6",
      "policynumber": 6,
      "type": "EdgePort",
      "user": "SDXCTLR"
    },
    "policy7": {
      "href": "http://atlanticwave-sdx-controller.renci.ben:5000/api/v1/policies/number/7",
      "policynumber": 7,
      "type": "EdgePort",
      "user": "SDXCTLR"
    },
    "policy8": {
      "href": "http://atlanticwave-sdx-controller.renci.ben:5000/api/v1/policies/number/8",
      "policynumber": 8,
      "type": "L2Multipoint",
      "user": "mcevik"
    }
  }
}
[root@atlanticwave-sdx-controller script-sdx]# ./curl-0.sh -c cookie-mcevik.txt -o get_policy -N 8

============================================================================== 
--- Get POLICY - http://atlanticwave-sdx-controller.renci.ben:5000/api/v1/policies/number/8
============================================================================== 
{
  "policy8": {
    "href": "http://atlanticwave-sdx-controller.renci.ben:5000/api/v1/policies/number/8",
    "json": {
      "L2Multipoint": {
        "bandwidth": 8000,
        "endpoints": [
          {
            "port": 12,
            "switch": "uncs1",
            "vlan": 1423
          },
          {
            "port": 8,
            "switch": "rencis2",
            "vlan": 1425
          },
          {
            "port": 12,
            "switch": "ncsus1",
            "vlan": 1424
          },
          {
            "port": 12,
            "switch": "dukes1",
            "vlan": 1422
          }
        ],
        "endtime": "2025-10-09T15:59:00",
        "starttime": "2020-09-04T10:00:00"
      }
    },
    "policynumber": "8",
    "type": "L2Multipoint",
    "user": "mcevik"
  }
}

Flow Tables

# RENCI-1

corsa-2# show openflow flow br21
                                                                 count : 27
  +-------+-------+---------------------------+---------------------------+
  | table | prio  |           match           |          actions          |
  +-------+-------+---------------------------+---------------------------+
  |     0 |     0 | -                         | goto_table:1              |
  +-------+-------+---------------------------+---------------------------+
  |     0 |   100 | in_port=1,dl_vlan=1411    | output:2,output:11,       |
  |       |       |                           | output:23,output:30       |
  +-------+-------+---------------------------+---------------------------+
  |     0 |   100 | in_port=2,dl_vlan=1411    | output:1,output:11,       |
  |       |       |                           | output:23,output:30       |
  +-------+-------+---------------------------+---------------------------+
  |     0 |   100 | in_port=11,dl_vlan=1411   | output:1,output:2,        |
  |       |       |                           | output:23,output:30       |
  +-------+-------+---------------------------+---------------------------+
  |     0 |   100 | in_port=23,dl_vlan=1411   | output:1,output:2,        |
  |       |       |                           | output:11,output:30       |
  +-------+-------+---------------------------+---------------------------+
  |     0 |   100 | in_port=30,dl_vlan=1411   | output:1,output:2,        |
  |       |       |                           | output:11,output:23       |
  +-------+-------+---------------------------+---------------------------+
  |     1 |     0 | -                         | goto_table:2              |
  +-------+-------+---------------------------+---------------------------+
  |     2 |     0 | -                         | goto_table:3              |
  +-------+-------+---------------------------+---------------------------+
  |     3 |     0 | -                         | goto_table:4              |
  +-------+-------+---------------------------+---------------------------+
  |     3 |     0 | in_port=11                | CONTROLLER:65509,         |
  |       |       |                           | goto_table:4              |
  +-------+-------+---------------------------+---------------------------+
  |     3 |     0 | in_port=12                | CONTROLLER:65509,         |
  |       |       |                           | goto_table:4              |
  +-------+-------+---------------------------+---------------------------+
  |     3 |     0 | in_port=30                | CONTROLLER:65509,         |
  |       |       |                           | goto_table:4              |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     0 | -                         | clear_actions             |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     0 | in_port=1                 | clear_actions             |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     1 | in_port=1,                | output:2,output:11,       |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  | output:12,output:30,      |
  |       |       |                           | output:23                 |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     3 | in_port=1,dl_vlan=1       | output:23,output:2        |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     0 | in_port=2                 | clear_actions             |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     1 | in_port=2,                | output:11,output:12,      |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  | output:30,output:23,      |
  |       |       |                           | output:1                  |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     3 | in_port=2,dl_vlan=1       | output:23,output:1        |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     0 | in_port=11                | clear_actions             |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     1 | in_port=11,               | output:2,output:12,       |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  | output:30,output:23,      |
  |       |       |                           | output:1                  |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     1 | in_port=12,               | output:2,output:11,       |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  | output:30,output:23,      |
  |       |       |                           | output:1                  |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     0 | in_port=23                | clear_actions             |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     1 | in_port=23,               | output:2,output:11,       |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  | output:12,output:30,      |
  |       |       |                           | output:1                  |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     3 | in_port=23,dl_vlan=1      | output:2,output:1         |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     0 | in_port=30                | clear_actions             |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     1 | in_port=30,               | output:2,output:11,       |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  | output:12,output:23,      |
  |       |       |                           | output:1                  |
  +-------+-------+---------------------------+---------------------------+
corsa-2# show bridge br21 tunnel 
                                                                                       count : 8
  +--------+---------+-------------+-------+-------+--------+--------+-------+-------+---------+
  | ofport | ifdescr |    type     | port  | vlan  | tclass |  tpid  | inner | oper  | v-range |
  +--------+---------+-------------+-------+-------+--------+--------+-------+-------+---------+
  |      1 |         | vlan-range  |     1 |     - |      0 | 0x8100 |     - |  up   | 1-1499  |
  |      2 |         | vlan-range  |     2 |     - |      0 | 0x8100 |     - |  up   | 1-1499  |
  |     11 |         | vlan-range  |    11 |     - |      0 | 0x8100 |     - |  up   | 1-1499  |
  |     12 |         | vlan-range  |    12 |     - |      0 | 0x8100 |     - |  up   | 1-1499  |
  |     19 |         | passthrough |    19 |     - |      0 |      - |     - |  up   | -       |
  |     20 |         | passthrough |    20 |     - |      0 |      - |     - |  up   | -       |
  |     23 |         | vlan-range  |    23 |     - |      0 | 0x8100 |     - |  up   | 1-1499  |
  |     30 |         | vlan-range  |    30 |     - |      0 | 0x8100 |     - |  up   | 1-1499  |
  +--------+---------+-------------+-------+-------+--------+--------+-------+-------+---------+

corsa-2# show bridge br19 tunnel 
Info: There are no tunnels present.

# RENCI-2

cdp2100-a00001233# show openflow flow br25
                                                                 count : 16
  +-------+-------+---------------------------+---------------------------+
  | table | prio  |           match           |          actions          |
  +-------+-------+---------------------------+---------------------------+
  |     0 |     0 | -                         | goto_table:1              |
  +-------+-------+---------------------------+---------------------------+
  |     0 |     1 | in_port=8,dl_vlan=1425    | set_field:4097->vlan_vid, |
  |       |       |                           | goto_table:1              |
  +-------+-------+---------------------------+---------------------------+
  |     0 |   100 | in_port=23,dl_vlan=1411   | drop                      |
  +-------+-------+---------------------------+---------------------------+
  |     1 |     0 | -                         | goto_table:2              |
  +-------+-------+---------------------------+---------------------------+
  |     2 |     0 | -                         | goto_table:3              |
  +-------+-------+---------------------------+---------------------------+
  |     3 |     0 | -                         | goto_table:4              |
  +-------+-------+---------------------------+---------------------------+
  |     3 |     0 | in_port=8                 | CONTROLLER:65509,         |
  |       |       |                           | goto_table:4              |
  +-------+-------+---------------------------+---------------------------+
  |     3 |     2 | in_port=8,dl_vlan=1       | CONTROLLER:65509,         |
  |       |       |                           | goto_table:4              |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     0 | -                         | clear_actions             |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     1 | in_port=8,                | output:23                 |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  |                           |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     3 | in_port=8,dl_vlan=1       | output:23                 |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     4 | in_port=8,dl_vlan=1,      | output:23                 |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  |                           |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     0 | in_port=23                | clear_actions             |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     1 | in_port=23,               | output:8                  |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  |                           |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     3 | in_port=23,dl_vlan=1      | set_field:5521->vlan_vid, |
  |       |       |                           | output:8                  |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     4 | in_port=23,dl_vlan=1,     | set_field:5521->vlan_vid, |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  | output:8                  |
  +-------+-------+---------------------------+---------------------------+
cdp2100-a00001233# show bridge br25 tunnel 
                                                                                       count : 6
  +--------+---------+-------------+-------+-------+--------+--------+-------+-------+---------+
  | ofport | ifdescr |    type     | port  | vlan  | tclass |  tpid  | inner | oper  | v-range |
  +--------+---------+-------------+-------+-------+--------+--------+-------+-------+---------+
  |      8 |         | passthrough |     2 |     - |      0 |      - |     - |  up   | -       |
  |     23 |         | vlan-range  |    23 |     - |      0 | 0x8100 |     - |  up   | 1-1499  |
  |     25 |         | passthrough |    25 |     - |      0 |      - |     - |  up   | -       |
  |     26 |         | passthrough |    26 |     - |      0 |      - |     - |  up   | -       |
  |   1425 |         | ctag        |     3 |  1425 |      0 | 0x8100 |     - |  up   | -       |
  |  10001 |         | ctag        |     4 |  1425 |      0 | 0x8100 |     - |  up   | -       |
  +--------+---------+-------------+-------+-------+--------+--------+-------+-------+---------+

cdp2100-a00001233# show bridge br19 tunnel 
                                                                                 count : 2
  +--------+---------+-------+-------+-------+--------+--------+-------+-------+---------+
  | ofport | ifdescr | type  | port  | vlan  | tclass |  tpid  | inner | oper  | v-range |
  +--------+---------+-------+-------+-------+--------+--------+-------+-------+---------+
  |   1425 |         | ctag  |    13 |  1425 |      0 | 0x8100 |     - |  up   | -       |
  |  10001 |         | ctag  |    14 |  1425 |      0 | 0x8100 |     - |  up   | -       |
  +--------+---------+-------+-------+-------+--------+--------+-------+-------+---------+

# DUKE
corsa-1# show openflow flow br22
                                                                 count : 25
  +-------+-------+---------------------------+---------------------------+
  | table | prio  |           match           |          actions          |
  +-------+-------+---------------------------+---------------------------+
  |     0 |     0 | -                         | goto_table:1              |
  +-------+-------+---------------------------+---------------------------+
  |     0 |   100 | in_port=1,dl_vlan=1411    | output:2,output:11        |
  +-------+-------+---------------------------+---------------------------+
  |     0 |   100 | in_port=2,dl_vlan=1411    | output:1,output:11        |
  +-------+-------+---------------------------+---------------------------+
  |     0 |   100 | in_port=11,dl_vlan=1411   | output:1,output:2         |
  +-------+-------+---------------------------+---------------------------+
  |     0 |     1 | in_port=12,dl_vlan=1422   | set_field:4097->vlan_vid, |
  |       |       |                           | goto_table:1              |
  +-------+-------+---------------------------+---------------------------+
  |     1 |     0 | -                         | goto_table:2              |
  +-------+-------+---------------------------+---------------------------+
  |     2 |     0 | -                         | goto_table:3              |
  +-------+-------+---------------------------+---------------------------+
  |     3 |     0 | -                         | goto_table:4              |
  +-------+-------+---------------------------+---------------------------+
  |     3 |     0 | in_port=11                | CONTROLLER:65509,         |
  |       |       |                           | goto_table:4              |
  +-------+-------+---------------------------+---------------------------+
  |     3 |     0 | in_port=12                | CONTROLLER:65509,         |
  |       |       |                           | goto_table:4              |
  +-------+-------+---------------------------+---------------------------+
  |     3 |     2 | in_port=12,dl_vlan=1      | CONTROLLER:65509,         |
  |       |       |                           | goto_table:4              |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     0 | -                         | clear_actions             |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     0 | in_port=1                 | clear_actions             |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     1 | in_port=1,                | output:2,output:11,       |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  | output:12                 |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     3 | in_port=1,dl_vlan=1       | output:2,                 |
  |       |       |                           | set_field:5518->vlan_vid, |
  |       |       |                           | output:12                 |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     4 | in_port=1,dl_vlan=1,      | output:2,                 |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  | set_field:5518->vlan_vid, |
  |       |       |                           | output:12                 |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     0 | in_port=2                 | clear_actions             |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     1 | in_port=2,                | output:1,output:11,       |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  | output:12                 |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     3 | in_port=2,dl_vlan=1       | output:1,                 |
  |       |       |                           | set_field:5518->vlan_vid, |
  |       |       |                           | output:12                 |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     4 | in_port=2,dl_vlan=1,      | output:1,                 |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  | set_field:5518->vlan_vid, |
  |       |       |                           | output:12                 |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     0 | in_port=11                | clear_actions             |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     1 | in_port=11,               | output:1,output:2,        |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  | output:12                 |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     1 | in_port=12,               | output:1,output:2,        |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  | output:11                 |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     3 | in_port=12,dl_vlan=1      | output:1,output:2         |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     4 | in_port=12,dl_vlan=1,     | output:1,output:2         |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  |                           |
  +-------+-------+---------------------------+---------------------------+
corsa-1# show bridge br22 tunnel 
                                                                                       count : 8
  +--------+---------+-------------+-------+-------+--------+--------+-------+-------+---------+
  | ofport | ifdescr |    type     | port  | vlan  | tclass |  tpid  | inner | oper  | v-range |
  +--------+---------+-------------+-------+-------+--------+--------+-------+-------+---------+
  |      1 |         | vlan-range  |     1 |     - |      0 | 0x8100 |     - |  up   | 1-1499  |
  |      2 |         | vlan-range  |     2 |     - |      0 | 0x8100 |     - |  up   | 1-1499  |
  |     11 |         | vlan-range  |    11 |     - |      0 | 0x8100 |     - |  up   | 1-1499  |
  |     12 |         | vlan-range  |    12 |     - |      0 | 0x8100 |     - |  up   | 1-1499  |
  |     19 |         | passthrough |    19 |     - |      0 |      - |     - |  up   | -       |
  |     20 |         | passthrough |    20 |     - |      0 |      - |     - |  up   | -       |
  |   1422 |         | ctag        |     5 |  1422 |      0 | 0x8100 |     - |  up   | -       |
  |  10001 |         | ctag        |     6 |  1422 |      0 | 0x8100 |     - |  up   | -       |
  +--------+---------+-------------+-------+-------+--------+--------+-------+-------+---------+

corsa-1# show bridge br19 tunnel 
                                                                                 count : 2
  +--------+---------+-------+-------+-------+--------+--------+-------+-------+---------+
  | ofport | ifdescr | type  | port  | vlan  | tclass |  tpid  | inner | oper  | v-range |
  +--------+---------+-------+-------+-------+--------+--------+-------+-------+---------+
  |   1422 |         | ctag  |     7 |  1422 |      0 | 0x8100 |     - |  up   | -       |
  |  10001 |         | ctag  |     8 |  1422 |      0 | 0x8100 |     - |  up   | -       |
  +--------+---------+-------+-------+-------+--------+--------+-------+-------+---------+

# UNC
corsa-1# show openflow flow br23
                                                                 count : 20
  +-------+-------+---------------------------+---------------------------+
  | table | prio  |           match           |          actions          |
  +-------+-------+---------------------------+---------------------------+
  |     0 |     0 | -                         | goto_table:1              |
  +-------+-------+---------------------------+---------------------------+
  |     0 |   100 | in_port=1,dl_vlan=1411    | output:11                 |
  +-------+-------+---------------------------+---------------------------+
  |     0 |   100 | in_port=11,dl_vlan=1411   | output:1                  |
  +-------+-------+---------------------------+---------------------------+
  |     0 |     1 | in_port=12,dl_vlan=1423   | set_field:4097->vlan_vid, |
  |       |       |                           | goto_table:1              |
  +-------+-------+---------------------------+---------------------------+
  |     1 |     0 | -                         | goto_table:2              |
  +-------+-------+---------------------------+---------------------------+
  |     2 |     0 | -                         | goto_table:3              |
  +-------+-------+---------------------------+---------------------------+
  |     3 |     0 | -                         | goto_table:4              |
  +-------+-------+---------------------------+---------------------------+
  |     3 |     0 | in_port=11                | CONTROLLER:65509,         |
  |       |       |                           | goto_table:4              |
  +-------+-------+---------------------------+---------------------------+
  |     3 |     0 | in_port=12                | CONTROLLER:65509,         |
  |       |       |                           | goto_table:4              |
  +-------+-------+---------------------------+---------------------------+
  |     3 |     2 | in_port=12,dl_vlan=1      | CONTROLLER:65509,         |
  |       |       |                           | goto_table:4              |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     0 | -                         | clear_actions             |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     0 | in_port=1                 | clear_actions             |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     1 | in_port=1,                | output:12,output:11       |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  |                           |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     3 | in_port=1,dl_vlan=1       | set_field:5519->vlan_vid, |
  |       |       |                           | output:12                 |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     4 | in_port=1,dl_vlan=1,      | set_field:5519->vlan_vid, |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  | output:12                 |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     0 | in_port=11                | clear_actions             |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     1 | in_port=11,               | output:1,output:12        |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  |                           |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     1 | in_port=12,               | output:1,output:11        |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  |                           |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     3 | in_port=12,dl_vlan=1      | output:1                  |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     4 | in_port=12,dl_vlan=1,     | output:1                  |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  |                           |
  +-------+-------+---------------------------+---------------------------+
corsa-1# show bridge br23 tunnel 
                                                                                       count : 8
  +--------+---------+-------------+-------+-------+--------+--------+-------+-------+---------+
  | ofport | ifdescr |    type     | port  | vlan  | tclass |  tpid  | inner | oper  | v-range |
  +--------+---------+-------------+-------+-------+--------+--------+-------+-------+---------+
  |      1 |         | vlan-range  |     1 |     - |      0 | 0x8100 |     - |  up   | 1-1499  |
  |      2 |         | vlan-range  |     2 |     - |      0 | 0x8100 |     - |  up   | 1-1499  |
  |     11 |         | vlan-range  |    11 |     - |      0 | 0x8100 |     - |  up   | 1-1499  |
  |     12 |         | vlan-range  |    12 |     - |      0 | 0x8100 |     - |  up   | 1-1499  |
  |     19 |         | passthrough |    19 |     - |      0 |      - |     - |  up   | -       |
  |     20 |         | passthrough |    20 |     - |      0 |      - |     - |  up   | -       |
  |   1423 |         | ctag        |     5 |  1423 |      0 | 0x8100 |     - |  up   | -       |
  |  10001 |         | ctag        |     6 |  1423 |      0 | 0x8100 |     - |  up   | -       |
  +--------+---------+-------------+-------+-------+--------+--------+-------+-------+---------+

corsa-1# show bridge br19 tunnel
                                                                                 count : 2
  +--------+---------+-------+-------+-------+--------+--------+-------+-------+---------+
  | ofport | ifdescr | type  | port  | vlan  | tclass |  tpid  | inner | oper  | v-range |
  +--------+---------+-------+-------+-------+--------+--------+-------+-------+---------+
  |   1423 |         | ctag  |     7 |  1423 |      0 | 0x8100 |     - |  up   | -       |
  |  10001 |         | ctag  |     8 |  1423 |      0 | 0x8100 |     - |  up   | -       |
  +--------+---------+-------+-------+-------+--------+--------+-------+-------+---------+

# NCSU
corsa-1# show openflow flow br24 
                                                                 count : 20
  +-------+-------+---------------------------+---------------------------+
  | table | prio  |           match           |          actions          |
  +-------+-------+---------------------------+---------------------------+
  |     0 |     0 | -                         | goto_table:1              |
  +-------+-------+---------------------------+---------------------------+
  |     0 |   100 | in_port=1,dl_vlan=1411    | output:11                 |
  +-------+-------+---------------------------+---------------------------+
  |     0 |   100 | in_port=11,dl_vlan=1411   | output:1                  |
  +-------+-------+---------------------------+---------------------------+
  |     0 |     1 | in_port=12,dl_vlan=1424   | set_field:4097->vlan_vid, |
  |       |       |                           | goto_table:1              |
  +-------+-------+---------------------------+---------------------------+
  |     1 |     0 | -                         | goto_table:2              |
  +-------+-------+---------------------------+---------------------------+
  |     2 |     0 | -                         | goto_table:3              |
  +-------+-------+---------------------------+---------------------------+
  |     3 |     0 | -                         | goto_table:4              |
  +-------+-------+---------------------------+---------------------------+
  |     3 |     0 | in_port=11                | CONTROLLER:65509,         |
  |       |       |                           | goto_table:4              |
  +-------+-------+---------------------------+---------------------------+
  |     3 |     0 | in_port=12                | CONTROLLER:65509,         |
  |       |       |                           | goto_table:4              |
  +-------+-------+---------------------------+---------------------------+
  |     3 |     2 | in_port=12,dl_vlan=1      | CONTROLLER:65509,         |
  |       |       |                           | goto_table:4              |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     0 | -                         | clear_actions             |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     0 | in_port=1                 | clear_actions             |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     1 | in_port=1,                | output:11,output:12       |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  |                           |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     3 | in_port=1,dl_vlan=1       | set_field:5520->vlan_vid, |
  |       |       |                           | output:12                 |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     4 | in_port=1,dl_vlan=1,      | set_field:5520->vlan_vid, |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  | output:12                 |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     0 | in_port=11                | clear_actions             |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     1 | in_port=11,               | output:1,output:12        |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  |                           |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     1 | in_port=12,               | output:1,output:11        |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  |                           |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     3 | in_port=12,dl_vlan=1      | output:1                  |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     4 | in_port=12,dl_vlan=1,     | output:1                  |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  |                           |
  +-------+-------+---------------------------+---------------------------+
corsa-1# show bridge br24 tunnel
                                                                                       count : 7
  +--------+---------+-------------+-------+-------+--------+--------+-------+-------+---------+
  | ofport | ifdescr |    type     | port  | vlan  | tclass |  tpid  | inner | oper  | v-range |
  +--------+---------+-------------+-------+-------+--------+--------+-------+-------+---------+
  |      1 |         | vlan-range  |     1 |     - |      0 | 0x8100 |     - |  up   | 1-1499  |
  |     11 |         | vlan-range  |    11 |     - |      0 | 0x8100 |     - |  up   | 1-1499  |
  |     12 |         | vlan-range  |    12 |     - |      0 | 0x8100 |     - |  up   | 1-1499  |
  |     19 |         | passthrough |    19 |     - |      0 |      - |     - |  up   | -       |
  |     20 |         | passthrough |    20 |     - |      0 |      - |     - |  up   | -       |
  |   1424 |         | ctag        |     5 |  1424 |      0 | 0x8100 |     - |  up   | -       |
  |  10001 |         | ctag        |     6 |  1424 |      0 | 0x8100 |     - |  up   | -       |
  +--------+---------+-------------+-------+-------+--------+--------+-------+-------+---------+
corsa-1# show bridge br19 tunnel
                                                                                 count : 2
  +--------+---------+-------+-------+-------+--------+--------+-------+-------+---------+
  | ofport | ifdescr | type  | port  | vlan  | tclass |  tpid  | inner | oper  | v-range |
  +--------+---------+-------+-------+-------+--------+--------+-------+-------+---------+
  |   1424 |         | ctag  |     7 |  1424 |      0 | 0x8100 |     - |  up   | -       |
  |  10001 |         | ctag  |     8 |  1424 |      0 | 0x8100 |     - |  up   | -       |
  +--------+---------+-------+-------+-------+--------+--------+-------+-------+---------+

[mcevik0]

Note that flow tables seen on the outputs above show that L2MP-Ratelimiting-VFC is not used during the tests (although dynamic tunnels are seen). Current flow tables are different (include extra flows) for ratelimiting.

[mcevik0]

The problem is about the flows that forward traffic to multiple ports AND do VLAN translation.

This is the output from a packet capture.

NCSU-DTN is running ping 10.14.20.3
On UNC-DTN tcpdump -i em2 -nnS -e vlan shows the packets below. Note the VLAN tag 1422 which is supposed to be 1423

19:24:35.888677 c8:1f:66:f0:9c:cd > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 1422, p 0, ethertype ARP, Request who-has 10.14.20.3 tell 10.14.20.4, length 46
19:24:36.890033 c8:1f:66:f0:9c:cd > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 1422, p 0, ethertype ARP, Request who-has 10.14.20.3 tell 10.14.20.4, length 46
19:24:37.892032 c8:1f:66:f0:9c:cd > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 1422, p 0, ethertype ARP, Request who-has 10.14.20.3 tell 10.14.20.4, length 46

Next, NCSU-DTN is running ping 10.14.20.2

On UNC-DTN tcpdump -i em2 -nnS -e vlan shows the packets below. Note the VLAN tag which is 1423

1423, p 0, ethertype IPv4, 10.14.20.2 > 10.14.20.4: ICMP echo reply, id 21325, seq 1, length 64
19:40:54.378829 c8:1f:66:e7:21:3b > c8:1f:66:f0:9c:cd, ethertype 802.1Q (0x8100), length 102: vlan 1423, p 0, ethertype IPv4, 10.14.20.2 > 10.14.20.4: ICMP echo reply, id 21325, seq 2, length 64
19:40:55.378972 c8:1f:66:e7:21:3b > c8:1f:66:f0:9c:cd, ethertype 802.1Q (0x8100), length 102: vlan 1423, p 0, ethertype IPv4, 10.14.20.2 > 10.14.20.4: ICMP echo reply, id 21325, seq 3, length 64
19:40:56.379974 c8:1f:66:e7:21:3b > c8:1f:66:f0:9c:cd, ethertype 802.1Q (0x8100), length 102: vlan 

This traffic is forwarded through DUKE switch and hitting the flows below.

  +-------+-------+---------------------------+---------------------------+
  |     4 |     3 | in_port=1,dl_vlan=1       | output:2,                 |
  |       |       |                           | set_field:5518->vlan_vid, |
  |       |       |                           | output:12                 |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     4 | in_port=1,dl_vlan=1,      | output:2,                 |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  | set_field:5518->vlan_vid, |
  |       |       |                           | output:12                 |
  +-------+-------+---------------------------+---------------------------+

and 
  +-------+-------+---------------------------+---------------------------+
  |     4 |     3 | in_port=2,dl_vlan=1       | output:1,                 |
  |       |       |                           | set_field:5518->vlan_vid, |
  |       |       |                           | output:12                 |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     4 | in_port=2,dl_vlan=1,      | output:1,                 |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  | set_field:5518->vlan_vid, |
  |       |       |                           | output:12                 |
  +-------+-------+---------------------------+---------------------------+

Note the flows that forward traffic and do VLAN translation. This is not working as assumed whereas without VLAN translation it works. I changed the flows and used a group entry, and it worked well.

OpenFlow specification has some information about Group tables, but I haven't seen a specific item that matches this case yet, probably the most relevant explanation can be on Page 98 of the OpenFlow Specification on B.9.2 Groups

This is not about rate limiting, but the way forwarding to multiple ports is implemented (with VLAN translation case)

So, when the code is run without rate limiting the working state of flows is below (on DUKE switch which is forwarding to multiple ports and doing VLAN translation for its edge DTN. The same case should be valid for RENCI-1 switch and its DTN).

corsa-1# show openflow group br22 
                                                      count : 2
  +----+----------+-------------------------------------------+
  | id |   type   |                  buckets                  |
  +----+----------+-------------------------------------------+
  |  0 | indirect | bucket=set_field:5518->vlan_vid,output:12 |
  |  1 | indirect | bucket=set_field:5518->vlan_vid,output:12 |
  +----+----------+-------------------------------------------+

# 
# Modified flows
# 

 +-------+-------+---------------------------+---------------------------+  
  |     4 |     3 | in_port=1,dl_vlan=1       | group:1,output:2          |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     4 | in_port=1,dl_vlan=1,      | group:1,output:2          |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  |                           |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     3 | in_port=2,dl_vlan=1       | group:0,output:1          |
  +-------+-------+---------------------------+---------------------------+
  |     4 |     4 | in_port=2,dl_vlan=1,      | group:0,output:1          |
  |       |       | dl_dst=ff:ff:ff:ff:ff:ff  |                           |
  +-------+-------+---------------------------+---------------------------+

All sites can be reached from NCSU-DTN

[root@bf40g1 ~]# hostname
bf40g1.ncsu.ben
[root@bf40g1 ~]# fping 10.14.20.3 10.14.20.2 10.14.20.5 
10.14.20.3 is alive
10.14.20.2 is alive
10.14.20.5 is alive

Same ping test performed for all sites, works well to reach the rest of the DTNs.

When L2MP ratelimiting is enabled, changes below are applied to the flow table to implement forwarding to multiple ports

openflow group br22 add 0 indirect buckets bucket=set_field:5518->vlan_vid,output:10001

openflow flow br22 modify 4 priority 3 match in_port=1,dl_vlan=1 action group:0,output:2
openflow flow br22 modify 4 priority 4 match in_port=1,dl_vlan=1,dl_dst=ff:ff:ff:ff:ff:ff action group:0,output:2
openflow flow br22 modify 4 priority 3 match in_port=2,dl_vlan=1 action group:0,output:1
openflow flow br22 modify 4 priority 4 match in_port=2,dl_vlan=1,dl_dst=ff:ff:ff:ff:ff:ff action group:0,output:1

openflow flow br22 modify 4 priority 3 match in_port=10001,dl_vlan=1 action output:2,output:1
openflow flow br22 modify 4 priority 4 match in_port=10001,dl_vlan=1,dl_dst=ff:ff:ff:ff:ff:ff action output:2,output:1

For the last 2 flows, code change is made on 86da8e3
(Reference doc that describes flow numbers is AW-SDX - Design - L2Multipoint Rate-Limiting)

I think that the remaining code change will be for implementing the group table in localctlr/RyuTranslateInterface.py

[mcevik0]

Some initial thoughts I shared on Slack and I want to post here

It seems to me that Flow.4 and Flow.5 can be wrapped into a group table. (I already changed Flow.8 and Flow.9)
https://ryu.readthedocs.io/en/latest/ofproto_v1_3_ref.html#ryu.ofproto.ofproto_v1_3_parser.OFPGroupMod can be used similar to add_flow. Group table ID can be selected as the intermediate_vlan tag to identify it.

[mcevik0]

Another reference for groups https://floodlight.atlassian.net/wiki/spaces/floodlightcontroller/pages/7995427/How+to+Work+with+Fast-Failover+OpenFlow+Groups

[YufengXin]

-----Seeing this error msg at duke Corsa:

show log openflow

2020-11-01T20:33:24.678162-05:00 corsa-1 ovs-vswitchd: br22 - openflow
2020-11-01T20:33:24.678169-05:00 corsa-1 ovs-vswitchd: group add - group: 0 - type indirect, bucket[0]: set_field:5518->vlan_vid,output:12,output:1
2020-11-01T20:33:24.678172-05:00 corsa-1 ovs-vswitchd: action_error (OFPBAC_BAD_OUT_PORT) - only a single output action is supported