ofport 10001 and 142x is not properly removed after a P2MP connection.

[congwang09]

The ofport 10001 and 142x is not properly removed after a P2MP connection.

These ports will have to be manually removed. Otherwise an error will occur for the next request:

2020-10-27 17:48:11,002 localcontroller: 140111241176896 DEBUG    Received a INSTL message from 0x7f6e29df4c88
Received a INSTL message from 0x7f6e29df4c88
2020-10-27 17:48:11,002 localcontroller: 140111241176896 DEBUG    install_rule_sdxmsg: 7:203:EdgePortLCRule: switch 203, 7:11
install_rule_sdxmsg: 7:203:EdgePortLCRule: switch 203, 7:11
2020-10-27 17:48:11,009 localcontroller.ryucontrollerinterface: 140111241176896 DEBUG    Sending  new cmd to RTI: 203:EdgePortLCRule: switch 203, 7:11
Sending  new cmd to RTI: 203:EdgePortLCRule: switch 203, 7:11
Cookie 0x02 used in datapath 203 for EdgePortLCRule: switch 203, 7:11
SDX Rule EdgePortLCRule: switch 203, 7:11 being installed in datapath 203
Registering cookie 0x02 to function <bound method RyuTranslateInterface.unknown_source_cb of <RyuTranslateInterface.RyuTranslateInterface object at 0x7f7243e9afd0>> for packet_in handling
Inserting into switch table 3 switch rules [2:OFPMatch(oxm_fields={'in_port': 11}):[OFPInstructionGotoTable(len=8,table_id=4,type=1), OFPInstructionActions(actions=[OFPActionOutput(len=16,max_len=65509,port=4294967293,type=0)],type=4)]]
2020-10-27 17:48:11,014 localcontroller: 140111241176896 DEBUG    Received a INSTL message from 0x7f6e29df4c88
Received a INSTL message from 0x7f6e29df4c88
2020-10-27 17:48:11,014 localcontroller: 140111241176896 DEBUG    install_rule_sdxmsg: 7:203:EdgePortLCRule: switch 203, 7:12
install_rule_sdxmsg: 7:203:EdgePortLCRule: switch 203, 7:12
Calling add_flow on the following:
  2:3:0
OFPMatch(oxm_fields={'in_port': 11})
[OFPInstructionGotoTable(len=8,table_id=4,type=1), OFPInstructionActions(actions=[OFPActionOutput(len=16,max_len=65509,port=4294967293,type=0)],type=4)]
None:0:0
add_flow for 2:3:0:OFPMatch(oxm_fields={'in_port': 11}):[OFPInstructionGotoTable(len=8,table_id=4,type=1), OFPInstructionActions(actions=[OFPActionOutput(len=16,max_len=65509,port=4294967293,type=0)],type=4)]
2020-10-27 17:48:11,019 localcontroller.ryucontrollerinterface: 140111241176896 DEBUG    Sending  new cmd to RTI: 203:EdgePortLCRule: switch 203, 7:12
Sending  new cmd to RTI: 203:EdgePortLCRule: switch 203, 7:12
2020-10-27 17:48:11,021 localcontroller.ryutranslateinterface: 140128675304712 ERROR    Existing sdxcookie: 7, switchid: 203
Existing sdxcookie: 7, switchid: 203
Cookie 0x03 used in datapath 203 for EdgePortLCRule: switch 203, 7:12
SDX Rule EdgePortLCRule: switch 203, 7:12 being installed in datapath 203
Registering cookie 0x03 to function <bound method RyuTranslateInterface.unknown_source_cb of <RyuTranslateInterface.RyuTranslateInterface object at 0x7f7243e9afd0>> for packet_in handling
Inserting into switch table 3 switch rules [3:OFPMatch(oxm_fields={'in_port': 12}):[OFPInstructionGotoTable(len=8,table_id=4,type=1), OFPInstructionActions(actions=[OFPActionOutput(len=16,max_len=65509,port=4294967293,type=0)],type=4)]]
Calling add_flow on the following:
  3:3:0
OFPMatch(oxm_fields={'in_port': 12})
[OFPInstructionGotoTable(len=8,table_id=4,type=1), OFPInstructionActions(actions=[OFPActionOutput(len=16,max_len=65509,port=4294967293,type=0)],type=4)]
None:0:0
add_flow for 3:3:0:OFPMatch(oxm_fields={'in_port': 12}):[OFPInstructionGotoTable(len=8,table_id=4,type=1), OFPInstructionActions(actions=[OFPActionOutput(len=16,max_len=65509,port=4294967293,type=0)],type=4)]
140111241176896 hb_response_handler: HBRESP: ['MAIN_PHASE']-None-None
140111241176896 hb_request_handler: HBREQ: ['MAIN_PHASE']-None-None
140111241176896 hb_response_handler: HBRESP: ['MAIN_PHASE']-None-None
2020-10-27 17:48:26,669 localcontroller: 140111241176896 DEBUG    Received a INSTL message from 0x7f6e29df4c88
Received a INSTL message from 0x7f6e29df4c88
2020-10-27 17:48:26,669 localcontroller: 140111241176896 DEBUG    install_rule_sdxmsg: 8:203:L2MultipointEndpointLCRule: switch 203, 8:([1]), ([(12, 1423)]), 1, 8000
install_rule_sdxmsg: 8:203:L2MultipointEndpointLCRule: switch 203, 8:([1]), ([(12, 1423)]), 1, 8000
2020-10-27 17:48:26,673 localcontroller.ryucontrollerinterface: 140111241176896 DEBUG    Sending  new cmd to RTI: 203:L2MultipointEndpointLCRule: switch 203, 8:([1]), ([(12, 1423)]), 1, 8000
Sending  new cmd to RTI: 203:L2MultipointEndpointLCRule: switch 203, 8:([1]), ([(12, 1423)]), 1, 8000
Cookie 0x04 used in datapath 203 for L2MultipointEndpointLCRule: switch 203, 8:([1]), ([(12, 1423)]), 1, 8000
SDX Rule L2MultipointEndpointLCRule: switch 203, 8:([1]), ([(12, 1423)]), 1, 8000 being installed in datapath 203
L2MultipointEndpointLCRule: 0,4:4:L2MultipointEndpointLCRule: switch 203, 8:([1]), ([(12, 1423)]), 1, 8000
L2MultipointEndpointLCRule: Corsa Case L2MULTIPOINTCORSABWDISABLED False 
L2MultipointEndpointLCRule: port: 12 -  vlan: 1423
L2MultipointEndpointLCRule: IN_PORT: in_port:12 -  VLAN_VID: vlan_vid:1423,1
L2MultipointEndpointLCRule: l2mp_bw_in_port: 1423 -  l2mp_bw_out_port: 10001
L2MultipointEndpointLCRule: Tunnel attach https://corsa-1.unc.ben/api/v1/bridges/br23/tunnels:{'ofport': 10001, 'port': 6, 'vlan-id': 1423, 'shaped-rate': 8000}
L2MultipointEndpointLCRule: Tunnel attach https://corsa-1.unc.ben/api/v1/bridges/br23/tunnels:{'ofport': 1423, 'port': 5, 'vlan-id': 1423, 'shaped-rate': 8000}
L2MultipointEndpointLCRule: Tunnel attach https://corsa-1.unc.ben/api/v1/bridges/br19/tunnels:{'ofport': 1423, 'port': 7, 'vlan-id': 1423, 'shaped-rate': 8000}
L2MultipointEndpointLCRule: Tunnel attach https://corsa-1.unc.ben/api/v1/bridges/br19/tunnels:{'ofport': 10001, 'port': 8, 'vlan-id': 1423, 'shaped-rate': 8000}
L2MultipointEndpointLCRule: mperule.get_flooding_ports           : [1]
L2MultipointEndpointLCRule: mperule.get_endpoint_ports_and_vlans : [(12, 1423)]
L2MultipointEndpointLCRule: ENDPOINT TABLE and LEARNING TABLE
L2MultipointEndpointLCRule: FLOOD TABLE
L2MultipointEndpointLCRule -1- : port: 1 
L2MultipointEndpointLCRule -2- : outport: 1 
L2MultipointEndpointLCRule -3- : outport: 12 
L2MultipointEndpointLCRule -4- : port: 12 
Registering cookie 0x04 to function <bound method RyuTranslateInterface.l2multipoint_unknown_source_cb of <RyuTranslateInterface.RyuTranslateInterface object at 0x7f7243e9afd0>> for packet_in handling
Inserting into switch table 0 switch rules [<RyuTranslateInterface.TranslatedCorsaRuleContainer object at 0x7f723f8fe278>, <RyuTranslateInterface.TranslatedCorsaRuleContainer object at 0x7f723f8feba8>, <RyuTranslateInterface.TranslatedCorsaRuleContainer object at 0x7f723f8be7b8>, <RyuTranslateInterface.TranslatedCorsaRuleContainer object at 0x7f723f8be780>, 4:OFPMatch(oxm_fields={'in_port': 12, 'vlan_vid': 5519}):[OFPInstructionActions(actions=[OFPActionOutput(len=16,max_len=65509,port=1423,type=0)],type=4)], 4:OFPMatch(oxm_fields={'in_port': 10001, 'vlan_vid': 5519}):[OFPInstructionActions(actions=[OFPActionSetField(vlan_vid=4097)],type=4), OFPInstructionGotoTable(len=8,table_id=1,type=1)], 4:OFPMatch(oxm_fields={'in_port': 10001, 'vlan_vid': 4097}):[OFPInstructionGotoTable(len=8,table_id=4,type=1), OFPInstructionActions(actions=[OFPActionOutput(len=16,max_len=65509,port=4294967293,type=0)],type=4)], 4:OFPMatch(oxm_fields={'in_port': 1, 'vlan_vid': 4097}):[OFPInstructionActions(actions=[OFPActionSetField(vlan_vid=5519), OFPActionOutput(len=16,max_len=65509,port=10001,type=0)],type=4)], 4:OFPMatch(oxm_fields={'in_port': 1, 'eth_dst': 'ff:ff:ff:ff:ff:ff', 'vlan_vid': 4097}):[OFPInstructionActions(actions=[OFPActionSetField(vlan_vid=5519), OFPActionOutput(len=16,max_len=65509,port=10001,type=0)],type=4)], 4:OFPMatch(oxm_fields={'in_port': 10001, 'vlan_vid': 4097}):[OFPInstructionActions(actions=[OFPActionOutput(len=16,max_len=65509,port=1,type=0)],type=4)], 4:OFPMatch(oxm_fields={'in_port': 10001, 'eth_dst': 'ff:ff:ff:ff:ff:ff', 'vlan_vid': 4097}):[OFPInstructionActions(actions=[OFPActionOutput(len=16,max_len=65509,port=1,type=0)],type=4)], 4:OFPMatch(oxm_fields={'in_port': 1423, 'vlan_vid': 5519}):[OFPInstructionActions(actions=[OFPActionOutput(len=16,max_len=65509,port=12,type=0)],type=4)], 4:OFPMatch(oxm_fields={'in_port': 1423, 'eth_dst': 'ff:ff:ff:ff:ff:ff', 'vlan_vid': 5519}):[OFPInstructionActions(actions=[OFPActionOutput(len=16,max_len=65509,port=12,type=0)],type=4)]]
Calling add_flow on the following:
  https://corsa-1.unc.ben/api/v1/bridges/br23/tunnels:{'ofport': 10001, 'port': 6, 'vlan-id': 1423, 'shaped-rate': 8000}:[201]:fa18b3d4d84507dba6568678a45fcecdec03247d7b5c18e45c5f288066a52d970cf8ee0bcf7759c698a7b56b92824963c8c03acf77f0a3aa91ad4f64c3aa7b15 - CORSA_REST_CMD
Starting new HTTPS connection (1): corsa-1.unc.ben:443
/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py:988: InsecureRequestWarning: Unverified HTTPS request is being made to host 'corsa-1.unc.ben'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
https://corsa-1.unc.ben:443 "POST /api/v1/bridges/br23/tunnels HTTP/1.1" 409 34
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/atlanticwave-proto/localctlr/RyuTranslateInterface.py", line 482, in main_loop
    self.install_rule(datapath, event)
  File "/atlanticwave-proto/localctlr/RyuTranslateInterface.py", line 1836, in install_rule
    self.corsa_rest_cmd(rule)
  File "/atlanticwave-proto/localctlr/RyuTranslateInterface.py", line 1611, in corsa_rest_cmd
    response.json()))
Exception: REST command failed post:{'ofport': 10001, 'port': 6, 'vlan-id': 1423, 'shaped-rate': 8000}
    409
    {'error': 'Entry already exists'}

Corsa bridge tunnel:

corsa-1# show bridge br24 tunnel 
                                                                                         count : 8
  +--------+---------+-------------+-------+-------+--------+--------+-------+-------+-----------+
  | ofport | ifdescr |    type     | port  | vlan  | tclass |  tpid  | inner | oper  |  v-range  |
  +--------+---------+-------------+-------+-------+--------+--------+-------+-------+-----------+
  |      1 |         | vlan-range  |     1 |     - |      0 | 0x8100 |     - | down  | 1-1599    |
  |      2 |         | vlan-range  |     2 |     - |      0 | 0x8100 |     - |  up   | 1-1599    |
  |     11 |         | vlan-range  |    11 |     - |      0 | 0x8100 |     - |  up   | 1-1599    |
  |     12 |         | vlan-range  |    12 |     - |      0 | 0x8100 |     - |  up   | 1420-1599 |
  |     19 |         | passthrough |    19 |     - |      0 |      - |     - |  up   | -         |
  |     20 |         | passthrough |    20 |     - |      0 |      - |     - |  up   | -         |
  |   1424 |         | ctag        |     5 |  1424 |      0 | 0x8100 |     - |  up   | -         |
  |  10001 |         | ctag        |     6 |  1424 |      0 | 0x8100 |     - |  up   | -         |
  +--------+---------+-------------+-------+-------+--------+--------+-------+-------+-----------+

For now, to manually remove the ports:

corsa-1# configure bridge br24 tunnel detach ofport 1424
corsa-1# configure bridge br24 tunnel detach ofport 10001
corsa-1# show bridge br24 tunnel
                                                                                         count : 6
  +--------+---------+-------------+-------+-------+--------+--------+-------+-------+-----------+
  | ofport | ifdescr |    type     | port  | vlan  | tclass |  tpid  | inner | oper  |  v-range  |
  +--------+---------+-------------+-------+-------+--------+--------+-------+-------+-----------+
  |      1 |         | vlan-range  |     1 |     - |      0 | 0x8100 |     - | down  | 1-1599    |
  |      2 |         | vlan-range  |     2 |     - |      0 | 0x8100 |     - |  up   | 1-1599    |
  |     11 |         | vlan-range  |    11 |     - |      0 | 0x8100 |     - |  up   | 1-1599    |
  |     12 |         | vlan-range  |    12 |     - |      0 | 0x8100 |     - |  up   | 1420-1599 |
  |     19 |         | passthrough |    19 |     - |      0 |      - |     - |  up   | -         |
  |     20 |         | passthrough |    20 |     - |      0 |      - |     - |  up   | -         |
  +--------+---------+-------------+-------+-------+--------+--------+-------+-------+-----------+

[mcevik0]

This was tested extensively when policies are removed those ports were being removed as well.
If the SDX/Local controllers are restarted while active polices exist, next run will encounter the reported problem.
@congwang09 - Can you confirm that SDX/Local controllers were not restarted/redeployed while any P2MP connections were active ?

[mcevik0]

A better approach can be to add a code piece to the remove all dynamic P2MP ports on the switch when LC is restarted. I can show you the places where to add that code if you need me to do that.