About unmanaging devices

[dann1]

Hi @sk4zuzu 👋

So, while testing the change e2a454f on the staging env we saw some issues

The vlan subinterface didn't get removed, the unamangement rule was created for the parent device correctly though.

[root@nfhhvmadlb05 ~]# nmcli con show
NAME           UUID                                  TYPE      DEVICE
vmnic6.741     4d6a1665-2089-4d1f-a455-b42733c8ec33  vlan      vmnic6.741
lo             a2b776aa-2a67-4fcd-8df1-aac9ba729122  loopback  lo
eno2           c8e5b936-8fb3-43c1-864c-a6e2a451aae3  ethernet  --
eno3           f3058b56-651f-4670-bd14-8f4ba88ddf79  ethernet  --
eno4           98426a7f-69a0-43a2-8c20-88e2f9c41777  ethernet  --
enp0s26u1u6u3  b2dff766-e9f7-4c23-bba8-a873e32f4e27  ethernet  --
enp131s0f0     6e12b7b6-f993-422d-adb6-445cdf4a9bc8  ethernet  --
enp131s0f1     ad9419b9-b0fc-440c-9aab-e098e40e91d3  ethernet  --
enp133s0f0     72b551e5-1dda-4a9c-9c38-bd0903610376  ethernet  --
enp133s0f1     8a039257-be30-4bb2-bb1d-37b938f4a005  ethernet  --
enp5s0f0       4a859122-28fb-4957-851f-f8b0ce6f84a9  ethernet  --
enp5s0f1       915f4fa7-272a-4728-a664-6f24e8d66b16  ethernet  --
vmnic6         41809029-122d-45c8-bbeb-db03ff0b9a12  ethernet  --
[root@nfhhvmadlb05 ~]# ethtool -i vmnic6
driver: i40e
version: 5.14.0-687.30.1.el9_8.x86_64
firmware-version: 8.40 0x8000af80 20.5.13
expansion-rom-version:
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
[root@nfhhvmadlb05 ~]# grep 0000:01:00.0 /etc/udev/rules.d/99-unmanage.rules
# 0000:01:00.0
SUBSYSTEM=="net", ACTION=="add|move", ENV{ID_PATH}=="pci-0000:01:00.0", \
SUBSYSTEM=="net", ENV{ID_PATH}=="pci-0000:01:00.0", ENV{FORCE_NM_UNMANAGED}=="1", TEST=="/usr/bin/nmcli", \
SUBSYSTEM=="net", ENV{ID_PATH}=="pci-0000:01:00.0", ENV{FORCE_NM_UNMANAGED}=="1", TEST=="/usr/sbin/netplan", \
SUBSYSTEM=="net", ACTION=="add|move", ENV{ID_PATH}=="pci-0000:01:00.0", TEST=="/usr/sbin/ip", \

When I tested this change on the dev env, unamanging the parent device led to the subinterface being unmanaged automatically, but perhaps this is because I didn't create the vlan subinterface with a network manager profile, only the parent interface.

The result is the we don't actually migrate the connection to the ovs internal interface

14: vmnic6.741@vmnic6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f8:bc:12:1c:80:10 brd ff:ff:ff:ff:ff:ff
    inet 192.168.115.52/24 brd 192.168.115.255 scope global noprefixroute vmnic6.741
       valid_lft forever preferred_lft forever
    inet6 fe80::fabc:12ff:fe1c:8010/64 scope link
       valid_lft forever preferred_lft forever
15: ovsbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 3c:fd:fe:19:4c:60 brd ff:ff:ff:ff:ff:ff
    inet 192.168.115.52/24 metric 400 scope global ovsbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::3efd:feff:fe19:4c60/64 scope link

Having said that, I think that if that device would have been managed correctly, we probably lose the SSH connection right after the udev rules trigger.

So we need to somehow delay (maybe move it) the unamangement of the OVS interfaces to the opennebula-ovs execution. The nuance comes from the fact that we reference those interfaces earlier to rename them

      pci_devices:
        - address: "0000:83:00.0"
          set_driver: omit
          set_name: vmnic0
          unguarded: true
          unlisted: true
        - address: "0000:01:00.0"
          set_driver: omit
          set_name: vmnic6
          unlisted: true
.
.
.
   ovs:
     set:
     br:
       ovsbr0:
         # set:
         ports: [bond0]
         addrs:
           - cidr: "{{ ansible_default_ipv4.address ~ '/' ~ ansible_default_ipv4.prefix }}"
             metric: 400
         gw: "{{ ansible_default_ipv4.gateway }}"

     bond:
       bond0:
         ifaces: [vmnic0, vmnic6]
         set:
           - bond_mode: balance-slb

     port:
       ovsbr0:
         set:
           - tag: 741
      iface:
       ovsbr0:
         set:
           - mtu_request: 1500
       vmnic0:
         set:
           - mtu_request: 9126
       vmnic6:
         set:
           - mtu_request: 9126

Perhaps we need, yet another flag to set unmanage: false/true to decide if these interfaces should be unmanaged or not and leave the creation of that rule to the opennebula-ovs execution after the IPs are set on the interfaces.

Then we might need to extend that dedicated unmanagement in opennebula-ovs to also unmanage VLAN subinterface connections associated to the iface used in ovs

So, as a summary

• we might want control over what gets unmanaged
  • when unmanaging, we need to check vlan subinterface connections
• the unmanagement of the interfaces used in OVS could be done by OVS
  • to cover the case of management interface migration

WDYT ? I think on this particular deployment we didn't lose the SSH connection because the vlan subinterface didn't get unmanaged, but if that had been the case, we probably would have never gotten to the OVS bit.

[sk4zuzu]

@dann1 Hi, since we need this "unmanagement" stuff in general not only for OVS, we need to keep current code. But that doesn't mean that we cannot run it again and still keep it idempotent. Let me research it. 👍😇

[sk4zuzu]

@dann1 After thinking about it a bit my conclusion is that we need:

• unmanage: t/f flag, otherwise we'd have to implement recursive processing of devices and every possible derived subdevice in UDEV (not only VLANs), which is complete overengineering (stupidity for no reason). IMHO the answer here should really be "fix your architecture".
• Then in opennebula-ovs.sh since we can extract all VLAN tags from the ovs structure and just remove all existing VLAN interfaces in the simplest/dumbest way possible.
• Then also unmanage all interfaces opennebula-ovs.sh sees in NM or NP, but without involving UDEV.

I will proceed to do exactly those things. 🤗