Skip to content

Commit db3fee6

Browse files
bcavnvidiabcavnvidia
authored
fix(api,agent): Added additional route-leaking controls (NVIDIA#1433)
## Description Adds granular route-leaking option to VPC profiles so that deployment can configure certain VPCs to leak more than just default-route from the underlay into the VPC VRF. There are user-friendly options for leaking tenant routes to the underlay and for leaking a default-route from the underlay into tenant overlays, but some use-cases need more control over leaking from default VRF to tenant VRF. Specifically, users who plan to run a very large number of number of anycast announcements for a single IP might exceed the limits of ECMP groups in HBN and even cumulus. The work-around is leaking those routes to the underlay and allowing the ingress path to such VIPs to take the underlay. On the "client" side, those VIPs routes would need to be leaked into the tenant VRF from the default VRF to prevent the client traffic from taking a default route in the overlay and hair-pinning at a border device VRF, or possibly getting dropped completely. ## Type of Change <!-- Check one that best describes this PR --> - [x] **Add** - New feature or capability - [ ] **Change** - Changes in existing functionality - [ ] **Fix** - Bug fixes - [ ] **Remove** - Removed features or deprecated functionality - [ ] **Internal** - Internal changes (refactoring, tests, docs, etc.) ## Related Issues (Optional) <!-- If applicable, provide GitHub Issue. --> ## Breaking Changes - [ ] This PR contains breaking changes <!-- If checked above, describe the breaking changes and migration steps --> ## Testing <!-- How was this tested? Check all that apply --> - [x] Unit tests added/updated - [ ] Integration tests added/updated - [ ] Manual testing performed - [ ] No testing required (docs, internal refactor, etc.) ## Additional Notes Satisfies NVIDIA#1313
1 parent fa3f208 commit db3fee6

13 files changed

Lines changed: 273 additions & 20 deletions

File tree

‎crates/agent/src/ethernet_virtualization.rs‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -465,6 +465,11 @@ pub async fn update_nvue(
465465
vni: rt.vni,
466466
})
467467
.collect(),
468+
accepted_leaks_from_underlay: rp
469+
.accepted_leaks_from_underlay
470+
.iter()
471+
.map(|l| l.prefix.to_owned())
472+
.collect(),
468473
})
469474
},
470475
bgp_leaf_session_password: nc.bgp_leaf_session_password.clone(),
@@ -2575,6 +2580,13 @@ mod tests {
25752580
leak_default_route_from_underlay: include_network_host_route_and_default_leaking,
25762581
leak_tenant_host_routes_to_underlay: include_network_host_route_and_default_leaking,
25772582
tenant_leak_communities_accepted: include_network_host_route_and_default_leaking,
2583+
accepted_leaks_from_underlay: if include_network_host_route_and_default_leaking {
2584+
vec![rpc::PrefixFilterPolicyEntry {
2585+
prefix: "10.255.0.0/24".to_string(),
2586+
}]
2587+
} else {
2588+
vec![]
2589+
},
25782590
route_target_imports: vec![rpc_common::RouteTarget {
25792591
asn: 44444,
25802592
vni: 55555,
@@ -2828,6 +2840,7 @@ mod tests {
28282840
tenant_leak_communities_accepted: false,
28292841
leak_default_route_from_underlay: false,
28302842
leak_tenant_host_routes_to_underlay: false,
2843+
accepted_leaks_from_underlay: vec![],
28312844
route_target_imports: vec![nvue::RouteTargetConfig {
28322845
asn: 44444,
28332846
vni: 55555,
@@ -3062,6 +3075,7 @@ mod tests {
30623075
tenant_leak_communities_accepted: false,
30633076
leak_default_route_from_underlay: false,
30643077
leak_tenant_host_routes_to_underlay: false,
3078+
accepted_leaks_from_underlay: vec![],
30653079
route_target_imports: vec![rpc_common::RouteTarget {
30663080
asn: 44444,
30673081
vni: 55555,

‎crates/agent/src/nvue.rs‎

Lines changed: 38 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,9 @@ const NETWORK_SECURITY_GROUP_RULE_COUNT_MAX: usize = 10000;
7171
/// into IPv4 and IPv6 buckets. Each bucket gets sequential indices
7272
/// starting at `start_index`. Unparseable prefixes are warned and
7373
/// dropped (because NVUE would fail on invalid addresses anyway).
74+
///
75+
/// start_index should usually be set to 1 to avoid nvue config complaints when
76+
/// using the output as part of policy generation.
7477
fn split_prefixes_by_family(prefixes: &[String], start_index: usize) -> (Vec<Prefix>, Vec<Prefix>) {
7578
let valid: Vec<_> = prefixes
7679
.iter()
@@ -122,13 +125,15 @@ pub fn build(conf: NvueConfig) -> eyre::Result<String> {
122125
// profiles into FlatInterfaceConfig could change that.
123126
// For now, we clone later so that we can put this into each TmplVpc
124127
// and make a later transition easier.
125-
let routing_profile = conf
126-
.ct_routing_profile
127-
.as_ref()
128-
.map(|rt| TmplRoutingProfile {
128+
let routing_profile = conf.ct_routing_profile.as_ref().map(|rt| {
129+
let (v4leaks, v6leaks) = split_prefixes_by_family(&rt.accepted_leaks_from_underlay, 1);
130+
131+
TmplRoutingProfile {
129132
TenantLeakCommunitiesAccepted: rt.tenant_leak_communities_accepted,
130133
LeakDefaultRouteFromUnderlay: rt.leak_default_route_from_underlay,
131134
LeakTenantHostRoutesToUnderlay: rt.leak_tenant_host_routes_to_underlay,
135+
AcceptedLeaksFromUnderlayIpv4: v4leaks,
136+
AcceptedLeaksFromUnderlayIpv6: v6leaks,
132137
RouteTargetImports: rt
133138
.route_target_imports
134139
.iter()
@@ -145,7 +150,8 @@ pub fn build(conf: NvueConfig) -> eyre::Result<String> {
145150
VNI: rt.vni,
146151
})
147152
.collect(),
148-
});
153+
}
154+
});
149155

150156
// There are two assumptions about pre-FNN...
151157
// 1 - ManagedHostNetworkConfigResponse only has rules inherited either from VPC or Instance,
@@ -338,6 +344,20 @@ pub fn build(conf: NvueConfig) -> eyre::Result<String> {
338344
RoutingProfile: routing_profile.clone(),
339345
PortPrefixes: port.VpcPrefixes.clone(),
340346
PortPrefixesIpv6: port.VpcPrefixesIpv6.clone(),
347+
HasLeaksFromUnderlayIpv4: routing_profile
348+
.as_ref()
349+
.map(|p| {
350+
p.LeakDefaultRouteFromUnderlay
351+
|| !p.AcceptedLeaksFromUnderlayIpv4.is_empty()
352+
})
353+
.unwrap_or_default(),
354+
HasLeaksFromUnderlayIpv6: routing_profile
355+
.as_ref()
356+
.map(|p| {
357+
p.LeakDefaultRouteFromUnderlay
358+
|| !p.AcceptedLeaksFromUnderlayIpv6.is_empty()
359+
})
360+
.unwrap_or_default(),
341361
});
342362

343363
port_configs.push(port);
@@ -981,6 +1001,7 @@ pub struct RoutingProfile {
9811001
pub route_target_imports: Vec<RouteTargetConfig>,
9821002
pub route_targets_on_exports: Vec<RouteTargetConfig>,
9831003
pub tenant_leak_communities_accepted: bool,
1004+
pub accepted_leaks_from_underlay: Vec<String>,
9841005
}
9851006

9861007
#[derive(Clone, Deserialize, Debug)]
@@ -1265,6 +1286,8 @@ struct TmplRoutingProfile {
12651286
RouteTargetImports: Vec<TmplRouteTargetConfig>,
12661287
RouteTargetsOnExports: Vec<TmplRouteTargetConfig>,
12671288
TenantLeakCommunitiesAccepted: bool,
1289+
AcceptedLeaksFromUnderlayIpv4: Vec<Prefix>,
1290+
AcceptedLeaksFromUnderlayIpv6: Vec<Prefix>,
12681291
}
12691292

12701293
#[allow(non_snake_case)]
@@ -1359,6 +1382,11 @@ struct TmplVpc {
13591382
HasVpcPeerPrefixesIpv6: bool,
13601383
VpcPeerPrefixesIpv6: Vec<Prefix>,
13611384

1385+
/// Whether there are _any_ leaks from the default
1386+
/// VRF into the tenant VRF being used.
1387+
HasLeaksFromUnderlayIpv4: bool,
1388+
HasLeaksFromUnderlayIpv6: bool,
1389+
13621390
// The relationship between interface:VPC is 1:1 but VPC:interface is 1:M.
13631391
// So, a single VPC could have multiple, per-port, VpcPrefixes. We can
13641392
// accumulate these and pass them into the template for ease-of-use.
@@ -1678,6 +1706,7 @@ mod tests {
16781706
leak_tenant_host_routes_to_underlay: false,
16791707
route_target_imports: vec![],
16801708
route_targets_on_exports: vec![],
1709+
accepted_leaks_from_underlay: vec![],
16811710
});
16821711
conf.ct_port_configs = vec![PortConfig {
16831712
interface_name: "pf0vf0_if".into(),
@@ -1758,6 +1787,7 @@ mod tests {
17581787

17591788
route_target_imports: vec![],
17601789
route_targets_on_exports: vec![],
1790+
accepted_leaks_from_underlay: vec![],
17611791
});
17621792
conf.ct_port_configs = vec![PortConfig {
17631793
interface_name: "pf0vf0_if".into(),
@@ -1841,6 +1871,7 @@ mod tests {
18411871
leak_tenant_host_routes_to_underlay: false,
18421872
route_target_imports: vec![],
18431873
route_targets_on_exports: vec![],
1874+
accepted_leaks_from_underlay: vec![],
18441875
});
18451876
conf.ct_port_configs = vec![
18461877
PortConfig {
@@ -1911,6 +1942,7 @@ mod tests {
19111942
tenant_leak_communities_accepted: false,
19121943
route_target_imports: vec![],
19131944
route_targets_on_exports: vec![],
1945+
accepted_leaks_from_underlay: vec![],
19141946
});
19151947
conf.ct_port_configs = vec![PortConfig {
19161948
interface_name: "pf0vf0_if".into(),
@@ -2072,6 +2104,7 @@ mod tests {
20722104
leak_tenant_host_routes_to_underlay: false,
20732105
route_target_imports: vec![],
20742106
route_targets_on_exports: vec![],
2107+
accepted_leaks_from_underlay: vec![],
20752108
}
20762109
}
20772110

‎crates/agent/src/tests/full.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -838,6 +838,7 @@ async fn handle_netconf(AxumState(state): AxumState<Arc<Mutex<State>>>) -> impl
838838
tenant_leak_communities_accepted: false,
839839
leak_default_route_from_underlay: false,
840840
leak_tenant_host_routes_to_underlay: false,
841+
accepted_leaks_from_underlay: vec![],
841842
route_target_imports: vec![rpc_common::RouteTarget {
842843
asn: 44444,
843844
vni: 55555,

‎crates/agent/templates/nvue_startup_fnn.conf‎

Lines changed: 33 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -296,24 +296,42 @@
296296
match:
297297
any: {}
298298
{{- range $vpc := $tenant.Vpcs }}
299-
{{- if $vpc.RoutingProfile.LeakDefaultRouteFromUnderlay }}
299+
{{- if $vpc.HasLeaksFromUnderlayIpv4 }}
300300
FROM_DEFAULT_VRF_TO_{{ $vpc.VrfName }}_PREFIX_LIST:
301301
rule:
302+
{{- range $leak := $vpc.RoutingProfile.AcceptedLeaksFromUnderlayIpv4 }}
303+
'{{ $leak.Index }}':
304+
action: permit
305+
match:
306+
'{{ $leak.Prefix }}':
307+
max-prefix-len: 32{{/* Any */}}
308+
{{- end }}
309+
{{- if $vpc.RoutingProfile.LeakDefaultRouteFromUnderlay }}
302310
'65534':
303311
action: permit
304312
match:
305313
0.0.0.0/0: {}{{/* Exact match */}}
314+
{{- end }}
306315
{{- end }}
307316
{{- end }}
308317
{{- range $vpc := $tenant.Vpcs }}
309-
{{- if $vpc.RoutingProfile.LeakDefaultRouteFromUnderlay }}
318+
{{- if $vpc.HasLeaksFromUnderlayIpv6 }}
310319
FROM_DEFAULT_VRF_TO_{{ $vpc.VrfName }}_PREFIX_LIST_IPV6:
311320
type: ipv6
312321
rule:
322+
{{- range $leak := $vpc.RoutingProfile.AcceptedLeaksFromUnderlayIpv6 }}
323+
'{{ $leak.Index }}':
324+
action: permit
325+
match:
326+
'{{ $leak.Prefix }}':
327+
max-prefix-len: 128{{/* Any */}}
328+
{{- end }}
329+
{{- if $vpc.RoutingProfile.LeakDefaultRouteFromUnderlay }}
313330
'65534':
314331
action: permit
315332
match:
316333
'::/0': {}{{/* Exact match */}}
334+
{{- end }}
317335
{{- end }}
318336
{{- end }}
319337
route-map:
@@ -544,7 +562,7 @@
544562
action:
545563
deny: {}
546564
{{- range $vpc := $tenant.Vpcs }}
547-
{{- if $vpc.RoutingProfile.LeakDefaultRouteFromUnderlay }}
565+
{{- if $vpc.HasLeaksFromUnderlayIpv4 }}
548566
FROM_DEFAULT_VRF_TO_{{ $vpc.VrfName }}_MAP:
549567
rule:
550568
'10':
@@ -556,8 +574,8 @@
556574
{{- end }}
557575
{{- end }}
558576
{{- range $vpc := $tenant.Vpcs }}
559-
{{- if $vpc.RoutingProfile.LeakDefaultRouteFromUnderlay }}
560-
FROM_DEFAULT_VRF_TO_{{ $vpc.VrfName }}_MAP_IPV6:
577+
{{- if $vpc.HasLeaksFromUnderlayIpv6 }}
578+
FROM_DEFAULT_VRF_TO_{{ $vpc.VrfName }}_MAP_V6:
561579
rule:
562580
'10':
563581
action:
@@ -773,13 +791,13 @@
773791
route-export:
774792
to-evpn:
775793
enable: on
776-
{{- if $vpc.RoutingProfile.LeakDefaultRouteFromUnderlay }}
794+
{{- if $vpc.HasLeaksFromUnderlayIpv4 }}
777795
route-import:
778796
from-vrf:
779797
route-map: FROM_DEFAULT_VRF_TO_{{ $vpc.VrfName }}_MAP
780798
enable: on
781799
list:
782-
default: {}
800+
default: {}
783801
{{- end}}
784802
ipv6-unicast:
785803
enable: on
@@ -789,6 +807,14 @@
789807
route-export:
790808
to-evpn:
791809
enable: on
810+
{{- if $vpc.HasLeaksFromUnderlayIpv6 }}
811+
route-import:
812+
from-vrf:
813+
route-map: FROM_DEFAULT_VRF_TO_{{ $vpc.VrfName }}_MAP_V6
814+
enable: on
815+
list:
816+
default: {}
817+
{{- end}}
792818
enable: on
793819
neighbor:
794820
{{- range $vpc.HostInterfaces}}

‎crates/agent/templates/tests/nvue_startup_fnn_with_leaks.yaml.expected‎

Lines changed: 24 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -180,12 +180,22 @@
180180
any: {}
181181
FROM_DEFAULT_VRF_TO_vpc_1025186_PREFIX_LIST:
182182
rule:
183+
'1':
184+
action: permit
185+
match:
186+
'10.255.0.0/24':
187+
max-prefix-len: 32
183188
'65534':
184189
action: permit
185190
match:
186191
0.0.0.0/0: {}
187192
FROM_DEFAULT_VRF_TO_vpc_1025197_PREFIX_LIST:
188193
rule:
194+
'1':
195+
action: permit
196+
match:
197+
'10.255.0.0/24':
198+
max-prefix-len: 32
189199
'65534':
190200
action: permit
191201
match:
@@ -527,15 +537,15 @@
527537
match:
528538
ip-prefix-list: FROM_DEFAULT_VRF_TO_vpc_1025197_PREFIX_LIST
529539
type: ipv4
530-
FROM_DEFAULT_VRF_TO_vpc_1025186_MAP_IPV6:
540+
FROM_DEFAULT_VRF_TO_vpc_1025186_MAP_V6:
531541
rule:
532542
'10':
533543
action:
534544
permit: {}
535545
match:
536546
ip-prefix-list: FROM_DEFAULT_VRF_TO_vpc_1025186_PREFIX_LIST_IPV6
537547
type: ipv6
538-
FROM_DEFAULT_VRF_TO_vpc_1025197_MAP_IPV6:
548+
FROM_DEFAULT_VRF_TO_vpc_1025197_MAP_V6:
539549
rule:
540550
'10':
541551
action:
@@ -709,6 +719,12 @@
709719
route-export:
710720
to-evpn:
711721
enable: on
722+
route-import:
723+
from-vrf:
724+
route-map: FROM_DEFAULT_VRF_TO_vpc_1025186_MAP_V6
725+
enable: on
726+
list:
727+
default: {}
712728
enable: on
713729
neighbor:
714730
10.217.5.170:
@@ -818,6 +834,12 @@
818834
route-export:
819835
to-evpn:
820836
enable: on
837+
route-import:
838+
from-vrf:
839+
route-map: FROM_DEFAULT_VRF_TO_vpc_1025197_MAP_V6
840+
enable: on
841+
list:
842+
default: {}
821843
enable: on
822844
neighbor:
823845
10.217.5.170:

‎crates/api/src/cfg/README.md‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -260,7 +260,26 @@ Extends `StateControllerConfig` with:
260260
| `admin_vpc` | `Option<AdminFnnConfig>` | — | FNN configuration for the admin network VPC. |
261261
| `common_internal_route_target` | `Option<RouteTargetConfig>` | — | Double-tag for internal tenant routes (consumed by the network infrastructure). |
262262
| `additional_route_target_imports` | `Vec<RouteTargetConfig>` | `[]` | Extra route targets imported on DPU VRFs. |
263-
| `routing_profiles` | `HashMap<String, FnnRoutingProfileConfig>` | `{}` | Named per-VPC routing profiles. |
263+
| `routing_profiles` | `HashMap<String, FnnRoutingProfileConfig>` | `{}` | Named per-VPC routing profiles (see [FnnRoutingProfileConfig](#fnnroutingprofileconfig)). |
264+
265+
### `FnnRoutingProfileConfig`
266+
267+
| Field | Type | Default | Description |
268+
|-------|------|---------|-------------|
269+
| `route_target_imports` | `Vec<RouteTargetConfig>` | `[]` | Route targets imported into DPU VRFs for VPC routes. |
270+
| `route_targets_on_exports` | `Vec<RouteTargetConfig>` | `[]` | Route targets added to routes exported by the DPU. |
271+
| `internal` | `bool` | `false` | Whether the profile uses internal VNI allocation. |
272+
| `leak_default_route_from_underlay` | `bool` | `false` | Leak the default route from the underlay/default VRF into tenant VRFs. |
273+
| `leak_tenant_host_routes_to_underlay` | `bool` | `false` | Leak tenant host routes into the underlay/default VRF. |
274+
| `tenant_leak_communities_accepted` | `bool` | `false` | Honor route-leak communities sent by the tenant host OS. |
275+
| `accepted_leaks_from_underlay` | `Vec<PrefixFilterPolicyEntry>` | `[]` | Specific underlay/default VRF prefixes allowed to leak into tenant VRFs. Routing only; does not affect ACLs. |
276+
| `access_tier` | `u32` | `0` | Routing profile access tier. Lower values grant broader access. |
277+
278+
### `PrefixFilterPolicyEntry`
279+
280+
| Field | Type | Default | Description |
281+
|-------|------|---------|-------------|
282+
| `prefix` | `IpNetwork` | **required** | IPv4 or IPv6 CIDR prefix accepted by a prefix-list policy. |
264283

265284
### `DpaConfig`
266285

0 commit comments

Comments
 (0)