route configuration cleanup and fixes #94

nmeyerhans · 2023-09-19T18:22:48Z

Issue #, if available: n/a

Description of changes:

This change adds a rule for ENI traffic egressing with the primary NIC's address in the source field. This was left out previously in keeping with AL2's behavior, but leads to traffic loss.

Test results will follow in the comments.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

nmeyerhans · 2023-09-19T19:36:01Z

IPv4 route validation

AL2 results for comparison

AL2023:

Address and route config

[ec2-user@ip-10-0-1-90 ~]$ ip -4 addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    altname enp0s5
    altname eni-00a83dc1dc7657d4f
    altname device-number-0
    inet 10.0.1.90/24 metric 512 brd 10.0.1.255 scope global dynamic ens5
       valid_lft 2418sec preferred_lft 2418sec
3: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    altname enp0s6
    altname eni-04beacb132d06dbf5
    altname device-number-1
    inet 10.0.10.193/24 metric 522 brd 10.0.10.255 scope global dynamic ens6
       valid_lft 2458sec preferred_lft 2458sec

[ec2-user@ip-10-0-1-90 ~]$ ip -4 ro
default via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.90 metric 512 
default via 10.0.10.1 dev ens6 proto dhcp src 10.0.10.193 metric 522 
10.0.0.2 via 10.0.1.1 dev ens5 proto dhcp src 10.0.1.90 metric 512 
10.0.0.2 via 10.0.10.1 dev ens6 proto dhcp src 10.0.10.193 metric 522 
10.0.1.0/24 dev ens5 proto kernel scope link src 10.0.1.90 metric 512 
10.0.1.1 dev ens5 proto dhcp scope link src 10.0.1.90 metric 512 
10.0.10.0/24 dev ens6 proto kernel scope link src 10.0.10.193 metric 522 
10.0.10.1 dev ens6 proto dhcp scope link src 10.0.10.193 metric 522

AL2:

[ec2-user@ip-10-0-1-81 ~]$ ip -4 addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    inet 10.0.1.81/24 brd 10.0.1.255 scope global dynamic eth0
       valid_lft 3316sec preferred_lft 3316sec
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    inet 10.0.10.40/24 brd 10.0.10.255 scope global dynamic eth1
       valid_lft 3293sec preferred_lft 3293sec

[ec2-user@ip-10-0-1-81 ~]$ ip -4 ro
default via 10.0.1.1 dev eth0 
default via 10.0.10.1 dev eth1 metric 10001 
10.0.1.0/24 dev eth0 proto kernel scope link src 10.0.1.81 
10.0.10.0/24 dev eth1 proto kernel scope link src 10.0.10.40 
169.254.169.254 dev eth0

Routes to gateways

AL2023:

[ec2-user@ip-10-0-1-90 ~]$ ip -4 ro get 10.0.1.1
10.0.1.1 dev ens5 src 10.0.1.90 uid 1000 
    cache 
[ec2-user@ip-10-0-1-90 ~]$ ip -4 ro get 10.0.10.1
10.0.10.1 dev ens6 src 10.0.10.193 uid 1000 
    cache

AL2:

[ec2-user@ip-10-0-1-81 ~]$ ip -4 ro get 10.0.1.1
10.0.1.1 dev eth0 src 10.0.1.81 uid 1000 
    cache 
[ec2-user@ip-10-0-1-81 ~]$ ip -4 ro get 10.0.10.1
10.0.10.1 dev eth1 src 10.0.10.40 uid 1000 
    cache

routes to IMDS

AL2023:

[ec2-user@ip-10-0-1-90 ~]$ ip -4 ro get 169.254.169.254
169.254.169.254 via 10.0.1.1 dev ens5 src 10.0.1.90 uid 1000 
    cache

AL2:

[ec2-user@ip-10-0-1-81 ~]$ ip -4 ro get 169.254.169.254
169.254.169.254 dev eth0 src 10.0.1.81 uid 1000 
    cache

device-number-0 subnet peer with device-number-1 src ip

AL2023:

[ec2-user@ip-10-0-1-90 ~]$ ip -4 ro get 10.0.1.99 from 10.0.10.193
10.0.1.99 from 10.0.10.193 via 10.0.10.1 dev ens6 table 10001 uid 1000 
    cache

AL2:

[ec2-user@ip-10-0-1-81 ~]$ ip -4 ro get 10.0.1.99 from 10.0.10.40
10.0.1.99 from 10.0.10.40 via 10.0.10.1 dev eth1 table 10001 uid 1000 
    cache

device-number-1 gateway from device-number-0 src ip

AL2023:

[ec2-user@ip-10-0-1-90 ~]$ ip -4 ro get 10.0.10.1 from 10.0.1.90
10.0.10.1 from 10.0.1.90 via 10.0.1.1 dev ens5 table 10000 uid 1000 
    cache

AL2:

Note This is broken on AL2! Should be routed via device-number-0, aka eth0!

[ec2-user@ip-10-0-1-81 ~]$ ip -4 ro get 10.0.10.1 from 10.0.1.81
10.0.10.1 from 10.0.1.81 dev eth1 uid 1000 
    cache

nmeyerhans · 2023-09-19T20:58:41Z

IPv6 route validation

Address and route config

AL2023:

[ec2-user@ip-10-0-1-90 ~]$ ip -6 addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 state UP qlen 1000
    inet6 2600:1f14:eeb:2202::f3bd/128 scope global dynamic noprefixroute 
       valid_lft 427sec preferred_lft 117sec
    inet6 fe80::45d:7aff:fed3:fb77/64 scope link 
       valid_lft forever preferred_lft forever
3: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 state UP qlen 1000
    inet6 2600:1f14:eeb:2210:4b4e:d138:6a1a:a4b5/128 scope global dynamic noprefixroute 
       valid_lft 397sec preferred_lft 87sec
    inet6 fe80::4c2:70ff:fe93:5295/64 scope link 
       valid_lft forever preferred_lft forever

[ec2-user@ip-10-0-1-90 ~]$ ip -6 ro show
2600:1f14:eeb:2202::/64 dev ens5 proto ra metric 512 pref medium
2600:1f14:eeb:2210::/64 dev ens6 proto ra metric 522 pref medium
fe80::/64 dev ens5 proto kernel metric 256 pref medium
fe80::/64 dev ens6 proto kernel metric 256 pref medium
default via fe80::4ad:f9ff:fe4f:ee1 dev ens5 proto ra metric 512 expires 1794sec pref medium
default via fe80::47a:12ff:fe95:11a3 dev ens6 proto ra metric 522 expires 1796sec pref medium

AL2:

[ec2-user@ip-10-0-1-81 ~]$ ip -6 addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 state UP qlen 1000
    inet6 2600:1f14:eeb:2202::9194/128 scope global dynamic 
       valid_lft 421sec preferred_lft 111sec
    inet6 fe80::487:e1ff:fe88:71d9/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 state UP qlen 1000
    inet6 2600:1f14:eeb:2210:49:1223:5a92:8b80/128 scope global dynamic 
       valid_lft 409sec preferred_lft 99sec
    inet6 fe80::422:fbff:fe89:744b/64 scope link 
       valid_lft forever preferred_lft forever

[ec2-user@ip-10-0-1-81 ~]$ ip -6 ro
unreachable ::/96 dev lo metric 1024 error 4294967183 pref medium
unreachable ::ffff:0.0.0.0/96 dev lo metric 1024 error 4294967183 pref medium
unreachable 2002:a00::/24 dev lo metric 1024 error 4294967183 pref medium
unreachable 2002:7f00::/24 dev lo metric 1024 error 4294967183 pref medium
unreachable 2002:a9fe::/32 dev lo metric 1024 error 4294967183 pref medium
unreachable 2002:ac10::/28 dev lo metric 1024 error 4294967183 pref medium
unreachable 2002:c0a8::/32 dev lo metric 1024 error 4294967183 pref medium
unreachable 2002:e000::/19 dev lo metric 1024 error 4294967183 pref medium
2600:1f14:eeb:2202::9194 dev eth0 proto kernel metric 256 expires 418sec pref medium
2600:1f14:eeb:2202::/64 dev eth0 proto kernel metric 256 pref medium
2600:1f14:eeb:2210:49:1223:5a92:8b80 dev eth1 proto kernel metric 256 expires 406sec pref medium
2600:1f14:eeb:2210::/64 dev eth1 proto kernel metric 256 pref medium
unreachable 3ffe:ffff::/32 dev lo metric 1024 error 4294967183 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev eth1 proto kernel metric 256 pref medium
default via fe80::4ad:f9ff:fe4f:ee1 dev eth0 proto ra metric 1024 expires 1791sec hoplimit 255 pref medium
default via fe80::47a:12ff:fe95:11a3 dev eth1 proto ra metric 1024 expires 1795sec hoplimit 255 pref medium

Routes to gateways

Note that AL2023 configures different metrics here to ensure that the
routes over devices with a lower device-number are preferred
regardless of the order in which the devices are configured. This is
relevant at (re)boot because the order is not guaranteed.

AL2023:

[ec2-user@ip-10-0-1-90 ~]$ ip -6 ro show default
default via fe80::4ad:f9ff:fe4f:ee1 dev ens5 proto ra metric 512 expires 1799sec pref medium
default via fe80::47a:12ff:fe95:11a3 dev ens6 proto ra metric 522 expires 1790sec pref medium

AL2:

[ec2-user@ip-10-0-1-81 ~]$ ip -6 ro show default
default via fe80::4ad:f9ff:fe4f:ee1 dev eth0 proto ra metric 1024 expires 1796sec hoplimit 255 pref medium
default via fe80::47a:12ff:fe95:11a3 dev eth1 proto ra metric 1024 expires 1790sec hoplimit 255 pref medium

routes to IMDS

AL2023:

[ec2-user@ip-10-0-1-90 ~]$ ip -6 ro get fd00:ec2::254
fd00:ec2::254 from :: via fe80::4ad:f9ff:fe4f:ee1 dev ens5 proto ra src 2600:1f14:eeb:2202::f3bd metric 512 pref medium

AL2:

[ec2-user@ip-10-0-1-81 ~]$ ip -6 ro get fd00:ec2::254
fd00:ec2::254 from :: via fe80::4ad:f9ff:fe4f:ee1 dev eth0 proto ra src 2600:1f14:eeb:2202::9194 metric 1024 hoplimit 255 pref medium

device-number-0 subnet peer with device-number-1 src ip

Validate that traffic is routed via device-number-1 per policy rules.

AL2023:

[ec2-user@ip-10-0-1-90 ~]$ ip -6 ro get 2600:1f14:eeb:2202::99 from 2600:1f14:eeb:2210:4b4e:d138:6a1a:a4b5
2600:1f14:eeb:2202::99 from 2600:1f14:eeb:2210:4b4e:d138:6a1a:a4b5 via fe80::47a:12ff:fe95:11a3 dev ens6 table 10001 proto ra src 2600:1f14:eeb:2210:4b4e:d138:6a1a:a4b5 metric 522 pref medium

AL2:

[ec2-user@ip-10-0-1-81 ~]$ ip -6 ro get 2600:1f14:eeb:2202::99 from 2600:1f14:eeb:2210:49:1223:5a92:8b80
2600:1f14:eeb:2202::99 from 2600:1f14:eeb:2210:49:1223:5a92:8b80 via fe80::47a:12ff:fe95:11a3 dev eth1 table 10001 src 2600:1f14:eeb:2210:49:1223:5a92:8b80 metric 1024 pref medium

device-number-1 subnet peer from device-number-0 src ip

AL2023:

[ec2-user@ip-10-0-1-90 ~]$ ip -6 ro get 2600:1f14:eeb:2210::99 from 2600:1f14:eeb:2202::f3bd
2600:1f14:eeb:2210::99 from 2600:1f14:eeb:2202::f3bd via fe80::4ad:f9ff:fe4f:ee1 dev ens5 table 10000 proto ra src 2600:1f14:eeb:2202::f3bd metric 512 pref medium

AL2:

Note AL2 is broken here. This traffic from the device-number-0 IP
must egress via device-number-0, otherwise it is dropped by the VPC
src/dest check

[ec2-user@ip-10-0-1-81 ~]$ ip -6 ro get 2600:1f14:eeb:2210::99 from 2600:1f14:eeb:2202::9194
2600:1f14:eeb:2210::99 from 2600:1f14:eeb:2202::9194 dev eth1 proto kernel src 2600:1f14:eeb:2210:49:1223:5a92:8b80 metric 256 pref medium

Previously the code would disable errexit in a certain block, and then unconditionally re-enable it at the end, regardless of whether it had originally been enabled. This change ensures that it's only re-enabled if it originally had been.

Install routes from DHCP in the main table with increasing metrics Install static routes in secondary tables Formatting changes to the generated config drop-in for readability Fixes: 33b68bb ("Install prefix routes in private tables")

nmeyerhans force-pushed the route-fix branch from 84f571c to 662691d Compare September 19, 2023 18:24

Kemotaha approved these changes Sep 19, 2023

View reviewed changes

vigh-m approved these changes Sep 21, 2023

View reviewed changes

Noah Meyerhans added 2 commits September 21, 2023 15:00

Route configuration simplification

c04b3e5

Install routes from DHCP in the main table with increasing metrics Install static routes in secondary tables Formatting changes to the generated config drop-in for readability Fixes: 33b68bb ("Install prefix routes in private tables")

nmeyerhans force-pushed the route-fix branch from 662691d to c04b3e5 Compare September 21, 2023 22:02

nmeyerhans merged commit 0a57c62 into amazonlinux:main Sep 21, 2023
3 checks passed

nmeyerhans deleted the route-fix branch September 21, 2023 22:03

krzysztof-bronk mentioned this pull request Oct 11, 2023

2.4.0-1.amzn2023.0.1 breaks docker connectivity inside host #97

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

route configuration cleanup and fixes #94

route configuration cleanup and fixes #94

nmeyerhans commented Sep 19, 2023

nmeyerhans commented Sep 19, 2023

nmeyerhans commented Sep 19, 2023

route configuration cleanup and fixes #94

route configuration cleanup and fixes #94

Conversation

nmeyerhans commented Sep 19, 2023

nmeyerhans commented Sep 19, 2023

IPv4 route validation

Address and route config

Routes to gateways

routes to IMDS

device-number-0 subnet peer with device-number-1 src ip

device-number-1 gateway from device-number-0 src ip

nmeyerhans commented Sep 19, 2023

IPv6 route validation

Address and route config

Routes to gateways

routes to IMDS

device-number-0 subnet peer with device-number-1 src ip

device-number-1 subnet peer from device-number-0 src ip