multiroute multiple ISP, individual snat and multiple routing tables

[andyb2000]

I may be asking too much from this excellent firewall script that I've used for so many years! But here is what i'm doing so to see if I'm trying to push it too far!

Ubuntu server 22.04 LTS
iptables v1.8.7 (nf_tables)
Linux version 5.15.0-86-generic
Arno's Iptables Firewall(AIF) v2.1.1

I have two external connections via two different providers, the second has multiple IPs assigned:
eno1.15 - DHCP obtained public IP address 1.1.1.2, d/g/ 1.1.1.1
ens5 - static public IP address 2.2.2.2/29, d/g/ 2.2.2.1
ens5:1 - static public IP address 2.2.2.3/29
ens5:2 - static public IP address 2.2.2.4/29

I also have a LAN:
ens1 - static 192.168.1.1/24 (all clients on the lan sit behind this and have nat'd internet via the ens5 public IP so hide behind 2.2.2.2)

To facilitate the linux server being able to answer queries from both public external connections I use linux multiple routing tables with set naming (defined in /etc/iproute2/rt_tables):
1 rt1
2 rt2

So my two routing tables:

root:~# ip route show table rt1
default via 2.2.2.1 dev ens5
2.2.2.0/29 dev ens5 scope link
root:~# ip route show table rt2
default via 1.1.1.1 dev eno1.15
1.1.1.0/23 dev eno1.15 scope link

This allows me to ssh inbound to my server to either of it's public IP addresses.
Now aif configuration:

EXT_IF="ens5 eno1.15"
EXT_IF_DHCP_IP=1
EXTERNAL_NET="2.2.2.0/29 1.1.1.0/23"
INT_IF="eno1"
INTERNAL_NET="192.168.1.0/24"
NAT=1
NAT_INTERNAL_NET="192.168.1.0/24"
OPEN_TCP="53 443 22"
OPEN_UDP="53 161"

The rest is pretty much unchanged, a few host_open entries, etc, but the rest is stock.

Plugin I have enabled:
outbound-snat.conf

ENABLED=1
OUTBOUND_SNAT_NET_HOST="192.168.1.5>2.2.2.3 192.168.1.133>1.1.1.2"

This is so that device 192.168.1.5 is always hidden behind public IP 2.2.2.3
In theory 192.168.1.133 should hide behind the other external of 1.1.1.2 but this doesn't appear to work.

multiroute.conf - I have tried this on and off to see how this changes behaviour, it does appear to work, however it seems to cause problems for the OUTBOUND_SNAT entry 192.168.1.5>2.2.2.3
multiroute.conf:

ENABLED=1
RP_FILTER_DISABLE=1
MULTIROUTE_EXT_IF1=ens5
MULTIROUTE_EXT_ROUTER1=2.2.2.1
MULTIROUTE_EXT_IP1=2.2.2.2
MULTIROUTE_EXT_WEIGHT1=15
MULTIROUTE_EXT_IF2=eno1.15
MULTIROUTE_EXT_ROUTER2=1.1.1.1
MULTIROUTE_EXT_IP2=1.1.1.2
MULTIROUTE_EXT_WEIGHT2=5
MULTIROUTE_INT_IP=192.168.1.1
MULTIROUTE_INT_NET=192.168.1.0/24

I also did a bit of custom routing to allow loopback for internal hosts hitting that outbound snat entry, so this is
custom-rules:

/sbin/iptables -t nat -A PREROUTING -s 0.0.0.0/0 -d 2.2.2.3 -p tcp --dport 80 -j DNAT --to-destination 192.168.1.5
/sbin/iptables -t nat -A PREROUTING -s 0.0.0.0/0 -d 2.2.2.3 -p tcp --dport 8123 -j DNAT --to-destination 192.168.1.5
/sbin/iptables -t nat -A PREROUTING -s 0.0.0.0/0 -d 2.2.2.3 -p tcp --dport 443 -j DNAT --to-destination 192.168.1.5:8123

So I suspect I'm trying to get it to do too much!

Any suggestions on trying to achieve what I want here:

• Allow multiple inbound connections on my two ISP connections
• do SNAT for those two special LAN IPs to go out the specific public IP addresses
• Allow standard NAT for everything else going out via 2.2.2.2
• Allow loopback/shortcut for any internal hosts hitting 2.2.2.3 to hit the correct 192.168.1.5 address

Thanks in advance and apologies, this isn't a bug as such!
Andy

[andyb2000]

To follow this up.
I do believe it's working, however I'm not sure the weight in multiroute is being honoured correctly.

Testing multiple times, often I see my connection going out over the second interface (1.1.1.2) which should have the weight of 5 (lower than the primary of 15).

Is there a way to force this so that the second is not used?

ENABLED=1
RP_FILTER_DISABLE=1
MULTIROUTE_EXT_IF1=ens5
MULTIROUTE_EXT_ROUTER1=2.2.2.1
MULTIROUTE_EXT_IP1=2.2.2.2
MULTIROUTE_EXT_WEIGHT1=15
MULTIROUTE_EXT_IF2=eno1.15
MULTIROUTE_EXT_ROUTER2=1.1.1.1
MULTIROUTE_EXT_IP2=1.1.1.2
MULTIROUTE_EXT_WEIGHT2=5
MULTIROUTE_INT_IP=192.168.1.1
MULTIROUTE_INT_NET=192.168.1.0/24