I’ve wanted to play with multi-WAN routing setups for awhile. I thought about buying Starlink, but it’s a bit pricey as a purely backup solution, and I’m in a situation where my primary internet is very reliable for about the same price. So I settled for LTE backup. I looked around for awhile, trying to decide what the right solution is for me. I really wanted to pass through the IPv4 address and IPv6 prefix from the ISP through to my OPNsense router, since I don’t want to deal with triple-NAT and dual layer firewall. A lot of LTE hotspots, even those that support wired Ethernet, still act as firewall routers, and I didn’t want that hassle.

Oh, and also a rant on how we should stop encouraging people to continue using IPv4 and deploying IPv4-only networks because it’s easier. IPv4 is legacy and should be for backwards compatibility only when you are traversing the internet.

The Video⌗

Feel free to watch my video on this topic! Click the thumbnail to view it Video Thumbnail I made a big oof in the video and used fast.t-mobile.net instead of fast.t-mobile.com. oops.

Slight Background on Mobile Networking, and IPv6 Rant⌗

Back in the early days of cellular data, there was a lot of overhead in encapsulating IP frames within cellular data channels. The oldest digital cellular system in use, GSM, was originally designed to carry circuit switched voice traffic and had the option for circuit switched data (basically using the fixed-bandwidth voice channel as an extremely expensive modem). Packet traffic was eventually layered on top, and packets had to be carried across the T1/E1 network which was really not designed to carry packets. GSM used TDMA, ‘Time Division Multiple Access’, so each client had a certain time slot in the sequence to transmit a fixed amount of data.

Eventually 3G came, and it could do variable-bandwidth channels using its CDMA scheme. Both the worldwide UMTS standard and the US/CA CDMA2000 standard used Code Division Multiple Access, where transmissions between all clients and the tower were done at the same time and encoded using different orthogonal codes (Walsh/Hadamard codes). This basically added a ton of error correction code and then relied on the fact that the error correction encoding was different for each client to decipher the message intended for this specific client. The upside to this scheme was the variable bandwidth of the channel, so instead of transferring the client from the RACH channel (random access channel - mobile requests a control channel) to the low-bandwidth control channel to a different higher bandwidth data channel, the tower could change the channel bandwidth as needed. But the system was still primarily circuit-based, with better support for packets.

Now we have LTE. LTE is a purely packet based network. The entire system is based around IPv6. The RAN (radio access network) translates radio waves to/from IPv6 right at the tower - the eNodeB. From there, it’s all networking gear and specialized software.

The upside to this is we get native IPv6 subnets routed to the client device, and the use of IPv6-only mobile networks has really improved IPv6 uptake (and we desperately need to improve IPv6 uptake). The downside is we don’t have native IPv4, like at all. No dual-stack like the cable and fiber companies. You just don’t get IPv4.

The solution mobile operators have to deal with legacy IPv4 services is to use a technique called 464XLAT as well as CGNAT, and also DNS64 is an optional solution as well.

464XLAT is a way to rewrite the IP header of an IPv4 packet into an IPv6 one, using a special 96-bit prefix and filling the remainder with the IPv4 address. The mobile device (CLAT - client / customer side) will rewrite IPv4 packets into IPv6 ones, send them over the RAN and throught the mobile network, where they are terminated by the PLAT (provider-side), which is the gateway to the IPv4 world.

DNS64 is a method which can replace the CLAT by rewriting A records (IPv4) with AAAA records (IPv6) using the 96-bit xlat prefix when the site has no native AAAA records, so the application which requested access will then do a native IPv6 connction to the PLAT. For sites which have native AAAA records, the application can stay on IPv6 the entire time.

Due to IPv4 address exhaustion, at this point we really need to stop giving out public IPv4s to home internet users. If your game server can’t handle IPv6 the devs have only had two decades to fix it. So, anyway, mobile ISPs can’t allocate a public IPv4 to each device so it can talk to the legacy IPv4 world, so the PLAT will also do CGNAT - network address translation. In theory the devices should be assigned an IPv4 in the 100.64/10 prefix, but T-Mobile seems to be re-using the 6/8 prefix that belongs to the DoD, presumably because the 100.64/10 prefix hadn’t been formalized and they were already doing IPv4 CGNAT long before LTE and IPv6 for their GSM network dating back to the late 90s.

The Hardware⌗

I ended up selecting the Mikrotik SXT US Kit, which includes a US band LTE modem, decently directional antenna, wired Ethernet with PoE, and RouterOS. RouterOS is …. dense with configuration options, and I’m not that good at managing it, but it’s supposed to be able to do a pure passthrough of the LTE modem to an interface without interfering with the network at all.

I was able to configure this in RouterOS eventually. I’m happy with the hardware so far. I stuck it in a window facing roughly the direction of my nearest T-mobile tower and it just worked. I’m sure if I was more on the fringes of cell service I would have needed an actual pole and to actually aim it.

RouterOS⌗

I started with the default config that the SXT had. It really wanted to be a firewall router, but I gave it a static IP on my internal LAN network instead and turned off DHCP and firewalling. I then created a new VLAN interface in RouterOS for the LTE modem, and bridged the LTE modem to the VLAN using this command:

/interface lte apn
add apn=fast.t-mobile.com ip-type=ipv4-ipv6 passthrough-interface=LTEWAN

For some reason RouterOS defaulted to ipv4 only, no idea why. adding ip-type fixed that.

I then plugged it into my Mikrotik CRS328-24P-4S+ switch, it got 802.3af PoE, used the untagged network with its LAN IP for management, and has a tagged network for the internet connection back to OPNsense.

Here’s the full config I ended up with:

[admin@MikroTik] > export 
# apr/26/2022 20:06:56 by RouterOS 6.45.9
# software id = XXXX-XXXX
#
# model = RBSXTR
# serial number = XXXXXXXXX
/interface bridge
add admin-mac=xx:xx:xx:xx:xx:xx auto-mac=no comment=defconf name=bridge
/interface vlan
add interface=ether1 name=LTEWAN vlan-id=2
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface lte apn
add apn=fast.t-mobile.com ip-type=ipv4-ipv6 passthrough-interface=LTEWAN passthrough-mac=auto
/interface lte
set [ find ] apn-profiles=fast.t-mobile.com mac-address=xx:xx:xx:xx:xx:xx name=lte1
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile
set [ find default=yes ] html-directory=flash/hotspot
/ip pool
add name=default-dhcp ranges=192.168.88.10-192.168.88.254
/ip dhcp-server
add address-pool=default-dhcp interface=bridge name=defconf
/interface bridge port
add bridge=bridge comment=defconf interface=ether1
add bridge=bridge comment=defconf interface=ether2
/interface list member
add comment=defconf interface=bridge list=LAN
add interface=lte1 list=WAN
/ip address
add address=172.27.0.9/20 comment=defconf interface=ether1 network=172.27.0.0
/ip dhcp-server network
add address=192.168.88.0/24 comment=defconf gateway=192.168.88.1 netmask=24
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=172.27.0.9 comment=defconf name=router.lan
/ip firewall filter
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" disabled=yes ipsec-policy=out,none out-interface-list=WAN
/system clock
set time-zone-name=America/Chicago

OPNsense⌗

I was able to get DHCP (IPv4) and SLAAC (IPv6) working. T-Mobile doesn’t do IPv6 PD via DHCPv6, so SLAAC is what I get. It seems like the SLAAC prefix is stable as long as the modem is powered on, and it gets a new one when the modem reboots. This correlates with phone users reporting a stable prefix through tower handover and phone sleep unless they reboot the phone. Not totally unexpected. Unlike DHCPv6 there isn’t a native IPv6 way to request a static SLAAC prefix since the DUID is only used in DHCPv6 (although they could tie prefix to IMEI for a mobile-specific solution). The lack of a delegated prefix means we can either have exactly one IPv6 subnet in the LAN side (passing through the prefix we got from upstream) OR we can use exclusively DHCPv6 and smaller than /64 subnets internally. Android’s core devs are stupidly opposed to using DHCPv6 in Android, but every other OS supports it.

I tested with a temporary server instead of OPNsense and was able to successfully receive inbound connections to my IPv6 address (with the server connected directly to the LTE modem), although I’ve heard T-mobile blocks ICMP (ping) packets. HTTP inbound worked just fine. There’s sometimes a huge latency in initiating the connection (up to 1 sec for the TCP socket to connect). Presumably it has to wait for the mobile side to wake / check for notifications from the tower, and LTE is designed to conserve battery power so it has a relatively lengthy wake up interval compared to WiFi. Once the session is up it seems like latency isn’t bad at all.