After struggling for over 20 hours, I wanted to share the results of my investigation regarding very poor Internet upload erformance.

Setup

• Proxmox Server with a Single 10GbE NIC
• OPNsense VM on Proxmox
• OPNsense uses VirtIO NICs tied to the 10GbE Linux Bridge
• upstream Gateway is a OpenWRT router with 1GbE uplink
• Zyxel XS1930 Switch connecting Proxmox Host and Gateway

Problem

Internet download speeds are fine (900Mbit/s) but upload speeds are not (5-15MBit/s instead of 50MBit/s)

Solution

Various OPNsense tunables (configured for 8 CPU cores)

• hw.ibrs_disable = 1
• net.isr.maxthreads = -1
• net.isr.bindthreads = 1
• net.isr.dispatch = deferred
• net.inet.rss.enabled = 1
• net.inet.rss.bits = 6
• kern.ipc.maxsockbuf = 16777216
• net.inet.tcp.recvbuf_max = 4194304
• net.inet.tcp.recvspace = 262144
• net.inet.tcp.sendbuf_inc = 16384
• net.inet.tcp.sendbuf_max = 4194304
• net.inet.tcp.sendspace = 262144
• net.inet.tcp.soreceive_stream = 1
• net.pf.source_nodes_hashsize = 1048576
• net.inet.tcp.mssdflt = 1240
• net.inet.tcp.abc_l_var = 52
• net.inet.tcp.minmss = 536
• kern.random.fortuna.minpoolsize = 128
• net.isr.defaultqlimit = 2048

Enabling Multiqueue in Proxmox for the VirtIO NICs

(binary stepping, 1 Queue for 2 cores, 2 Queues for 4 cores, 3 Queue for 8 cores ect, total amount of all Queues mustn’t be greater then the VMs CPU cores)

Enabling Flow Control on all involved Network devices

• Proxmox hardware NIC: ethtool -K nic0 rx on tx on
• OpenWRT lan interfnace:

uci set network.lan.txpause='1'

uci set network.lan.rxpause='1'

uci commit

reload_config

• Zyxel Switch:

Port -> Port Setup - Checked all Ports

Enabling Port Buffering

Zyxel Switch:

Port -> Port Buffer - Checked the Port with the Gateway

Reason

The Main reason for this problem seems to be the down-stepping of 10Gbit traffic to 1Gbit devices. Without Flow control enabled on all involved devices, the sending rate can't be adjusted. But without enabling Port Buffering, the Switch won't allocate resources for adjusting the traffic flow rate for slower devices.

This Problem should only affect people who use devices with different link speeds on the same switch.

sauce

Wie hosted ihr eure Instanzen?

I tried debugging this issue for hours now but I'm out of ideas. I'm running WireGuard on my OPNsense firewall. It worked flawlessly for about a year but now I've gotten a really strange issue.

Here is the Client config:

As you can see, the Client got assigned the IP 10.10.10.11/32

I can ping this IP and the Client can access all Server in the network when connected with the VPN.

BUT when it connects to hosts in the LAN, it doesn't use it's assigned 10.10.10.11 IP but the public IP of the OPNsense firewall instead.

This also doesn't happen every time, but most of the time. I assume that it's perhaps a ARP issue, but I don't know why the OPNsense firewall sends its public IP (WireGuard Endpoint IP) instead of the Clients assigned IP at all.

The IP the Client should use in the LAN (virtual VPN IP):

The IP which the Client actually uses (Endpoint IP):

Every help would be greatly appreciated!

EDIT: I removed and re-added the peers in OPNsense and it works again, at least for now. Maybe something broke during an update(?). I will report back if this already fixed the issue (the problem can sometimes be hard to replicate)

EDIT 2: The issue reappeared. But I noticed, that I now have the problem only with Gecko based browsers, chromium works fine (tested on Android).

EDIT 3: The issue only appeared with Gecko based browsers because mine are configured to use some public DOH DNS, which resolved my internal host FQDNs to public IPs, not private ones from the LAN

https://www.pixiv.net/member_illust.php?illust_id=113828731&