IPsec MTU is determined during the negotiation process and represents the maximum size of packets as they traverse the IPSec tunnel. The MTU across the internet is generally 1500 bytes, but can be lower. The default IPsec MTU (internet MTU minus encryption overhead) is therefore usually around 1420 bytes, depending on the encryption in use.
If the connecting device or application has MTU restrictions, you may have to set the IPsec MTU manually either via the "extra configuration parameters" box on the IPsec endpoint page or via the VNS3 Firewall.
In the "extra configuration parameters" box, you can use the "mtu=" parameter to set the IPSec MTU in bytes.
The other way to alleviate MTU issues is to set an artificial TCP MSS for IPsec traffic traffic prior to it being encapsulated and sent over the tunnel. Here is an example of those MSS/MTU clamping firewall rules:
FORWARD_CUST -s 192.168.220.0/24 -d 172.16.0.0/24 -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1300
FORWARD_CUST -d 192.168.220.0/24 -s 172.16.0.0/24 -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1300
When using firewall rules to set path MSS, you will need to account for the TCP overhead, IPsec encryption overhead, and the internet path MTU, as you are operating on the size of the TCP payload. For example, if your internet path MTU is 1400, you might start with an MSS of 1200 and work up from there. (That's 1400 minus 80 for IPsec overhead minus 60 for TCP headers and rounded down for good luck.)
We usually find that 1100 is a good starting place for an unknown path MTU, but occasionally we have found that we need to go even lower. The MSS value can be set arbitrarily, but should ideally be as high as possible without causing MTU issues.
The advantage to using the MSS firewall rules over the mtu= IPsec parameter is that the firewall rules take effect immediately and do not trigger an IPsec renegotiation. This leads to faster troubleshooting and fewer complications. Once an appropriate MSS has been discovered, you may choose to calculate an appropriate path MTU and set the "mtu=" parameter. Be aware that this may cause fragmentation of packets across the link (degrading performance) unless the application host's MTU is also changed. The disadvantage to using the firewall rules is that it only affects TCP traffic traversing the tunnel.
NOTE: If you are using a version of VNS3 prior to 4.4.x, you will need to add "compat:" in front of the "mtu=" parameter like this: compat:mtu=1300