Building a 10Gb Capable OPNsense Machine

Posted

in

by

For better network observability and security it’s a great idea to run a professional/enterprise firewall that can greatly expand the capabilities of your network beyond the functionality of router/modems supplied by your ISP.

Components

Buying a prebuilt appliance can get you up and running quickly with manufacturer support, however, 10Gb capable machines can run many thousands of dollars (a FortiGate 100-F I run in production was around $3,500 just for the appliance). Using some common off the shelf parts and used server grade hardware we can achieve excellent performance with very low power consumption.

ComponentPartSourcePrice
Main ChassisLenovo M920q TinyUsed/eBay$200
PCIe Riser AdapterLenovo 01AJ940 RiserNew/eBay$26
NICIntel X550-T2Used/eBay$210
All pricing is in CAD including shipping and duties.

Lenovo M920q Tiny

The magic of this build is made possible with the Lenovo M920q Tiny – a 1L PC with an amazing featureset that will be the powerhouse of OPNsense. The “q” version of the Lenovo Tiny denotes PCIe functionality which will be required to install a NIC. Using the 01AJ940 PCIe riser adapter, you can add any half-length half-height PCIe component, restricted only buy the power draw. The whole package uses very little power and is practically silent under load.

This M920q came with an i5-8500T and 16GB of memory. The 8500T supports AES-NI which is required for good firewall performance.

M920q with Intel x550-T2 installed in the PCIe slot

Intel X550-T2

It’s surprising to see how many fake Intel X520/X540/X550 cards there are on the market. Mostly shipping from China at a very low price, you can hopefully tell which cards are real by the quality of the silkscreening, and the Intel logo printed nicely on the PCB, among others.

The X550-T2 was chosen over the X540-T2 as it supports NBASE-T for 1G/2.5G/5G/10G functionality at about half the power draw of the older X540. The card will automatically negotiate with the upstream ISP gateway, in this case 10Gbase-T:

Putting it All Together

BIOS

Ensure that VTd is enabled for PCIe passthrough support, and set ICE Performance Mode to Better Thermal Performance under Power.

Proxmox

The OPNsense software will run virtualized in Proxmox which gives us future expandabilty options if we wish to run other VMs on the machine. The OPNsense VM uses under 10% CPU idle so there is lots of headroom for future projects.

With Proxmox installed, we can setup PCIe passthrough in /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
Bash

Then, apply GRUB settings and reboot:

update-grub
reboot -h now
Bash

When creating the VM, make sure to set the following:

  1. System > check Qemu Agent
  2. Disks > when using an SSD check the following:
    • Discard
    • IO Thread
    • SSD emulation
  3. CPU > select type host for best performance, and assign all cores (6 cores in this case)
  4. Memory > I assigned half the available memory (8192MB) as a ballooning device

Next, we can add the X550-T2 as a passthrough device so our VM has full, direct access to the card. Under Hardware, select Add > PCI Device > Raw Device and choose the first ID for the card, in this case 0000:01:00.0 for our Intel X550:

Check All Functions and make sure ROM-Bar is select as well under Advanced.

Configuring OPNsense

In OPNsense, assign the NIC ports to LAN and WAN, in this case ix0 and ix1:

We will keep vmbr0 assigned by Proxmox to the VM as the management port for OPNsense. This way, any settings on the interface ports will not impact connections to the VM.

Under Interfaces > Settings, we can enable (uncheck) all CRC, TSO, and LRO offloading features for the card since they should be fully supported, and will help alleviate some strain on the CPU:

Install the QEMU Guest agent plugin in OPNsense so Proxmox has insights to our VM:

Final Results

Testing with 3Gb WAN, the build performs very well providing expected throughput and latency:

And CPU never breaks a sweat:

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *