pfSense - pfSense is a free and open source firewall and router that also features unified threat management, load balancing, multi WAN, and more
Sophos UTM - Sophos UTM integrates complete security software within a single appliance.
OPNsense - OPNsense® you next open source firewall. Free Download. High-end Security Made Easy™. Offers Intrusion Prevention, Captive Portal, Traffic Shaping and more.
IPFire - IPFire is a server distribution with intended to use as a firewall.
ClearOS - A very good alternative for Windows Server, for free
MikroTik RouterOS - The main product of MikroTik is a Linux-based operating system known as MikroTik RouterOS.