iptables and Multiple NICs (masq/fwd)

[theoaf]

Hi everyone

Any Linux networking types around please? I've got a (probably easy!) question for you:

I've got a dual Xeon machine running Centos 5 (effectively RedHat) which I've been using for firewall duties with two of five network cards in use (eth0 - external internet and eth1 my trusted home network 192.168.0.1/24). I've set up my firewall using iptables and forwarding/masquerading and all of my internal machines can ping each other and machines on the internet - works fine.

I now need to use a third NIC in the machine to connect other "internal" machines to it, with the aim of all machines on both eth1 and eth2 being able to communicate with each other (and the internet). I had decided that the easiest approach was to set up eth2 as a separate subnet (192.168.1.1/24) and masq the machines in that subnet as I have been doing for those hanging off eth1.

I'm nearly there, but not quite...
- the server with the NICs in it can ping everything (int/int2/ext)
- the machines on int (existing network) can ping the server but NOT int2 ("new") machines
- the machines on int2 can ping the server but NOT int1 machines

Everyone on the two internal networks can get to the internet (external) so forwarding is working.

Is it some kind of routing issue (well, yes... but how/what?) and do I fix it by changing my firewall or by configuring the routing table, or something else?

I've included as much info as I can below. Any comments on whether the firewall looks sensible (from a security point of view)/and changes I need to make to finish the last part of this (both subnets working) would be much appreciated

Here is the output of ifconfig (ext IP blanked with x.x.x.x)

eth0      Link encap:Ethernet  HWaddr 00:B0:D0:20:E5:4F
          inet addr:x.x.x.x  Bcast:255.255.255.255 Mask:255.255.252.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2169439 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1433083 errors:0 dropped:0 overruns:0 carrier:0
          collisions:1790 txqueuelen:1000
          RX bytes:3102962411 (2.8 GiB)  TX bytes:88647610 (84.5 MiB)

eth1      Link encap:Ethernet  HWaddr 00:90:27:FC:90:B0
          inet addr:192.168.0.1  Bcast:192.168.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:5374956 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10010489 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:330970541 (315.6 MiB)  TX bytes:596826515 (569.1 MiB)

eth2      Link encap:Ethernet  HWaddr 00:90:27:FC:90:B1
          inet addr:192.168.1.1  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:7912 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3669 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:929420 (907.6 KiB)  TX bytes:215114 (210.0 KiB)

...and route:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.1.0     *               255.255.255.0   U     0      0        0 eth2
192.168.0.0     *               255.255.255.0   U     0      0        0 eth1
82.9.236.0      *               255.255.252.0   U     0      0        0 eth0
169.254.0.0     *               255.255.0.0     U     0      0        0 eth2
default         abc2-mexch-0-0- 0.0.0.0         UG    0      0        0 eth0

Here's the new firewall script, based on (my understanding of) the Linux HOWTO example.

#!/bin/sh
#
# rc.firewall-iptables-stronger
#
FWVER=0.88s

IPTABLES=/sbin/iptables
LSMOD=/sbin/lsmod
DEPMOD=/sbin/depmod
MODPROBE=/sbin/modprobe
GREP=/bin/grep
AWK=/bin/awk
IFCONFIG=/sbin/ifconfig

#  For this example, "eth0" is external and "eth1+2" are internal
EXTIF="eth0"
INTIF="eth1"
INTIF2="eth2"

echo "  External Interface:  $EXTIF"
echo "  Internal Interface:  $INTIF"
echo "  2nd Intl Interface:  $INTIF2"
echo "  ---"

# Determine the external IP automatically:
EXTIP="`$IFCONFIG $EXTIF | $AWK \
 /$EXTIF/'{next}//{split($0,a,":");split(a[2],a," ");print a[1];exit}'`"

echo "  External IP: $EXTIP"
echo "  ---"

# Assign the internal TCP/IP network and IP address
INTNET="192.168.0.0/24"
INTNET2="192.168.1.0/24"
INTIP="192.168.0.1/32"
INTIP2="192.168.1.1/32"

echo "  Internal Network:  $INTNET"
echo "  Internal IP:       $INTIP"
echo "  Internal Network2: $INTNET2"
echo "  Internal IP2:      $INTIP2"
echo "  ---"

# Setting a few other local variables
#
UNIVERSE="0.0.0.0/0"

# Need to verify that all modules have all required dependencies
#
echo "  - Verifying that all kernel modules are ok"
$DEPMOD -a

echo -en "    Loading kernel modules: "

# With the new IPTABLES code, the core MASQ functionality is now either
# modular or compiled into the kernel.  This HOWTO shows ALL IPTABLES
# options as MODULES.  If your kernel is compiled correctly, there is
# NO need to load the kernel modules manually.  
#Load the main body of the IPTABLES module - "ip_tables"
#  - Loaded automatically when the "iptables" command is invoked
#
#  - Loaded manually to clean up kernel auto-loading timing issues
#
echo -en "ip_tables, "
#
#Verify the module isn't loaded.  If it is, skip it
#
if [ -z "` $LSMOD | $GREP ip_tables | $AWK {'print $1'} `" ]; then
   $MODPROBE ip_tables
fi


#Load the IPTABLES filtering module - "iptable_filter" 
#
#  - Loaded automatically when filter policies are activated


#Load the stateful connection tracking framework - "ip_conntrack"
#
# The conntrack  module in itself does nothing without other specific 
# conntrack modules being loaded afterwards such as the "ip_conntrack_ftp"
# module
#
#  - This module is loaded automatically when MASQ functionality is 
#    enabled 
#
#  - Loaded manually to clean up kernel auto-loading timing issues
#
echo -en "ip_conntrack, "
#Verify the module isn't loaded.  If it is, skip it
if [ -z "` $LSMOD | $GREP ip_conntrack | $AWK {'print $1'} `" ]; then
   $MODPROBE ip_conntrack
fi

#Load the FTP tracking mechanism for full FTP tracking
#
# Enabled by default -- insert a "#" on the next line to deactivate
#
echo -e "ip_conntrack_ftp, "
#Verify the module isn't loaded.  If it is, skip it
if [ -z "` $LSMOD | $GREP ip_conntrack_ftp | $AWK {'print $1'} `" ]; then
   $MODPROBE ip_conntrack_ftp
fi

#Load the general IPTABLES NAT code - "iptable_nat"
#  - Loaded automatically when MASQ functionality is turned on
# 
#  - Loaded manually to clean up kernel auto-loading timing issues
#
echo -en "iptable_nat, "
#Verify the module isn't loaded.  If it is, skip it
if [ -z "` $LSMOD | $GREP iptable_nat | $AWK {'print $1'} `" ]; then
   $MODPROBE iptable_nat
fi

#Loads the FTP NAT functionality into the core IPTABLES code
# Required to support non-PASV FTP.
echo -e "ip_nat_ftp"
#Verify the module isn't loaded.  If it is, skip it
if [ -z "` $LSMOD | $GREP ip_nat_ftp | $AWK {'print $1'} `" ]; then
   $MODPROBE ip_nat_ftp
fi

echo "  ---"

#CRITICAL:  Enable IP forwarding since it is disabled by default since
echo "  Enabling forwarding.."
echo "1" > /proc/sys/net/ipv4/ip_forward

echo "  Enabling DynamicAddr.."
echo "1" > /proc/sys/net/ipv4/ip_dynaddr

echo "  ---"

#Clearing any previous configuration
#
#  Unless specified, the defaults for INPUT, OUTPUT, and FORWARD to DROP
#
#    You CANNOT change this to REJECT as it isn't a vaild policy setting.
#    If you want REJECT, you must explictly REJECT at the end of a giving 
#    INPUT, OUTPUT, or FORWARD chain
#
echo "  Clearing any existing rules and setting default policy to DROP.."
$IPTABLES -P INPUT DROP
$IPTABLES -F INPUT 
$IPTABLES -P OUTPUT DROP
$IPTABLES -F OUTPUT 
$IPTABLES -P FORWARD DROP
$IPTABLES -F FORWARD 
$IPTABLES -F -t nat

#Not needed and it will only load the unneeded kernel module
#
#$IPTABLES -F -t mangle

# Delete all User-specified chains
$IPTABLES -X

# Reset all IPTABLES counters
$IPTABLES -Z

#Configuring specific CHAINS for later use in the ruleset
#
#  NOTE:  Some users prefer to have their firewall silently
#         "DROP" packets while others prefer to use "REJECT"
#         to send ICMP error messages back to the remote 
#         machine.  The default is "REJECT" but feel free to
#         change this below.
#
# NOTE: Without the --log-level set to "info", every single
#       firewall hit will goto ALL vtys.  This is a very big
#       pain.
#
echo "  Creating a DROP chain.."
$IPTABLES -N reject-and-log-it
$IPTABLES -A reject-and-log-it -j LOG --log-level info 
$IPTABLES -A reject-and-log-it -j REJECT

echo -e "\n   - Loading INPUT rulesets"

#######################################################################
# INPUT: Incoming traffic from various interfaces.  All rulesets are 
#        already flushed and set to a default policy of DROP. 
# loopback interfaces are valid.
#
$IPTABLES -A INPUT -i lo -s $UNIVERSE -d $UNIVERSE -j ACCEPT


# local interface, local machines, going anywhere is valid
#
$IPTABLES -A INPUT -i $INTIF -s $INTNET -d $UNIVERSE -j ACCEPT
#Oaf: second NIC:
$IPTABLES -A INPUT -i $INTIF2 -s $INTNET2 -d $UNIVERSE -j ACCEPT

# remote interface, claiming to be local machines, IP spoofing, get lost
$IPTABLES -A INPUT -i $EXTIF -s $INTNET -d $UNIVERSE -j reject-and-log-it

# external interface, from any source, for ICMP traffic is valid
$IPTABLES -A INPUT -i $EXTIF -p ICMP -s $UNIVERSE -d $EXTIP -j ACCEPT


# Allow any related traffic coming back to the MASQ server in.
#  STATEFULLY TRACKED
$IPTABLES -A INPUT -i $EXTIF -s $UNIVERSE -d $EXTIP -m state --state \
 ESTABLISHED,RELATED -j ACCEPT

#
# ----- End OPTIONAL INPUT Section -----

# Catch all rule, all other incoming is denied and logged. 
#
$IPTABLES -A INPUT -s $UNIVERSE -d $UNIVERSE -j reject-and-log-it


# ---------------------------------------------------------------------

echo -e "   - Loading OUTPUT rulesets"

#######################################################################
# OUTPUT: Outgoing traffic from various interfaces.  All rulesets are 
#         already flushed and set to a default policy of DROP. 
# Workaround bug in netfilter
# See http://www.netfilter.org/security/2002-04-02-icmp-dnat.html
#
$IPTABLES -A OUTPUT -m state -p icmp --state INVALID -j DROP

# loopback interface is valid.
#
$IPTABLES -A OUTPUT -o lo -s $UNIVERSE -d $UNIVERSE -j ACCEPT

# local interfaces, any source going to local net is valid
#
$IPTABLES -A OUTPUT -o $INTIF -s $EXTIP -d $INTNET -j ACCEPT
# Oaf eth2 interface
$IPTABLES -A OUTPUT -o $INTIF2 -s $EXTIP -d $INTNET2 -j ACCEPT
$IPTABLES -A OUTPUT -o $INTIF -s $EXTIP -d $INTNET2 -j ACCEPT
$IPTABLES -A OUTPUT -o $INTIF2 -s $EXTIP -d $INTNET -j ACCEPT

# local interface, MASQ server source going to the local net is valid
#
$IPTABLES -A OUTPUT -o $INTIF -s $INTIP -d $INTNET -j ACCEPT
# Oaf eth2 interface

$IPTABLES -A OUTPUT -o $INTIF2 -s $INTIP2 -d $INTNET2 -j ACCEPT
# outgoing to local net on remote interface, stuffed routing, deny
#
$IPTABLES -A OUTPUT -o $EXTIF -s $UNIVERSE -d $INTNET -j reject-and-log-it

# anything else outgoing on remote interface is valid
#
$IPTABLES -A OUTPUT -o $EXTIF -s $EXTIP -d $UNIVERSE -j ACCEPT

# ----- Begin OPTIONAL OUTPUT Section -----
#
# DHCPd - Enable the following lines if you run an INTERNAL DHCPd server
#         - Remove BOTH #s all the #s if you need this functionality.
#
#$IPTABLES -A OUTPUT -o $INTIF -p tcp -s $INTIP --sport 67 \
# -d 255.255.255.255 --dport 68 -j ACCEPT
#$IPTABLES -A OUTPUT -o $INTIF -p udp -s $INTIP --sport 67 \
# -d 255.255.255.255 --dport 68 -j ACCEPT

#
# ----- End OPTIONAL OUTPUT Section -----


# Catch all rule, all other outgoing is denied and logged. 
#
$IPTABLES -A OUTPUT -s $UNIVERSE -d $UNIVERSE -j reject-and-log-it


echo -e "   - Loading FORWARD rulesets"

#######################################################################
# FORWARD: Enable Forwarding and thus IPMASQ
#

# ----- Begin OPTIONAL FORWARD Section -----
#
#  Put PORTFW commands here
#
# ----- End OPTIONAL FORWARD Section -----


echo "     - FWD: Allow all connections OUT and only existing/related IN"
$IPTABLES -A FORWARD -i $EXTIF -o $INTIF -m state --state ESTABLISHED,RELATED \
 -j ACCEPT

$IPTABLES -A FORWARD -i $INTIF -o $EXTIF -j ACCEPT

#Oaf eth2
$IPTABLES -A FORWARD -i $EXTIF -o $INTIF2 -m state --state ESTABLISHED,RELATED \
 -j ACCEPT
$IPTABLES -A FORWARD -i $INTIF2 -o $EXTIF -j ACCEPT


# Catch all rule, all other forwarding is denied and logged. 
#
$IPTABLES -A FORWARD -j reject-and-log-it


echo "     - NAT: Enabling SNAT (MASQUERADE) functionality on $EXTIF"
#
#More liberal form
#$IPTABLES -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE
#
#Stricter form
$IPTABLES -t nat -A POSTROUTING -o $EXTIF -j SNAT --to $EXTIP


#################################################################

Any ideas please?

I just want to be able to get the two subnets talking to each other

Thanks for reading this far

Oaf

[alexkoon]

Not a direct answer - but if I read you correctly, wouldn't it have been easier to setup the second NIC to be on the same network is your eth1? This way there should be no iptables routing. All machines on the new eth2 will just have to route through that NIC to hit eth1 and its routing. Or have I read this wrong?

[theoaf]

...wouldn't it have been easier to setup the second NIC to be on the same network is your eth1? This way there should be no iptables routing. All machines on the new eth2 will just have to route through that NIC to hit eth1 and its routing. Or have I read this wrong?

Hi

Thanks for your reply. That had been my original idea (I guess really it's the third NIC) but then I thought separate subnets for eth1 and eth2 would mean I didn't have to worry about masks... beginning to wonder now

I'll have another look later and see if I can progress it further. Networking isn't really my thing but I have a suspicion that I'm not THAT far from sorting it!

Cheers

Oaf

[directhex]

alexkoon is right about it being easier to simply set up a better subnet mask than 255.255.255.0 - the catch-all solution being 255.255.0.0 which means 192.168.*.* can talk to each other

[theoaf]

Ah that sounds like a good plan. Will try it later today and let you know how I get on.

Thanks

Oaf