IPv4 is running out, get ’em while they’re hot!

If you are in business of building and configuring Linux servers, there will come a day where you need to configure a server with multiple Network Interface Cards and/or multiple IP Address.
I had just such a task.

Before we go on, the rest of this article is going to focus on Ethernet Network Interfaces of Red Hat Enterprise Linux 7.5 servers running on AWS EC2.

I was tasked with building a server in the AWS cloud running on an EC2 instance; and this server needed to have a NICs in multiple subnets.

When you spin up an instance in EC2 with multiple ENIs, the OS will be automatically configured to use them; but only for Amazon Linux and Windows, even then it is not totally reliable… and I was using RHEL7.5 besides. Hence the configuration must be done manually.

There are many tutorials online that tell you how to configure multiple NICs or IPs, either by running a series of commands or editing a few files.

Run ‘route add 1.2.3.4/5 6.7.8.9’, Create file ‘ifcfg-eth1’, etc.

But what are those commands doing? How to you extrapolate or automate the process?

lets break it down.

What files are needed?

Each attached network interface will be displayed when you run ‘ifconfig’

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

$ sudo ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
  inet 10.42.8.87  netmask 255.255.255.0  broadcast 10.41.8.255
  inet6 fe81::f4:29ff:fefc:9f9e  prefixlen 64  scopeid 0x20<link>
  ether 03:f4:29:fc:9f:9e  txqueuelen 1000  (Ethernet)
  RX packets 5343008  bytes 959117490 (914.6 MiB)
  RX errors 0  dropped 0  overruns 0  frame 0
  TX packets 9066476  bytes 1671391051 (1.5 GiB)
  TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
  inet 127.0.0.1  netmask 255.0.0.0
  inet6 ::1  prefixlen 128  scopeid 0x10<host>
  loop  txqueuelen 1000  (Local Loopback)
  RX packets 0  bytes 0 (0.0 B)
  RX errors 0  dropped 0  overruns 0  frame 0
  TX packets 0  bytes 0 (0.0 B)
  TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Each interface that is configured will have the following files in /etc/sysconfig/network-scripts/:
1 or multiple ‘ifcfg-ethX:Y’ (1 per IP per NIC)
A ‘route-ethX’

Note that the ethX starts at 0, and :Y is only required for non primary IPs
e.g. if eth1 had multiple IPs, then it would be eth1 (note that absence of :Y for the primary IP), eth1:1 (2nd IP), eth1:2 (3rd IP), etc.

The various commands that other tutorials tell you to run, are generally just creating these files in the background.

An example of these file:
A server has 2 NICs and the second NIC has 3 IP addresses.

It would therefore have the following files:
ifcfg-eth0, ifcfg-eth1, ifcfg-eth1:1, ifcfg-eth1:2, route-eth0, an route-eth1

Now you know the pattern of the files that need to be created, all that is left is the contents of those files.

What do the files contain?

‘ifcfg-ethX’ files will contain the following:

1
2
3
4
5
6

BOOTPROTO=dhcp
DEVICE=eth<X>
HWADDR=<MAC>
ONBOOT=yes
TYPE=Ethernet
USERCTL=no

1
2
3
4
5
6
7
8
9

'ifcfg-ethX:Y' files contain:
BOOTPROTO=static
DEVICE=eth<X>:<Y>
HWADDR=<MAC>
IPADDR=<IP>
NETMASK=255.255.255.255
ONBOOT=yes
TYPE=Ethernet
USERCTL=no

‘route-ethX’ files contain:

1

<CIDR> via <GW> dev eth<X>

<X> is the interface number e.g. eth0, eth1, eth2, etc.

<Y> is the IP number e.g. eth1, eth1:1, eth1:2, etc.

<MAC> is the MAC address of the interface.

This can be found in 2 ways: 

1. The ’ether’ value for that interface when running ‘ifconfig ethX’
2. The contents of the file’ /sys/class/net/eth<X>/address’

<IP> is the IP address that this file is assigning to this interface

<CIDR> this is the range of IPs that when being communicated with, this interface is to be used (in CIDR notation)

<GW> this is the Gateway of the subnet this NIC is in, typically the first IP of the subnet

I will not be going into detail on all the possible settings but what I have given is for a fairly typical build.

Now that you know what files need to be created and what they should contain, lets work through an example.

Example Scenario, please?

Sure!

Say you have a server with 2 NICs:

• A primary interface (eth0) with one IP address of 10.0.0.10 and a mac address of 01:01:01:01:01:01
• A secondary NIC with a mac address of 02:02:02:02:02:02 and 2 IP addresses: 10.0.64.10 (primary) and 10.0.64.11 (secondary)

Note that there is a single ‘primary’ IP address, all additional IP addresses are called ‘secondary’ IPs.

Also, the different NICs are in different subnets. The primary NIC is used to talk the subnet 10.0.0.0/18 and the second NIC is used when talking to servers in the subnet range 10.0.64.0/18.

In this situation, you would have to create the following files with the accompanying contents:

1
2
3
4
5
6
7

ifcfg-eth0
BOOTPROTO=dhcp
DEVICE=eth0
HWADDR=01:01:01:01:01:01
ONBOOT=yes
TYPE=Ethernet
USERCTL=no

ifcfg-eth1

1
2
3
4
5
6

BOOTPROTO=dhcp
DEVICE=eth1
HWADDR=02:02:02:02:02:02
ONBOOT=yes
TYPE=Ethernet
USERCTL=no

ifcfg-eth1:1

1
2
3
4
5
6
7
8

BOOTPROTO=static
DEVICE=eth1:1
HWADDR=02:02:02:02:02:02
IPADDR=10.0.64.11
NETMASK=255.255.255.255
ONBOOT=yes
TYPE=Ethernet
USERCTL=no

route-eth0

1

10.0.0.0/18 via 10.0.0.1 dev eth0

route-eth1

1

10.0.64.0/18 via 10.0.64.1 dev eth1

Now you know what files to create, what to put in them and have worked through an example.

What’s next?

That’s it for this article but hopefully you learnt something. Not just told what commands to run, but why and what they do.
If you want to learn more, there are a lot of other tutorials and documentation on the internet so feel free to dive as deep as you want.