Build a Dedicated Hardware Ethernet Sniffer (100FDX)

Background

Ethernet is by far the most popular local area communications system, and starting to pick up speed in wide area communications too. Ethernet can operate over various hardware options, including a wide range of encapsulations. For example, most home ADSL Internet connections use IP (Internet Protocol) encapsulated with PPP (Point-to-Point Prototol) encapsulated with Ethernet which in turn is encapsulated on ATM (Asynchronous Transfer Mode). However, despite the exotic possibilities, you will generally find Ethernet running on two twisted pairs at 100Mbit in most homes and most datacenters. Gradually the use of four twisted pairs running gigabit Ethernet is becoming popular, but for most situations 100Mbit is enough.

Why sniff ethernet traffic?

Sniffing traffic means taking a copy of the information on the wire and examining the data. The primary use of this technique is for debugging network software -- sniffing provides visibility into what it happening, so you can figure out what works and what doesn't. Another important use is for security purposes. If someone is breaking into your system (or has already broken in) they will always produce some additional packets on your Ethernet connection. Detecting these packets allows you to find the footprints of the culprit and know what they are doing, maybe even discover who they are. It can also be useful to generate a summary of data for accounting the usage of your link.

Software sniffing

There are a number of options for software sniffing. The simplest is to jump onto your linux router and run either the tcpdump command or the wireshark command depending on your preference. Sadly, not all routers provide these facilities. You may be working with a Cisco router, where the ability to capture and examine packets is absolutely woeful. You may also be using proprietary devices that don't allow command line access, or embedded linux routers that do allow command line access but have been stripped back such that no debugging tools are available.

Another option for software sniffing is the "man in the middle". This involves a dedicated sniffer box with two ports configured as a software "bridge". Actually, the Linux bridging code tries to work a bit like a smart switch with a MAC table that is learned over time and optional spanning tree protocol.

There are good things and bad things about this design. On the good side, it only requires standard hardware with no modification, and nearly any PC-style system can do it. On the bad side, it does interfere with the traffic, partly caused by the somewhat arbitrary time delay across the software bridge, partly because it tries to emulate a real switch, and further because sooner or later something will trigger the bridge to transmit additional packets into the network and reveal the fact that it is listening. Worse than that, it can't really do 100FDX because of the limitations of the bridging. Finally, something many people don't consider is that if the sniffer breaks down, or some software bug opens a security hole, then the link will be doomed. This may not even be directly a problem with the bridge, it could be any other software that happens to be on the same box, potentially causing a problem with the bridging.

Hardware sniffing

To get beyond the limitations of the software bridge, a hard-wired crossover will guarantee that packet timing and bandwidth is perfectly preserved. Cutting the TX lines also guarantees that no additional packets will be injected into the network, no matter what the circumstances. Software crash or even complete power failure on the sniffer box will not have any effect noticable to the network being sniffed. The data is received on two different NICs so it either requires a "bonding" configuration of the ports or suitable software that can sniff two ports simultaneously and merge the output (e.g. mergecap).

This block diagram gives the basic outline of the hardware modification below.

Suitable hardware

There are a great many makes and models of PC board out there. The basic desirable properties are:

The hardware option used here is a Soekris net5501. You can buy them in Australia from Yawarra Information Appliances, or order them direct from Soekris in the USA. The net5501 has four Ethernet ports, each one using a different IRQ line (for better performance) and a 500MHz AMD Geode chip (not great, but good enough and runs at low power with neither heatsink nor fan).

The net5501 also has protection diode networks both before and after the transformer so this provides plenty of available space for wiring on top of the board, and it is easy to get to the tracks (which are also on the top of the board). Thus, it is reasonably easy to modify (if you have either good eyes or a microscope but that's true of all modern computer hardware).

Crossover wiring

In the photograph you can see one port wired to another port with fine wire wrap wire. Careful examination reveals that the polarity of each link remains consistent on either port (red is always on the left, blue on the right). The link between the ports is wired as a crossover, there are several reasons for this: it ensures that each signal goes to an RX circuit; and it means that external wiring for these ports will be identical as if the same ports were configured as a software bridge.

Cutting tool

For cutting tracks, you can use either a knife, or a Dremel tool using a fine dentist-drill type of cutting bit.

Cut TX lines and remove RX termination resistors

Most important is to ensure that the transmit circuits on the sniffer ports are completely disconnected from the cables. The best way to do that is to cut the track. The small pink "X" near the top (just above the "Pulse" transformer) shows where the tracks are cut (four cuts need to be made in total).

As well as cutting TX lines, it is also a good idea to remove the termination resistors on the RX circuit. These are just underneath the "Pulse" transformer, also marked with a pink "X". Termination resistors are each 50 ohm surface mount resistors, to a center tap. That makes a total of four resistors, working in pairs to provide 100 ohm termination for each of the RX circuits. As per the usual rule, termination must only exist at the ends of the circuit, since our receivers are the "man in the middle" in this configuration, the ideal situation is for this circuit to have no termination (and as little influence on the signal as possible).

A fine soldering iron is required to desolder these termination resistors. With care, they can be stored and replaced on the board at a future date if necessary. Note that the RX input the NIC chip itself loads the circuit with approximately 1k ohm due to its intrinsic input impedance. Although not ideal, the 1k impedance is ten times as big as the characteristic line impedance (100 ohm) and the proper termination impedance (also 100 ohm). This makes it unlikely to interfere with the signal, but in cases where the cables are long or otherwise near the limit, this problem may be noticable.

Software consideration

The software collecting traffic depends very much on what you want to do with what you collect. Presuming the main purpose is debugging, you will need a program such as wireshark to examine the packets in meaningful format. You can also use wireshark to collect data directly from the interface, but there's a bit of a trick here. We have two interfaces to collect -- each one gives half of the conversation. This is completely necessary if we are to be able to collect data from 100FDX communication.

There is a handy program called mergecap, that merges two PCAP files together into a single output. This is a good way to put the whole conversation into a single file. There are times when you do not want to merge the traffic, such as when you have an ethernet broadcast (e.g. STP) and you are trying to figure out which side it is coming from.