If you've been in the IT business for any length of time, you're most likely familiar with the good old TCP three-way handshake. If you're not, no worries. Here's the simplified explanation.

Let's say that you're sitting at your workstation, and you pull up Firefox to visit a website. To access that website, your workstation and the web server have to set up the connection. Here's what happens:

• Your workstation sends a packet with only the SYN flag set to the web server. This is your workstation's way of saying, "Hello, Mr. Server. I'd like to make a connection with you."
• After receiving your workstation's SYN packet, the web server sends back a packet with the SYN and ACK flags set. With this, the server is saying, "Yeah, bro. I'm here, and I'm willing to connect with you."
• Upon receipt of the SYN-ACK packet, the workstation sends back a packet with only the ACK flag set. With this, the workstation is saying, "Cool deal, dude. I'm glad to connect with you."
• Upon receipt of the ACK packet, the server sets up the connection with the workstation so that they can exchange information.

This sequence works the same way for setting up any kind of TCP connection. This includes connections that involve Secure Shell, telnet, and the various mail server protocols, among other things.

However, clever people can use a variety of tools to craft TCP packets with some very weird combinations of flags. With these so-called invalid packets, a few things could happen:

• The invalid packets could be used to elicit responses from the target machine in order to find out what operating system it's running, what services are running on it, and which versions of the services are running.
• The invalid packets could be used to trigger certain sorts of security vulnerabilities on the target machine.
• Some of these invalid packets require more processing power than what normal packets require, which could make them useful for performing DoS attacks.

Now, in truth, the DROP all rule at the end of the filter table's INPUT chain will block some of these invalid packets. However, there are some things that this rule might miss. And even if we could count on it to block all the invalid stuff, this still isn't the most efficient way of doing things. By depending on this DROP all rule, we're allowing these invalid packets to travel through the entire INPUT chain, looking for a rule that will let them through. When no ALLOW rule is found for them, they'll finally get blocked by the DROP all rule, which is the last rule in the chain. So, what if we could find a more efficient solution?

Ideally, we'd like to block these invalid packets before they travel through the entire INPUT chain. We could do that with a PREROUTING chain, but the filter table doesn't have a PREROUTING chain. Therefore, we need to use the PREROUTING chain of the mangle table instead. Let's start by adding these two rules:

 donnie@ubuntu:~$ sudo iptables -t mangle -A PREROUTING -m conntrack --ctstate INVALID -j DROP
 [sudo] password for donnie:
 donnie@ubuntu:~$ sudo iptables -t mangle -A PREROUTING -p tcp ! --syn -m conntrack --ctstate NEW -j DROP
 donnie@ubuntu:~$

The first of these rules will block most of what we would consider invalid. However, there are still some things that it misses. Due to this, we added the second rule, which blocks all NEW packets that are not SYN packets. Now, let's see what we have:

 donnie@ubuntu:~$ sudo iptables -L
 Chain INPUT (policy ACCEPT)
 target     prot opt source               destination         
 ACCEPT     all  --  anywhere             anywhere            
 ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
 ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh
 DROP       all  --  anywhere             anywhere            
 
 Chain FORWARD (policy ACCEPT)
 target     prot opt source               destination         
 
 Chain OUTPUT (policy ACCEPT)
 target     prot opt source               destination         
 donnie@ubuntu:~$

Hmm...

We can't see our new rules, can we? That's because, by default, iptables -L only shows rules for the filter table. We need to see what we've just placed into the mangle table, so let's do this, instead:

 donnie@ubuntu:~$ sudo iptables -t mangle -L
 Chain PREROUTING (policy ACCEPT)
 target     prot opt source               destination         
 DROP       all  --  anywhere             anywhere             ctstate INVALID
 DROP       tcp  --  anywhere             anywhere             tcp flags:!FIN,SYN,RST,ACK/SYN ctstate NEW
 
 Chain INPUT (policy ACCEPT)
 target     prot opt source               destination         
 Chain FORWARD (policy ACCEPT)
 target     prot opt source               destination         
 Chain OUTPUT (policy ACCEPT)
 target     prot opt source               destination         
 Chain POSTROUTING (policy ACCEPT)
 target     prot opt source               destination         
 donnie@ubuntu:~$

Here, we used the -t mangle option to indicate that we want to see the configuration for the mangle table. Something rather curious that you may have noticed is how iptables renders the rule that was created by the sudo iptables -t mangle -A PREROUTING -p tcp ! --syn -m conntrack --ctstate NEW -j DROP command. For some reason, it renders as follows:

DROP       tcp  --  anywhere             anywhere             tcp flags:!FIN,SYN,RST,ACK/SYN ctstate NEW

It looks strange, but don't let that throw you. It still means that it's blocking NEW packets that aren't SYN packets.

Previously, I mentioned that the iptables-persistent package won't save subsequent changes to your iptables rules. So, as things stand now, the mangle table rules that we just added will disappear once I reboot this virtual machine. To make these changes permanent, I'll use the iptables-save command to save a new file in my own home directory. Then, I'll copy the file over to the /etc/iptables directory, replacing the original one:

 donnie@ubuntu:~$ sudo iptables-save > rules.v4
 [sudo] password for donnie:
 donnie@ubuntu:~$ sudo cp rules.v4 /etc/iptables/
 donnie@ubuntu:~$

To test this, we'll use a handy utility called Nmap. It's a free utility that you can install on your Windows, Mac, or Linux workstation. Or, if you don't want to install it on your host machine, you can install it on one of your Linux virtual machines. It's in the normal repositories of Debian/Ubuntu, RHEL/CentOS 7, and RHEL/CentOS 8. So, just install the Nmap package with the appropriate install command for your distro. If you want to install Nmap on your Windows or Mac host machine, you'll need to download it from the Nmap website.

With our new mangle table rules in place, let's perform an XMAS scan of our Ubuntu machine. I have Nmap installed here on the Fedora workstation that I'm currently using, so I'll just use this for now. I can do this like so:

[donnie@fedora-teaching ~]$ sudo nmap -sX 192.168.0.15
 [sudo] password for donnie:
 Starting Nmap 7.70 ( https://nmap.org ) at 2019-07-26 21:20 EDT
 Nmap scan report for 192.168.0.15
 Host is up (0.00052s latency).
 All 1000 scanned ports on 192.168.0.15 are open|filtered
 MAC Address: 08:00:27:A4:95:1A (Oracle VirtualBox virtual NIC)
 
 Nmap done: 1 IP address (1 host up) scanned in 21.41 seconds
 [donnie@fedora-teaching ~]$

By default, Nmap only scans the most commonly used 1,000 ports. The XMAS scan sends invalid packets that consist of the FIN, PSH, and URG flags. The fact that all 1,000 scanned ports show as open|filtered means that the scan was blocked, and that Nmap can't determine the true state of the ports. (In reality, port 22 is open.) We can view the result to see which rule did the blocking. (To simplify things a bit, I'll only show the output for the PREROUTING chain since it's the only mangle table chain that's doing anything):

donnie@ubuntu:~$ sudo iptables -t mangle -L -v
 Chain PREROUTING (policy ACCEPT 2898 packets, 434K bytes)
  pkts bytes target     prot opt in     out     source               destination    
     
  2000 80000 DROP  all  --  any    any     anywhere             anywhere             ctstate INVALID
     
  0     0 DROP       tcp  --  any    any     anywhere             anywhere             tcp flags:!FIN,SYN,RST,ACK/SYN ctstate NEW
 
 . . .
 . . .
 donnie@ubuntu:~$

Here, you can see that the first rule – the INVALID rule – blocked 2,000 packets and 80,000 bytes. Now, let's zero out the counter so that we can do another scan:

 donnie@ubuntu:~$ sudo iptables -t mangle -Z
 donnie@ubuntu:~$ sudo iptables -t mangle -L -v
 Chain PREROUTING (policy ACCEPT 22 packets, 2296 bytes)
 pkts bytes target prot opt in out source destination
 0 0 DROP all -- any any anywhere anywhere ctstate INVALID
 0 0 DROP tcp -- any any anywhere anywhere tcp flags:!FIN,SYN,RST,ACK/SYN ctstate NEW
 
 . . .
 . . .
 donnie@ubuntu:~$

This time, let's do a Window scan, which bombards the target machine with ACK packets:

[donnie@fedora-teaching ~]$ sudo nmap -sW 192.168.0.15
 Starting Nmap 7.70 ( https://nmap.org ) at 2019-07-26 21:39 EDT
 Nmap scan report for 192.168.0.15
 Host is up (0.00049s latency).
 All 1000 scanned ports on 192.168.0.15 are filtered
 MAC Address: 08:00:27:A4:95:1A (Oracle VirtualBox virtual NIC)
 
 Nmap done: 1 IP address (1 host up) scanned in 21.44 seconds
 [donnie@fedora-teaching ~]$

As before, the scan was blocked, as indicated by the message that all 1,000 scanned ports have been filtered. Now, let's view what we have on the target Ubuntu machine:

 donnie@ubuntu:~$ sudo iptables -t mangle -L -v
 Chain PREROUTING (policy ACCEPT 45 packets, 6398 bytes)
  pkts bytes target     prot opt in     out     source               destination         
     
  0     0 DROP       all  --  any    any     anywhere             anywhere             ctstate INVALID
 
  2000 80000 DROP       tcp  --  any    any     anywhere             anywhere             tcp flags:!FIN,SYN,RST,ACK/SYN ctstate NEW
 
 . . .
 . . .      
 donnie@ubuntu:~$

This time, we can see that our invalid packets got past the first rule, but were blocked by the second rule. 

Now, just for fun, let's clear out the mangle table rules and do our scans again. We'll use the -D option to delete the two rules from the mangle table:

 donnie@ubuntu:~$ sudo iptables -t mangle -D PREROUTING 1
 donnie@ubuntu:~$ sudo iptables -t mangle -D PREROUTING 1
 donnie@ubuntu:~$

When you delete a rule, you have to specify the rule number, just like you do when you insert a rule. Here, I specified rule 1 twice, because deleting the first rule moved the second rule up to first place. Now, let's verify that the rules are gone:

 donnie@ubuntu:~$ sudo iptables -t mangle -L
 Chain PREROUTING (policy ACCEPT)
 target prot opt source destination
 
 . . .
 . . .
 donnie@ubuntu:~$

Yes, they are. Cool. Now, let's see what we get when we perform another XMAS scan:

 [donnie@fedora-teaching ~]$ sudo nmap -sX 192.168.0.15
 [sudo] password for donnie:
 Starting Nmap 7.70 ( https://nmap.org ) at 2019-07-26 21:48 EDT
 Nmap scan report for 192.168.0.15
 Host is up (0.00043s latency).
 All 1000 scanned ports on 192.168.0.15 are open|filtered
 MAC Address: 08:00:27:A4:95:1A (Oracle VirtualBox virtual NIC)
 
 Nmap done: 1 IP address (1 host up) scanned in 21.41 seconds
 [donnie@fedora-teaching ~]$

Even without the mangle table rules, it shows that my scan is still blocked. What's up with that? This is happening because I still have the DROP all rule at the end of the INPUT table. Let's disable that and see what we get with another scan.

First, I need to see what the rule number is:

donnie@ubuntu:~$ sudo iptables -L
 Chain INPUT (policy ACCEPT)
 target     prot opt source               destination         
 ACCEPT     all  --  anywhere             anywhere            
 ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
 ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh
 DROP       all  --  anywhere             anywhere            
 
 Chain FORWARD (policy ACCEPT)
 target     prot opt source               destination         
 
 Chain OUTPUT (policy ACCEPT)
 target     prot opt source               destination         
 donnie@ubuntu:~$

Counting down, I can see that it's rule number 4, so I'll delete it:

donnie@ubuntu:~$ sudo iptables -D INPUT 4
donnie@ubuntu:~$donnie@ubuntu:~$ sudo iptables -L
 Chain INPUT (policy ACCEPT)
 target     prot opt source               destination         
 ACCEPT     all  --  anywhere             anywhere            
 ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
 ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh
 
 Chain FORWARD (policy ACCEPT)
 target     prot opt source               destination         
 
 Chain OUTPUT (policy ACCEPT)
 target     prot opt source               destination         
 donnie@ubuntu:~$

Now, for the XMAS scan:

[donnie@fedora-teaching ~]$ sudo nmap -sX 192.168.0.15
 Starting Nmap 7.70 ( https://nmap.org ) at 2019-07-26 21:49 EDT
 Nmap scan report for 192.168.0.15
 Host is up (0.00047s latency).
 Not shown: 999 closed ports
 PORT STATE SERVICE
 22/tcp open|filtered ssh
 MAC Address: 08:00:27:A4:95:1A (Oracle VirtualBox virtual NIC)
 
 Nmap done: 1 IP address (1 host up) scanned in 98.76 seconds
 [donnie@fedora-teaching ~]$

This time, the scan shows that 999 ports are closed and that port 22, the SSH port, is either open or filtered. This shows that the scan is no longer being blocked by anything.