Maliciously malformed packets take advantage of vulnerabilities in operating systems and applications by intentionally altering the content of data fields in network protocols. These vulnerabilities may include causing a system crash (a form of denial of service) or forcing the system to execute the arbitrary code.

An example of malformed packet vulnerability is Cisco Security Advisory cisco-sa-20140611-ipv6, wherein vulnerability in parsing malformed IPv6 packets in a certain series of routers could cause a reload (reboot) of a certain card that carries network traffic, which could intermittently cause service outages.

Another example of this kind of vulnerability is in some unpatched Windows or Linux systems that will crash if they receive a series of fragmented packets where the fragments overlap each other.

The types and possibilities of malformed packets are endless, but vulnerabilities are usually announced as they are discovered and some may provide packet details. You can build display filters and/or build coloring rules in Wireshark to detect these packets. It also helps to study and understand what range of values the different protocol fields normally and legally contain, and what TCP and other protocol sequences normally look like so you can spot suspicious contents in packet flows.