Security analysis methodology

Security analysis at the packet level is based on detecting and analyzing suspect traffic, that is, the traffic that does not match normal patterns because of the presence of unusual protocol types or ports, or unusual requests, responses, or packet frequency. Suspicious traffic may include reconnaissance (discovery) sweeps, phone home behavior, denial of service attacks, botnet commands, or other types of behavior from direct attacks or virus- or botnet-based agents.

Wireshark captures strategic points in the network to investigate suspicious packets from specific hosts or on network segments and egress points can also complement any Intrusion Detection System (IDS) systems that may be in place to alert the IT staff about the suspicious traffic.

The importance of baselining

The ability to identify abnormal traffic patterns that bear investigation versus traffic caused by poorly behaving applications, misconfigurations, or faulty devices can be made much easier if you have a baseline of what is normal. A baseline is a snapshot capture of typical conversations with your primary applications and servers and the background traffic on the network segments that they reside on. In a potential security breach situation, you can compare the normal protocols, traffic patterns, and user sessions from a baseline with a current capture, filter out the normal traffic, and then inspect the differences.

To allow the comparison of baselines in your security analysis, you need to periodically capture and store packet trace files that cover a sufficient period of time to provide a good sample of typical user and background traffic patterns while keeping the file sizes manageable for use within Wireshark, for example, 100 MB to 1 GB per file. You can configure the Ring Buffer option within Wireshark's Capture Options window to save a series of reasonably sized files for longer captures or busier network segments.

Although your baselining needs and practices will depend on your environment, some of the traffic aspects that you should inspect include:

• Broadcast and multicast types and rates:
  • What devices and applications are using broadcasts and multicasts?
  • What are the typical broadcast and multicast packet rates?
• Applications and protocols:
  • What applications are running over the network?
  • What protocols and ports are they using?
  • Application launch sequences and typical tasks
  • Are application sessions encrypted?
  • Are all users forced to use encryption? Any exceptions?
  • What are the login/logout sequences and dependencies?
• Routing protocol(s) and routing updates
• ICMP traffic
• Boot-up sequences
• Name resolution sessions
• Wireless connectivity includes normal management, control, and data frame contents
• VoIP and video communications
• Idle time traffic is the host communicating with other hosts when there are no users logged in
• What backup processes are running at night and for how long?
• Are there any suspect protocols or broadcasts/scans taking place?

As you inspect your baseline captures, it is helpful to view a summary of the protocols being used by selecting Protocol Hierarchy from the Wireshark's Statistics menu. In the following screenshot, for example, you can see that there is some Internet Relay Chat (IRC) traffic, as well as the Trivial File Transfer Protocol (TFTP) traffic, neither of which might be normal on your network and could be an indication of rogue communications with outside entities:

Analyzing baselines of normal traffic levels and patterns is also an excellent way of getting familiar with your network environment and its typical packet flows and protocols, which better prepares you to spot abnormal traffic.