17. Components » Flow Sensor

Most routers and many enterprise switches are able to collect IP traffic statistics and export them as flow records to Flow Sensor. Since the flow protocol performs pre-aggregation of traffic data, the flows sent to Flow Sensor are much smaller than the monitored traffic. This makes Flow Sensor an excellent option for monitoring remote or high-traffic networks. The advantages and disadvantages of flow-based monitoring versus packet-based monitoring are listed in the Choosing a Method of Traffic Monitoring chapter.

Appendix 2 shows how to enable NetFlow, sFlow, or IPFIX on a number of devices, but the best and most up-to-date instructions can only be found in the vendor’s documentation.

To add a Flow Sensor, click the [+] button from the title bar of the Configuration » Components panel. To modify an existing Flow Sensor, go to Configuration » Components and click its name.

Flow Sensor Configuration parameters:

● Sensor Name – A short name to help you identify the Flow Sensor

● Sensor Visibility – Toggles the listing inside the Reports » Devices panel

● Device Group – Enter a description if you wish to organize components (e.g. by location, characteristics) or to permit fine-grained access for roles

● Sensor Server – Select a server that fulfills the minimum system requirements for running the Flow Sensor. If this is not the Console server, follow the NFS configuration steps to make the flow data visible in the web interface

● Sensor License – The license used by the Flow Sensor. Wanguard provides all features; Wansight does not provide traffic anomaly detection and reaction

● Listener IP:Port – The IP address (IPv4 or IPv6) of the network interface that receives flow packets, and the destination port set on the flow exporter. The destination port can be used only by a single flow exporter, so when multiple flow exporters are used, each of them must be configured to send flows to a different port

● Flow Protocol – The flow protocol used by the flow exporter:

▪ Netflow or IPFIX – Select if your router exports NetFlow v5, v7, v9 or IPFIX, then click on the options button on the right-hand side of the field for Netflow-specific options:

• Long Flows Timeout – Juniper MX maintains the start time of previously exported flows. If you’re using such a router then you need to set here the flow active/inactive timeout value defined in its configuration, the most common value being 60 seconds. For other flow exporters, leave to None

• Sampling (1/N) – Enter the sampling rate configured on the flow exporter, or leave the default value unchanged when no sampling rate is configured

• Force Sampling Value – Overrides the sampling value reported by the flow exporter, in case of misreadings

▪ sFlow – Select if your router exports flows using the sFlow protocol

● Exporter IP – IP address of the flow exporter (router, switch, probe). Usually, it is the loopback address of the router. For sFlow exporters, enter the IP that sends the flow packets, not the Agent IP

● Exporter SNMP – You must enable SNMP on the flow exporter to allow Console to extract interface information automatically

▪ Enabled – Click the button on the right-hand side of the Exporter SNMP field to configure the following options:

▪ Disabled – When SNMP is not configured, you must add each interface manually, including its SNMP index, speed, etc.

● IP Zone – Flow Sensor learns from the IP Zone the network’s boundaries, and extracts per-subnet settings

● Flow Collector – When enabled, the raw flow data is saved on disk. You can query flow records in Reports » Tools » Flows and limit the disk space used by flows in General Settings » Data Retention

• Flow Filtering Expression - If set, it will restrict the flows that are analyzed by Flow Sensor. Click on the bookmark button to see the correct syntax

• Repeater IP:Port – Flow Sensor can retransmit flow packets to another flow collector. This feature is enabled if the field contains the IP of another flow collector and a destination port

• Compression Algorithm - Flow Sensor can compress the raw flow data using several algorithms. LZO offers the fastest compression. BZ2 offers the best compression rate, but it’s around 30 times slower than LZO. LZ4 and ZSTD offer a balance between speed and efficiency

• Compression Threads - Flow Sensor can use multiple threads to compress the raw flow data. On most systems, a single thread will suffice. If Flow Sensor consumes more than 50% CPU, consider increasing the thread number

• Compression Level - LZ4 and ZSTD can be configured with custom compression levels. If the field is empty or 0, the default compression level of the compression algorithm will be applied

● IP Validation – This parameter is frequently used for distinguishing the traffic’s direction (relative to the monitored network):

▪ Off – Disables IP Validation

▪ On – Flow Sensor examines only the flows that have the source IP and/or destination IP inside the selected IP Zone. When a flow has the destination IP inside the IP Zone, that traffic is considered inbound. When the source IP is inside the IP Zone, that traffic is considered outbound. This option simplifies the configuration of interfaces because the direction of each interface can remain set to Auto, but it doesn’t show inbound/outbound traffic as traffic entering/exiting the interface (the same way as SNMP Sensor and any other SNMP-based tool), but traffic entering/exiting the network

• Log Invalidated Flows – When set to Periodically, the event log will show the percentage of invalidated flows and ten flows failing IP validation, once every ten ticks

▪ Strict – Flow Sensor examines only the flows that have either the source IP or the destination IP inside the IP Zone

▪ Exclusive – Flow Sensor examines only the flows that have the destination IP inside the IP Zone

● AS Validation – BGP-enabled routers can export flows that contain the source and destination ASN (Autonomous System Number). In most cases, if the AS number is set to 0, then the IP address belongs to the local ASN. This option is rarely used for establishing traffic direction. AS Validation provides three choices:

▪ Off – Disables AS validation

▪ On – Flow Sensor examines only the flows that have the source ASN and/or the destination ASN inside the local AS list (defined below)

• Local AS List – Enter here your AS numbers, separated by space

• Log Invalidated Flows – When set to Periodically, the event log will show the percentage of invalidated flows and ten flows failing AS validation, once every ten ticks

▪ Strict – Flow Sensor examines only the flows that have either the source ASN or the destination ASN inside the local AS list (defined below)

● Granularity – Low values increase the accuracy of Sensor graphs at the expense of RAM usage. The default value of 20 seconds is recommended in most cases

● Time Zone – Set the time offset between the time zone of the Flow Sensor’s server and the time zone of the flow exporter. Running NTP on both devices to keep their clocks synchronized is extremely important

● Monitored Interfaces – This grid contains the interfaces that need to be monitored. To avoid producing duplicate flow entries, it is recommended to add only the upstream interfaces. To add interfaces one by one, click the Add Interface button. To add interfaces in bulk, click the Manage Interfaces button. The following parameters define each monitored interface:

▪ SNMP Index – Each interface is internally identifiable by its SNMP index. You can configure SNMP and have this number auto-filled, or you must extract it manually from the flow exporter and enter it for each interface

▪ Interface Name – A short description that identifies the monitored interface. Descriptions longer than ten characters might clutter some reports

▪ Interface Color – The color used in graphs for the interface. The default color is a random one. You can change it by clicking the drop-down menu

▪ Traffic Direction – Direction of the traffic entering the interface, relative to your network:

• Auto – This is the recommended option in most cases. When selected, the direction of traffic is established by IP and/or AS Validation, or by the router itself (some flow protocols send the direction inside flow data)

• Upstream – Set for upstream interfaces (e.g., peering interfaces, interfaces connected to the Internet)

• Downstream – Set for downstream interfaces (e.g., customer interfaces, interfaces connected to your network)

• Null – Traffic entering Null interfaces is discarded by the router so it’s also ignored by Flow Sensor

▪ Stats Engine – Collects various traffic tops and AS (Autonomous System) data:

• Basic – Enables tops for internal IPs (IPs included in the IP Zone), IP protocols, TCP/UDP ports, and IP versions

• Extended – Enables the tops from Basic as well as tops and graphs for Upstream ASNs and countries. It adds a minimal performance penalty. When the router is not exporting AS information in flows (e.g., non-BGP router), Flow Sensor uses an internal GeoIP database to obtain AS data, making live stats for autonomous systems slightly inaccurate if the GeoIP database contains obsolete information. This is the recommended value

◦ Refresh Interval – Set how often the MRT file should be reloaded in RAM. If it is set to Auto, then the file will be refreshed every time it is modified. It it is set to Never, the file will only be loaded when the Sensor starts

◦ BGP Dump File – If it contains the path to an existing file exported by BGPd in MRT format, it enables tops and graphs for Transit ASNs

◦ BGP Router IPv4/IPv6 – If the MRT file was entered, then you must also enter the IPv4 and/or IPv6 address of the next-hop BGP router. You can find the next-hop by observing the NEXT_HOP parameter from the bgpdump’s output

• Full – Enables the tops from Extended as well as tops for external IPs (IPs not included in the IP Zone). It increases the RAM usage several times over, especially during spoofed attacks from randomized sources. Live stats for autonomous systems and countries are very accurate. Only this option permits the detection of threshold violations for external IPs

▪ Link Speed In & Link Speed Out – Enter the interface’s speed (bandwidth, capacity). The values are used for percentage-based reports and percentage-based bits/s thresholds

● Comments – Comments about the Flow Sensor can be saved here. These observations are not visible elsewhere

To start the Flow Sensor, click the small on/off button displayed next to its name in the Configuration » Components panel. Make sure that the Flow Sensor starts correctly by watching the event log and the traffic values from Reports » Devices » Overview. If after 5 minutes the traffic values are not correct, follow the troubleshooting steps listed below.

Note

To obtain detailed information about the sources of the attacks detected by Flow Sensor, add a Flow Filter (the default parameters can be left unchanged), and then activate it in the “When an anomaly is detected…” panel of the Response. If the network supports BGP Flowspec, you could also perform DDoS mitigation directly on the router by configuring an ExaBGP Connector.

17.1. Flow Sensor Troubleshooting

✔ Look for warnings or errors produced by the Flow Sensor in the event log

✔ Go to Help » Software Updates to make sure that you are running the latest version of the software

✔ Check if you have correctly configured the Flow Sensor. Each configuration parameter is described in the previous section

✔ Check if the server is receiving flow packets on the configured Listener IP:Port by executing the following command which shows the first 100 packets received from the flow exporter:

[root@localhost ~]# tcpdump -i <interface_eth0_p1p1_etc> -n -c 100 host <flow_exporter_ip> and udp and port <destination_port>

✔ Check if the local firewall permits the Flow Sensor to receive flow packets:

[root@localhost ~]# ufw status || firewall-cmd --list-all || iptables -L -n -v && iptables -t raw -L -n -v

✔ Make sure that the clocks of the server and of the flow exporter are synchronized, preferably to the same NTP server. When both devices don’t reside in the same time zone, adjust the Time Zone parameter accordingly. To verify that the server is synchronized by NTP, execute:

[root@localhost ~]# ntpq -p || chronyc tracking || timedatectl status

✔ Check if the reverse path filtering mechanism is not dropping the packets:

[root@localhost ~]# netstat -s | grep Filter

✔ If the Flow Sensor takes too long to detect attacks, ensure that the flow exporter is configured to export flows in the shortest time possible. The maximum flow export time detected since the Flow Sensor started is shown in the Flow Delay column from Reports » Devices » Overview. Setting the Granularity parameter to 5 seconds will also help detect attacks faster in some cases. Not all flow exporters can export flows in a timely manner, some needing tens of seconds to compose and export flows. To detect attacks faster in this case it’s best to use Packet Sensor which can detect attacks in under 1 second.

✔ If the event log receives a warning like Received flow <starting/ending> <X> seconds ago, check the following:

▪ When the warning refers to the starting time, make sure that the server’s clock and flow exporter’s clock are synchronized and that the time zone is set correctly on both devices. For some routers, such as Juniper MX, it is necessary to set the Flows Timeout parameter to the same value (usually 30 seconds) as the one configured on the router. These routers maintain the start time of exported flows

▪ When the warning refers to the ending time, make sure that the clocks are synchronized, the time zone is set correctly, the flow exporter is properly configured, and the PFC PIC is not overloaded (on Juniper in particular). If the number of seconds is around a multiple of 8600 seconds (1 hour), then the time zones may not be the same on both devices

▪ In some JunOS versions, there is a flow export rate limit with a default of 1k pps, which leads to flow aging errors. To raise the limit to 40k pps you need to execute:

set forwarding-options sampling instance NETFLOW family inet output inline-jflow flow-export-rate 40

▪ Some Cisco IOS XE devices do not export flows using NetFlow version 5 in under 5 minutes, even when configured to do so. In this case, switch to using Flexible NetFlow

▪ In order to provide fast and up-to-date traffic statistics, Flow Sensor accepts only flows describing traffic that started and ended in the last 5 minutes. All flows aged and exported with a delay exceeding 300 seconds are therefore ignored. This is a design decision that can’t be changed

▪ Flow Sensor does not misinterpret the start/end time of flows. Some flow exporters are known to have bugs, limitations, or inconsistencies regarding flow aging and stamping flow packets with the correct time. In this case, contact your vendor to ensure that the flow exporter is correctly configured, runs the latest firmware, and can expire flows in under 5 minutes. In some cases, a router reboot will fix all these issues

▪ You can double-check whether the Flow Sensor’s time and the flow’s start/end time differ by more than 300 seconds by inspecting the raw flow data. In Reports » Tools » Flows » Flow Records, select any interface of the Flow Sensor, set Display to Extended, and generate a listing for the last 5 minutes:

◦ Column “Received Time” indicates the time when the Flow Sensor received the flow packet, according to the clock of the server

◦ Column “Start Time” indicates the time when the flow started, according to the clock of the flow exporter

◦ Column “Stop Time” indicates the time when the flow ended, according to the clock of the flow exporter

✔ The event log warning Sensor frozen for <X> seconds. Restarting the collector can have several causes. It is generated when the flow packets are too scarce (1 every few seconds), or when flow packets are not received for tens of seconds (e.g., due to a network outage or router reload). Another cause indicates a performance issue, with the Flow Sensor not having enough CPU and I/O resources to analyze traffic and send data to the SQL server in a timely manner. In this case, use a physical server instead of a virtual machine, or decrease from the IP Zone the IP graphs and IP accounting data that need to be collected

✔ If you don’t see traffic on some/all of your monitored interfaces, but you see in Reports » Devices » Overview that the Flow Sensor is receiving flows (in the “Flows/s” column), you need to check if you have correctly configured the flow exporter to send flows to the server for each of the monitored interfaces. To list the interfaces that are actually sending flows, go to Reports » Tools » Flows » Flow Tops, select any Flow Sensor interface, set Top Type to Any Interface, check Include Unmonitored Ifs in the Display Options selector, and generate a top for the last 10 minutes. The column “In/Out If” lists the SNMP index of every interface that exports flows, even if it wasn’t configured as a monitored interface in the Flow Sensor configuration

✔ When you add interfaces with the Traffic Direction parameter set to Auto, and IP Validation is being used, make sure that the IP Zone you have selected contains all your IP blocks. To capture a sample of flows failing validation in the event log, set the Log Invalidated Flows parameter to Periodically

✔ The traffic readings of the Flow Sensor may differ from other SNMP-based monitoring tools. If IP Validation is enabled, Flow Sensor counts In/Out traffic as traffic entering/exiting the IP Zone, unlike SNMP-based tools which show In/Out traffic as traffic entering/exiting the interface. To see if the traffic readings differ, add an SNMP Sensor and configure it to monitor the same flow exporter and the same interfaces (the Interface Discovery parameter from the Components » SNMP Sensor section will make this very easy)

✔ If the Flow Sensor does not show the correct statistics after upgrading the router’s firmware, the SNMP index of each interface has probably changed. In this case, adjust the SNMP indexes of the monitored interfaces manually, or redefine them

✔ If you only see statistics for a single traffic direction, either inbound or outbound, go to Reports » Tools » Flows » Flow Records and generate a listing for the last 10 minutes. If all your IPs are listed in a single column, check the flow exporter’s configuration and feature list. Not all devices can export flows in both directions or with the same SNMP index. Some Brocade routers are known to generate only inbound sFlow

✔ Flow Sensor could crash during spoofed attacks for not having enough RAM if a monitored interface has the Stats Engine parameter set to Full. It is highly recommended to set this parameter to Extended, especially on systems that don’t have enough RAM

✔ If the registered traffic is too low after upgrading to JunOS 15.1F2 or 16.1R1, execute:

set chassis fpc inline-services flow-table-size ipv4-flow-table-size 15

✔ To troubleshoot Sensor graph or IP graph issues, follow the Graphs Troubleshooting guide

✔ The event log error License key not compatible with the existing server indicates that the server is unregistered and you need to send the string from Configuration » Servers » [Server] » Hardware Key to sales@andrisoft.com