Understanding Disk Space Utilization


Like any other system resource, disk space is limited. Andrisoft Wanguard can collect and store very large quantities of data, depending on its configuration. If you think it's using up too much disk space, read through this article to understand what might be the cause and how to prevent the disk being filled in the future. When the disk is full on the Console server, it's no longer possible to log in to the application and some components will fail to run!


What is using up your precious disk space?

Unless you changed the default path for storing flows, graphs and dump files, you can use the ncdu utility or the following command to see what is using up your disk space (this may take up to a few minutes, depending on how many files you have):
du -ch -d 1 /var/lib/mysql/andrisoft /opt/andrisoft /var/lib/influxdb/data/andrisoft

Database

If the /var/lib/mysql/andrisoft directory occupies too much disk space, lower the parameters from Configuration » General Settings » Data Retention. The database is purged from old data once every hour, so the changes in Data Retention will not be immediately visible. Also, on some distributions, MySQL/MariaDB will not free up the deleted data and instead it will reserve the resulted disk space for future use. To reclaim the newly freed up disk space, execute the following command on the Console server (it might take a very long time to execute):
/opt/andrisoft/bin/WANmaintenance optimize_db
If the disk got filled then it's very likely that the database got corrupted and cannot be started. If this is the case, free up some disk space and execute:
/opt/andrisoft/bin/WANmaintenance repair_db

Packet Dumps (traces)

If the /opt/andrisoft/dumps directory occupies too much disk space, this means you have either done large packet dumps manually, or you have configured the software to capture large packet dumps in Configuration » Network & Policy » Responses.

  • Configure the software to delete old pcap files automatically in Configuration » General Settings » Data Retention » Packet Dumps.
  • Recheck the configuration of the Traffic Sample Capturing Action, going side by side with the User Manual, which explains each field in detail. Configuring the Max. Packets parameter is the best way to avoid unnecessarily large dump files.
  • If you need to clear up some disk space immediately, you can manually delete any files/directories from within the /opt/andrisoft/dumps directory but not the dumps directory itself. If you decide to delete directories or files manually, please allow the software to create them back automatically and do not attempt to recreate them by yourself.

Flows

If the /opt/andrisoft/flows directory occupies too much disk space, this means the Flow Sensor is receiving lots of flows and you have chosen to save them by enabling the Flow Collector option.

  • As a first attempt to lower the disk space consumption of flows, select the "Saved compressed flows" option within the Flow Sensor Configuration window. More information can be found here.
  • Configure the software to delete old flows automatically in Configuration » General Settings » Data Retention » Flow Collectors.
  • If you need to clear up some disk space immediately, you can delete any files/directories manually from within the /opt/andrisoft/flows directory but not the flows directory itself. If you decide to manually delete directories or files, please allow the software to create them back automatically and do not attempt to recreate them by yourself.

IP Graphs (RRDTool)

If the /opt/andrisoft/graphs directory occupies too much disk space, this means you have either configured the software to generate graph data files for a very large number of IPs, or you have configured graph data files to contain an unnecessarily large amount of information.

  • Since version 7.2 is possible to use InfluxDB instead of RRD files to store graph data. InfluxDB uses less disk space, so you should seriously consider using it.
  • To find out how much disk space each IP graph file requires, go to Configuration » General Settings » Storage & Graphs. Tweaking the Accuracy of Round Robin Archive, Decoders, Stored Units or the Consolidation Functions modifies the IP graph file size, which is dynamically displayed in the bottom section of the Configuration window. It is possible that a lot of unnecessary information is stored for each IP and therefore it is best to reconsider the necessity of each option while consulting the User Manual here. The configuration is applied to all IP graph files, and it is currently not possible to choose different configuration options for different IP prefixes.
  • We strongly suggest disabling IP graphing for very large IPv4 prefixes or IPv6 prefixes from the IP Zone Configuration window (Configuration » Network & Policy » IP Zone). Depending on what you have configured in the Storage & Graphs window, the software calculates how much storage is required in case you choose to turn on IP graphing for any IPv4 prefix. This is displayed in the "Storage Requirements" column of the IP Settings section, in the IP Zone Configuration window. You can immediately see that turning IP graphing on for large IP prefixes is not a good idea.
  • *IMPORTANT* It is important to know that the software generates IP Graph files at their maximum size and updates them accordingly without modifying the size afterward. Therefore, you cannot rely on the fact that an IP has little traffic and won't occupy much space. An IP graph file is generated at the first packet that has that IP as a source or a destination address. A simple network scan on a /8 prefix for which IP graphing is enabled can flood your hard disk with full-sized IP graphing files.
  • You can aggregate IP graphing information collected by multiple Packet Sensors or Flow Sensor interfaces by using the Sensor Cluster component.
  • If you need to clear up some disk space immediately, you can manually delete any files/directories from within the /opt/andrisoft/graphs directory but not the graphs directory itself. If you decide to delete directories or files manually, please allow the software to create them back automatically and do not attempt to recreate them by yourself. To delete the IP graph files which were not updated in the last 90 days, execute:
    find /opt/andrisoft/graphs/ips -mtime +90 -type f -exec /bin/rm {} \;
    

IP Graphs (Influxdb)

If the /var/lib/influxdb/data/andrisoft directory occupies too much disk space, then you probably configured the software to generate graph data files for a very large number of IPs.
  • The software can be configured to store graph data locally or on a remote server. If the existing Console doesn't have enough disk space, you could use another server just for storing graphs.
  • By default, the graph data is saved in influxdb under the database "andrisoft". You can delete the existing database by executing the "DROP DATABASE andrisoft" command from within influxdb CLI. Then, you can recreate the database in Configuration » General Settings » Storage & Graphs.

If you require further assistance, please contact support andrisoft.com.



Author
Andrisoft Team
Date Created
2015-06-19 11:27:37
Date Updated
2023-08-06 00:03:18
Views
13129