OpenVAS Change Request #52: Tighter integration of nmap
Status: Voted +5. Done as of SVN rev. 11026-11029,11033,11057,11067-11070.
Run nmap only once per network for port scanning, service and OS detection, traceroute and NSE, thus highly increasing performances and maintainability.
This change request proposes a complete rewriting of the way nmap is used by OpenVAS. Nmap is currently executed once per target for portscanning and OS detection, once per target for service detection and once per NSE script per target if you enable them (for a total of about 80 executions per host).
In case network wide scanning is enabled, then portscanning is performed only once, but there is still one nmap instance per NSE per target. (We have 77 supported NSE in OpenVAS, but the number of available NSE scripts is increasing rapidly). These NSE wrappers perform new port scans (either partial or complete ones).
Nmap has been designed to handle parallelism and is more efficient when launched once against multiple targets than multiple times against single targets. This statement is especially true in term of memory usage. This waste of memory, time and bandwidth could be avoided by having nmap executed only once, at the beginning of the vulnerability scan. This can be achieved by using the network wide scan phase and rewriting the related NVTs.
The most obvious effect is a massive performances increase (strongly reduced scan time and memory footprint) but these changes also means a cleanup of the network (unauthenticated) scanning stack and possibly more accurate results. Proposed changes are also intended to ease the transition to newer nmap releases.
- alive hosts detection (don't exclusively use ICMP)
- port scanning
- service/version detection
- OS fingerprinting
- traceroute (parallel, smart, adaptive and multiprotocol)
- NSE scans
- misc low level informations (incremental IP ID's, broken TCP seq generation...)
During the testing phase, a standard NSE scan for a /24 network (255 addresses, about 20 hosts up) ran 60 times faster than when using the current system.
Design and Implementation
The main change is the introduction of a C wrapper for nmap, exporting a new function run_nmap() to the NASL scripts.
This function builds a nmap command line according to the scan configuration, runs nmap against the whole network and stores the results in the knowledge base (KB). The ability to parse XML files is also implemented.
A C implementation was chosen because it allows to use (via the Glib XML parser) the XML output of Nmap. OpenVAS currently uses the greppable one, which is deprecated and not updated anymore. Therefore, the plugin can be easily extended and migration to support newer versions of nmap should be simpler.
When setting network_scan="yes", ACT_INIT, ACT_SCANNER and ACT_SETTINGS NVT's are executed twice. The first execution is done against the whole network, the second one against each host. The proposed changes heavily rely upon this mechanism (already implemented after was CR #49 accepted).
It is necessary to change the scanner behaviour regarding ACT_INIT scripts in order to handle dependencies correctly for the new NSE wrappers. These wrappers need to be executed a first time before nmap_exp.nasl, to register themselves, but only if they have been explicitely selected. ACT_INIT (and ACT_SETTINGS) scripts were always scheduled for execution. This was problematical as the new NSE wrappers were then automatically selected with this system. Decision has been taken to keep the new NSE wrappers in this category and change the scanner so that ACT_INIT scripts are only executed if they have been selected in the scan configuration. The implementation requires therefore to move the last ACT_INIT NVT to ACT_SETTINGS and then make the category non-autoselect.
Results read from nmap output (or imported XML file) are stored in the KB. The NVT's simply need to read the results and report them.
Changes on NVT's necessary for this change:
Exports a registration system, so that NSE wrappers can register themselves for execution, with their arguments.
- additional set of NSE wrappers:
- 1st step: register themselves with their arguments if they are scheduled for execution.
- 2nd step: read the results from the KB and report them.
As they're simplified, new NSE wrappers can be generated from the original NSE scripts. These new NSE wrappers would be stored as a separate set from the existing ones, in a nmap_nse_net directory for instance.
- 1st step: rebuild command line from scan config and registered scripts, then run nmap.
- 2nd step: read port scanning, OS detection... results from the KB and report them.
- 2011-07-04 Henri Doreau <henri.doreau at greenbone dot net>:
Marked as done.
- 2011-06-01 Henri Doreau <henri.doreau at greenbone dot net>:
Updated text with description of required scanner changes (non-autoselect ACT_INIT).
- 2011-05-30 Michael Wiegand <michael.wiegand at greenbone dot net>:
Updated status with voting results
- 2011-04-27 Jan-Oliver Wagner <jan-oliver.wagner at greenbone dot net>:
slight rework of text
- 2011-04-27 Henri Doreau <henri.doreau at greenbone dot net>: