OpenVAS Change Request #60: NVT directory structure
Status: Voted +10. In developement.
Avoid huge number of files in a single directory by using subdirectories for each year where NVTs are placed created in that year.
- Discussion on mailing list openvas-plugins from February 2012
- Continued discussion on mailing list openvas-plugins from October 2012
- OpenVAS CR #24: OpenVAS-Server: Reorganize NVTs in Subdirectories: The CR that introduced subdirectories and covered early ideas how to organize structure.
Currently, the main NVT directory contains almost 30,000 NVTs. The Feed directory with one detached signature file per NVT contains therefore already almost 60,000 files.
The large number of files in a single directory creates several problems:
Some file systems may get into trouble with such large number of files in a single directory. This affects OpenVAS installations using the feed service.
It is problematic to handle such a directory manually, for example to find a special file.
Subversion gets into performance problems with this situation.
When segregated by year of creation we will never have to worry about excess scripts in any year. It is easy to manage and not tied to category or any other metric.
Early ideas were about using the OID structure for segregation. This may not ensure a more or less equal distribution across subdirectories and also the OID strategy has not settled yet.
A main effect would be that the OpenVAS NVT Feed synchronisation script does not delete files not present in the feed service. This makes it difficult to move files into subdirectories because it would create numerous duplicates. Starting in 2013 with a subdirectory 2013/ would mean no problem.
For OpenVAS-6, the removal can be made default, so that OpenVAS-6 users will not have problems once later on we move files. Of course, the moving of files of 2012 and earlier could only be done after OpenVAS-5 is retired.
It would be very important then to inform users migrating to OpenVAS-6 that any NVTs in their plugins directory that are not part of the feed service will be removed.
A downside is that finding a filename to a given OID or family is not straight-forward possible via the directory tree. However, this search is offered via the OpenVAS website.
Design and Implementation
The following steps would need to be done:
Create directory openvas-plugins/scripts/2013 and start committing new scripts in 2013 into that directory. Only exceptions: a) If they belong to one of the groups already collected in another directory such as GSHB or the NSE wrappers. b) include files, c) NVTs that are used as dependencies.
Adjust the sync script to create a directory "private" if it does not exist yet and to exclude removal of it and its content during synchronisation. Make it part of OpenVAS-6.
Create a script which help users to figure out which of their NVTs are not part of the OpenVAS NVT Feed and offer to move them to the subdirectory "private/". This should be as simple check where NVTs are lacking an OpenVAS signature. It needs to be taken care to only check the OpenVAS signature and not other valid signatures a user might have configured and used. Make this part of OpenVAS-6, helper routine probably best as part of openvas-nvt-sync when called with a special parameter (like --migrate-to-private).
- 2013-03-19 Jan-Oliver Wagner <jan-oliver.wagner at greenbone dot net>:
All items are implemented with OpenVAS-6. Once OpenVAS-5 is retired, directories 2012/, 2011/ etc. can be created and the files be moved there.
- 2012-12-18 Jan-Oliver Wagner <jan-oliver.wagner at greenbone dot net>:
Added voting results and setting to "in development".
- 2012-12-12 Jan-Oliver Wagner <jan-oliver.wagner at greenbone dot net>: