English | Deutsch
Home »

Minutes of the Second Developer Conference (DevCon2, 2009)

This page hosts the minutes as provided by the participants.

Following topics were covered:
OpenVAS DevCon2
July 10, 2009
Discussion: Base Data Structures / Get-Rid-Of-List

Moderator: Jan-Oliver Wagner
Minutes:   Michael Wiegand
Attending: Matthew Mundell
           Felix Wolfsteller
           Tobias von Keler
           Ahmad Rooyani
           Michael Wiegand

Along with the Nessus code base, the OpenVAS project has inherited a number of
bad design choices regarding the code structure from the Nessus project. Some
of the bad design choices are easy to identify, while others are not.

Work has already begun on replacing of badly implemented and hard-to-maintain
code parts with glib (http://library.gnome.org/devel/glib/); one example for
this the command line parsing in openvas-client, openvas-server and
openvas-nasl.

The goal of these efforts is to reduce the code base and to get clean and well
documented code which will be easier to maintain. Easy maintenance was agreed
to be an import goal because even though computing power and capacity always
increases, developer's minds generally don't.

One way towards this goal is to replace individual, home-grown and often
multiple implementations of generic concepts with implementations from standard
libraries. Glib was chosen as a basis for this because it has a large user base
(e.g. through the GNOME and GTK+ projects) and is widely available. The
implementations in glib are much more likely to get multiple reviews for code
quality and security than individual implementations in the Nessus/OpenVAS
source code.

The first steps on this way is to identify existing data structures in the
OpenVAS code base and evaluate the use of those data structures and potential
benefits of using glib data structures in their place.

One of the most widely used data structures in the OpenVAS code base is the so
called "arglist" structure, which is roughly similar to a hash table. It is
extremely flexible and useful for dynamic data structures, but tends to be used
as well in places where static structures would be more reasonable. In those
places, the use of arglists makes maintaining and debugging the existing code
unnecessarily difficult.

It was agreed that work in this area will begin by evaluating the use of
arglists in the different OpenVAS modules and to identify places where the use
of simple structs or other established structures would make more sense.

One of the places where improving the data structures has started is the server
side NVT meta data cache. This has already resulted in reduced disk usage for
the server cache. Good starting points for future work are most likely the
scheduler_plugin and name_cache structures on the server side and the
listnotebook structure on the client side.

The automatically generated Doxygen documentation provides a good collection of
existing data structures. Especially for the existing structures, the
documentation should be improved to understand where changes make the most
sense.

Another point is the creation and implementation of guidelines for internal API
functionality. The goal here is a more layered approach and to get rid of many
instances of bypassing, duplication and cross-referencing existing in the code
right now. This would also make developing a testing infrastructure easier.

As for the "get-rid-of-list", it was agreed that the top two priorities should
be replacing home-grown implementations with established functionality as
described above on one side and removing no longer needed functionality on the
other side. Work has begun on identifying these areas. The following files have
been identified as prime targets for removal since they contain obsolete or
redundant functionality or can be more or less easily replaced by established
data structures.

In openvas-libraries:
  - harglists.c, harglists.h
  - hlst.c, hlst.h
  - rand.c, rand.h
Back to overview.
OpenVAS DevCon2
July 9, 2009
Discussion: Community/PR/Marketing

Moderator: Jan-Oliver Wagner
Minutes:   Felix Wolfsteller, merged with Tim Browns
Attending: all

Slides from Kost and Jan-Oliver Wagner were taken as an entry point.

We have good tools, code review, ChangeLogs etc, but a lot of work
that needs ongoing effort is done by only a few hands.
There are many other things and kinds of involvements that we would like to do,
but we simply cannot because of the lack of (free) hands.



Work to be done
---------------

Generally we identified three types of work that partly could also be done
by non-coders or people who are not completely familiar with all the details
of OpenVAS:

(Of course, "Which task belongs to which category?" can be discussed endlessly)

There is
 * work that needs constant attention like
    Mailing list administration
    Certain web site updates
    Bugtracker maintenance
    Testing before (daily) NVT feed updates
    IRC channel administration
 * Work that has to be done in intervals/from time to time (e.g. before releases)
    Translations
    Certain web site updates
    Posting news and updates on online communities like freshmeat, ohloh, twitter, identi.ca
    FAQ maintenance
    Build (and probably functional) testing before releases
    Talks on fairs and conferences
 * Work that has to be done just once
    New website design
    Setup of new facilities, e.g. for build testing
    Certain web site updates
    Videos tutorials
    Write articles, tutorials

There are two questions: 'How to recruit somebody to do all this?'
and 'Do we need all this?'.
Where there is the need for help, we have to communicate the need better
(e.g. has to be stated on the website).

Also, a list of responsible persons should be maintained on the website.
Ideally, a vice is found and mentioned there too.

Tim Brown proposed to raise all who have svn commit access to IRC channel
(#openvas) OPS to allow channel maintenance to be better shared.

A list of possible jobs (Junior Jobs) could be created and posted to the
OpenVAS web site, this might include the tasks listed above.

We recognized that we do speak on conferences, but are not nicely organized
about that. As k0st states, it lacks a website area for it.


Fundraising/Donations
---------------------

If nobody can be found volunteering for the tasks mentioned above, it might be
worth looking into fundraising to get these things done.

Tim informed that we can take donations via SPI (this includes online
donations via Click&Pledge), this has been set up but is not used.
Tim will clarify ways funds can be used as he has concerns about work
for hire with respect to SPIs legal status.
Jan notes that SPIs partner organisation can also take donations in Europe and
highlights first kind-of-fundraising success with the Workshop that preceeded
the OpenVAS Developer Conference 2.


(Automated) Testing
-------------------

Regarding testing before Feed update there was a discussion whether automatized
tests are possible here.

Coming from there, we realized that there are two kinds of needs and
expectations: who runs more bleeding edge open source software probably does
not expect that everything always works perfectly and might accept changes more
easy than the one who uses OpenVAS as part of a toolchain.

It was discussed whether the openSUSE build service should be used for testing
before release. As it looks as if automated functional tests could be
integrated as well and packaging for many systems and distributions could be
created 'with one click', it is worth looking deeper into it.

Some expertise about automated testing is available via dn-systems (Dirk).


Website
-------

Most community work would be reflected on the webpage, so here a
wishlist of unsorted webpage edits:
    Maintained 'fairs and conference list'
    Linklist (do we need additional grouping here,
              e.g. tutorials, discussions, related tools?)
    List of volunteers/responsibles
    A 'we need help' statement
    A proper definition of the OpenVAS NVT Feed
    New design?
    FAQ [recently picked up and initiated by Geoff Galitz]
    Junior Jobs?


Connections
-----------

It was agreed to apply for membership at ocert[.org].

Connections to other communities and projects were discussed, they follow in
an unsorted list

* oCERT (http://www.ocert.org/)
    Tim to contact about OpenVAS becoming a member project.
* OWASP (http://www.owasp.org/)
    Jan will follow this up with .de OWASP folk.
        Tim noted that he knows several of the OWASP local chapter leaders
        (.tr, .it) as well as a number of OWASP project leaders, one of the
        authors of the OWASP testing guide and a number of the founders &
        current board.
* Planet OpenVAS (http://planet.openvas.org/)
    Needed fixing (now done), any developer, contributing organisation can
    have their RSS feed added?  Speak to Tim about this.
* Twitter/identi.ca (http://twitter.com/openvas & http://identi.ca/openvas)
    Kost and Tim to work on integrating it with the IRC bot.  Kost provides
    bot source to Tim.
* ohloh (http://www.ohloh.net/p/openvas/)
    Felix and Kost to persue this.  Looks like a success. [it was shortly
    after the DevCon]


Off Topic
---------

Tim mentioned Phrasendrescher
(http://labs.portcullis.co.uk/application/phrasen-drescher/) as a
possible replacement for hydra (developed by Tim's colleague at Portcullis and
BSD licensed).
Not as much support for other protocols yet, but API is clean, so easy to add.
Back to overview.
OpenVAS DevCon2
July 9, 2009
Discussion: External Library Integration

Moderator: Tim Brown
Minutes:   Chandrashekhar Basavanna
Attending: Lukas Grunwald
           Thomas Rotter
           Sven Wurth
           Thomas Reinke
           Chandrashekhar Basavanna
           Tim Brown
           Geoff Galitz
           Goran Licina
           Goran Zivkovic

(see also OVDC2_nasl.txt)

SSH Library Integration

SSH protocol is currently implemented in NASL. It is not easily understandable
and extensibility is hard.
There was a proposal to integrate with the standard implementation instead of
re-inventing the wheel.

There was a question raised by one of the member specifically about the reason
for implementing a change and what are the problems that the project members
are currently facing.
However it was made clear that the change is more from the point of moving to
standards and also to ease the maintainability.
It was agreed by all the members present that such integration would help the
project focus on core things.
Making use of the existing implementation would assist in easier feature
integration and less maintenance.
An exercise to evaluate various existing SSH implementation must be carried out.
The following well known libraries are available currently,
 - OpenSSH (www.openssh.org)
 - libssh2 (http://www.libssh2.org/wiki/index.php/Main_Page)
 - libssh (www.libssh.org)
 - LSH (http://www.lysator.liu.se/~nisse/lsh/)

The libraries have to be evaluated based on the below criteria,
 - Support for various distribution
 - Development support, user and development community base
 - Size of the library
 - License compatibility

Decision:
Geoff agreed to evaluate each library and come out with results that will be
put forward in the form of CR.

SMB support:
Currently WMI support is under implementation as per CR #25.
There's still a drawback exists to write Windows based remote/local checks that
are based on SMB/DCERPC packet crafting.
There is no means currently available and smb_nt.inc is an outdated
implementation and doesn't work with the newer authentication methods.

The WMI implementation is based on the Samba code base.
Since there's an ongoing work and also since there's no suitable alternative
available to provide packet crafting level API's for NASL writers, it was
decided by all members that it is easier to integrate Samba and expose the low
level functionality.

Decision:
Chandra to create/update the CR for the Samba implementation.

Other protocols
Other implementations that are complex in nature could follow the same model as
SSH and SMB. Each such proposals have to be taken through the Change Request
process.
Back to overview.
OpenVAS DevCon2
July 11, 2009
Discussion: Library Consolidation

Moderator: Jan-Oliver Wagner
Minutes:   Jan-Oliver Wagner
Attending: Matthew Mundell
           Felix Wolfsteller
           Tobias von Keler
           Ahmad Rooyani
           Michael Wiegand

(Results in CR#38  http://www.openvas.org/openvas-cr-38.html)
Back to overview.
OpenVAS DevCon2
July 10, 2009
Discussion: Consolidation NASL and NASL libraries

Moderator: Thomas Reinke
Minutes:   Tim Brown
Attending: Lukas Grunwald
           Thomas Rotter
           Sven Wurth
           Thomas Reinke
           Chandrashekhar Basavanna
           Tim Brown
           Geoff Galitz
           Goran Licina
           Goran Zivkovic

Internal infrastructure

* API calls
    For now we use script_id, script_oid is moving forward... communication
    regarding moving to use it.  Jan and Thomas to coordinate
* File system
    Scripts to be ordered in (sub)directories.
    Different approaches were discussed.
    Agreement on directory names equalling family names + an include
    directory (both feed and svn).
* Obsolete plugins
    Why?
     - Legal reasons
     - Potentially malicious code
    How?
     - Minimum note (log_warn?)
     - We will not remove plugins for outdated systems
* Avoiding duplication
    Sometimes what seems to be duplicates are none (e.g. remote vs local).
    -> Make a note in textfile managed under svn for each new.
* Coordinated effort of implementing NVTs for CVEs
    Currently a plain text file is used.
    Communication of process (svn), mailing for others.



External integration

* CPE, OSVDB, CVSS, BID, CVE
    Should meta data be moved to a database or external files?
    Besides a general agreement to separate the meta data from
    the NVTs, many open questions were found:
    How? What about the client? In the first stage just the NASLs
    themselves? Gives i18n, easier updates.
    Chandra (was?) volunteered to create a Change Request to handle this
    change.
    References to CVE etc are recommended but not mandatory.


Protocol implementations: in NASL or part of NASL API?

With the prime example of SSH, there is and will be functionality implemented
in NASL that could be handled by other already existing libraries.
This can come with obvious advantages (better support, implemented by experts
in their fields, etc. etc.), but also raises a couple of issues:

* SSH
    Questions about a non-NASL ssh implementation:
    What about Licensing?
    Support?
    Library?
    Distros?
    Geoff (was?) volunteered to investigate the issue.
* SMB
    Chandra will raise a Change Request to integrate libsmb.
* General guidelines
    Where they are used as a transport, it will be recommended that
    providing the protocol is of sufficient complexity a CR will be raised
    to use the appropriate domain expert developed solution.
        (see OVDC2_external_libs.txt)


NASL harmonisation

Moderator: Thomas Reinke

* Unit tests
    All .inc files should be documented.
    Thomas is going to do some analysis of what functions are most
    important (hack the NASL compiler) both by static and runtime.
    (Results were posted on the mailing list [1]).

* Documentation
    Jan will look at ways to use doxygen for NASL files.

* Vhost
    This is a killer feature, as a first step we will look at how to specify
    a vhost for a given IP, but intention is to support multiple vhosts.
    (see OVDC_vhosts.txt)


[1] http://lists.wald.intevation.org/pipermail/openvas-devel/2009-July/001629.html
Back to overview.
OpenVAS DevCon2
July 10, 2009
Discussion: Nmap integration

Moderator: Jan-Oliver Wagner
Minutes:   Jan-Oliver Wagner, Matthew Mundell
Attending: all


* Slides submitted by Kost to the openvas-devel mailing
  list were used to initiate the discussion.

* Standard usage of Nmap for OpenVAS at the momement is port scanning.

* Efficiency/Memory consumption of Nmap
  * efficiency depends on usage
  * experienced: nmap bad at multiple-processes, bad for
    specific (multiple) IPs (thats how OpenVAS uses Nmap).
    Better on Class C networks.

* Executable vs. library
  * This question is discussed in Eric's document and some
    more emails on openvas-devel.
  * This question will not necessarily solve the memory problem.

* Prepass of Nmap
  * This is already practiced e.g. by Tim and Tom.
  * It does not solve the memory problem.
  * This introduces a time window where the network can change.
    Of course, the network can change during test, whatever
    method is chosen.
  * Perhaps make the prepass optional alternative to per-host.
  * Scans can hang for a while and prevent prepass from finishing fast.

* Chunking Nmap scan
  * To cope with the memory problem it seems
    to be a valid option to chunk the task for
    Nmap, e.g. chunks of 1000 ports.
  * It appears not doable to handle such chunks in
    the plugin scheduler.
  * Can the chunking size be a preference for nmap NASL
    script? This might make sense but will not
    allow to start some NASL scripts earlier (because
    of the way the scheduler works).
  * Agreed: Try out chunked version of Nmap NASL-wrapper
    to see whether it improves performance.

* Separate Portscan out of OpenVAS
  * An even more radical approach than prepass would
    be to separate portscanning out of OpenVAS.
  * It was agreed that this does bring any real advantage
    and introduce more problems for the handling.
  * After all it is already possible (i.e. you
    can import a nmap scan result).

* Network discovery
  * Host discovery could preced port scan phase.
  * Have to wait for it before scan starts.
  * Reliability is debatable, but fun for a start
  * in general this could only be informational.
  * Results can go into KB and be used by nasl scripts
    and especially refined by nasl scripts (-> inventory).
  * Could provide a start for scan target list.
  * Can be problem with hosts you shouldn't scan.

* Integration NSE
  * There are about 60 LUA scripts from the Nmap project
  * Question is wether we want to add yet another language
    but in fact it is only about offering Nmap scripts to
    OpenVAS users and not have OpenVAS NVT team use LUA.
  * The NSE scripts are of little use for pen testers.
  * The output should be parsable.
  * General agreement to postpone until stable.

* Service Detection
  * find_services.c: C-module, hopeless out of date,
    broken concept in terms of upating patterns.
    Has ca. 100 signatures while nmap has > 5000.
  * Generally agreed we need a NASL script to
    handle service detection.
  * nmap can do the service detection and fill
    the KB accordingly.
  * Essential: should the signatures be distributed
    via Feed to deliver most uptodate easily?
    If so, it needs to be clearified which sigfile
    will be used (system provided one, the feed one,
    individual one). After all, should be configurable.
  * The OS's sigfile might be very much out of date -
    on the other hand it might be specially crafted
    (does this happen in reality?).
  * How do nmap people handle to keep signatures
    uptodate?
  * Need to talk to the nmap people on this idea.
Back to overview.
OpenVAS DevCon2
July 11, 2009
Discussion: OpenVAS Manager

Moderator: Jan-Oliver Wagner
Minutes:   Geoff Galitz
Attending: all


New model introduced:

Graphic to be supplied.


The primary change proposed here is to add a new layer called the OpenVAS
Manager.

This would sit between the client and the scan server.
The primary purpose is to offload administrative tasks and other non scanning
related tasks from the openvas-server component.
At the same time this is a good time to remove old un-used code and
plan for more flexible scanner management.

Informational points follow:


Proposed benefits:
-  security:     move bulk of operations to non-priviledged binary
-  scalability:  manage multiple scanners from central location
-  scalability/reliability: store reports manager/server side
-  reliability/efficiency:  reports stored in SQL database
-  reliability:  manager can sanity check requests and client connections
-  reliability/efficiency:  remove un-used code from current openvasd
-  reliability/efficiency:  reduce OTP complexity

Concerns/Drawbacks:
-  Workflow:  New model interferes with workflow in customized environments
(for example, users and script that interact diretly with the server, bypassing
the current client (see note on compatibility).
-  Workflow:  Highly customized environments that implement the current OTP
protocol directly at any point (see note on compatibility).
-  Security/privacy:  With multiple data stores (reports saved on the manager
and optionally on the client) data privacy becomes more complicated.
How many versions of the same data must be secured or securely deleted?
How can this be verified?


It was agreed that the concerns were not significant enough to warrant
any changes to the architecture at this time.


Details:

- Miscellaneous: Reducing OTP complexity ultimately means reducing the
supported command set to four commands: long attack, preferences, plugins order
and attached file.
- Compatibility:  The current client/server model is still supported
temporarily. When a client connects to the manager using the OTP protocol, the
manager acts as a proxy to the openvas server.
Hence backward compatibility is maintained until the current form of the OTP
protocol is ultimately retired.
Ultimately any customized environments that use the OTP protocol directly will
need to be ported to the new model, which is more work for the affected users.

- Near term goal:  report management, removing un-used/bad code
- Mid term goals:  distributed scanner management, enhanced user management,
                   OTP2

TBD Items:  NVT syncing across a distributed configuration

Back to overview.
OpenVAS DevCon2
July 11, 2009
Discussion: Tool-chain Reliability Policy

Moderator: Jan-Oliver Wagner
Minutes:   Jan-Oliver Wagner
Attending: Matthew Mundell
           Felix Wolfsteller
           Michael Wiegand


Summary:

* The OpenVAS Team already practices a systematic
  method on release management for all components
  of the OpenVAS tool chain.

* However, it is not really documented and
  transparent for the public. But this could
  attract more users who do care about
  release management, especially when
  they connect their tool chains to OpenVAS.

* It was proposed to have a web page that
  collects information about:

  * Procedure for a new release
  * How long is a release supported
    (not necessarily a time, it could
    be dynamic criteria)
  * Procedure to retire a release.
  * Name all interfaces that are taken
    care of in terms of being unchanged/
    compatible). Definitely NBE, OMP, OTP,
    openvasrc belongs to this group.
    However, there might be more interfaces
    like the command line options of OpenVAS-Client
    and all these need to be named and described
    explicitely.

* Stability can be suported from the source code
  side as well, especially with unit testing.
  We can not implement unit testing for all of the
  code rapidly, it likely will take a very long time.
  Those elements of OpenVAS that are the most
  important interfaces to other tools should be covered
  first when starting to implement unit testing.

* The procedures should be applied and practiced
  the first time with the version that retires
  OpenVAS 1.0.
Back to overview.
OpenVAS DevCon2
July 9, 2009
Discussion: Virtual host scanning

Moderator: Thomas Reinke
Minutes:   Chandrashekhar Basavanna
Attending: Lukas Grunewald
           Thomas Rotter
           Sven Wurth
           Thomas Reinke
           Chandrashekhar Basavanna
           Tim Brown
           Geoff Galitz
           Goran Licina
           Goran Zivkovic


(see also OVDC2_nasl.txt)

OpenVAS doesn't support scanning virtual host, web servers, it scans only the
given IP or the Host.

It was agreed by all members in the discussion group that this feature is a
real value add and this needs to be supported.

Three requirements for implementing virtual hosts support were discussed,
 - Virtual Host discovery
 - Update plugins to support virutal host scanning
 - Reporting

The following approaches were discussed and debated,

I. Virtual Host discovery

1.Automatically discovering all the virtual hosts through web mirroring
techniques:
Though there are techniques available, they are not accurate in listing all the
virtual hosts.

2.Manual method of entering all the virtual hosts in a text box or provide
through the file.

    Decision: Option #2


II. Update plugins to support virtual hosts scanning
1.Loop through all virtual hosts and perform the scanning for each host.
Ideally each web related plugin should be updated to include,

    vhost = get_kb_item(¿VirtualHosts¿);
    foreach dir cgi_dir
        DO_CHECK

    Also wherever HTTP requests are being constructed, we should update the
    header to contain

    Host: virtual_host

    If vhost value returned is large in number, the plugins might run for
    a very long time. Plugin timeout value might need to be set to
    a large number.
    When you call get_kb_item and if it returns multiple values, openvas
    will fork mutliple processes. Fork limit might need to be set in order
    to limit the number of forks.

2.Instead of looping through all the virtual hosts and scanning, administrator
can enter the virtual hosts directly for scanning in the scan_target.

    In this case, plugins need not be updated to scan through all virtual
    hosts. But, each plugin need to be audited to check if,

    Host: virtual_host is handled appropriately.

    Decision: Option #2


III. Reporting
When multiple virtual hosts are scanned on a single system, there is a
possibility that same report might be listed multiple times.

1.Reports have to be created separately for each scanned virtual host. This can
be achieved by automatically prefixing virtual host name to the report
description or by enhancing security_ API's to include an additional parameter
called v_host

Example: security_warning(port, data, v_host)

2.  the user enters the virtual hosts as scan targets, reports will anyway be
created separately.

    Decision: Option #2
Back to overview.