Merge pull request #3 from austin-taylor/master

sync

.travis.yml

@@ -0,0 +1,24 @@
+group: travis_latest
+language: python
+cache: pip
+python:
+    - 2.7
+
+#    - 3.6
+#matrix:
+#    allow_failures:
+#        - python: 3.6 - Commenting out testing for Python 3.6 until ready
+
+install:
+    - pip install -r requirements.txt
+    - pip install flake8  # pytest  # add another testing frameworks later
+before_script:
+    # stop the build if there are Python syntax errors or undefined names
+    - flake8 . --count --exclude=deps/qualysapi --select=E901,E999,F821,F822,F823 --show-source --statistics
+    # exit-zero treats all errors as warnings.  The GitHub editor is 127 chars wide
+    - flake8 . --count --exit-zero --exclude=deps/qualysapi --max-complexity=10 --max-line-length=127 --statistics
+script:
+    - true  # pytest --capture=sys  # add other tests here
+notifications:
+    on_success: change
+    on_failure: change  # `always` will be the setting once code changes slow down

CNAME

@@ -0,0 +1 @@
+www.vulnwhisperer.com

README.md

@@ -2,28 +2,37 @@
 
 <p align="center"> <i>Create <u><b>actionable data</b></u> from your vulnerability scans </i> </p> 
 
-<p align="center" style="width:400px"><img src="https://github.com/austin-taylor/vulnwhisperer/blob/master/docs/source/vulnwhisp_dashboard.jpg" style="width:400px"></p>
+<p align="center" style="width:400px"><img src="https://github.com/austin-taylor/vulnwhisperer/blob/master/docs/source/vulnWhispererWebApplications.png" style="width:400px"></p>
 
 
-VulnWhisperer is a vulnerability report aggregator. VulnWhisperer will pull all the reports
+VulnWhisperer is a vulnerability data and report aggregator. VulnWhisperer will pull all the reports
  and create a file with a unique filename which is then fed into logstash. Logstash extracts data from the filename and tags all of the information inside the report (see logstash_vulnwhisp.conf file). Data is then shipped to elasticsearch to be indexed.
 
+[![Build Status](https://travis-ci.org/austin-taylor/VulnWhisperer.svg?branch=master)](https://travis-ci.org/austin-taylor/VulnWhisperer)
+[![MIT License](https://img.shields.io/badge/license-MIT-blue.svg?style=flat)](http://choosealicense.com/licenses/mit/)
+
 
 Requirements
 -------------
 ####
-*   ElasticStack
+*   ElasticStack 5.x
 *   Python 2.7
 *   Vulnerability Scanner
 *   Optional: Message broker such as Kafka or RabbitMQ 
 
 Currently Supports
--------------
+-----------------
-####
+
-*   Elasticsearch 2.x
+### Vulnerability Frameworks
-*   Python 2.7
+
-*   Nessus
+- [X] Nessus V6
-*   Qualys - Web Application Scanner
+- [X] Qualys Web Applications
+- [ ] Qualys Vulnerability Management (_in progress_)
+- [ ] OpenVAS
+- [ ] Nexpose
+- [ ] Insight VM
+- [ ] NMAP
+- [ ] More to come
 
 
 Setup
@@ -55,6 +64,10 @@ There are a few configuration steps to setting up VulnWhisperer:
 *   Import ElasticSearch Templates
 *   Import Kibana Dashboards
 
+<a href="https://github.com/austin-taylor/VulnWhisperer/blob/master/configs/frameworks_example.ini">example.ini file</a>
+<p align="left" style="width:200px"><img src="https://github.com/austin-taylor/vulnwhisperer/blob/master/docs/source/config_example.png" style="width:200px"></p>
+
+
 Run
 -----
 To run, fill out the configuration file with your vulnerability scanner settings. Then you can execute from the command line.
@@ -65,6 +78,7 @@ or
 vuln_whisperer -c configs/example.ini -s qualys
 
 ```
+<p align="center" style="width:300px"><img src="https://github.com/austin-taylor/vulnwhisperer/blob/master/docs/source/running_vuln_whisperer.png" style="width:400px"></p>
 Next you'll need to import the visualizations into Kibana and setup your logstash config. A more thorough README is underway with setup instructions.
 
 _For windows, you may need to type the full path of the binary in vulnWhisperer located in the bin directory._
@@ -73,6 +87,7 @@ Credit
 ------
 Big thank you to <a href="https://github.com/SMAPPER">Justin Henderson</a> for his contributions to vulnWhisperer!
 
+
 AS SEEN ON TV
 -------------
-<p align="center" style="width:400px"><a href="https://twitter.com/MalwareJake/status/935654519471353856"><img src="https://github.com/austin-taylor/vulnwhisperer/blob/master/docs/source/as_seen_on_tv.png" style="width:400px"></a></p>
+<p align="center" style="width:400px"><a href="https://twitter.com/MalwareJake/status/935654519471353856"><img src="https://github.com/austin-taylor/vulnwhisperer/blob/master/docs/source/as_seen_on_tv.png" style="width:400px"></a></p>

_config.yml

@@ -0,0 +1 @@
+theme: jekyll-theme-leap-day

kibana/vuln_whisp_kibana/1000_vulnWhispererBaseVisuals.json

@@ -144,7 +144,7 @@
     "_type": "visualization",
     "_source": {
       "title": "VulnWhisperer - ScanName",
-      "visState": "{\"title\":\"VulnWhisperer - ScanName\",\"type\":\"table\",\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMeticsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":false,\"totalFunc\":\"sum\"},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"bucket\",\"params\":{\"field\":\"plugin_name.keyword\",\"size\":20,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"Scan Name\"}}],\"listeners\":{}}",
+      "visState": "{\"title\":\"VulnWhisperer - ScanName\",\"type\":\"table\",\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMeticsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":false,\"totalFunc\":\"sum\"},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"bucket\",\"params\":{\"field\":\"scan_name.keyword\",\"size\":20,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"Scan Name\"}}],\"listeners\":{}}",
       "uiStateJSON": "{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":null,\"direction\":null}}}}",
       "description": "",
       "version": 1,

vulnwhisp/frameworks/qualys.py

@@ -25,7 +25,6 @@ class qualysWhisperAPI(object):
     DELETE_REPORT = '/delete/was/report/{report_id}'
     GET_WEBAPP_DETAILS = '/get/was/webapp/{was_id}'
     QPS_REST_3 = '/qps/rest/3.0'
-
     REPORT_DETAILS = '/get/was/report/{report_id}'
     REPORT_STATUS = '/status/was/report/{report_id}'
     REPORT_CREATE = '/create/was/report'

vulnwhisp/vulnwhisp.py

@@ -531,7 +531,8 @@ class vulnWhispererQualys(vulnWhispererBase):
 
                     vuln_ready = self.qualys_scan.process_data(path=self.write_path, file_id=str(generated_report_id))
 
-                    vuln_ready.to_csv(relative_path_name, index=False, header=True)  # add when timestamp occured
+                    vuln_ready['scan_name'] = scan_name
+                    vuln_ready['scan_reference'] = scan_reference
                     vuln_ready.rename(columns=self.COLUMN_MAPPING, inplace=True)
 
                     record_meta = (