Merge pull request 'Traduction du modèle de rapport de bug' (#1) from mes-modifs into master

Reviewed-on: #1

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,41 @@
+---
+name: Rapport de bug
+about: Créez un rapport pour nous aider à nous améliorer
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Décrivez le bug**
+Une description claire et concise de ce qu'est le bug.
+
+**Module affecté**
+Lequel des modules ne fonctionne pas comme prévu, par exemple, Nessus, Qualys WAS, Qualys VM, OpenVAS, ELK, Jira...
+
+**Trace de débogage de VulnWhisperer**
+Si possible, veuillez joindre la trace de débogage de l'exécution pour une enquête plus approfondie (exécuter avec l'option `-d`).
+
+**Pour reproduire**
+Étapes pour reproduire le comportement :
+1. Allez à '...'
+2. Cliquez sur '....'
+3. Voir l'erreur
+
+**Comportement attendu**
+Une description claire et concise de ce à quoi vous vous attendiez.
+
+**Captures d'écran**
+Si applicable, ajoutez des captures d'écran pour aider à expliquer votre problème.
+
+**Système sur lequel VulnWhisperer s'exécute (veuillez compléter les informations suivantes) :**
+ - OS : [ex. Ubuntu Server]
+ - Version : [ex. 18.04.2 LTS]
+ - Version de VulnWhisperer : [ex. 1.7.1]
+
+**Contexte additionnel**
+Ajoutez tout autre contexte sur le problème ici.
+
+**Note importante**
+Comme VulnWhisperer s'appuie sur ELK pour l'agrégation de données, il est attendu que vous ayez déjà une instance ELK ou les connaissances pour en déployer une.
+Pour accélérer le déploiement, nous fournissons un fichier docker-compose à jour et testé qui déploie toute l'infrastructure nécessaire et nous supporterons son déploiement, mais nous ne donnerons pas de support pour les instances ELK.

.gitignore

@@ -1,3 +1,11 @@
+# Vulnwhisperer stuff
+data/
+logs/
+elk6/vulnwhisperer.ini
+resources/elk6/vulnwhisperer.ini
+configs/frameworks_example.ini
+tests/data
+
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[cod]
@@ -100,3 +108,6 @@ ENV/
 
 # mypy
 .mypy_cache/
+
+# Mac
+.DS_Store

.gitmodules

@@ -1,3 +1,3 @@
-[submodule "qualysapi"]
-	path = deps/qualysapi
-	url = git@github.com:austin-taylor/qualysapi.git
+[submodule "tests/data"]
+	path = tests/data
+	url = https://github.com/HASecuritySolutions/VulnWhisperer-tests.git

.travis.yml

@@ -3,12 +3,22 @@ language: python
 cache: pip
 python:
     - 2.7
+env:
+    - TEST_PATH=tests/data
 
+services:
+  - docker
 #    - 3.6
 #matrix:
 #    allow_failures:
 #        - python: 3.6 - Commenting out testing for Python 3.6 until ready
 
+before_install:
+    - mkdir -p ./data/esdata1
+    - mkdir -p ./data/es_snapshots
+    - sudo chown -R 1000:1000 ./data/es*
+    - docker build -t vulnwhisperer-local .
+    - docker-compose -f docker-compose-test.yml up -d
 install:
     - pip install -r requirements.txt
     - pip install flake8  # pytest  # add another testing frameworks later
@@ -18,7 +28,9 @@ before_script:
     # exit-zero treats all errors as warnings.  The GitHub editor is 127 chars wide
     - flake8 . --count --exit-zero --exclude=deps/qualysapi --max-complexity=10 --max-line-length=127 --statistics
 script:
-    - true  # pytest --capture=sys  # add other tests here
+    - python setup.py install
+    - bash tests/test-vuln_whisperer.sh
+    - bash tests/test-docker.sh
 notifications:
     on_success: change
     on_failure: change  # `always` will be the setting once code changes slow down

Dockerfile

@@ -0,0 +1,26 @@
+FROM centos:7
+
+MAINTAINER Justin Henderson justin@hasecuritysolutions.com
+
+RUN yum update -y && \
+    yum install -y python python-devel git gcc && \
+    curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
+    python get-pip.py
+
+WORKDIR /opt/VulnWhisperer
+
+COPY requirements.txt requirements.txt
+COPY setup.py setup.py
+COPY vulnwhisp/ vulnwhisp/
+COPY bin/ bin/
+COPY configs/frameworks_example.ini frameworks_example.ini
+
+RUN python setup.py clean --all && \
+    pip install -r requirements.txt
+
+
+WORKDIR /opt/VulnWhisperer
+RUN python setup.py install
+
+
+CMD  vuln_whisperer -c /opt/VulnWhisperer/frameworks_example.ini

README.md

@@ -1,260 +1,138 @@
-<p align="center"><img src="https://github.com/austin-taylor/vulnwhisperer/blob/master/docs/source/vuln_whisperer_logo_s.png" width="400px"></p>
-<p align="center"> <i>Create <u><b>actionable data</b></u> from your vulnerability scans </i> </p> 
+<p align="center"><img src="https://git.gudita.com/Cyberdefense/VulnWhisperer/raw/branch/master/docs/source/vuln_whisperer_logo_s.png" width="400px"></p>
+<p align="center"> <i>Créez des <u><b>données exploitables</b></u> à partir de vos scans de vulnérabilités</i> </p> 
 
-<p align="center" style="width:400px"><img src="https://github.com/austin-taylor/vulnwhisperer/blob/master/docs/source/vulnWhispererWebApplications.png" style="width:400px"></p>
+<p align="center" style="width:400px"><img src="https://git.gudita.com/Cyberdefense/VulnWhisperer/raw/branch/master/docs/source/vulnWhispererWebApplications.png" style="width:400px"></p>
 
 
-VulnWhisperer is a vulnerability data and report aggregator. VulnWhisperer will pull all the reports
- and create a file with a unique filename which is then fed into logstash. Logstash extracts data from the filename and tags all of the information inside the report (see logstash_vulnwhisp.conf file). Data is then shipped to elasticsearch to be indexed.
+VulnWhisperer est un outil de gestion des vulnérabilités et un agrégateur de rapports. VulnWhisperer récupère tous les rapports des différents scanners de vulnérabilités et crée un fichier avec un nom unique pour chacun, utilisant ensuite ces données pour se synchroniser avec Jira et alimenter Logstash. Jira effectue une synchronisation complète en cycle fermé avec les données fournies par les scanners, tandis que Logstash indexe et étiquette toutes les informations contenues dans le rapport (voir les fichiers logstash dans `/resources/elk6/pipeline/`). Les données sont ensuite envoyées à ElasticSearch pour être indexées, et finissent dans un format visuel et consultable dans Kibana avec des tableaux de bord déjà définis.
 
-[![Build Status](https://travis-ci.org/austin-taylor/VulnWhisperer.svg?branch=master)](https://travis-ci.org/austin-taylor/VulnWhisperer)
-[![MIT License](https://img.shields.io/badge/license-MIT-blue.svg?style=flat)](http://choosealicense.com/licenses/mit/)
-[![Twitter](https://img.shields.io/twitter/follow/VulnWhisperer.svg?style=social&label=Follow)](https://twitter.com/VulnWhisperer)
+VulnWhisperer est un projet open-source financé par la communauté. VulnWhisperer est actuellement fonctionnel mais nécessite une refonte de la documentation et une revue de code. Si vous souhaitez de l'aide, si vous êtes intéressé par de nouvelles fonctionnalités, ou si vous recherchez un support payant, veuillez nous contacter à **info@sahelcyber.com**.
 
-Currently Supports
------------------
 
-### Vulnerability Frameworks
+### Scanners de Vulnérabilités Supportés
 
-- [X] [Nessus (v6 & **v7**)](https://www.tenable.com/products/nessus/nessus-professional)
+- [X] [Nessus (**v6**/**v7**/**v8**)](https://www.tenable.com/products/nessus/nessus-professional)
 - [X] [Qualys Web Applications](https://www.qualys.com/apps/web-app-scanning/)
 - [X] [Qualys Vulnerability Management](https://www.qualys.com/apps/vulnerability-management/)
-- [X] [OpenVAS](http://www.openvas.org/)
+- [X] [OpenVAS (**v7**/**v8**/**v9**)](http://www.openvas.org/)
 - [X] [Tenable.io](https://www.tenable.com/products/tenable-io)
 - [ ] [Detectify](https://detectify.com/)
 - [ ] [Nexpose](https://www.rapid7.com/products/nexpose/)
 - [ ] [Insight VM](https://www.rapid7.com/products/insightvm/)
 - [ ] [NMAP](https://nmap.org/)
-- [ ] More to come
+- [ ] [Burp Suite](https://portswigger.net/burp)
+- [ ] [OWASP ZAP](https://www.zaproxy.org/)
+- [ ] Et d'autres à venir
 
-Getting Started
-===============
+### Plateformes de Reporting Supportées
 
-1) Follow the [install requirements](#installreq)
-2) Fill out the section you want to process in <a href="https://github.com/austin-taylor/VulnWhisperer/blob/master/configs/frameworks_example.ini">example.ini file</a>
-3) Modify the IP settings in the <a href="https://github.com/austin-taylor/VulnWhisperer/tree/master/logstash">logstash files to accomodate your environment</a> and import them to your logstash conf directory (default is /etc/logstash/conf.d/)
-4) Import the <a href="https://github.com/austin-taylor/VulnWhisperer/tree/master/kibana/vuln_whisp_kibana">kibana visualizations</a>
-5) [Run Vulnwhisperer](#run)
+- [X] [Elastic Stack (**v6**/**v7**)](https://www.elastic.co/elk-stack)
+- [ ] [OpenSearch - Envisagé pour la prochaine mise à jour](https://opensearch.org/)
+- [X] [Jira](https://www.atlassian.com/software/jira)
+- [ ] [Splunk](https://www.splunk.com/)
 
-Need assistance or just want to chat? Join our [slack channel](https://t.co/xlrqzLb3vY)
+## Démarrage
 
-Requirements
--------------
-####
-*   ElasticStack 5.x
-*   Python 2.7
-*   Vulnerability Scanner
-*   Optional: Message broker such as Kafka or RabbitMQ 
+1) Suivez les [prérequis d'installation](#installreq)
+2) Remplissez la section que vous souhaitez traiter dans le fichier <a href="https://git.gudita.com/Cyberdefense/VulnWhisperer/src/branch/master/configs/frameworks_example.ini">frameworks_example.ini</a>
+3) [JIRA] Si vous utilisez Jira, remplissez la configuration Jira dans le fichier de configuration mentionné ci-dessus.
+3) [ELK] Modifiez les paramètres IP dans les <a href="https://git.gudita.com/Cyberdefense/VulnWhisperer/src/branch/master/resources/elk6/pipeline">fichiers Logstash pour correspondre à votre environnement</a> et importez-les dans votre répertoire de configuration logstash (par défaut `/etc/logstash/conf.d/`)
+4) [ELK] Importez les <a href="https://git.gudita.com/Cyberdefense/VulnWhisperer/src/branch/master/resources/elk6/kibana.json">visualisations Kibana</a>
+5) [Exécutez Vulnwhisperer](#run)
 
-<a id="installreq">Install Requirements-VulnWhisperer(may require sudo)</a>
---------------------
-**First, install requirement dependencies**
+> **Note importante concernant les liens du Wiki :** La migration de Gitea ne transfère pas toujours le Wiki d'un projet GitHub (qui est techniquement un dépôt séparé). Si les liens vers le Wiki (comme le guide de déploiement ELK) ne fonctionnent pas, vous devrez peut-être recréer ces pages manuellement dans l'onglet "Wiki" de votre dépôt sur Gitea.
+
+Besoin d'aide ou juste envie de discuter ? Rejoignez notre [canal Slack](https://join.slack.com/t/vulnwhisperer/shared_invite/enQtNDQ5MzE4OTIyODU0LWQxZTcxYTY0MWUwYzA4MTlmMWZlYWY2Y2ZmM2EzNDFmNWVlOTM4MzNjYzI0YzdkMDA0YmQyYWRhZGI2NGUxNGI)
+
+## Prérequis
+* Python 2.7
+* Un Scanner de Vulnérabilités
+* Un Système de Reporting : Jira / ElasticStack 6.6
+
+<a id="installreq"></a>
+## Prérequis d'Installation - VulnWhisperer (peut nécessiter sudo)
+**Installez les dépendances des paquets du système d'exploitation** (pour les distributions basées sur Debian, CentOS n'en a pas besoin)
 ```shell
+sudo apt-get install zlib1g-dev libxml2-dev libxslt1-dev
 
-sudo apt-get install  zlib1g-dev libxml2-dev libxslt1-dev 
-```
+(Optionnel) Utilisez un environnement virtuel python pour ne pas perturber les bibliothèques python de l'hôte
 
-**Second, install dependant modules**
-```python
+virtualenv venv # créera l'environnement virtuel python 2.7
 
-cd deps/qualysapi
+source venv/bin/activate # démarre l'environnement, pip s'exécutera ici et devrait installer les bibliothèques sans sudo
+
+deactivate # pour quitter l'environnement virtuel une fois que vous avez terminé
+
+Installez les dépendances des bibliothèques python
+
+pip install -r /chemin/vers/VulnWhisperer/requirements.txt
+cd /chemin/vers/VulnWhisperer
 python setup.py install
-```
 
+(Optionnel) Si vous utilisez un proxy, ajoutez l'URL du proxy comme variable d'environnement au PATH
 
-**Third, install requirements**
+export HTTP_PROXY=[http://exemple.com:8080](http://exemple.com:8080)
+export HTTPS_PROXY=[http://exemple.com:8080](http://exemple.com:8080)
 
-```python
-pip install -r /path/to/VulnWhisperer/requirements.txt
-cd /path/to/VulnWhisperer
-python setup.py install
-```
-
-Now you're ready to pull down scans. (see <a href="#run">run section</a>)
-
-
-Install Requirements-ELK Node **\*SAMPLE\***
---------------------
-The following instructions should be utilized as a **Sample Guide** in the absence of an existing ELK Cluster/Node. This will cover a Debian example install guide of a stand-alone node of Elasticsearch & Kibana.
-
-While Logstash is included in this install guide, it it recommended that a seperate host pulling the VulnWhisperer data is utilized with Logstash to ship the data to the Elasticsearch node.
-
-*Please note there is a docker-compose.yml available as well.*
-
-**Debian:** *(https://www.elastic.co/guide/en/elasticsearch/reference/5.6/deb.html)*
-```shell
-sudo apt-get install -y default-jre
-wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
-sudo apt-get install apt-transport-https
-echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
-sudo apt-get update && sudo apt-get install elasticsearch kibana logstash
-sudo /bin/systemctl daemon-reload
-sudo /bin/systemctl enable elasticsearch.service
-sudo /bin/systemctl enable kibana.service
-sudo /bin/systemctl enable logstash.service
-```
-
-**Elasticsearch & Kibana Sample Config Notes** 
-
-Utilizing your favorite text editor:
-*   Grab your host IP and change the IP of your /etc/elasticsearch/elasticsearch.yml file. (This defaults to 'localhost')
-*   Validate Elasticsearch is set to run on port 9200 (Default)
-*   Grab your host IP and change the IP of your /etc/kibana/kibana.yml file. (This defaults to 'localhost') *Validate that Kibana is pointing to the correct Elasticsearch IP (This was set in the previous step)*
-*   Validate Kibana is set to run on port 5601 (Default)
-
-*Start elasticsearch and validate they are running/communicating with one another:* 
-```shell
-sudo service elasticsearch start
-sudo service kibana start
-```
-OR
-```shell
-sudo systemctl start elasticsearch.service
-sudo systemctl start kibana.service
-```
-
-**Logstash Sample Config Notes**
-
-*   Copy/Move the Logstash .conf files from */VulnWhisperer/logstash/* to */etc/logstash/conf.d/*
-*   Validate the Logstash.conf files *input* contains the correct location of VulnWhisper Scans in the *input.file.path* directory identified below:
-```
-input {
-  file {
-    path => "/opt/vulnwhisperer/nessus/**/*"
-    start_position => "beginning"
-    tags => "nessus"
-    type => "nessus"
-  }
-}
-```
-*   Validate the Logstash.conf files *output* contains the correct Elasticsearch IP set during the previous step above: (This will default to localhost)
-```
-output {
-  if "nessus" in [tags] or [type] == "nessus" {
-    #stdout { codec => rubydebug }
-    elasticsearch {
-      hosts => [ "localhost:9200" ]
-      index => "logstash-vulnwhisperer-%{+YYYY.MM}"
-    }
-  }
-```
-*   Validate logstash has the correct file permissions to read the location of the VulnWhisperer Scans
-
-Once configured run Logstash: (Running Logstash as a service will pick up all the files in */etc/logstash/conf.d/* If you would like to run only one logstash file please reference the command below):
-
-Logstash as a service:
-```shell
-sudo service logstash start
-```
-*OR*
-```shell
-sudo systemctl start logstash.service
-```
-Single Logstash file:
-```shell
-sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/1000_nessus_process_file.conf
-```
+Vous êtes maintenant prêt à télécharger les scans.
 
 Configuration
------
+Il y a quelques étapes de configuration pour mettre en place VulnWhisperer :
 
-There are a few configuration steps to setting up VulnWhisperer:
-*   Configure Ini file
-*   Setup Logstash File
-*   Import ElasticSearch Templates
-*   Import Kibana Dashboards
+Configurer le fichier Ini
 
-<a href="https://github.com/austin-taylor/VulnWhisperer/blob/master/configs/frameworks_example.ini">frameworks_example.ini file</a>
-<p align="left" style="width:200px"><img src="https://github.com/austin-taylor/vulnwhisperer/blob/master/docs/source/config_example.png" style="width:200px"></p>
+Configurer le fichier Logstash
 
+Importer les modèles ElasticSearch
 
-<a id="run">Run</a>
------
-To run, fill out the configuration file with your vulnerability scanner settings. Then you can execute from the command line.
-```python
+Importer les tableaux de bord Kibana
 
-vuln_whisperer -c configs/frameworks_example.ini -s nessus
-or
+Exécution
+Pour exécuter, remplissez le fichier de configuration avec les paramètres de votre scanner de vulnérabilités. Ensuite, vous pouvez l'exécuter depuis la ligne de commande.
+
+# (optionnel : -F -> fournit une coloration "Fantaisie" des logs, utile pour la compréhension lors de l'exécution manuelle de VulnWhisperer)
+vuln_whisperer -c configs/frameworks_example.ini -s nessus 
+# ou
 vuln_whisperer -c configs/frameworks_example.ini -s qualys
 
-```
-If no section is specified (e.g. -s nessus), vulnwhisperer will check on the config file for the modules that have the property enabled=true and run them sequentially.
-
-<p align="center" style="width:300px"><img src="https://github.com/austin-taylor/vulnwhisperer/blob/master/docs/source/running_vuln_whisperer.png" style="width:400px"></p>
-Next you'll need to import the visualizations into Kibana and setup your logstash config. A more thorough README is underway with setup instructions.
-
+Si aucune section n'est spécifiée (ex. -s nessus), vulnwhisperer vérifiera dans le fichier de configuration les modules ayant la propriété enabled=true et les exécutera séquentiellement.
 
 Docker-compose
------
-The docker-compose file has been tested and running on a Ubuntu 18.04 environment, with docker-ce v.18.06. The structure's purpose is to store locally the data from the scanners, letting vulnwhisperer update the records and Logstash feed them to ElasticSearch, so it requires a local storage folder.
+ELK est un monde en soi, et pour les nouveaux venus sur la plateforme, cela nécessite des compétences de base sous Linux et généralement un peu de dépannage jusqu'à ce qu'il soit déployé et fonctionne comme prévu. Comme nous ne sommes pas en mesure de fournir un support pour les problèmes ELK de chaque utilisateur, nous avons mis en place un docker-compose qui inclut :
 
-- It will run out of the box if you create on the root directory of VulnWhisperer a folder named "data", which needs permissions for other users to read/write/execute in order to sync:
-```shell
- mkdir data && chmod -R 666 data #data/database/report_tracker.db will need 777 to use with local vulnwhisperer
-```
-otherwise the users running inside the docker containers will not be able to work with it properly. If you don't apply chmod recursively, it will still work to sync the data, but only root use in localhost will have access to the created data (if you run local vulnwhisperer with the same data will break).
-- docker/logstash.yml file will need other read/write permissions in order for logstash container to use the configuration file; youll need to run:
-```shell
-chmod 666 docker/logstash.yml
-```
-- You will need to rebuild the vulnwhisperer Dockerfile before launching the docker-compose, as by the way it is created right now it doesn't pull the last version of the VulnWhisperer code from Github, due to docker layering inner workings. To do this, the best way is to:
-```shell
-wget https://raw.githubusercontent.com/qmontal/docker_vulnwhisperer/master/Dockerfile 
-docker build --no-cache -t hasecuritysolutions/docker_vulnwhisperer -f Dockerfile . --network=host
-```
-This will create the image hasecuritysolutions/docker_vulnwhisperer:latest from scratch with the latest updates. Will soon fix that with the next VulnWhisperer version.
-- The vulnwhisperer container inside of docker-compose is using network_mode=host instead of the bridge mode by default; this is due to issues encountered when the container is trying to pull data from your scanners from a different VLAN than the one you currently are. The host network mode uses the DNS and interface from the host itself, fixing those issues, but it breaks the network isolation from the container (this is due to docker creating bridge interfaces to route the traffic, blocking both container's and host's network). If you change this to bridge, you might need to add your DNS to the config in order to resolve internal hostnames.
-- ElasticSearch requires having the value vm.max_map_count with a minimum of 262144; otherwise, it will probably break at launch. Please check https://elk-docker.readthedocs.io/#prerequisites to solve that.
-- If you want to change the "data" folder for storing the results, remember to change it from both the docker-compose.yml file and the logstash files that are in the root "docker/" folder.
-- Hostnames do NOT allow _ (underscores) on it, if you change the hostname configuration from the docker-compose file and add underscores, config files from logstash will fail.
-- If you are having issues with the connection between hosts, to troubleshoot them you can spawn a shell in said host doing the following:
-```shell
-docker ps #check the images from the containers
-docker exec -i -t 665b4a1e17b6 /bin/bash #where 665b4a1e17b6 is the container image you want to troubleshoot
-```
-You can also make sure that all ELK components are working by doing "curl -i host:9200 (elastic)/ host:5601 (kibana) /host:9600 (logstash). WARNING! It is possible that logstash is not exposing to the external network the port but it does to its internal docker network "esnet".
-- If Kibana is not showing the results, check that you are searching on the whole ES range, as by default it shows logs for the last 15 minutes (you can choose up to last 5 years)
-- X-Pack has been disabled by default due to the noise, plus being a trial version. You can enable it modifying the docker-compose.yml and docker/logstash.conf files. Logstash.conf contains the default credentials for the X-Pack enabled ES.
-- On Logstash container, "/usr/share/logstash/pipeline/" is the default path for pipelines and "/usr/share/logstash/config/" for logstash.yml file, instead of "/etc/logstash/conf.d/" and "/etc/logstash/".
-- In order to make vulnwhisperer run periodically, add to crontab the following:
-```shell
-0 8 * * * /usr/bin/docker-compose run vulnwhisp-vulnwhisperer
-``` 
+VulnWhisperer
 
-To launch docker-compose, do:
-```shell
-docker-compose -f docker-compose.yml up
-```
+Logstash 6.6
 
+ElasticSearch 6.6
 
-Running Nightly
----------------
-If you're running linux, be sure to setup a cronjob to remove old files that get stored in the database. Be sure to change .csv if you're using json.
+Kibana 6.6
 
-Setup crontab -e with the following config (modify to your environment) - this will run vulnwhisperer each night at 0130:
+Le docker-compose nécessite simplement de spécifier les chemins où les données de VulnWhisperer seront sauvegardées, et où se trouvent les fichiers de configuration. S'il est exécuté directement après un git clone, en ajoutant simplement la configuration du scanner au fichier de configuration de VulnWhisperer (/resources/elk6/vulnwhisperer.ini), il fonctionnera immédiatement.
 
-`00 1 * * * /usr/bin/find /opt/vulnwhisp/ -type f -name '*.csv' -ctime +3 -exec rm {} \;`
+Il se charge également de charger automatiquement les tableaux de bord et les visualisations Kibana via l'API, ce qui doit être fait manuellement autrement au démarrage de Kibana.
 
-`30 1 * * * /usr/local/bin/vuln_whisperer -c /opt/vulnwhisp/configs/example.ini`
+Pour plus d'informations sur le docker-compose, consultez le wiki docker-compose ou la FAQ.
 
+Feuille de route
+Notre feuille de route actuelle est la suivante :
 
-_For windows, you may need to type the full path of the binary in vulnWhisperer located in the bin directory._
+[ ] Créer un standard de vulnérabilité
 
-Video Walkthrough -- Featured on ElasticWebinar
-----------------------------------------------
-<a href="http://www.youtube.com/watch?feature=player_embedded&v=zrEuTtRUfNw?start=30
-" target="_blank"><img src="https://github.com/austin-taylor/vulnwhisperer/blob/master/docs/source/elastic_webinar.png" 
-alt="Elastic presentation on VulnWhisperer" border="10" /></a>
+[ ] Mapper les résultats de chaque scanner au standard
 
-Authors
-------
-   - [Austin Taylor (@HuntOperator)](https://github.com/austin-taylor)
-   - [Justin Henderson (@smapper)](https://github.com/SMAPPER)
-   
-Contributors
-------------
-   - [Quim Montal (@qmontal)](https://github.com/qmontal)
+[ ] Créer des directives de module de scanner pour une intégration facile de nouveaux scanners
 
-AS SEEN ON TV
--------------
-<p align="center" style="width:400px"><a href="https://twitter.com/MalwareJake/status/935654519471353856"><img src="https://github.com/austin-taylor/vulnwhisperer/blob/master/docs/source/as_seen_on_tv.png" style="width:400px"></a></p>
+[ ] Refactoriser le code pour réutiliser les fonctions et permettre une compatibilité totale entre les modules
+
+[ ] Changer Nessus CSV en JSON
+
+[ ] Adapter le Logstash unique au standard et aux tableaux de bord Kibana
+
+[ ] Implémenter le scanner Detectify
+
+[ ] Implémenter le reporting/tableaux de bord Splunk
+
+En plus de cela, nous essayons de nous concentrer sur la correction des bugs dès que possible, ce qui peut retarder le développement. Nous accueillons également très volontiers les PR (Pull Requests), et une fois que nous aurons implémenté le nouveau standard, il sera très facile d'ajouter la compatibilité avec de nouveaux scanners.
+
+Le standard de vulnérabilité sera initialement un nouveau JSON simple à un niveau avec toutes les informations correspondantes des différents scanners ayant des noms de variables standardisés, tout en conservant le reste des variables telles quelles.

bin/vuln_whisperer

@@ -4,11 +4,13 @@ __author__ = 'Austin Taylor'
 
 
 from vulnwhisp.vulnwhisp import vulnWhisperer
-from vulnwhisp.utils.cli import bcolors
 from vulnwhisp.base.config import vwConfig
+from vulnwhisp.test.mock import mockAPI
 import os
 import argparse
 import sys
+import logging
+
 
 def isFileValid(parser, arg):
     if not os.path.exists(arg):
@@ -16,6 +18,7 @@ def isFileValid(parser, arg):
     else:
         return arg
 
+
 def main():
 
     parser = argparse.ArgumentParser(description=""" VulnWhisperer is designed to create actionable data from\
@@ -24,48 +27,102 @@ def main():
                         help='Path of config file', type=lambda x: isFileValid(parser, x.strip()))
     parser.add_argument('-s', '--section', dest='section', required=False,
                         help='Section in config')
+    parser.add_argument('--source', dest='source', required=False,
+                        help='JIRA required only! Source scanner to report')
+    parser.add_argument('-n', '--scanname', dest='scanname', required=False,
+                        help='JIRA required only! Scan name from scan to report')
     parser.add_argument('-v', '--verbose', dest='verbose', action='store_true', default=True,
                         help='Prints status out to screen (defaults to True)')
-    parser.add_argument('-u', '--username', dest='username', required=False, default=None, type=lambda x: x.strip(), help='The NESSUS username')
-    parser.add_argument('-p', '--password', dest='password', required=False, default=None, type=lambda x: x.strip(), help='The NESSUS password')
+    parser.add_argument('-u', '--username', dest='username', required=False, default=None,
+                        help='The NESSUS username', type=lambda x: x.strip())
+    parser.add_argument('-p', '--password', dest='password', required=False, default=None,
+                        help='The NESSUS password', type=lambda x: x.strip())
+    parser.add_argument('-F', '--fancy', action='store_true',
+                        help='Enable colourful logging output')
+    parser.add_argument('-d', '--debug', action='store_true',
+                        help='Enable debugging messages')
+    parser.add_argument('--mock', action='store_true',
+                        help='Enable mocked API responses')
+    parser.add_argument('--mock_dir', dest='mock_dir', required=False, default=None,
+                        help='Path of test directory')
     args = parser.parse_args()
 
+    # First setup logging
+    logging.basicConfig(
+        stream=sys.stdout,
+        #format only applies when not using -F flag for colouring
+        format='%(levelname)s:%(name)s:%(funcName)s:%(message)s',
+        level=logging.DEBUG if args.debug else logging.INFO
+    )
+    logger = logging.getLogger()
+    # we set up the logger to log as well to file
+    fh = logging.FileHandler('vulnwhisperer.log')
+    fh.setLevel(logging.DEBUG if args.debug else logging.INFO)
+    fh.setFormatter(logging.Formatter("%(asctime)s %(levelname)s %(name)s - %(funcName)s:%(message)s", "%Y-%m-%d %H:%M:%S"))
+    logger.addHandler(fh)
+
+    if args.fancy:
+        import coloredlogs
+        coloredlogs.install(level='DEBUG' if args.debug else 'INFO')
+
+    if args.mock:
+        mock_api = mockAPI(args.mock_dir, args.verbose)
+        mock_api.mock_endpoints()
+
+    exit_code = 0
+
     try:
         if args.config and not args.section:
-	    
-            print('{yellow}WARNING: {warning}{endc}'.format(yellow=bcolors.WARNING,
-                                                      warning='No section was specified, vulnwhisperer will scrape enabled modules from config file. \
-						      \nPlease specify a section using -s. \
-                                                      \nExample vuln_whisperer -c config.ini -s nessus',
-                                                      endc=bcolors.ENDC))
-	    config = vwConfig(config_in=args.config)
-	    enabled_sections = config.get_enabled()
-		
-	    for section in enabled_sections:
-		vw = vulnWhisperer(config=args.config,
-                               profile=section,
-                               verbose=args.verbose,
-                               username=args.username,
-                               password=args.password)
-
-            	vw.whisper_vulnerabilities()
-	    sys.exit(1)
+            # this remains a print since we are in the main binary
+            print('WARNING: {warning}'.format(warning='No section was specified, vulnwhisperer will scrape enabled modules from config file. \
+                            \nPlease specify a section using -s. \
+                                                    \nExample vuln_whisperer -c config.ini -s nessus'))
+            logger.info('No section was specified, vulnwhisperer will scrape enabled modules from the config file.')
+            
+            config = vwConfig(config_in=args.config)
+            enabled_sections = config.get_sections_with_attribute('enabled')
 
+            for section in enabled_sections:
+                try:
+                    vw = vulnWhisperer(config=args.config,
+                                       profile=section,
+                                       verbose=args.verbose,
+                                       username=args.username,
+                                       password=args.password,
+                                       source=args.source,
+                                       scanname=args.scanname)
+                    exit_code += vw.whisper_vulnerabilities()
+                except Exception as e:
+                    logger.error("VulnWhisperer was unable to perform the processing on '{}'".format(args.source))
         else:
+            logger.info('Running vulnwhisperer for section {}'.format(args.section))
             vw = vulnWhisperer(config=args.config,
                                profile=args.section,
                                verbose=args.verbose,
                                username=args.username,
-                               password=args.password)
+                               password=args.password,
+                               source=args.source,
+                               scanname=args.scanname)
+            exit_code += vw.whisper_vulnerabilities()
 
-            vw.whisper_vulnerabilities()
-            sys.exit(1)
+        close_logging_handlers(logger)
+        sys.exit(exit_code)
 
     except Exception as e:
         if args.verbose:
-            print('{red} ERROR: {error}{endc}'.format(red=bcolors.FAIL, error=e, endc=bcolors.ENDC))
+            # this will remain a print since we are in the main binary
+            logger.error('{}'.format(str(e)))
+            print('ERROR: {error}'.format(error=e))
+        # TODO: fix this to NOT be exit 2 unless in error
+        close_logging_handlers(logger)
         sys.exit(2)
 
+    close_logging_handlers(logger)
+
+def close_logging_handlers(logger):
+    for handler in logger.handlers:
+        handler.close()
+        logger.removeFilter(handler)
 
 if __name__ == '__main__':
     main()

configs/frameworks_example.ini

@@ -2,6 +2,8 @@
 enabled=true
 hostname=localhost
 port=8834
+access_key=
+secret_key=
 username=nessus_username
 password=nessus_password
 write_path=/opt/VulnWhisperer/data/nessus/
@@ -13,6 +15,8 @@ verbose=true
 enabled=true
 hostname=cloud.tenable.com
 port=443
+access_key=
+secret_key=
 username=tenable.io_username
 password=tenable.io_password
 write_path=/opt/VulnWhisperer/data/tenable/
@@ -20,13 +24,13 @@ db_path=/opt/VulnWhisperer/data/database
 trash=false
 verbose=true
 
-[qualys]
+[qualys_web]
 #Reference https://www.qualys.com/docs/qualys-was-api-user-guide.pdf to find your API
 enabled = true
 hostname = qualysapi.qg2.apps.qualys.com
 username = exampleuser
 password = examplepass
-write_path=/opt/VulnWhisperer/data/qualys/
+write_path=/opt/VulnWhisperer/data/qualys_web/
 db_path=/opt/VulnWhisperer/data/database
 verbose=true
 
@@ -37,21 +41,15 @@ max_retries = 10
 template_id = 126024
 
 [qualys_vuln]
-#Reference https://www.qualys.com/docs/qualys-was-api-user-guide.pdf to find your API
+#Reference https://www.qualys.com/docs/qualys-api-vmpc-user-guide.pdf to find your API
 enabled = true
 hostname = qualysapi.qg2.apps.qualys.com
 username = exampleuser
 password = examplepass
-write_path=/opt/VulnWhisperer/data/qualys/
+write_path=/opt/VulnWhisperer/data/qualys_vuln/
 db_path=/opt/VulnWhisperer/data/database
 verbose=true
 
-# Set the maximum number of retries each connection should attempt.
-#Note, this applies only to failed connections and timeouts, never to requests where the server returns a response.
-max_retries = 10
-# Template ID will need to be retrieved for each document. Please follow the reference guide above for instructions on how to get your template ID.
-template_id = 126024
-
 [detectify]
 #Reference https://developer.detectify.com/
 enabled = false
@@ -74,16 +72,23 @@ write_path=/opt/VulnWhisperer/data/openvas/
 db_path=/opt/VulnWhisperer/data/database
 verbose=true
 
-#[proxy]
-; This section is optional. Leave it out if you're not using a proxy.
-; You can use environmental variables as well: http://www.python-requests.org/en/latest/user/advanced/#proxies
+[jira]
+enabled = false
+hostname = jira-host
+username = username
+password = password
+write_path = /opt/VulnWhisperer/data/jira/
+db_path = /opt/VulnWhisperer/data/database
+verbose = true
+dns_resolv = False
 
-; proxy_protocol set to https, if not specified.
-#proxy_url = proxy.mycorp.com
+#Sample jira report scan, will automatically be created for existent scans
+#[jira.qualys_vuln.test_scan]
+#source = qualys_vuln
+#scan_name = Test Scan
+#jira_project = PROJECT
+; if multiple components, separate by "," = None
+#components = 
+; minimum criticality to report (low, medium, high or critical) = None
+#min_critical_to_report = high
 
-; proxy_port will override any port specified in proxy_url
-#proxy_port = 8080
-
-; proxy authentication
-#proxy_username = proxyuser
-#proxy_password = proxypass

configs/test.ini

@@ -0,0 +1,94 @@
+[nessus]
+enabled=true
+hostname=nessus
+port=443
+access_key=
+secret_key=
+username=nessus_username
+password=nessus_password
+write_path=/opt/VulnWhisperer/data/nessus/
+db_path=/opt/VulnWhisperer/data/database
+trash=false
+verbose=true
+
+[tenable]
+enabled=true
+hostname=tenable
+port=443
+access_key=
+secret_key=
+username=tenable.io_username
+password=tenable.io_password
+write_path=/opt/VulnWhisperer/data/tenable/
+db_path=/opt/VulnWhisperer/data/database
+trash=false
+verbose=true
+
+[qualys_web]
+#Reference https://www.qualys.com/docs/qualys-was-api-user-guide.pdf to find your API
+enabled = false
+hostname = qualys_web
+username = exampleuser
+password = examplepass
+write_path=/opt/VulnWhisperer/data/qualys_web/
+db_path=/opt/VulnWhisperer/data/database
+verbose=true
+
+# Set the maximum number of retries each connection should attempt.
+#Note, this applies only to failed connections and timeouts, never to requests where the server returns a response.
+max_retries = 10
+# Template ID will need to be retrieved for each document. Please follow the reference guide above for instructions on how to get your template ID.
+template_id = 126024
+
+[qualys_vuln]
+#Reference https://www.qualys.com/docs/qualys-was-api-user-guide.pdf to find your API
+enabled = true
+hostname = qualys_vuln
+username = exampleuser
+password = examplepass
+write_path=/opt/VulnWhisperer/data/qualys_vuln/
+db_path=/opt/VulnWhisperer/data/database
+verbose=true
+
+[detectify]
+#Reference https://developer.detectify.com/
+enabled = false
+hostname = detectify
+#username variable used as apiKey
+username = exampleuser
+#password variable used as secretKey
+password = examplepass
+write_path =/opt/VulnWhisperer/data/detectify/
+db_path = /opt/VulnWhisperer/data/database
+verbose = true
+
+[openvas]
+enabled = false
+hostname = openvas
+port = 4000
+username = exampleuser
+password = examplepass
+write_path=/opt/VulnWhisperer/data/openvas/
+db_path=/opt/VulnWhisperer/data/database
+verbose=true
+
+[jira]
+enabled = false
+hostname = jira-host
+username = username
+password = password
+write_path = /opt/VulnWhisperer/data/jira/
+db_path = /opt/VulnWhisperer/data/database
+verbose = true
+dns_resolv = False
+
+#Sample jira report scan, will automatically be created for existent scans
+#[jira.qualys_vuln.test_scan]
+#source = qualys_vuln
+#scan_name = Test Scan
+#jira_project = PROJECT
+; if multiple components, separate by "," = None
+#components = 
+; minimum criticality to report (low, medium, high or critical) = None
+#min_critical_to_report = high
+

deps/qualysapi/.gitignore

@@ -1,47 +0,0 @@
-*.py[cod]
-
-# C extensions
-*.so
-
-# Packages
-*.egg
-*.egg-info
-dist
-build
-eggs
-parts
-bin
-var
-sdist
-develop-eggs
-.installed.cfg
-lib
-lib64
-
-# Installer logs
-pip-log.txt
-
-# Unit test / coverage reports
-.coverage
-.tox
-nosetests.xml
-
-# Translations
-*.mo
-
-# Mr Developer
-.mr.developer.cfg
-.project
-.pydevproject
-
-# Mac
-.DS_Store
-
-# Authenticatin configuration
-*.qcrc
-config.qcrc
-config.ini
-
-# PyCharm
-.idea
-.qcrc.swp

deps/qualysapi/MANIFEST.in

@@ -1,2 +0,0 @@
-include README.md
-recursive-include examples *.py

deps/qualysapi/README.md

@@ -1,107 +0,0 @@
-qualysapi
-=========
-
-Python package, qualysapi, that makes calling any Qualys API very simple. Qualys API versions v1, v2, & WAS & AM (asset management) are all supported.
- 
-My focus was making the API super easy to use. The only parameters the user needs to provide is the call, and data (optional). It automates the following:
-* Automatically identifies API version through the call requested.
-* Automatically identifies url from the above step.
-* Automatically identifies http method as POST or GET for the request per Qualys documentation.
-
-Usage
-=====
-
-Check out the example scripts in the [/examples directory](https://github.com/paragbaxi/qualysapi/blob/master/examples/).
-
-Example
--------
-Detailed example found at [qualysapi-example.py](https://github.com/paragbaxi/qualysapi/blob/master/examples/qualysapi-example.py).
-
-Sample example below.
-
-```python
->>> import qualysapi
->>> a = qualysapi.connect()
-QualysGuard Username: my_username
-QualysGuard Password: 
->>> print a.request('about.php')
-<?xml version="1.0" encoding="UTF-8" ?>
-<!DOCTYPE ABOUT SYSTEM "https://qualysapi.qualys.com/about.dtd">
-<ABOUT>
-  <API-VERSION MAJOR="1" MINOR="4" />
-  <WEB-VERSION>7.10.61-1</WEB-VERSION>
-  <SCANNER-VERSION>7.1.10-1</SCANNER-VERSION>
-  <VULNSIGS-VERSION>2.2.475-2</VULNSIGS-VERSION>
-</ABOUT>
-<!-- Generated for username="my_username" date="2013-07-03T10:31:57Z" -->
-<!-- CONFIDENTIAL AND PROPRIETARY INFORMATION. Qualys provides the QualysGuard Service "As Is," without any warranty of any kind. Qualys makes no warranty that the information contained in this report is complete or error-free. Copyright 2013, Qualys, Inc. //--> 
-```
-
-Installation
-============
-
-Use pip to install:
-```Shell
-pip install qualysapi
-```
-
-NOTE: If you would like to experiment without installing globally, look into 'virtualenv'.
-
-Requirements
-------------
-
-* requests (http://docs.python-requests.org)
-* lxml (http://lxml.de/)
-
-Tested successfully on Python 2.7.
-
-Configuration
-=============
-
-By default, the package will ask at the command prompt for username and password. By default, the package connects to the Qualys documented host (qualysapi.qualys.com).
-
-You can override these settings and prevent yourself from typing credentials by doing any of the following:
-
-1. By running the following Python, `qualysapi.connect(remember_me=True)`. This automatically generates a .qcrc file in your current working directory, scoping the configuration to that directory.
-2. By running the following Python, `qualysapi.connect(remember_me_always=True)`. This automatically generates a .qcrc file in your home directory, scoping the configuratoin to all calls to qualysapi, regardless of the directory.
-3. By creating a file called '.qcrc' (for Windows, the default filename is 'config.ini') in your home directory or directory of the Python script.
-4. This supports multiple configuration files. Just add the filename in your call to qualysapi.connect('config.txt').
-
-Example config file
--------------------
-```INI
-; Note, it should be possible to omit any of these entries.
-
-[info]
-hostname = qualysapi.serviceprovider.com
-username = jerry
-password = I<3Elaine
-
-# Set the maximum number of retries each connection should attempt. Note, this applies only to failed connections and timeouts, never to requests where the server returns a response.
-max_retries = 10
-
-[proxy]
-; This section is optional. Leave it out if you're not using a proxy.
-; You can use environmental variables as well: http://www.python-requests.org/en/latest/user/advanced/#proxies
-
-; proxy_protocol set to https, if not specified.
-proxy_url = proxy.mycorp.com
-
-; proxy_port will override any port specified in proxy_url
-proxy_port = 8080
-
-; proxy authentication
-proxy_username = kramer
-proxy_password = giddy up!
-```
-
-
-License
-=======
-Apache License, Version 2.0
-http://www.apache.org/licenses/LICENSE-2.0.html
-
-Acknowledgements
-================
-
-Special thank you to Colin Bell for qualysconnect.

deps/qualysapi/changelog.txt

@@ -1,12 +0,0 @@
-3.5.0
-- Retooled authentication.
-
-3.4.0
-- Allows choice of configuration filenames. Easy to support those with multiple Qualys accounts, and need to automate tasks.
-
-3.3.0
-- Remove curl capability. Requests 2.0 and latest urllib3 can handle https proxy.
-- Workaround for audience that does not have lxml. Warning: cannot handle lxml.builder E objects for AM & WAS APIs.
-
-3.0.0
-Proxy support.

deps/qualysapi/examples/__init__.py

@@ -1 +0,0 @@
-__author__ = 'pbaxi'

deps/qualysapi/examples/qualysapi-example.py

@@ -1,113 +0,0 @@
-__author__ = 'Parag Baxi <parag.baxi@gmail.com>'
-__license__ = 'Apache License 2.0'
-
-import qualysapi
-from lxml import objectify
-from lxml.builder import E
-
-# Setup connection to QualysGuard API.
-qgc = qualysapi.connect('config.txt')
-#
-# API v1 call: Scan the New York & Las Vegas asset groups
-# The call is our request's first parameter.
-call = 'scan.php'
-# The parameters to append to the url is our request's second parameter.
-parameters = {'scan_title': 'Go big or go home', 'asset_groups': 'New York&Las Vegas', 'option': 'Initial+Options'}
-# Note qualysapi will automatically convert spaces into plus signs for API v1 & v2.
-# Let's call the API and store the result in xml_output.
-xml_output = qgc.request(call, parameters, concurrent_scans_retries=2, concurrent_scans_retry_delay=600)
-# concurrent_retries: Retry the call this many times if your subscription hits the concurrent scans limit.
-# concurrent_retries: Delay in seconds between retrying when subscription hits the concurrent scans limit.
-# Example XML response when this happens below:
-#     <?xml version="1.0" encoding="UTF-8"?>
-#     <ServiceResponse xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://localhost:50205/qps/rest/app//xsd/3.0/was/wasscan.xsd">
-#       <responseCode>INVALID_REQUEST</responseCode>
-#       <responseErrorDetails>
-#         <errorMessage>You have reached the maximum number of concurrent running scans (10) for your account</errorMessage>
-#         <errorResolution>Please wait until your previous scans have completed</errorResolution>
-#       </responseErrorDetails>
-#
-print(xml_output)
-#
-# API v1 call: Print out all IPs associated with asset group "Looneyville Texas".
-# Note that the question mark at the end is optional.
-call = 'asset_group_list.php?'
-# We can still use strings for the data (not recommended).
-parameters = 'title=Looneyville Texas'
-# Let's call the API and store the result in xml_output.
-xml_output = qgc.request(call, parameters)
-# Let's objectify the xml_output string.
-root = objectify.fromstring(xml_output)
-# Print out the IPs.
-print(root.ASSET_GROUP.SCANIPS.IP.text)
-# Prints out:
-# 10.0.0.102
-#
-# API v2 call: Print out DNS name for a range of IPs.
-call = '/api/2.0/fo/asset/host/'
-parameters = {'action': 'list', 'ips': '10.0.0.10-10.0.0.11'}
-xml_output = qgc.request(call, parameters)
-root = objectify.fromstring(xml_output)
-# Iterate hosts and print out DNS name.
-for host in root.RESPONSE.HOST_LIST.HOST:
-    print(host.IP.text, host.DNS.text)
-# Prints out:
-# 10.0.0.10 mydns1.qualys.com
-# 10.0.0.11 mydns2.qualys.com
-#
-# API v3 WAS call: Print out number of webapps.
-call = '/count/was/webapp'
-# Note that this call does not have a payload so we don't send any data parameters.
-xml_output = qgc.request(call)
-root = objectify.fromstring(xml_output)
-# Print out count of webapps.
-print(root.count.text)
-# Prints out:
-# 89
-#
-# API v3 WAS call: Print out number of webapps containing title 'Supafly'.
-call = '/count/was/webapp'
-# We can send a string XML for the data.
-parameters = '<ServiceRequest><filters><Criteria operator="CONTAINS" field="name">Supafly</Criteria></filters></ServiceRequest>'
-xml_output = qgc.request(call, parameters)
-root = objectify.fromstring(xml_output)
-# Print out count of webapps.
-print(root.count.text)
-# Prints out:
-# 3
-#
-# API v3 WAS call: Print out number of webapps containing title 'Lightsabertooth Tiger'.
-call = '/count/was/webapp'
-# We can also send an lxml.builder E object.
-parameters = (
-    E.ServiceRequest(
-        E.filters(
-            E.Criteria('Lightsabertooth Tiger', field='name',operator='CONTAINS'))))
-xml_output = qgc.request(call, parameters)
-root = objectify.fromstring(xml_output)
-# Print out count of webapps.
-print(root.count.text)
-# Prints out:
-# 0
-# Too bad, because that is an awesome webapp name!
-#
-# API v3 Asset Management call: Count tags.
-call = '/count/am/tag'
-xml_output = qgc.request(call)
-root = objectify.fromstring(xml_output)
-# We can use XPATH to find the count.
-print(root.xpath('count')[0].text)
-# Prints out:
-# 840
-#
-# API v3 Asset Management call: Find asset by name.
-call = '/search/am/tag'
-parameters = '''<ServiceRequest>
-        <preferences>
-            <limitResults>10</limitResults>
-        </preferences>
-        <filters>
-            <Criteria field="name" operator="CONTAINS">PB</Criteria>
-        </filters>
-    </ServiceRequest>'''
-xml_output = qgc.request(call, parameters)

deps/qualysapi/examples/qualysapi-simple-v1.py

@@ -1,42 +0,0 @@
-#!/usr/bin/env python
-import sys
-import logging
-
-import qualysapi
-
-# Questions?  See:
-#  https://bitbucket.org/uWaterloo_IST_ISS/python-qualysconnect
-
-if __name__ == '__main__':
-    # Basic command line processing.
-    if len(sys.argv) != 2:
-        print('A single IPv4 address is expected as the only argument')
-        sys.exit(2)
-    
-    # Set the MAXIMUM level of log messages displayed @ runtime. 
-    logging.basicConfig(level=logging.INFO)
-    
-    # Call helper that creates a connection w/ HTTP-Basic to QualysGuard API.
-    qgs=qualysapi.connect()
-
-    # Logging must be set after instanciation of connector class.
-    logger = logging.getLogger('qualysapi.connector')
-    logger.setLevel(logging.DEBUG)
-
-    # Log to sys.out.
-    logger_console = logging.StreamHandler()
-    logger_console.setLevel(logging.DEBUG)
-    formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s')
-    logging.getLogger(__name__).addHandler(logger)
-
-    # Formulate a request to the QualysGuard V1 API.
-    #  docs @
-    #  https://community.qualys.com/docs/DOC-1324
-    #  http://www.qualys.com/docs/QualysGuard_API_User_Guide.pdf
-    #
-    # Old way still works:
-    # ret = qgs.request(1,'asset_search.php', "target_ips=%s&"%(sys.argv[1]))
-    # New way is cleaner:
-    ret = qgs.request(1,'asset_search.php', {'target_ips': sys.argv[1]})
-
-    print(ret)

deps/qualysapi/examples/qualysapi-simple-v2-report.py

@@ -1,37 +0,0 @@
-#!/usr/bin/env python
-import sys
-import logging
-
-import qualysapi
-
-if __name__ == '__main__':
-    # Basic command line processing.
-    if len(sys.argv) != 3:
-        print('A report template and scan reference respectively are expected as the only arguments.')
-        sys.exit(2)
-    
-    # Set the MAXIMUM level of log messages displayed @ runtime. 
-    logging.basicConfig(level=logging.DEBUG)
-    
-    # Call helper that creates a connection w/ HTTP-Basic to QualysGuard v1 API
-    qgs=qualysapi.connect()
-
-    # Logging must be set after instanciation of connector class.
-    logger = logging.getLogger('qualysapi.connector')
-    logger.setLevel(logging.DEBUG)
-
-    # Log to sys.out.
-    logger_console = logging.StreamHandler()
-    logger_console.setLevel(logging.DEBUG)
-    formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s')
-    logging.getLogger(__name__).addHandler(logger)
-
-
-    # Formulate a request to the QualysGuard V1 API 
-    #  docs @
-    #  https://community.qualys.com/docs/DOC-1324
-    #  http://www.qualys.com/docs/QualysGuard_API_User_Guide.pdf
-    #
-    ret = qgs.request('/api/2.0/fo/report',{'action': 'launch', 'report_refs': sys.argv[2], 'output_format': 'xml', 'template_id': sys.argv[1], 'report_type': 'Scan'})
-
-    print(ret)

deps/qualysapi/examples/qualysapi-simple-v2.py

@@ -1,43 +0,0 @@
-#!/usr/bin/env python
-import sys
-import logging
-
-import qualysapi
-
-# Questions?  See:
-#  https://bitbucket.org/uWaterloo_IST_ISS/python-qualysconnect
-
-if __name__ == '__main__':
-    # Basic command line processing.
-    if len(sys.argv) != 2:
-        print('A single IPv4 address is expected as the only argument.')
-        sys.exit(2)
-    
-    # Set the MAXIMUM level of log messages displayed @ runtime. 
-    logging.basicConfig(level=logging.INFO)
-    
-    # Call helper that creates a connection w/ HTTP-Basic to QualysGuard v1 API
-    qgs=qualysapi.connect()
-
-    # Logging must be set after instanciation of connector class.
-    logger = logging.getLogger('qualysapi.connector')
-    logger.setLevel(logging.DEBUG)
-
-    # Log to sys.out.
-    logger_console = logging.StreamHandler()
-    logger_console.setLevel(logging.DEBUG)
-    formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s')
-    logging.getLogger(__name__).addHandler(logger)
-
-
-    # Formulate a request to the QualysGuard V1 API 
-    #  docs @
-    #  https://community.qualys.com/docs/DOC-1324
-    #  http://www.qualys.com/docs/QualysGuard_API_User_Guide.pdf
-    #
-    # Old way still works:
-    # ret = qgs.request(2, "asset/host","?action=list&ips=%s&"%(sys.argv[1]))
-    # New way is cleaner:
-    ret = qgs.request('/api/2.0/fo/asset/host',{'action': 'list', 'ips': sys.argv[1]})
-
-    print(ret)

deps/qualysapi/license

@@ -1,201 +0,0 @@
-                                 Apache License
-                           Version 2.0, January 2004
-                        http://www.apache.org/licenses/
-
-   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
-
-   1. Definitions.
-
-      "License" shall mean the terms and conditions for use, reproduction,
-      and distribution as defined by Sections 1 through 9 of this document.
-
-      "Licensor" shall mean the copyright owner or entity authorized by
-      the copyright owner that is granting the License.
-
-      "Legal Entity" shall mean the union of the acting entity and all
-      other entities that control, are controlled by, or are under common
-      control with that entity. For the purposes of this definition,
-      "control" means (i) the power, direct or indirect, to cause the
-      direction or management of such entity, whether by contract or
-      otherwise, or (ii) ownership of fifty percent (50%) or more of the
-      outstanding shares, or (iii) beneficial ownership of such entity.
-
-      "You" (or "Your") shall mean an individual or Legal Entity
-      exercising permissions granted by this License.
-
-      "Source" form shall mean the preferred form for making modifications,
-      including but not limited to software source code, documentation
-      source, and configuration files.
-
-      "Object" form shall mean any form resulting from mechanical
-      transformation or translation of a Source form, including but
-      not limited to compiled object code, generated documentation,
-      and conversions to other media types.
-
-      "Work" shall mean the work of authorship, whether in Source or
-      Object form, made available under the License, as indicated by a
-      copyright notice that is included in or attached to the work
-      (an example is provided in the Appendix below).
-
-      "Derivative Works" shall mean any work, whether in Source or Object
-      form, that is based on (or derived from) the Work and for which the
-      editorial revisions, annotations, elaborations, or other modifications
-      represent, as a whole, an original work of authorship. For the purposes
-      of this License, Derivative Works shall not include works that remain
-      separable from, or merely link (or bind by name) to the interfaces of,
-      the Work and Derivative Works thereof.
-
-      "Contribution" shall mean any work of authorship, including
-      the original version of the Work and any modifications or additions
-      to that Work or Derivative Works thereof, that is intentionally
-      submitted to Licensor for inclusion in the Work by the copyright owner
-      or by an individual or Legal Entity authorized to submit on behalf of
-      the copyright owner. For the purposes of this definition, "submitted"
-      means any form of electronic, verbal, or written communication sent
-      to the Licensor or its representatives, including but not limited to
-      communication on electronic mailing lists, source code control systems,
-      and issue tracking systems that are managed by, or on behalf of, the
-      Licensor for the purpose of discussing and improving the Work, but
-      excluding communication that is conspicuously marked or otherwise
-      designated in writing by the copyright owner as "Not a Contribution."
-
-      "Contributor" shall mean Licensor and any individual or Legal Entity
-      on behalf of whom a Contribution has been received by Licensor and
-      subsequently incorporated within the Work.
-
-   2. Grant of Copyright License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      copyright license to reproduce, prepare Derivative Works of,
-      publicly display, publicly perform, sublicense, and distribute the
-      Work and such Derivative Works in Source or Object form.
-
-   3. Grant of Patent License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      (except as stated in this section) patent license to make, have made,
-      use, offer to sell, sell, import, and otherwise transfer the Work,
-      where such license applies only to those patent claims licensable
-      by such Contributor that are necessarily infringed by their
-      Contribution(s) alone or by combination of their Contribution(s)
-      with the Work to which such Contribution(s) was submitted. If You
-      institute patent litigation against any entity (including a
-      cross-claim or counterclaim in a lawsuit) alleging that the Work
-      or a Contribution incorporated within the Work constitutes direct
-      or contributory patent infringement, then any patent licenses
-      granted to You under this License for that Work shall terminate
-      as of the date such litigation is filed.
-
-   4. Redistribution. You may reproduce and distribute copies of the
-      Work or Derivative Works thereof in any medium, with or without
-      modifications, and in Source or Object form, provided that You
-      meet the following conditions:
-
-      (a) You must give any other recipients of the Work or
-          Derivative Works a copy of this License; and
-
-      (b) You must cause any modified files to carry prominent notices
-          stating that You changed the files; and
-
-      (c) You must retain, in the Source form of any Derivative Works
-          that You distribute, all copyright, patent, trademark, and
-          attribution notices from the Source form of the Work,
-          excluding those notices that do not pertain to any part of
-          the Derivative Works; and
-
-      (d) If the Work includes a "NOTICE" text file as part of its
-          distribution, then any Derivative Works that You distribute must
-          include a readable copy of the attribution notices contained
-          within such NOTICE file, excluding those notices that do not
-          pertain to any part of the Derivative Works, in at least one
-          of the following places: within a NOTICE text file distributed
-          as part of the Derivative Works; within the Source form or
-          documentation, if provided along with the Derivative Works; or,
-          within a display generated by the Derivative Works, if and
-          wherever such third-party notices normally appear. The contents
-          of the NOTICE file are for informational purposes only and
-          do not modify the License. You may add Your own attribution
-          notices within Derivative Works that You distribute, alongside
-          or as an addendum to the NOTICE text from the Work, provided
-          that such additional attribution notices cannot be construed
-          as modifying the License.
-
-      You may add Your own copyright statement to Your modifications and
-      may provide additional or different license terms and conditions
-      for use, reproduction, or distribution of Your modifications, or
-      for any such Derivative Works as a whole, provided Your use,
-      reproduction, and distribution of the Work otherwise complies with
-      the conditions stated in this License.
-
-   5. Submission of Contributions. Unless You explicitly state otherwise,
-      any Contribution intentionally submitted for inclusion in the Work
-      by You to the Licensor shall be under the terms and conditions of
-      this License, without any additional terms or conditions.
-      Notwithstanding the above, nothing herein shall supersede or modify
-      the terms of any separate license agreement you may have executed
-      with Licensor regarding such Contributions.
-
-   6. Trademarks. This License does not grant permission to use the trade
-      names, trademarks, service marks, or product names of the Licensor,
-      except as required for reasonable and customary use in describing the
-      origin of the Work and reproducing the content of the NOTICE file.
-
-   7. Disclaimer of Warranty. Unless required by applicable law or
-      agreed to in writing, Licensor provides the Work (and each
-      Contributor provides its Contributions) on an "AS IS" BASIS,
-      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-      implied, including, without limitation, any warranties or conditions
-      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
-      PARTICULAR PURPOSE. You are solely responsible for determining the
-      appropriateness of using or redistributing the Work and assume any
-      risks associated with Your exercise of permissions under this License.
-
-   8. Limitation of Liability. In no event and under no legal theory,
-      whether in tort (including negligence), contract, or otherwise,
-      unless required by applicable law (such as deliberate and grossly
-      negligent acts) or agreed to in writing, shall any Contributor be
-      liable to You for damages, including any direct, indirect, special,
-      incidental, or consequential damages of any character arising as a
-      result of this License or out of the use or inability to use the
-      Work (including but not limited to damages for loss of goodwill,
-      work stoppage, computer failure or malfunction, or any and all
-      other commercial damages or losses), even if such Contributor
-      has been advised of the possibility of such damages.
-
-   9. Accepting Warranty or Additional Liability. While redistributing
-      the Work or Derivative Works thereof, You may choose to offer,
-      and charge a fee for, acceptance of support, warranty, indemnity,
-      or other liability obligations and/or rights consistent with this
-      License. However, in accepting such obligations, You may act only
-      on Your own behalf and on Your sole responsibility, not on behalf
-      of any other Contributor, and only if You agree to indemnify,
-      defend, and hold each Contributor harmless for any liability
-      incurred by, or claims asserted against, such Contributor by reason
-      of your accepting any such warranty or additional liability.
-
-   END OF TERMS AND CONDITIONS
-
-   APPENDIX: How to apply the Apache License to your work.
-
-      To apply the Apache License to your work, attach the following
-      boilerplate notice, with the fields enclosed by brackets "{}"
-      replaced with your own identifying information. (Don't include
-      the brackets!)  The text should be enclosed in the appropriate
-      comment syntax for the file format. We also recommend that a
-      file or class name and description of purpose be included on the
-      same "printed page" as the copyright notice for easier
-      identification within third-party archives.
-
-   Copyright 2017 Parag Baxi
-
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-
-       http://www.apache.org/licenses/LICENSE-2.0
-
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License.

deps/qualysapi/qualysapi/__init__.py

@@ -1,10 +0,0 @@
-# This is the version string assigned to the entire egg post
-# setup.py install
-
-# Ownership and Copyright Information.
-from __future__ import absolute_import
-__author__ = "Parag Baxi <parag.baxi@gmail.com>"
-__copyright__ = "Copyright 2011-2013, Parag Baxi"
-__license__ = "BSD-new"
-
-from qualysapi.util import connect

deps/qualysapi/qualysapi/api_actions.py

@@ -1,181 +0,0 @@
-from __future__ import absolute_import
-from lxml import objectify
-import qualysapi.api_objects
-from qualysapi.api_objects import *
-
-
-class QGActions(object):
-    def getHost(host):
-        call = '/api/2.0/fo/asset/host/'
-        parameters = {'action': 'list', 'ips': host, 'details': 'All'}
-        hostData = objectify.fromstring(self.request(call, parameters)).RESPONSE
-        try:
-            hostData = hostData.HOST_LIST.HOST
-            return Host(hostData.DNS, hostData.ID, hostData.IP, hostData.LAST_VULN_SCAN_DATETIME, hostData.NETBIOS, hostData.OS, hostData.TRACKING_METHOD)
-        except AttributeError:
-            return Host("", "", host, "never", "", "", "")
-
-    def getHostRange(self, start, end):
-        call = '/api/2.0/fo/asset/host/'
-        parameters = {'action': 'list', 'ips': start + '-' + end}
-        hostData = objectify.fromstring(self.request(call, parameters))
-        hostArray = []
-        for host in hostData.RESPONSE.HOST_LIST.HOST:
-            hostArray.append(Host(host.DNS, host.ID, host.IP, host.LAST_VULN_SCAN_DATETIME, host.NETBIOS, host.OS, host.TRACKING_METHOD))
-
-        return hostArray
-
-    def listAssetGroups(self, groupName=''):
-        call = 'asset_group_list.php'
-        if groupName == '':
-            agData = objectify.fromstring(self.request(call))
-        else:
-            agData = objectify.fromstring(self.request(call, 'title=' + groupName)).RESPONSE
-
-        groupsArray = []
-        scanipsArray = []
-        scandnsArray = []
-        scannersArray = []
-        for group in agData.ASSET_GROUP:
-            try:
-                for scanip in group.SCANIPS:
-                    scanipsArray.append(scanip.IP)
-            except AttributeError:
-                scanipsArray = []  # No IPs defined to scan.
-
-            try:
-                for scanner in group.SCANNER_APPLIANCES.SCANNER_APPLIANCE:
-                    scannersArray.append(scanner.SCANNER_APPLIANCE_NAME)
-            except AttributeError:
-                scannersArray = []  # No scanner appliances defined for this group.
-
-            try:
-                for dnsName in group.SCANDNS:
-                    scandnsArray.append(dnsName.DNS)
-            except AttributeError:
-                scandnsArray = []  # No DNS names assigned to group.
-
-            groupsArray.append(AssetGroup(group.BUSINESS_IMPACT, group.ID, group.LAST_UPDATE, scanipsArray, scandnsArray, scannersArray, group.TITLE))
-
-        return groupsArray
-
-    def listReportTemplates(self):
-        call = 'report_template_list.php'
-        rtData = objectify.fromstring(self.request(call))
-        templatesArray = []
-
-        for template in rtData.REPORT_TEMPLATE:
-            templatesArray.append(ReportTemplate(template.GLOBAL, template.ID, template.LAST_UPDATE, template.TEMPLATE_TYPE, template.TITLE, template.TYPE, template.USER))
-
-        return templatesArray
-
-    def listReports(self, id=0):
-        call = '/api/2.0/fo/report'
-
-        if id == 0:
-            parameters = {'action': 'list'}
-
-            repData = objectify.fromstring(self.request(call, parameters)).RESPONSE
-            reportsArray = []
-
-            for report in repData.REPORT_LIST.REPORT:
-                reportsArray.append(Report(report.EXPIRATION_DATETIME, report.ID, report.LAUNCH_DATETIME, report.OUTPUT_FORMAT, report.SIZE, report.STATUS, report.TYPE, report.USER_LOGIN))
-
-            return reportsArray
-
-        else:
-            parameters = {'action': 'list', 'id': id}
-            repData = objectify.fromstring(self.request(call, parameters)).RESPONSE.REPORT_LIST.REPORT
-            return Report(repData.EXPIRATION_DATETIME, repData.ID, repData.LAUNCH_DATETIME, repData.OUTPUT_FORMAT, repData.SIZE, repData.STATUS, repData.TYPE, repData.USER_LOGIN)
-
-    def notScannedSince(self, days):
-        call = '/api/2.0/fo/asset/host/'
-        parameters = {'action': 'list', 'details': 'All'}
-        hostData = objectify.fromstring(self.request(call, parameters))
-        hostArray = []
-        today = datetime.date.today()
-        for host in hostData.RESPONSE.HOST_LIST.HOST:
-            last_scan = str(host.LAST_VULN_SCAN_DATETIME).split('T')[0]
-            last_scan = datetime.date(int(last_scan.split('-')[0]), int(last_scan.split('-')[1]), int(last_scan.split('-')[2]))
-            if (today - last_scan).days >= days:
-                hostArray.append(Host(host.DNS, host.ID, host.IP, host.LAST_VULN_SCAN_DATETIME, host.NETBIOS, host.OS, host.TRACKING_METHOD))
-
-        return hostArray
-
-    def addIP(self, ips, vmpc):
-        # 'ips' parameter accepts comma-separated list of IP addresses.
-        # 'vmpc' parameter accepts 'vm', 'pc', or 'both'. (Vulnerability Managment, Policy Compliance, or both)
-        call = '/api/2.0/fo/asset/ip/'
-        enablevm = 1
-        enablepc = 0
-        if vmpc == 'pc':
-            enablevm = 0
-            enablepc = 1
-        elif vmpc == 'both':
-            enablevm = 1
-            enablepc = 1
-
-        parameters = {'action': 'add', 'ips': ips, 'enable_vm': enablevm, 'enable_pc': enablepc}
-        self.request(call, parameters)
-
-    def listScans(self, launched_after="", state="", target="", type="", user_login=""):
-        # 'launched_after' parameter accepts a date in the format: YYYY-MM-DD
-        # 'state' parameter accepts "Running", "Paused", "Canceled", "Finished", "Error", "Queued", and "Loading".
-        # 'title' parameter accepts a string
-        # 'type' parameter accepts "On-Demand", and "Scheduled".
-        # 'user_login' parameter accepts a user name (string)
-        call = '/api/2.0/fo/scan/'
-        parameters = {'action': 'list', 'show_ags': 1, 'show_op': 1, 'show_status': 1}
-        if launched_after != "":
-            parameters['launched_after_datetime'] = launched_after
-
-        if state != "":
-            parameters['state'] = state
-
-        if target != "":
-            parameters['target'] = target
-
-        if type != "":
-            parameters['type'] = type
-
-        if user_login != "":
-            parameters['user_login'] = user_login
-
-        scanlist = objectify.fromstring(self.request(call, parameters))
-        scanArray = []
-        for scan in scanlist.RESPONSE.SCAN_LIST.SCAN:
-            try:
-                agList = []
-                for ag in scan.ASSET_GROUP_TITLE_LIST.ASSET_GROUP_TITLE:
-                    agList.append(ag)
-            except AttributeError:
-                agList = []
-
-            scanArray.append(Scan(agList, scan.DURATION, scan.LAUNCH_DATETIME, scan.OPTION_PROFILE.TITLE, scan.PROCESSED, scan.REF, scan.STATUS, scan.TARGET, scan.TITLE, scan.TYPE, scan.USER_LOGIN))
-
-        return scanArray
-
-    def launchScan(self, title, option_title, iscanner_name, asset_groups="", ip=""):
-        # TODO: Add ability to scan by tag.
-        call = '/api/2.0/fo/scan/'
-        parameters = {'action': 'launch', 'scan_title': title, 'option_title': option_title, 'iscanner_name': iscanner_name, 'ip': ip, 'asset_groups': asset_groups}
-        if ip == "":
-            parameters.pop("ip")
-
-        if asset_groups == "":
-            parameters.pop("asset_groups")
-
-        scan_ref = objectify.fromstring(self.request(call, parameters)).RESPONSE.ITEM_LIST.ITEM[1].VALUE
-
-        call = '/api/2.0/fo/scan/'
-        parameters = {'action': 'list', 'scan_ref': scan_ref, 'show_status': 1, 'show_ags': 1, 'show_op': 1}
-
-        scan = objectify.fromstring(self.request(call, parameters)).RESPONSE.SCAN_LIST.SCAN
-        try:
-            agList = []
-            for ag in scan.ASSET_GROUP_TITLE_LIST.ASSET_GROUP_TITLE:
-                agList.append(ag)
-        except AttributeError:
-            agList = []
-
-        return Scan(agList, scan.DURATION, scan.LAUNCH_DATETIME, scan.OPTION_PROFILE.TITLE, scan.PROCESSED, scan.REF, scan.STATUS, scan.TARGET, scan.TITLE, scan.TYPE, scan.USER_LOGIN)

deps/qualysapi/qualysapi/api_methods.py

@@ -1,155 +0,0 @@
-from __future__ import absolute_import
-__author__ = 'pbaxi'
-
-from collections import defaultdict
-
-api_methods = defaultdict(set)
-api_methods['1'] = set([
-    'about.php',
-    'action_log_report.php',
-    'asset_data_report.php',
-    'asset_domain.php',
-    'asset_domain_list.php',
-    'asset_group_delete.php',
-    'asset_group_list.php',
-    'asset_ip_list.php',
-    'asset_range_info.php',
-    'asset_search.php',
-    'get_host_info.php',
-    'ignore_vuln.php',
-    'iscanner_list.php',
-    'knowledgebase_download.php',
-    'map-2.php',
-    'map.php',
-    'map_report.php',
-    'map_report_list.php',
-    'password_change.php',
-    'scan.php',
-    'scan_cancel.php',
-    'scan_options.php',
-    'scan_report.php',
-    'scan_report_delete.php',
-    'scan_report_list.php',
-    'scan_running_list.php',
-    'scan_target_history.php',
-    'scheduled_scans.php',
-    'ticket_delete.php',
-    'ticket_edit.php',
-    'ticket_list.php',
-    'ticket_list_deleted.php',
-    'time_zone_code.php',
-    'user.php',
-    'user_list.php',
-])
-# API v1 POST methods.
-api_methods['1 post'] = set([
-    'action_log_report.php',
-    'asset_group.php',
-    'asset_ip.php',
-    'ignore_vuln.php',
-    'knowledgebase_download.php',
-    'map-2.php',
-    'map.php',
-    'password_change.php',
-    'scan.php',
-    'scan_report.php',
-    'scan_target_history.php',
-    'scheduled_scans.php',
-    'ticket_delete.php',
-    'ticket_edit.php',
-    'ticket_list.php',
-    'ticket_list_deleted.php',
-    'user.php',
-    'user_list.php',
-])
-# API v2 methods (they're all POST).
-api_methods['2'] = set([
-    'api/2.0/fo/appliance/',
-    'api/2.0/fo/asset/excluded_ip/',
-    'api/2.0/fo/asset/excluded_ip/history/',
-    'api/2.0/fo/asset/host/',
-    'api/2.0/fo/asset/host/cyberscope/',
-    'api/2.0/fo/asset/host/cyberscope/fdcc/policy/',
-    'api/2.0/fo/asset/host/cyberscope/fdcc/scan/',
-    'api/2.0/fo/asset/host/vm/detection/',
-    'api/2.0/fo/asset/ip/',
-    'api/2.0/fo/asset/ip/v4_v6/',
-    'api/2.0/fo/asset/vhost/',
-    'api/2.0/fo/auth/',
-    # 'api/2.0/fo/auth/{type}/', # Added below.
-    'api/2.0/fo/compliance/',
-    'api/2.0/fo/compliance/control',
-    'api/2.0/fo/compliance/fdcc/policy',
-    'api/2.0/fo/compliance/policy/',
-    'api/2.0/fo/compliance/posture/info/',
-    'api/2.0/fo/compliance/scap/arf/',
-    'api/2.0/fo/knowledge_base/vuln/',
-    'api/2.0/fo/report/',
-    'api/2.0/fo/report/scorecard/',
-    'api/2.0/fo/scan/',
-    'api/2.0/fo/scan/compliance/',
-    'api/2.0/fo/session/',
-    'api/2.0/fo/setup/restricted_ips/',
-])
-for auth_type in set([
-    'ibm_db2',
-    'ms_sql',
-    'oracle',
-    'oracle_listener',
-    'snmp',
-    'unix',
-    'windows',
-]):
-    api_methods['2'].add('api/2.0/fo/auth/%s/' % auth_type)
-# WAS GET methods when no POST data.
-api_methods['was no data get'] = set([
-    'count/was/report',
-    'count/was/wasscan',
-    'count/was/wasscanschedule',
-    'count/was/webapp',
-    'download/was/report/',
-    'download/was/wasscan/',
-])
-# WAS GET methods.
-api_methods['was get'] = set([
-    'download/was/report/',
-    'download/was/wasscan/',
-    'get/was/report/',
-    'get/was/wasscan/',
-    'get/was/wasscanschedule/',
-    'get/was/webapp/',
-    'status/was/report/',
-    'status/was/wasscan/',
-])
-# Asset Management GET methods.
-api_methods['am get'] = set([
-    'count/am/asset',
-    'count/am/hostasset',
-    'count/am/tag',
-    'get/am/asset/',
-    'get/am/hostasset/',
-    'get/am/tag/',
-])
-# Asset Management v2 GET methods.
-api_methods['am2 get'] = set([
-    'get/am/asset/',
-    'get/am/hostasset/',
-    'get/am/tag/',
-    'get/am/hostinstancevuln/',
-    'get/am/assetdataconnector/',
-    'get/am/awsassetdataconnector/',
-    'get/am/awsauthrecord/',
-])
-# Keep track of methods with ending slashes to autocorrect user when they forgot slash.
-api_methods_with_trailing_slash = defaultdict(set)
-for method_group in set(['1', '2', 'was', 'am', 'am2']):
-    for method in api_methods[method_group]:
-        if method[-1] == '/':
-            # Add applicable method with api_version preceding it.
-            # Example:
-            # WAS API has 'get/was/webapp/'.
-            # method_group = 'was get'
-            # method_group.split()[0] = 'was'
-            # Take off slash to match user provided method.
-            # api_methods_with_trailing_slash['was'] contains 'get/was/webapp'
-            api_methods_with_trailing_slash[method_group.split()[0]].add(method[:-1])

deps/qualysapi/qualysapi/api_objects.py

@@ -1,120 +0,0 @@
-from __future__ import absolute_import
-import datetime
-from lxml import objectify
-
-
-class Host(object):
-    def __init__(self, dns, id, ip, last_scan, netbios, os, tracking_method):
-        self.dns = str(dns)
-        self.id = int(id)
-        self.ip = str(ip)
-        last_scan = str(last_scan).replace('T', ' ').replace('Z', '').split(' ')
-        date = last_scan[0].split('-')
-        time = last_scan[1].split(':')
-        self.last_scan = datetime.datetime(int(date[0]), int(date[1]), int(date[2]), int(time[0]), int(time[1]), int(time[2]))
-        self.netbios = str(netbios)
-        self.os = str(os)
-        self.tracking_method = str(tracking_method)
-
-
-class AssetGroup(object):
-    def __init__(self, business_impact, id, last_update, scanips, scandns, scanner_appliances, title):
-        self.business_impact = str(business_impact)
-        self.id = int(id)
-        self.last_update = str(last_update)
-        self.scanips = scanips
-        self.scandns = scandns
-        self.scanner_appliances = scanner_appliances
-        self.title = str(title)
-
-    def addAsset(conn, ip):
-        call = '/api/2.0/fo/asset/group/'
-        parameters = {'action': 'edit', 'id': self.id, 'add_ips': ip}
-        conn.request(call, parameters)
-        self.scanips.append(ip)
-
-    def setAssets(conn, ips):
-        call = '/api/2.0/fo/asset/group/'
-        parameters = {'action': 'edit', 'id': self.id, 'set_ips': ips}
-        conn.request(call, parameters)
-
-
-class ReportTemplate(object):
-    def __init__(self, isGlobal, id, last_update, template_type, title, type, user):
-        self.isGlobal = int(isGlobal)
-        self.id = int(id)
-        self.last_update = str(last_update).replace('T', ' ').replace('Z', '').split(' ')
-        self.template_type = template_type
-        self.title = title
-        self.type = type
-        self.user = user.LOGIN
-
-
-class Report(object):
-    def __init__(self, expiration_datetime, id, launch_datetime, output_format, size, status, type, user_login):
-        self.expiration_datetime = str(expiration_datetime).replace('T', ' ').replace('Z', '').split(' ')
-        self.id = int(id)
-        self.launch_datetime = str(launch_datetime).replace('T', ' ').replace('Z', '').split(' ')
-        self.output_format = output_format
-        self.size = size
-        self.status = status.STATE
-        self.type = type
-        self.user_login = user_login
-
-    def download(self, conn):
-        call = '/api/2.0/fo/report'
-        parameters = {'action': 'fetch', 'id': self.id}
-        if self.status == 'Finished':
-            return conn.request(call, parameters)
-
-
-class Scan(object):
-    def __init__(self, assetgroups, duration, launch_datetime, option_profile, processed, ref, status, target, title, type, user_login):
-        self.assetgroups = assetgroups
-        self.duration = str(duration)
-        launch_datetime = str(launch_datetime).replace('T', ' ').replace('Z', '').split(' ')
-        date = launch_datetime[0].split('-')
-        time = launch_datetime[1].split(':')
-        self.launch_datetime = datetime.datetime(int(date[0]), int(date[1]), int(date[2]), int(time[0]), int(time[1]), int(time[2]))
-        self.option_profile = str(option_profile)
-        self.processed = int(processed)
-        self.ref = str(ref)
-        self.status = str(status.STATE)
-        self.target = str(target).split(', ')
-        self.title = str(title)
-        self.type = str(type)
-        self.user_login = str(user_login)
-
-    def cancel(self, conn):
-        cancelled_statuses = ['Cancelled', 'Finished', 'Error']
-        if any(self.status in s for s in cancelled_statuses):
-            raise ValueError("Scan cannot be cancelled because its status is " + self.status)
-        else:
-            call = '/api/2.0/fo/scan/'
-            parameters = {'action': 'cancel', 'scan_ref': self.ref}
-            conn.request(call, parameters)
-
-            parameters = {'action': 'list', 'scan_ref': self.ref, 'show_status': 1}
-            self.status = objectify.fromstring(conn.request(call, parameters)).RESPONSE.SCAN_LIST.SCAN.STATUS.STATE
-
-    def pause(self, conn):
-        if self.status != "Running":
-            raise ValueError("Scan cannot be paused because its status is " + self.status)
-        else:
-            call = '/api/2.0/fo/scan/'
-            parameters = {'action': 'pause', 'scan_ref': self.ref}
-            conn.request(call, parameters)
-
-            parameters = {'action': 'list', 'scan_ref': self.ref, 'show_status': 1}
-            self.status = objectify.fromstring(conn.request(call, parameters)).RESPONSE.SCAN_LIST.SCAN.STATUS.STATE
-
-    def resume(self, conn):
-        if self.status != "Paused":
-            raise ValueError("Scan cannot be resumed because its status is " + self.status)
-        else:
-            call = '/api/2.0/fo/scan/'
-            parameters = {'action': 'resume', 'scan_ref': self.ref}
-            conn.request(call, parameters)
-
-            parameters = {'action': 'list', 'scan_ref': self.ref, 'show_status': 1}
-            self.status = objectify.fromstring(conn.request(call, parameters)).RESPONSE.SCAN_LIST.SCAN.STATUS.STATE

deps/qualysapi/qualysapi/config.py

@@ -1,221 +0,0 @@
-""" Module providing a single class (QualysConnectConfig) that parses a config
-file and provides the information required to build QualysGuard sessions.
-"""
-from __future__ import absolute_import
-from __future__ import print_function
-import os
-import stat
-import getpass
-import logging
-from six.moves import input
-from six.moves.configparser import *
-
-import qualysapi.settings as qcs
-# Setup module level logging.
-logger = logging.getLogger(__name__)
-
-# try:
-#    from requests_ntlm import HttpNtlmAuth
-# except ImportError, e:
-#    logger.warning('Warning: Cannot support NTML authentication.')
-
-
-__author__ = "Parag Baxi <parag.baxi@gmail.com> & Colin Bell <colin.bell@uwaterloo.ca>"
-__updated_by__ = "Austin Taylor <vulnWhisperer@austintaylor.io>"
-__copyright__ = "Copyright 2011-2013, Parag Baxi & University of Waterloo"
-__license__ = "BSD-new"
-
-
-class QualysConnectConfig:
-    """ Class to create a ConfigParser and read user/password details
-    from an ini file.
-    """
-
-    def __init__(self, filename=qcs.default_filename, remember_me=False, remember_me_always=False):
-
-        self._cfgfile = None
-
-        # Prioritize local directory filename.
-        # Check for file existence.
-        if os.path.exists(filename):
-            self._cfgfile = filename
-        elif os.path.exists(os.path.join(os.path.expanduser("~"), filename)):
-            # Set home path for file.
-            self._cfgfile = os.path.join(os.path.expanduser("~"), filename)
-
-        # create ConfigParser to combine defaults and input from config file.
-        self._cfgparse = ConfigParser(qcs.defaults)
-
-        if self._cfgfile:
-            self._cfgfile = os.path.realpath(self._cfgfile)
-
-            mode = stat.S_IMODE(os.stat(self._cfgfile)[stat.ST_MODE])
-
-            # apply bitmask to current mode to check ONLY user access permissions.
-            if (mode & (stat.S_IRWXG | stat.S_IRWXO)) != 0:
-                logging.warning('%s permissions allows more than user access.' % (filename,))
-
-            self._cfgparse.read(self._cfgfile)
-
-        # if 'info' doesn't exist, create the section.
-        if not self._cfgparse.has_section('qualys'):
-            self._cfgparse.add_section('qualys')
-
-        # Use default hostname (if one isn't provided).
-        if not self._cfgparse.has_option('qualys', 'hostname'):
-            if self._cfgparse.has_option('DEFAULT', 'hostname'):
-                hostname = self._cfgparse.get('DEFAULT', 'hostname')
-                self._cfgparse.set('qualys', 'hostname', hostname)
-            else:
-                raise Exception("No 'hostname' set. QualysConnect does not know who to connect to.")
-
-        # Use default max_retries (if one isn't provided).
-        if not self._cfgparse.has_option('qualys', 'max_retries'):
-            self.max_retries = qcs.defaults['max_retries']
-        else:
-            self.max_retries = self._cfgparse.get('qualys', 'max_retries')
-            try:
-                self.max_retries = int(self.max_retries)
-            except Exception:
-                logger.error('Value max_retries must be an integer.')
-                print('Value max_retries must be an integer.')
-                exit(1)
-            self._cfgparse.set('qualys', 'max_retries', str(self.max_retries))
-        self.max_retries = int(self.max_retries)
-
-        #Get template ID... user will need to set this to pull back CSV reports
-        if not self._cfgparse.has_option('qualys', 'template_id'):
-            self.report_template_id = qcs.defaults['template_id']
-        else:
-            self.report_template_id = self._cfgparse.get('qualys', 'template_id')
-            try:
-                self.report_template_id = int(self.report_template_id)
-            except Exception:
-                logger.error('Report Template ID Must be set and be an integer')
-                print('Value template ID must be an integer.')
-                exit(1)
-            self._cfgparse.set('qualys', 'template_id', str(self.report_template_id))
-        self.report_template_id = int(self.report_template_id)
-
-        # Proxy support
-        proxy_config = proxy_url = proxy_protocol = proxy_port = proxy_username = proxy_password = None
-        # User requires proxy?
-        if self._cfgparse.has_option('proxy', 'proxy_url'):
-            proxy_url = self._cfgparse.get('proxy', 'proxy_url')
-            # Remove protocol prefix from url if included.
-            for prefix in ('http://', 'https://'):
-                if proxy_url.startswith(prefix):
-                    proxy_protocol = prefix
-                    proxy_url = proxy_url[len(prefix):]
-            # Default proxy protocol is http.
-            if not proxy_protocol:
-                proxy_protocol = 'https://'
-            # Check for proxy port request.
-            if ':' in proxy_url:
-                # Proxy port already specified in url.
-                # Set proxy port.
-                proxy_port = proxy_url[proxy_url.index(':') + 1:]
-                # Remove proxy port from proxy url.
-                proxy_url = proxy_url[:proxy_url.index(':')]
-            if self._cfgparse.has_option('proxy', 'proxy_port'):
-                # Proxy requires specific port.
-                if proxy_port:
-                    # Warn that a proxy port was already specified in the url.
-                    proxy_port_url = proxy_port
-                    proxy_port = self._cfgparse.get('proxy', 'proxy_port')
-                    logger.warning('Proxy port from url overwritten by specified proxy_port from config:')
-                    logger.warning('%s --> %s' % (proxy_port_url, proxy_port))
-                else:
-                    proxy_port = self._cfgparse.get('proxy', 'proxy_port')
-            if not proxy_port:
-                # No proxy port specified.
-                if proxy_protocol == 'http://':
-                    # Use default HTTP Proxy port.
-                    proxy_port = '8080'
-                else:
-                    # Use default HTTPS Proxy port.
-                    proxy_port = '443'
-
-            # Check for proxy authentication request.
-            if self._cfgparse.has_option('proxy', 'proxy_username'):
-                # Proxy requires username & password.
-                proxy_username = self._cfgparse.get('proxy', 'proxy_username')
-                proxy_password = self._cfgparse.get('proxy', 'proxy_password')
-                # Not sure if this use case below is valid.
-                # # Support proxy with username and empty password.
-                # try:
-                #     proxy_password = self._cfgparse.get('proxy','proxy_password')
-                # except NoOptionError, e:
-                #     # Set empty password.
-                #     proxy_password = ''
-        # Sample proxy config:f
-        # 'http://user:pass@10.10.1.10:3128'
-        if proxy_url:
-            # Proxy requested.
-            proxy_config = proxy_url
-            if proxy_port:
-                # Proxy port requested.
-                proxy_config += ':' + proxy_port
-            if proxy_username:
-                # Proxy authentication requested.
-                proxy_config = proxy_username + ':' + proxy_password + '@' + proxy_config
-            # Prefix by proxy protocol.
-            proxy_config = proxy_protocol + proxy_config
-        # Set up proxy if applicable.
-        if proxy_config:
-            self.proxies = {'https': proxy_config}
-        else:
-            self.proxies = None
-
-        # ask username (if one doesn't exist)
-        if not self._cfgparse.has_option('qualys', 'username'):
-            username = input('QualysGuard Username: ')
-            self._cfgparse.set('qualys', 'username', username)
-
-        # ask password (if one doesn't exist)
-        if not self._cfgparse.has_option('qualys', 'password'):
-            password = getpass.getpass('QualysGuard Password: ')
-            self._cfgparse.set('qualys', 'password', password)
-
-
-
-        logging.debug(self._cfgparse.items('qualys'))
-
-        if remember_me or remember_me_always:
-            # Let's create that config file for next time...
-            # Where to store this?
-            if remember_me:
-                # Store in current working directory.
-                config_path = filename
-            if remember_me_always:
-                # Store in home directory.
-                config_path = os.path.expanduser("~")
-            if not os.path.exists(config_path):
-                # Write file only if it doesn't already exists.
-                # http://stackoverflow.com/questions/5624359/write-file-with-specific-permissions-in-python
-                mode = stat.S_IRUSR | stat.S_IWUSR  # This is 0o600 in octal and 384 in decimal.
-                umask_original = os.umask(0)
-                try:
-                    config_file = os.fdopen(os.open(config_path, os.O_WRONLY | os.O_CREAT, mode), 'w')
-                finally:
-                    os.umask(umask_original)
-                # Add the settings to the structure of the file, and lets write it out...
-                self._cfgparse.write(config_file)
-                config_file.close()
-
-    def get_config_filename(self):
-        return self._cfgfile
-
-    def get_config(self):
-        return self._cfgparse
-
-    def get_auth(self):
-        ''' Returns username from the configfile. '''
-        return (self._cfgparse.get('qualys', 'username'), self._cfgparse.get('qualys', 'password'))
-
-    def get_hostname(self):
-        ''' Returns hostname. '''
-        return self._cfgparse.get('qualys', 'hostname')
-
-    def get_template_id(self):
-        return self._cfgparse.get('qualys','template_id')

deps/qualysapi/qualysapi/connector.py

@@ -1,363 +0,0 @@
-from __future__ import absolute_import
-from __future__ import print_function
-__author__ = 'Parag Baxi <parag.baxi@gmail.com>'
-__copyright__ = 'Copyright 2013, Parag Baxi'
-__license__ = 'Apache License 2.0'
-
-""" Module that contains classes for setting up connections to QualysGuard API
-and requesting data from it.
-"""
-import logging
-import time
-
-try:
-    from urllib.parse import urlparse
-except ImportError:
-    from urlparse import urlparse
-
-from collections import defaultdict
-
-import requests
-
-import qualysapi.version
-import qualysapi.api_methods
-
-import qualysapi.api_actions
-import qualysapi.api_actions as api_actions
-
-# Setup module level logging.
-logger = logging.getLogger(__name__)
-
-try:
-    from lxml import etree
-except ImportError as e:
-    logger.warning(
-        'Warning: Cannot consume lxml.builder E objects without lxml. Send XML strings for AM & WAS API calls.')
-
-
-class QGConnector(api_actions.QGActions):
-    """ Qualys Connection class which allows requests to the QualysGuard API using HTTP-Basic Authentication (over SSL).
-
-    """
-
-    def __init__(self, auth, server='qualysapi.qualys.com', proxies=None, max_retries=3):
-        # Read username & password from file, if possible.
-        self.auth = auth
-        # Remember QualysGuard API server.
-        self.server = server
-        # Remember rate limits per call.
-        self.rate_limit_remaining = defaultdict(int)
-        # api_methods: Define method algorithm in a dict of set.
-        # Naming convention: api_methods[api_version optional_blah] due to api_methods_with_trailing_slash testing.
-        self.api_methods = qualysapi.api_methods.api_methods
-        #
-        # Keep track of methods with ending slashes to autocorrect user when they forgot slash.
-        self.api_methods_with_trailing_slash = qualysapi.api_methods.api_methods_with_trailing_slash
-        self.proxies = proxies
-        logger.debug('proxies = \n%s' % proxies)
-        # Set up requests max_retries.
-        logger.debug('max_retries = \n%s' % max_retries)
-        self.session = requests.Session()
-        http_max_retries = requests.adapters.HTTPAdapter(max_retries=max_retries)
-        https_max_retries = requests.adapters.HTTPAdapter(max_retries=max_retries)
-        self.session.mount('http://', http_max_retries)
-        self.session.mount('https://', https_max_retries)
-
-    def __call__(self):
-        return self
-
-    def format_api_version(self, api_version):
-        """ Return QualysGuard API version for api_version specified.
-
-        """
-        # Convert to int.
-        if type(api_version) == str:
-            api_version = api_version.lower()
-            if api_version[0] == 'v' and api_version[1].isdigit():
-                # Remove first 'v' in case the user typed 'v1' or 'v2', etc.
-                api_version = api_version[1:]
-            # Check for input matching Qualys modules.
-            if api_version in ('asset management', 'assets', 'tag', 'tagging', 'tags'):
-                # Convert to Asset Management API.
-                api_version = 'am'
-            elif api_version in ('am2'):
-                # Convert to Asset Management API v2
-                api_version = 'am2'
-            elif api_version in ('webapp', 'web application scanning', 'webapp scanning'):
-                # Convert to WAS API.
-                api_version = 'was'
-            elif api_version in ('pol', 'pc'):
-                # Convert PC module to API number 2.
-                api_version = 2
-            else:
-                api_version = int(api_version)
-        return api_version
-
-    def which_api_version(self, api_call):
-        """ Return QualysGuard API version for api_call specified.
-
-        """
-        # Leverage patterns of calls to API methods.
-        if api_call.endswith('.php'):
-            # API v1.
-            return 1
-        elif api_call.startswith('api/2.0/'):
-            # API v2.
-            return 2
-        elif '/am/' in api_call:
-            # Asset Management API.
-            return 'am'
-        elif '/was/' in api_call:
-            # WAS API.
-            return 'was'
-        return False
-
-    def url_api_version(self, api_version):
-        """ Return base API url string for the QualysGuard api_version and server.
-
-        """
-        # Set base url depending on API version.
-        if api_version == 1:
-            # QualysGuard API v1 url.
-            url = "https://%s/msp/" % (self.server,)
-        elif api_version == 2:
-            # QualysGuard API v2 url.
-            url = "https://%s/" % (self.server,)
-        elif api_version == 'was':
-            # QualysGuard REST v3 API url (Portal API).
-            url = "https://%s/qps/rest/3.0/" % (self.server,)
-        elif api_version == 'am':
-            # QualysGuard REST v1 API url (Portal API).
-            url = "https://%s/qps/rest/1.0/" % (self.server,)
-        elif api_version == 'am2':
-            # QualysGuard REST v1 API url (Portal API).
-            url = "https://%s/qps/rest/2.0/" % (self.server,)
-        else:
-            raise Exception("Unknown QualysGuard API Version Number (%s)" % (api_version,))
-        logger.debug("Base url =\n%s" % (url))
-        return url
-
-    def format_http_method(self, api_version, api_call, data):
-        """ Return QualysGuard API http method, with POST preferred..
-
-        """
-        # Define get methods for automatic http request methodology.
-        #
-        # All API v2 requests are POST methods.
-        if api_version == 2:
-            return 'post'
-        elif api_version == 1:
-            if api_call in self.api_methods['1 post']:
-                return 'post'
-            else:
-                return 'get'
-        elif api_version == 'was':
-            # WAS API call.
-            # Because WAS API enables user to GET API resources in URI, let's chop off the resource.
-            # '/download/was/report/18823' --> '/download/was/report/'
-            api_call_endpoint = api_call[:api_call.rfind('/') + 1]
-            if api_call_endpoint in self.api_methods['was get']:
-                return 'get'
-            # Post calls with no payload will result in HTTPError: 415 Client Error: Unsupported Media Type.
-            if data is None:
-                # No post data. Some calls change to GET with no post data.
-                if api_call_endpoint in self.api_methods['was no data get']:
-                    return 'get'
-                else:
-                    return 'post'
-            else:
-                # Call with post data.
-                return 'post'
-        else:
-            # Asset Management API call.
-            if api_call in self.api_methods['am get']:
-                return 'get'
-            else:
-                return 'post'
-
-    def preformat_call(self, api_call):
-        """ Return properly formatted QualysGuard API call.
-
-        """
-        # Remove possible starting slashes or trailing question marks in call.
-        api_call_formatted = api_call.lstrip('/')
-        api_call_formatted = api_call_formatted.rstrip('?')
-        if api_call != api_call_formatted:
-            # Show difference
-            logger.debug('api_call post strip =\n%s' % api_call_formatted)
-        return api_call_formatted
-
-    def format_call(self, api_version, api_call):
-        """ Return properly formatted QualysGuard API call according to api_version etiquette.
-
-        """
-        # Remove possible starting slashes or trailing question marks in call.
-        api_call = api_call.lstrip('/')
-        api_call = api_call.rstrip('?')
-        logger.debug('api_call post strip =\n%s' % api_call)
-        # Make sure call always ends in slash for API v2 calls.
-        if (api_version == 2 and api_call[-1] != '/'):
-            # Add slash.
-            logger.debug('Adding "/" to api_call.')
-            api_call += '/'
-        if api_call in self.api_methods_with_trailing_slash[api_version]:
-            # Add slash.
-            logger.debug('Adding "/" to api_call.')
-            api_call += '/'
-        return api_call
-
-    def format_payload(self, api_version, data):
-        """ Return appropriate QualysGuard API call.
-
-        """
-        # Check if payload is for API v1 or API v2.
-        if (api_version in (1, 2)):
-            # Check if string type.
-            if type(data) == str:
-                # Convert to dictionary.
-                logger.debug('Converting string to dict:\n%s' % data)
-                # Remove possible starting question mark & ending ampersands.
-                data = data.lstrip('?')
-                data = data.rstrip('&')
-                # Convert to dictionary.
-                #data = urllib.parse.parse_qs(data)
-                data = urlparse(data)
-                logger.debug('Converted:\n%s' % str(data))
-        elif api_version in ('am', 'was', 'am2'):
-            if type(data) == etree._Element:
-                logger.debug('Converting lxml.builder.E to string')
-                data = etree.tostring(data)
-                logger.debug('Converted:\n%s' % data)
-        return data
-
-    def request(self, api_call, data=None, api_version=None, http_method=None, concurrent_scans_retries=0,
-                concurrent_scans_retry_delay=0):
-        """ Return QualysGuard API response.
-
-        """
-        logger.debug('api_call =\n%s' % api_call)
-        logger.debug('api_version =\n%s' % api_version)
-        logger.debug('data %s =\n %s' % (type(data), str(data)))
-        logger.debug('http_method =\n%s' % http_method)
-        logger.debug('concurrent_scans_retries =\n%s' % str(concurrent_scans_retries))
-        logger.debug('concurrent_scans_retry_delay =\n%s' % str(concurrent_scans_retry_delay))
-        concurrent_scans_retries = int(concurrent_scans_retries)
-        concurrent_scans_retry_delay = int(concurrent_scans_retry_delay)
-        #
-        # Determine API version.
-        # Preformat call.
-        api_call = self.preformat_call(api_call)
-        if api_version:
-            # API version specified, format API version inputted.
-            api_version = self.format_api_version(api_version)
-        else:
-            # API version not specified, determine automatically.
-            api_version = self.which_api_version(api_call)
-        #
-        # Set up base url.
-        url = self.url_api_version(api_version)
-        #
-        # Set up headers.
-        headers = {"X-Requested-With": "QualysAPI (python) v%s - VulnWhisperer" % (qualysapi.version.__version__,)}
-        logger.debug('headers =\n%s' % (str(headers)))
-        # Portal API takes in XML text, requiring custom header.
-        if api_version in ('am', 'was', 'am2'):
-            headers['Content-type'] = 'text/xml'
-        #
-        # Set up http request method, if not specified.
-        if not http_method:
-            http_method = self.format_http_method(api_version, api_call, data)
-        logger.debug('http_method =\n%s' % http_method)
-        #
-        # Format API call.
-        api_call = self.format_call(api_version, api_call)
-        logger.debug('api_call =\n%s' % (api_call))
-        # Append api_call to url.
-        url += api_call
-        #
-        # Format data, if applicable.
-        if data is not None:
-            data = self.format_payload(api_version, data)
-        # Make request at least once (more if concurrent_retry is enabled).
-        retries = 0
-        #
-        # set a warning threshold for the rate limit
-        rate_warn_threshold = 10
-        while retries <= concurrent_scans_retries:
-            # Make request.
-            logger.debug('url =\n%s' % (str(url)))
-            logger.debug('data =\n%s' % (str(data)))
-            logger.debug('headers =\n%s' % (str(headers)))
-            if http_method == 'get':
-                # GET
-                logger.debug('GET request.')
-                request = self.session.get(url, params=data, auth=self.auth, headers=headers, proxies=self.proxies)
-            else:
-                # POST
-                logger.debug('POST request.')
-                # Make POST request.
-                request = self.session.post(url, data=data, auth=self.auth, headers=headers, proxies=self.proxies)
-            logger.debug('response headers =\n%s' % (str(request.headers)))
-            #
-            # Remember how many times left user can make against api_call.
-            try:
-                self.rate_limit_remaining[api_call] = int(request.headers['x-ratelimit-remaining'])
-                logger.debug('rate limit for api_call, %s = %s' % (api_call, self.rate_limit_remaining[api_call]))
-                if (self.rate_limit_remaining[api_call] > rate_warn_threshold):
-                    logger.debug('rate limit for api_call, %s = %s' % (api_call, self.rate_limit_remaining[api_call]))
-                elif (self.rate_limit_remaining[api_call] <= rate_warn_threshold) and (self.rate_limit_remaining[api_call] > 0):
-                    logger.warning('Rate limit is about to being reached (remaining api calls = %s)' % self.rate_limit_remaining[api_call])
-                elif self.rate_limit_remaining[api_call] <= 0:
-                    logger.critical('ATTENTION! RATE LIMIT HAS BEEN REACHED (remaining api calls = %s)!' % self.rate_limit_remaining[api_call])
-            except KeyError as e:
-                # Likely a bad api_call.
-                logger.debug(e)
-                pass
-            except TypeError as e:
-                # Likely an asset search api_call.
-                logger.debug(e)
-                pass
-            # Response received.
-            response = str(request.content)
-            logger.debug('response text =\n%s' % (response))
-            # Keep track of how many retries.
-            retries += 1
-            # Check for concurrent scans limit.
-            if not ('<responseCode>INVALID_REQUEST</responseCode>' in response and
-                    '<errorMessage>You have reached the maximum number of concurrent running scans' in response and
-                    '<errorResolution>Please wait until your previous scans have completed</errorResolution>' in response):
-                # Did not hit concurrent scan limit.
-                break
-            else:
-                # Hit concurrent scan limit.
-                logger.critical(response)
-                # If trying again, delay next try by concurrent_scans_retry_delay.
-                if retries <= concurrent_scans_retries:
-                    logger.warning('Waiting %d seconds until next try.' % concurrent_scans_retry_delay)
-                    time.sleep(concurrent_scans_retry_delay)
-                    # Inform user of how many retries.
-                    logger.critical('Retry #%d' % retries)
-                else:
-                    # Ran out of retries. Let user know.
-                    print('Alert! Ran out of concurrent_scans_retries!')
-                    logger.critical('Alert! Ran out of concurrent_scans_retries!')
-                    return False
-        # Check to see if there was an error.
-        try:
-            request.raise_for_status()
-        except requests.HTTPError as e:
-            # Error
-            print('Error! Received a 4XX client error or 5XX server error response.')
-            print('Content = \n', response)
-            logger.error('Content = \n%s' % response)
-            print('Headers = \n', request.headers)
-            logger.error('Headers = \n%s' % str(request.headers))
-            request.raise_for_status()
-        if '<RETURN status="FAILED" number="2007">' in response:
-            print('Error! Your IP address is not in the list of secure IPs. Manager must include this IP (QualysGuard VM > Users > Security).')
-            print('Content = \n', response)
-            logger.error('Content = \n%s' % response)
-            print('Headers = \n', request.headers)
-            logger.error('Headers = \n%s' % str(request.headers))
-            return False
-        return response

deps/qualysapi/qualysapi/contrib.py

@@ -1,290 +0,0 @@
-# File for 3rd party contributions.
-
-from __future__ import absolute_import
-from __future__ import print_function
-import six
-from six.moves import range
-
-__author__ = 'Parag Baxi <parag.baxi@gmail.com>'
-__license__ = 'Apache License 2.0'
-
-import logging
-import time
-import types
-import unicodedata
-from collections import defaultdict
-
-from lxml import etree, objectify
-
-
-# Set module level logger.
-logger = logging.getLogger(__name__)
-
-
-def generate_vm_report(self, report_details, startup_delay=60, polling_delay=30, max_checks=10):
-    ''' Spool and download QualysGuard VM report.
-
-    startup_delay: Time in seconds to wait before initially checking.
-    polling_delay: Time in seconds to wait between checks.
-    max_checks: Maximum number of times to check for report spooling completion.
-
-    '''
-    # Merge parameters.
-    report_details['action'] = 'launch'
-    logger.debug(report_details)
-    xml_output = qualysapi_instance.request(2, 'report', report_details)
-    report_id = etree.XML(xml_output).find('.//VALUE').text
-    logger.debug('report_id: %s' % (report_id))
-    # Wait for report to finish spooling.
-    # Maximum number of times to check for report.  About 10 minutes.
-    MAX_CHECKS = 10
-    logger.info('Report sent to spooler. Checking for report in %s seconds.' % (startup_delay))
-    time.sleep(startup_delay)
-    for n in range(0, max_checks):
-        # Check to see if report is done.
-        xml_output = qualysapi_instance.request(2, 'report', {'action': 'list', 'id': report_id})
-        tag_status = etree.XML(xml_output).findtext(".//STATE")
-        logger.debug('tag_status: %s' % (tag_status))
-        tag_status = etree.XML(xml_output).findtext(".//STATE")
-        logger.debug('tag_status: %s' % (tag_status))
-        if tag_status is not None:
-            # Report is showing up in the Report Center.
-            if tag_status == 'Finished':
-                # Report creation complete.
-                break
-        # Report not finished, wait.
-        logger.info('Report still spooling. Trying again in %s seconds.' % (polling_delay))
-        time.sleep(polling_delay)
-    # We now have to fetch the report.  Use the report id.
-    report_xml = qualysapi_instance.request(2, 'report', {'action': 'fetch', 'id': report_id})
-    return report_xml
-
-
-def qg_html_to_ascii(qg_html_text):
-    """Convert and return QualysGuard's quasi HTML text to ASCII text."""
-    text = qg_html_text
-    # Handle tagged line breaks (<p>, <br>)
-    text = re.sub(r'(?i)<br>[ ]*', '\n', text)
-    text = re.sub(r'(?i)<p>[ ]*', '\n', text)
-    # Remove consecutive line breaks
-    text = re.sub(r"^\s+", "", text, flags=re.MULTILINE)
-    # Remove empty lines at the end.
-    text = re.sub('[\n]+$', '$', text)
-    # Store anchor tags href attribute
-    links = list(lxml.html.iterlinks(text))
-    # Remove anchor tags
-    html_element = lxml.html.fromstring(text)
-    # Convert anchor tags to "link_text (link: link_url )".
-    logging.debug('Converting anchor tags...')
-    text = html_element.text_content().encode('ascii', 'ignore')
-    # Convert each link.
-    for l in links:
-        # Find and replace each link.
-        link_text = l[0].text_content().encode('ascii', 'ignore').strip()
-        link_url = l[2].strip()
-        # Replacing link_text
-        if link_text != link_url:
-            # Link text is different, most likely a description.
-            text = string.replace(text, link_text, '%s (link: %s )' % (link_text, link_url))
-        else:
-            # Link text is the same as the href.  No need to duplicate link.
-            text = string.replace(text, link_text, '%s' % (link_url))
-    logging.debug('Done.')
-    return text
-
-
-def qg_parse_informational_qids(xml_report):
-    """Return vulnerabilities of severity 1 and 2 levels due to a restriction of
-       QualysGuard's inability to report them in the internal ticketing system.
-    """
-    # asset_group's vulnerability data map:
-    #    {'qid_number': {
-    #                    # CSV info
-    #                    'hosts': [{'ip': '10.28.0.1', 'dns': 'hostname', 'netbios': 'blah', 'vuln_id': 'remediation_ticket_number'}, {'ip': '10.28.0.3', 'dns': 'hostname2', 'netbios': '', 'vuln_id': 'remediation_ticket_number'}, ...],
-    #                    'solution': '',
-    #                    'impact': '',
-    #                    'threat': '',
-    #                    'severity': '',
-    #                   }
-    #     'qid_number2': ...
-    #     }
-    # Add all vulnerabilities to list of dictionaries.
-    # Use defaultdict in case a new QID is encountered.
-    info_vulns = defaultdict(dict)
-    # Parse vulnerabilities in xml string.
-    tree = objectify.fromstring(xml_report)
-    # Write IP, DNS, & Result into each QID CSV file.
-    logging.debug('Parsing report...')
-    # TODO:  Check against c_args.max to prevent creating CSV content for QIDs that we won't use.
-    for host in tree.HOST_LIST.HOST:
-        # Extract possible extra hostname information.
-        try:
-            netbios = unicodedata.normalize('NFKD', six.text_type(host.NETBIOS)).encode('ascii', 'ignore').strip()
-        except AttributeError:
-            netbios = ''
-        try:
-            dns = unicodedata.normalize('NFKD', six.text_type(host.DNS)).encode('ascii', 'ignore').strip()
-        except AttributeError:
-            dns = ''
-        ip = unicodedata.normalize('NFKD', six.text_type(host.IP)).encode('ascii', 'ignore').strip()
-        # Extract vulnerabilities host is affected by.
-        for vuln in host.VULN_INFO_LIST.VULN_INFO:
-            try:
-                result = unicodedata.normalize('NFKD', six.text_type(vuln.RESULT)).encode('ascii', 'ignore').strip()
-            except AttributeError:
-                result = ''
-            qid = unicodedata.normalize('NFKD', six.text_type(vuln.QID)).encode('ascii', 'ignore').strip()
-            # Attempt to add host to QID's list of affected hosts.
-            try:
-                info_vulns[qid]['hosts'].append({'ip': '%s' % (ip),
-                                                 'dns': '%s' % (dns),
-                                                 'netbios': '%s' % (netbios),
-                                                 'vuln_id': '',
-                                                 # Informational QIDs do not have vuln_id numbers.  This is a flag to write the CSV file.
-                                                 'result': '%s' % (result), })
-            except KeyError:
-                # New QID.
-                logging.debug('New QID found: %s' % (qid))
-                info_vulns[qid]['hosts'] = []
-                info_vulns[qid]['hosts'].append({'ip': '%s' % (ip),
-                                                 'dns': '%s' % (dns),
-                                                 'netbios': '%s' % (netbios),
-                                                 'vuln_id': '',
-                                                 # Informational QIDs do not have vuln_id numbers.  This is a flag to write the CSV file.
-                                                 'result': '%s' % (result), })
-    # All vulnerabilities added.
-    # Add all vulnerabilty information.
-    for vuln_details in tree.GLOSSARY.VULN_DETAILS_LIST.VULN_DETAILS:
-        qid = unicodedata.normalize('NFKD', six.text_type(vuln_details.QID)).encode('ascii', 'ignore').strip()
-        info_vulns[qid]['title'] = unicodedata.normalize('NFKD', six.text_type(vuln_details.TITLE)).encode('ascii',
-                                                                                                           'ignore').strip()
-        info_vulns[qid]['severity'] = unicodedata.normalize('NFKD', six.text_type(vuln_details.SEVERITY)).encode('ascii',
-                                                                                                                 'ignore').strip()
-        info_vulns[qid]['solution'] = qg_html_to_ascii(
-            unicodedata.normalize('NFKD', six.text_type(vuln_details.SOLUTION)).encode('ascii', 'ignore').strip())
-        info_vulns[qid]['threat'] = qg_html_to_ascii(
-            unicodedata.normalize('NFKD', six.text_type(vuln_details.THREAT)).encode('ascii', 'ignore').strip())
-        info_vulns[qid]['impact'] = qg_html_to_ascii(
-            unicodedata.normalize('NFKD', six.text_type(vuln_details.IMPACT)).encode('ascii', 'ignore').strip())
-    # Ready to report informational vulnerabilities.
-    return info_vulns
-
-
-# TODO: Implement required function qg_remediation_tickets(asset_group, status, qids)
-# TODO: Remove static 'report_template' value.  Parameterize and document required report template.
-def qg_ticket_list(asset_group, severity, qids=None):
-    """Return dictionary of each vulnerability reported against asset_group of severity."""
-    global asset_group_details
-    # All vulnerabilities imported to list of dictionaries.
-    vulns = qg_remediation_tickets(asset_group, 'OPEN', qids)  # vulns now holds all open remediation tickets.
-    if not vulns:
-        # No tickets to report.
-        return False
-    #
-    # Sort the vulnerabilities in order of prevalence -- number of hosts affected.
-    vulns = OrderedDict(sorted(list(vulns.items()), key=lambda t: len(t[1]['hosts'])))
-    logging.debug('vulns sorted = %s' % (vulns))
-    #
-    # Remove QIDs that have duplicate patches.
-    #
-    # Read in patch report.
-    # TODO:  Allow for lookup of report_template.
-    # Report template is Patch report "Sev 5 confirmed patchable".
-    logging.debug('Retrieving patch report from QualysGuard.')
-    print('Retrieving patch report from QualysGuard.')
-    report_template = '1063695'
-    # Call QualysGuard for patch report.
-    csv_output = qg_command(2, 'report', {'action': 'launch', 'output_format': 'csv',
-                                          'asset_group_ids': asset_group_details['qg_asset_group_id'],
-                                          'template_id': report_template,
-                                          'report_title': 'QGIR Patch %s' % (asset_group)})
-    logging.debug('csv_output =')
-    logging.debug(csv_output)
-    logging.debug('Improving remediation efficiency by removing unneeded, redundant patches.')
-    print('Improving remediation efficiency by removing unneeded, redundant patches.')
-    # Find the line for Patches by Host data.
-    logging.debug('Header found at %s.' % (csv_output.find('Patch QID, IP, DNS, NetBIOS, OS, Vulnerability Count')))
-
-    starting_pos = csv_output.find('Patch QID, IP, DNS, NetBIOS, OS, Vulnerability Count') + 52
-    logging.debug('starting_pos = %s' % str(starting_pos))
-    # Data resides between line ending in 'Vulnerability Count' and a blank line.
-    patches_by_host = csv_output[starting_pos:csv_output[starting_pos:].find(
-        'Host Vulnerabilities Fixed by Patch') + starting_pos - 3]
-    logging.debug('patches_by_host =')
-    logging.debug(patches_by_host)
-    # Read in string patches_by_host csv to a dictionary.
-    f = patches_by_host.split(os.linesep)
-    reader = csv.DictReader(f, ['Patch QID', 'IP', 'DNS', 'NetBIOS', 'OS', 'Vulnerability Count'], delimiter=',')
-    # Mark Patch QIDs that fix multiple vulnerabilities with associated IP addresses.
-    redundant_qids = defaultdict(list)
-    for row in reader:
-        if int(row['Vulnerability Count']) > 1:
-            # Add to list of redundant QIDs.
-            redundant_qids[row['Patch QID']].append(row['IP'])
-            logging.debug('%s, %s, %s, %s' % (
-                row['Patch QID'],
-                row['IP'],
-                int(row['Vulnerability Count']),
-                redundant_qids[row['Patch QID']]))
-    # Log for debugging.
-    logging.debug('len(redundant_qids) = %s, redundant_qids =' % (len(redundant_qids)))
-    for patch_qid in list(redundant_qids.keys()):
-        logging.debug('%s, %s' % (str(patch_qid), str(redundant_qids[patch_qid])))
-    # Extract redundant QIDs with associated IP addresses.
-    # Find the line for Patches by Host data.
-    starting_pos = csv_output.find('Patch QID, IP, QID, Severity, Type, Title, Instance, Last Detected') + 66
-    # Data resides between line ending in 'Vulnerability Count' and end of string.
-    host_vulnerabilities_fixed_by_patch = csv_output[starting_pos:]
-    # Read in string host_vulnerabilities_fixed_by_patch csv to a dictionary.
-    f = host_vulnerabilities_fixed_by_patch.split(os.linesep)
-    reader = csv.DictReader(f, ['Patch QID', 'IP', 'QID', 'Severity', 'Type', 'Title', 'Instance', 'Last Detected'],
-                            delimiter=',')
-    # Remove IP addresses associated with redundant QIDs.
-    qids_to_remove = defaultdict(list)
-    for row in reader:
-        # If the row's IP address's Patch QID was found to have multiple vulnerabilities...
-        if len(redundant_qids[row['Patch QID']]) > 0 and redundant_qids[row['Patch QID']].count(row['IP']) > 0:
-            # Add the QID column to the list of dictionaries {QID: [IP address, IP address, ...], QID2: [IP address], ...}
-            qids_to_remove[row['QID']].append(row['IP'])
-    # Log for debugging.
-    logging.debug('len(qids_to_remove) = %s, qids_to_remove =' % (len(qids_to_remove)))
-    for a_qid in list(qids_to_remove.keys()):
-        logging.debug('%s, %s' % (str(a_qid), str(qids_to_remove[a_qid])))
-    #
-    # Diff vulns against qids_to_remove and against open incidents.
-    #
-    vulns_length = len(vulns)
-    # Iterate over list of keys rather than original dictionary as some keys may be deleted changing the size of the dictionary.
-    for a_qid in list(vulns.keys()):
-        # Debug log original qid's hosts.
-        logging.debug('Before diffing vulns[%s] =' % (a_qid))
-        logging.debug(vulns[a_qid]['hosts'])
-        # Pop each host.
-        # The [:] returns a "slice" of x, which happens to contain all its elements, and is thus effectively a copy of x.
-        for host in vulns[a_qid]['hosts'][:]:
-            # If the QID for the host is a dupe or if a there is an open Reaction incident.
-            if qids_to_remove[a_qid].count(host['ip']) > 0 or reaction_open_issue(host['vuln_id']):
-                # Remove the host from the QID's list of target hosts.
-                logging.debug('Removing remediation ticket %s.' % (host['vuln_id']))
-                vulns[a_qid]['hosts'].remove(host)
-            else:
-                # Do not remove this vuln
-                logging.debug('Will report remediation %s.' % (host['vuln_id']))
-        # Debug log diff'd qid's hosts.
-        logging.debug('After diffing vulns[%s]=' % (a_qid))
-        logging.debug(vulns[a_qid]['hosts'])
-        # If there are no more hosts left to patch for the qid.
-        if len(vulns[a_qid]['hosts']) == 0:
-            # Remove the QID.
-            logging.debug('Deleting vulns[%s].' % (a_qid))
-            del vulns[a_qid]
-    # Diff completed
-    if not vulns_length == len(vulns):
-        print('A count of %s vulnerabilities have been consolidated to %s vulnerabilities, a reduction of %s%%.' % (
-            int(vulns_length),
-            int(len(vulns)),
-            int(round((int(vulns_length) - int(len(vulns))) / float(vulns_length) * 100))))
-    # Return vulns to report.
-    logging.debug('vulns =')
-    logging.debug(vulns)
-    return vulns

deps/qualysapi/qualysapi/settings.py

@@ -1,21 +0,0 @@
-''' Module to hold global settings reused throughout qualysapi. '''
-
-from __future__ import absolute_import
-__author__ = "Colin Bell <colin.bell@uwaterloo.ca>"
-__copyright__ = "Copyright 2011-2013, University of Waterloo"
-__license__ = "BSD-new"
-
-import os
-
-global defaults
-global default_filename
-
-
-if os.name == 'nt':
-    default_filename = "config.ini"
-else:
-    default_filename = ".qcrc"
-
-defaults = {'hostname': 'qualysapi.qualys.com',
-            'max_retries': '3',
-            'template_id': '00000'}

deps/qualysapi/qualysapi/util.py

@@ -1,29 +0,0 @@
-""" A set of utility functions for QualysConnect module. """
-from __future__ import absolute_import
-import logging
-
-import qualysapi.config as qcconf
-import qualysapi.connector as qcconn
-import qualysapi.settings as qcs
-
-__author__ = "Parag Baxi <parag.baxi@gmail.com> & Colin Bell <colin.bell@uwaterloo.ca>"
-__copyright__ = "Copyright 2011-2013, Parag Baxi & University of Waterloo"
-__license__ = 'Apache License 2.0'
-
-# Set module level logger.
-logger = logging.getLogger(__name__)
-
-
-def connect(config_file=qcs.default_filename, remember_me=False, remember_me_always=False):
-    """ Return a QGAPIConnect object for v1 API pulling settings from config
-    file.
-    """
-    # Retrieve login credentials.
-    conf = qcconf.QualysConnectConfig(filename=config_file, remember_me=remember_me, 
-                                      remember_me_always=remember_me_always)
-    connect = qcconn.QGConnector(conf.get_auth(),
-                                 conf.get_hostname(),
-                                 conf.proxies,
-                                 conf.max_retries)
-    logger.info("Finished building connector.")
-    return connect

deps/qualysapi/qualysapi/version.py

@@ -1,3 +0,0 @@
-__author__ = 'Austin Taylor'
-__pkgname__ = 'qualysapi'
-__version__ = '4.1.0'

deps/qualysapi/setup.py

@@ -1,51 +0,0 @@
-#!/usr/bin/env python
-from __future__ import absolute_import
-import os
-import setuptools
-
-try:
-    from setuptools import setup
-except ImportError:
-    from distutils.core import setup
-
-__author__ = 'Austin Taylor <vulnWhisperer@austintaylor.io>'
-__copyright__ = 'Copyright 2017, Austin Taylor'
-__license__ = 'BSD-new'
-# Make pyflakes happy.
-__pkgname__ = None
-__version__ = None
-exec(compile(open('qualysapi/version.py').read(), 'qualysapi/version.py', 'exec'))
-
-
-# A utility function to read the README file into the long_description field.
-def read(fname):
-    """ Takes a filename and returns the contents of said file relative to
-    the current directory.
-    """
-    return open(os.path.join(os.path.dirname(__file__), fname)).read()
-
-
-setup(name=__pkgname__,
-      version=__version__,
-      author='Austin Taylor',
-      author_email='vulnWhisperer@austintaylor.io',
-      description='QualysGuard(R) Qualys API Package modified for VulnWhisperer',
-      license='BSD-new',
-      keywords='Qualys QualysGuard API helper network security',
-      url='https://github.com/austin-taylor/qualysapi',
-      package_dir={'': '.'},
-      #packages=setuptools.find_packages(),
-      packages=['qualysapi',],
-      # package_data={'qualysapi':['LICENSE']},
-      # scripts=['src/scripts/qhostinfo.py', 'src/scripts/qscanhist.py', 'src/scripts/qreports.py'],
-      long_description=read('README.md'),
-      classifiers=[
-          'Development Status :: 5 - Production/Stable',
-          'Topic :: Utilities',
-          'License :: OSI Approved :: Apache Software License',
-          'Intended Audience :: Developers',
-      ],
-      install_requires=[
-          'requests',
-      ],
-     )

docker-compose-test.yml

@@ -0,0 +1,97 @@
+version: '2'
+services:
+  elasticsearch:
+    image: docker.elastic.co/elasticsearch/elasticsearch:6.6.0
+    container_name: elasticsearch
+    environment:
+      - cluster.name=vulnwhisperer
+      - bootstrap.memory_lock=true
+      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
+      - xpack.security.enabled=false
+      - cluster.routing.allocation.disk.threshold_enabled=false
+    ulimits:
+      memlock:
+        soft: -1
+        hard: -1
+      nofile:
+        soft: 65536
+        hard: 65536
+    mem_limit: 8g
+    volumes:
+      - ./data/esdata1:/usr/share/elasticsearch/data
+      - ./data/es_snapshots:/snapshots
+    ports:
+      - 9200:9200
+    #restart: always
+    networks:
+      esnet:
+        aliases:
+          - elasticsearch.local
+
+  kibana:
+    image: docker.elastic.co/kibana/kibana:6.6.0
+    container_name: kibana
+    environment:
+      SERVER_NAME: kibana
+      ELASTICSEARCH_URL: http://elasticsearch:9200
+    ports:
+      - 5601:5601
+    depends_on:
+      - elasticsearch
+    networks:
+      esnet:
+        aliases:
+          - kibana.local
+
+  kibana-config:
+    image: alpine
+    container_name: kibana-config
+    volumes:
+      - ./resources/elk6/init_kibana.sh:/opt/init_kibana.sh
+      - ./resources/elk6/kibana_APIonly.json:/opt/kibana_APIonly.json
+      - ./resources/elk6/logstash-vulnwhisperer-template.json:/opt/index-template.json
+    command: sh -c "apk add --no-cache curl bash && chmod +x /opt/init_kibana.sh && chmod +r /opt/kibana_APIonly.json && cd /opt/ && /bin/bash /opt/init_kibana.sh" # /opt/kibana_APIonly.json"
+    networks:
+      esnet:
+        aliases:
+          - kibana-config.local
+
+  logstash:
+    image: docker.elastic.co/logstash/logstash:6.6.0
+    container_name: logstash
+    volumes:
+      - ./resources/elk6/pipeline/:/usr/share/logstash/pipeline
+      - ./data/vulnwhisperer/:/opt/VulnWhisperer/data
+      # - ./resources/elk6/logstash.yml:/usr/share/logstash/config/logstash.yml
+    environment:
+      - xpack.monitoring.enabled=false
+    depends_on:
+      - elasticsearch
+    ports:
+      - 9600:9600
+    networks:
+      esnet:
+        aliases:
+          - logstash.local
+  vulnwhisperer:
+    # image: hasecuritysolutions/vulnwhisperer:latest
+    image: vulnwhisperer-local
+    container_name: vulnwhisperer
+    entrypoint: [
+                  "vuln_whisperer",
+                  "-F",
+                  "-c",
+                  "/opt/VulnWhisperer/vulnwhisperer.ini",
+                  "--mock",
+                  "--mock_dir",
+                  "/tests/data"
+    ]    
+    volumes:
+      - ./data/vulnwhisperer/:/opt/VulnWhisperer/data
+      # - ./resources/elk6/vulnwhisperer.ini:/opt/VulnWhisperer/vulnwhisperer.ini
+      - ./configs/test.ini:/opt/VulnWhisperer/vulnwhisperer.ini
+      - ./tests/data/:/tests/data
+    network_mode: host
+
+networks:
+  esnet:

docker-compose.v6.yml

@@ -0,0 +1,86 @@
+version: '2'
+services:
+  elasticsearch:
+    image: docker.elastic.co/elasticsearch/elasticsearch:6.6.0
+    container_name: elasticsearch
+    environment:
+      - cluster.name=vulnwhisperer
+      - bootstrap.memory_lock=true
+      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
+      - xpack.security.enabled=false
+    ulimits:
+      memlock:
+        soft: -1
+        hard: -1
+      nofile:
+        soft: 65536
+        hard: 65536
+    mem_limit: 8g
+    volumes:
+      - esdata1:/usr/share/elasticsearch/data
+    ports:
+      - 9200:9200
+    #restart: always
+    networks:
+      esnet:
+        aliases:
+          - elasticsearch.local
+  kibana:
+    image: docker.elastic.co/kibana/kibana:6.6.0
+    container_name: kibana
+    environment:
+      SERVER_NAME: kibana
+      ELASTICSEARCH_URL: http://elasticsearch:9200
+    ports:
+      - 5601:5601
+    depends_on:
+      - elasticsearch
+    networks:
+      esnet:
+        aliases:
+          - kibana.local
+  kibana-config:
+    image: alpine
+    container_name: kibana-config
+    volumes:
+      - ./resources/elk6/init_kibana.sh:/opt/init_kibana.sh
+      - ./resources/elk6/kibana_APIonly.json:/opt/kibana_APIonly.json
+      - ./resources/elk6/logstash-vulnwhisperer-template.json:/opt/index-template.json
+    command: sh -c "apk add --no-cache curl bash && chmod +x /opt/init_kibana.sh && chmod +r /opt/kibana_APIonly.json && cd /opt/ && /bin/bash /opt/init_kibana.sh" # /opt/kibana_APIonly.json"
+    networks:
+      esnet:
+        aliases:
+          - kibana-config.local
+  logstash:
+    image: docker.elastic.co/logstash/logstash:6.6.0
+    container_name: logstash
+    volumes:
+      - ./resources/elk6/pipeline/:/usr/share/logstash/pipeline
+      - ./data/:/opt/VulnWhisperer/data
+      #- ./resources/elk6/logstash.yml:/usr/share/logstash/config/logstash.yml
+    environment:
+      - xpack.monitoring.enabled=false
+    depends_on:
+      - elasticsearch
+    networks:
+      esnet:
+        aliases:
+          - logstash.local
+  vulnwhisperer:
+    image: hasecuritysolutions/vulnwhisperer:latest
+    container_name: vulnwhisperer
+    entrypoint: [
+                  "vuln_whisperer",
+                  "-c",
+                  "/opt/VulnWhisperer/vulnwhisperer.ini"
+    ]    
+    volumes:
+      - ./data/:/opt/VulnWhisperer/data
+      - ./resources/elk6/vulnwhisperer.ini:/opt/VulnWhisperer/vulnwhisperer.ini
+    network_mode: host
+volumes:
+  esdata1:
+    driver: local
+
+networks:
+  esnet:

requirements.txt

@@ -1,7 +1,12 @@
 pandas==0.20.3
-setuptools==0.9.8
+setuptools==40.4.3
 pytz==2017.2
-Requests==2.18.3
-qualysapi==4.1.0
-lxml==4.1.1
-bs4
+Requests==2.20.0
+lxml==4.6.5
+future-fstrings
+bs4
+jira
+bottle
+coloredlogs
+qualysapi==6.0.0
+httpretty

resources/elk5-old_compatibility/docker-compose_ELK5_unsupported.yml

@@ -21,7 +21,7 @@ services:
       - 9200:9200
     environment:
       - xpack.security.enabled=false
-    restart: always
+    #restart: always
     networks:
       esnet:
         aliases:
@@ -46,6 +46,7 @@ services:
       - ./docker/1000_nessus_process_file.conf:/usr/share/logstash/pipeline/1000_nessus_process_file.conf
       - ./docker/2000_qualys_web_scans.conf:/usr/share/logstash/pipeline/2000_qualys_web_scans.conf
       - ./docker/3000_openvas.conf:/usr/share/logstash/pipeline/3000_openvas.conf
+      - ./docker/4000_jira.conf:/usr/share/logstash/pipeline/4000_jira.conf
       - ./docker/logstash.yml:/usr/share/logstash/config/logstash.yml
       - ./data/:/opt/VulnWhisperer/data
     environment:
@@ -57,7 +58,7 @@ services:
         aliases:
           - vulnwhisp-ls1.local
   vulnwhisp-vulnwhisperer:
-    image: hasecuritysolutions/docker_vulnwhisperer:latest
+    image: hasecuritysolutions/vulnwhisperer:latest
     container_name: vulnwhisp-vulnwhisperer
     volumes:
       - ./data/:/opt/VulnWhisperer/data

resources/elk5-old_compatibility/docker/3000_openvas.conf

@@ -2,7 +2,7 @@
 # Email: austin@hasecuritysolutions.com
 # Last Update: 03/04/2018
 # Version 0.3
-# Description: Take in qualys web scan reports from vulnWhisperer and pumps into logstash
+# Description: Take in Openvas web scan reports from vulnWhisperer and pumps into logstash
 
 input {
   file {

resources/elk5-old_compatibility/docker/4000_jira.conf

@@ -0,0 +1,21 @@
+# Description: Take in jira tickets from vulnWhisperer and pumps into logstash
+
+input {
+  file {
+    path => "/opt/Vulnwhisperer/jira/*.json"
+    type => json
+    codec => json
+    start_position => "beginning"
+    tags => [ "jira" ]
+  }
+}
+
+output {
+  if "jira" in [tags] {
+    stdout { codec => rubydebug }
+    elasticsearch {
+      hosts => [ "vulnwhisp-es1.local:9200" ]
+      index => "logstash-vulnwhisperer-%{+YYYY.MM}"
+    }
+  }
+}

resources/elk5-old_compatibility/elasticsearch/logstash-vulnwhisperer-template.json

@@ -53,7 +53,7 @@
       ],
       "properties": {
         "plugin_id": {
-          "type": "integer"
+          "type": "float"
         },
         "last_updated": {
           "type": "date"

resources/elk5-old_compatibility/kibana/vuln_whisp_kibana/1000_vulnWhispererBaseVisuals.json

@@ -60,7 +60,7 @@
     "_type": "visualization",
     "_source": {
       "title": "VulnWhisperer - Mitigation Readme",
-      "visState": "{\"title\":\"VulnWhisperer - Mitigation Readme\",\"type\":\"markdown\",\"params\":{\"markdown\":\"** Legend **\\n\\n* [Common Vulnerability Scoring System (CVSS)](https://nvd.nist.gov/vuln-metrics/cvss) is the NIST vulnerability scoring system\\n* Risk Number is residual risk score calculated from CVSS, which is adjusted to be specific to Heartland which accounts for services not in use such as Java and Flash\\n* Vulnerabilities by Tag are systems tagged with HIPAA and PCI identification.\\n\\n\\n** Workflow **\\n* Select 10.0 under Risk Number to identify Critical Vulnerabilities. \\n* For more information about a CVE, scroll down and click the CVE link.\\n* To filter by tags, use one of the following filters:\\n** tags:has_hipaa_data, tags:pci_asset, tags:hipaa_asset, tags:critical_asset**\"},\"aggs\":[],\"listeners\":{}}",
+      "visState": "{\"title\":\"VulnWhisperer - Mitigation Readme\",\"type\":\"markdown\",\"params\":{\"markdown\":\"** Legend **\\n\\n* [Common Vulnerability Scoring System (CVSS)](https://nvd.nist.gov/vuln-metrics/cvss) is the NIST vulnerability scoring system\\n* Risk Number is residual risk score calculated from CVSS, which is adjusted to be specific to the netowrk owner, which accounts for services not in use such as Java and Flash\\n* Vulnerabilities by Tag are systems tagged with HIPAA and PCI identification.\\n\\n\\n** Workflow **\\n* Select 10.0 under Risk Number to identify Critical Vulnerabilities. \\n* For more information about a CVE, scroll down and click the CVE link.\\n* To filter by tags, use one of the following filters:\\n** tags:has_hipaa_data, tags:pci_asset, tags:hipaa_asset, tags:critical_asset**\"},\"aggs\":[],\"listeners\":{}}",
       "uiStateJSON": "{}",
       "description": "",
       "version": 1,
@@ -88,7 +88,7 @@
     "_type": "visualization",
     "_source": {
       "title": "VulnWhisperer-CVSSHeatmap",
-      "visState": "{\"title\":\"VulnWhisperer-CVSSHeatmap\",\"type\":\"heatmap\",\"params\":{\"addTooltip\":true,\"addLegend\":true,\"enableHover\":false,\"legendPosition\":\"right\",\"times\":[],\"colorsNumber\":4,\"colorSchema\":\"Yellow to Red\",\"setColorRange\":false,\"colorsRange\":[],\"invertColors\":false,\"percentageMode\":false,\"valueAxes\":[{\"show\":false,\"id\":\"ValueAxis-1\",\"type\":\"value\",\"scale\":{\"type\":\"linear\",\"defaultYExtents\":false},\"labels\":{\"show\":false,\"rotate\":0,\"color\":\"#555\"}}]},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"host\",\"size\":50,\"order\":\"desc\",\"orderBy\":\"1\"}},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"group\",\"params\":{\"field\":\"cvss\",\"size\":50,\"order\":\"desc\",\"orderBy\":\"_term\"}}],\"listeners\":{}}",
+      "visState": "{\"title\":\"VulnWhisperer-CVSSHeatmap\",\"type\":\"heatmap\",\"params\":{\"addTooltip\":true,\"addLegend\":true,\"enableHover\":false,\"legendPosition\":\"right\",\"times\":[],\"colorsNumber\":4,\"colorSchema\":\"Yellow to Red\",\"setColorRange\":false,\"colorsRange\":[],\"invertColors\":false,\"percentageMode\":false,\"valueAxes\":[{\"show\":false,\"id\":\"ValueAxis-1\",\"type\":\"value\",\"scale\":{\"type\":\"linear\",\"defaultYExtents\":false},\"labels\":{\"show\":false,\"rotate\":0,\"color\":\"#555\"}}]},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"host.keyword\",\"size\":50,\"order\":\"desc\",\"orderBy\":\"1\"}},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"group\",\"params\":{\"field\":\"cvss.keyword\",\"size\":50,\"order\":\"desc\",\"orderBy\":\"_term\"}}],\"listeners\":{}}",
       "uiStateJSON": "{\"vis\":{\"defaultColors\":{\"0 - 3500\":\"rgb(255,255,204)\",\"3500 - 7000\":\"rgb(254,217,118)\",\"7000 - 10500\":\"rgb(253,141,60)\",\"10500 - 14000\":\"rgb(227,27,28)\"}}}",
       "description": "",
       "version": 1,
@@ -354,7 +354,7 @@
     "_type": "visualization",
     "_source": {
       "title": "VulnWhisperer-CVSS",
-      "visState": "{\"title\":\"VulnWhisperer-CVSS\",\"type\":\"table\",\"params\":{\"perPage\":10,\"showMeticsAtAllLevels\":false,\"showPartialRows\":false,\"showTotal\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"totalFunc\":\"sum\",\"type\":\"table\"},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"bucket\",\"params\":{\"field\":\"cvss\",\"size\":20,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"CVSS Score\"}},{\"id\":\"4\",\"enabled\":true,\"type\":\"cardinality\",\"schema\":\"metric\",\"params\":{\"field\":\"asset.keyword\",\"customLabel\":\"# of Assets\"}}],\"listeners\":{}}",
+      "visState": "{\"title\":\"VulnWhisperer-CVSS\",\"type\":\"table\",\"params\":{\"perPage\":10,\"showMeticsAtAllLevels\":false,\"showPartialRows\":false,\"showTotal\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"totalFunc\":\"sum\",\"type\":\"table\"},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"bucket\",\"params\":{\"field\":\"cvss.keyword\",\"size\":20,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"CVSS Score\"}},{\"id\":\"4\",\"enabled\":true,\"type\":\"cardinality\",\"schema\":\"metric\",\"params\":{\"field\":\"asset.keyword\",\"customLabel\":\"# of Assets\"}}],\"listeners\":{}}",
       "uiStateJSON": "{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":0,\"direction\":\"desc\"}}}}",
       "description": "",
       "version": 1,
@@ -447,4 +447,4 @@
       }
     }
   }
-]
+]

resources/elk5-old_compatibility/logstash/1000_nessus_process_file.conf

@@ -7,13 +7,13 @@
 
 input {
   file {
-    path => "/opt/vulnwhisperer/nessus/**/*"
+    path => "/opt/VulnWhisperer/nessus/**/*"
     start_position => "beginning"
     tags => "nessus"
     type => "nessus"
   }
   file {
-    path => "/opt/vulnwhisperer/tenable/*.csv"
+    path => "/opt/VulnWhisperer/tenable/*.csv"
     start_position => "beginning"
     tags => "tenable"
     type => "tenable"
@@ -27,7 +27,7 @@ filter {
 
     csv {
       # columns => ["plugin_id", "cve", "cvss", "risk", "asset", "protocol", "port", "plugin_name", "synopsis", "description", "solution", "see_also", "plugin_output"]
-      columns => ["plugin_id", "cve", "cvss", "risk", "asset", "protocol", "port", "plugin_name", "synopsis", "description", "solution", "see_also", "plugin_output", "asset_uuid", "vulnerability_state", "ip", "fqdn", "netbios", "operating_system", "mac_address", "plugin_family", "cvss_base", "cvss_temporal", "cvss_temporal_vector", "cvss_vector", "cvss3_base", "cvss3_temporal", "cvss3_temporal_vector", "cvss_vector", "system_type", "host_start", "host_end"]
+      columns => ["plugin_id", "cve", "cvss", "risk", "asset", "protocol", "port", "plugin_name", "synopsis", "description", "solution", "see_also", "plugin_output", "asset_uuid", "vulnerability_state", "ip", "fqdn", "netbios", "operating_system", "mac_address", "plugin_family", "cvss_base", "cvss_temporal", "cvss_temporal_vector", "cvss_vector", "cvss3_base", "cvss3_temporal", "cvss3_temporal_vector", "cvss3_vector", "system_type", "host_start", "host_end"]
       separator => ","
       source => "message"
     }

resources/elk5-old_compatibility/logstash/2000_qualys_web_scans.conf

@@ -6,7 +6,7 @@
 
 input {
   file {
-    path => "/opt/vulnwhisperer/qualys/*.json"
+    path => [ "/opt/VulnWhisperer/data/qualys/*.json" , "/opt/VulnWhisperer/data/qualys_web/*.json", "/opt/VulnWhisperer/data/qualys_vuln/*.json" ]
     type => json
     codec => json
     start_position => "beginning"

resources/elk5-old_compatibility/logstash/3000_openvas.conf

@@ -6,7 +6,7 @@
 
 input {
   file {
-    path => "/opt/vulnwhisperer/openvas/*.json"
+    path => "/opt/VulnWhisperer/openvas/*.json"
     type => json
     codec => json
     start_position => "beginning"

resources/elk5-old_compatibility/logstash/4000_jira.conf

@@ -0,0 +1,21 @@
+# Description: Take in jira tickets from vulnWhisperer and pumps into logstash
+
+input {
+  file {
+    path => "/opt/VulnWhisperer/jira/*.json"
+    type => json
+    codec => json
+    start_position => "beginning"
+    tags => [ "jira" ]
+  }
+}
+
+output {
+  if "jira" in [tags] {
+    stdout { codec => rubydebug }
+    elasticsearch {
+      hosts => [ "localhost:9200" ]
+      index => "logstash-vulnwhisperer-%{+YYYY.MM}"
+    }
+  }
+}

resources/elk6/filebeat.yml

@@ -0,0 +1,116 @@
+###################### Filebeat Configuration Example #########################
+
+# This file is an example configuration file highlighting only the most common
+# options. The filebeat.full.yml file from the same directory contains all the
+# supported options with more comments. You can use it as a reference.
+#
+# You can find the full configuration reference here:
+# https://www.elastic.co/guide/en/beats/filebeat/index.html
+
+#=========================== Filebeat prospectors =============================
+
+filebeat.prospectors:
+
+# Each - is a prospector. Most options can be set at the prospector level, so
+# you can use different prospectors for various configurations.
+# Below are the prospector specific configurations.
+
+- input_type: log
+  # Paths that should be crawled and fetched. Glob based paths.
+  paths:
+    # Linux Example
+    #- /var/log/*.log
+
+    #Windows Example
+    - c:\nessus\My Scans\*
+
+  # Exclude lines. A list of regular expressions to match. It drops the lines that are
+  # matching any regular expression from the list.
+  #exclude_lines: ["^DBG"]
+
+  # Include lines. A list of regular expressions to match. It exports the lines that are
+  # matching any regular expression from the list.
+  #include_lines: ["^ERR", "^WARN"]
+
+  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
+  # are matching any regular expression from the list. By default, no files are dropped.
+  #exclude_files: [".gz$"]
+
+  # Optional additional fields. These field can be freely picked
+  # to add additional information to the crawled log files for filtering
+  #fields:
+  #  level: debug
+  #  review: 1
+
+  ### Multiline options
+
+  # Mutiline can be used for log messages spanning multiple lines. This is common
+  # for Java Stack Traces or C-Line Continuation
+
+  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
+  #multiline.pattern: ^\[
+
+  # Defines if the pattern set under pattern should be negated or not. Default is false.
+  #multiline.negate: false
+
+  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
+  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
+  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
+  #multiline.match: after
+
+
+#================================ General =====================================
+
+# The name of the shipper that publishes the network data. It can be used to group
+# all the transactions sent by a single shipper in the web interface.
+#name:
+
+# The tags of the shipper are included in their own field with each
+# transaction published.
+#tags: ["service-X", "web-tier"]
+
+# Optional fields that you can specify to add additional information to the
+# output.
+#fields:
+#  env: staging
+
+#================================ Outputs =====================================
+
+# Configure what outputs to use when sending the data collected by the beat.
+# Multiple outputs may be used.
+
+#-------------------------- Elasticsearch output ------------------------------
+#output.elasticsearch:
+  # Array of hosts to connect to.
+#  hosts: ["logstash01:9200"]
+
+  # Optional protocol and basic auth credentials.
+  #protocol: "https"
+  #username: "elastic"
+  #password: "changeme"
+
+#----------------------------- Logstash output --------------------------------
+output.logstash:
+  # The Logstash hosts
+  hosts: ["logstashserver1:5044", "logstashserver2:5044", "logstashserver3:5044"]
+
+  # Optional SSL. By default is off.
+  # List of root certificates for HTTPS server verifications
+  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
+
+  # Certificate for SSL client authentication
+  #ssl.certificate: "/etc/pki/client/cert.pem"
+
+  # Client Certificate Key
+  #ssl.key: "/etc/pki/client/cert.key"
+
+#================================ Logging =====================================
+
+# Sets log level. The default log level is info.
+# Available log levels are: critical, error, warning, info, debug
+#logging.level: debug
+
+# At debug level, you can selectively enable logging only for some components.
+# To enable all selectors use ["*"]. Examples of other selectors are "beat",
+# "publish", "service".
+#logging.selectors: ["*"]

resources/elk6/init_kibana.sh

@@ -0,0 +1,52 @@
+#!/bin/bash
+
+#kibana_url="localhost:5601"
+kibana_url="kibana.local:5601"
+elasticsearch_url="elasticsearch.local:9200"
+add_saved_objects="curl -s -u elastic:changeme -k -XPOST 'http://"$kibana_url"/api/saved_objects/_bulk_create' -H 'Content-Type: application/json' -H \"kbn-xsrf: true\" -d @"
+
+#Create all saved objects - including index pattern
+saved_objects_file="kibana_APIonly.json"
+
+#if [ `curl -I localhost:5601/status | head -n1 |cut -d$' ' -f2` -eq '200' ]; then echo "Loading VulnWhisperer Saved Objects"; eval $(echo $add_saved_objects$saved_objects_file); else echo "waiting for kibana"; fi
+
+until curl -s "$elasticsearch_url/_cluster/health?pretty" | grep '"status"' | grep -qE "green|yellow"; do
+    curl -s "$elasticsearch_url/_cluster/health?pretty"
+    echo "Waiting for Elasticsearch..."
+    sleep 5
+done
+
+count=0
+until  curl -s --fail -XPUT "http://$elasticsearch_url/_template/vulnwhisperer" -H 'Content-Type: application/json' -d '@/opt/index-template.json'; do
+    echo "Loading VulnWhisperer index template..."
+    ((count++)) && ((count==60)) && break
+    sleep 1
+done
+
+if [[ count -le 60 && $(curl -s -I http://$elasticsearch_url/_template/vulnwhisperer | head -n1 |cut -d$' ' -f2) == "200" ]]; then
+    echo -e "\n✅ VulnWhisperer index template loaded"
+else
+    echo -e "\n❌ VulnWhisperer index template failed to load"
+fi
+
+until [ "`curl -s -I "$kibana_url"/status | head -n1 |cut -d$' ' -f2`" == "200" ]; do
+    curl -s -I "$kibana_url"/status
+    echo "Waiting for Kibana..."
+    sleep 5
+done
+
+echo "Loading VulnWhisperer Saved Objects"
+echo $add_saved_objects$saved_objects_file
+eval $(echo $add_saved_objects$saved_objects_file)
+
+#set "*" as default index
+#id_default_index="87f3bcc0-8b37-11e8-83be-afaed4786d8c"
+#os.system("curl -X POST -H \"Content-Type: application/json\" -H \"kbn-xsrf: true\" -d '{\"value\":\""+id_default_index+"\"}' http://elastic:changeme@"+kibana_url+"kibana/settings/defaultIndex")
+
+#Create vulnwhisperer index pattern
+#index_name = "logstash-vulnwhisperer-*"
+#os.system(add_index+index_name+"' '-d{\"attributes\":{\"title\":\""+index_name+"\",\"timeFieldName\":\"@timestamp\"}}'")
+
+#Create jira index pattern, separated for not fill of crap variables the Discover tab by default
+#index_name = "logstash-jira-*"
+#os.system(add_index+index_name+"' '-d{\"attributes\":{\"title\":\""+index_name+"\",\"timeFieldName\":\"@timestamp\"}}'")

resources/elk6/logstash-vulnwhisperer-template.json

@@ -0,0 +1,233 @@
+{
+  "index_patterns": "logstash-vulnwhisperer-*",
+  "mappings": {
+    "doc": {
+      "properties": {
+        "@timestamp": {
+          "type": "date"
+        },
+        "@version": {
+          "type": "keyword"
+        },
+        "asset": {
+          "type": "text",
+          "norms": false,
+          "fields": {
+            "keyword": {
+              "type": "keyword",
+              "ignore_above": 256
+            }
+          }
+        },
+        "asset_uuid": {
+          "type": "keyword"
+        },
+        "assign_ip": {
+          "type": "ip"
+        },
+        "category": {
+          "type": "keyword"
+        },
+        "cve": {
+          "type": "keyword"
+        },
+        "cvss_base": {
+          "type": "float"
+        },
+        "cvss_temporal_vector": {
+          "type": "keyword"
+        },
+        "cvss_temporal": {
+          "type": "float"
+        },
+        "cvss_vector": {
+          "type": "keyword"
+        },
+        "cvss": {
+          "type": "float"
+        },
+        "cvss3_base": {
+          "type": "float"
+        },
+        "cvss3_temporal_vector": {
+          "type": "keyword"
+        },
+        "cvss3_temporal": {
+          "type": "float"
+        },
+        "cvss3_vector": {
+          "type": "keyword"
+        },
+        "cvss3": {
+          "type": "float"
+        },
+        "description": {
+          "fields": {
+            "keyword": {
+              "ignore_above": 256,
+              "type": "keyword"
+            }
+          },
+          "norms": false,
+          "type": "text"
+        },
+        "dns": {
+          "type": "keyword"
+        },
+        "exploitability": {
+          "fields": {
+            "keyword": {
+              "ignore_above": 256,
+              "type": "keyword"
+            }
+          },
+          "norms": false,
+          "type": "text"
+        },
+        "fqdn": {
+          "type": "keyword"
+        },
+        "geoip": {
+          "dynamic": true,
+          "type": "object",
+          "properties": {
+            "ip": {
+              "type": "ip"
+            },
+            "latitude": {
+              "type": "float"
+            },
+            "location": {
+              "type": "geo_point"
+            },
+            "longitude": {
+              "type": "float"
+            }
+          }
+        },
+        "history_id": {
+          "type": "keyword"
+        },
+        "host": {
+          "type": "keyword"
+        },
+        "host_end": {
+          "type": "date"
+        },
+        "host_start": {
+          "type": "date"
+        },
+        "impact": {
+          "fields": {
+            "keyword": {
+              "ignore_above": 256,
+              "type": "keyword"
+            }
+          },
+          "norms": false,
+          "type": "text"
+        },
+        "ip_status": {
+          "type": "keyword"
+        },
+        "ip": {
+          "type": "ip"
+        },
+        "last_updated": {
+          "type": "date"
+        },
+        "operating_system": {
+          "type": "keyword"
+        },
+        "path": {
+          "type": "keyword"
+        },
+        "pci_vuln": {
+          "type": "keyword"
+        },
+        "plugin_family": {
+          "type": "keyword"
+        },
+        "plugin_id": {
+          "type": "keyword"
+        },
+        "plugin_name": {
+          "type": "keyword"
+        },
+        "plugin_output": {
+          "fields": {
+            "keyword": {
+              "ignore_above": 256,
+              "type": "keyword"
+            }
+          },
+          "norms": false,
+          "type": "text"
+        },
+        "port": {
+          "type": "integer"
+        },
+        "protocol": {
+          "type": "keyword"
+        },
+        "results": {
+          "type": "text"
+        },
+        "risk_number": {
+          "type": "integer"
+        },
+        "risk_score_name": {
+          "type": "keyword"
+        },
+        "risk_score": {
+          "type": "float"
+        },
+        "risk": {
+          "type": "keyword"
+        },
+        "scan_id": {
+          "type": "keyword"
+        },
+        "scan_name": {
+          "type": "keyword"
+        },
+        "scan_reference": {
+          "type": "keyword"
+        },
+        "see_also": {
+          "type": "keyword"
+        },
+        "solution": {
+          "type": "keyword"
+        },
+        "source": {
+          "type": "keyword"
+        },
+        "ssl": {
+          "type": "keyword"
+        },
+        "synopsis": {
+          "type": "keyword"
+        },
+        "system_type": {
+          "type": "keyword"
+        },
+        "tags": {
+          "type": "keyword"
+        },
+        "threat": {
+          "type": "text"
+        },
+        "type": {
+          "type": "keyword"
+        },
+        "vendor_reference": {
+          "type": "keyword"
+        },
+        "vulnerability_state": {
+          "type": "keyword"
+        }
+      }
+    }
+  }
+}

resources/elk6/logstash-vulnwhisperer-template_elk7.json

@@ -0,0 +1,231 @@
+{
+  "index_patterns": "logstash-vulnwhisperer-*",
+  "mappings": {
+    "properties": {
+      "@timestamp": {
+        "type": "date"
+      },
+      "@version": {
+        "type": "keyword"
+      },
+      "asset": {
+        "type": "text",
+        "norms": false,
+        "fields": {
+          "keyword": {
+            "type": "keyword",
+            "ignore_above": 256
+          }
+        }
+      },
+      "asset_uuid": {
+        "type": "keyword"
+      },
+      "assign_ip": {
+        "type": "ip"
+      },
+      "category": {
+        "type": "keyword"
+      },
+      "cve": {
+        "type": "keyword"
+      },
+      "cvss_base": {
+        "type": "float"
+      },
+      "cvss_temporal_vector": {
+        "type": "keyword"
+      },
+      "cvss_temporal": {
+        "type": "float"
+      },
+      "cvss_vector": {
+        "type": "keyword"
+      },
+      "cvss": {
+        "type": "float"
+      },
+      "cvss3_base": {
+        "type": "float"
+      },
+      "cvss3_temporal_vector": {
+        "type": "keyword"
+      },
+      "cvss3_temporal": {
+        "type": "float"
+      },
+      "cvss3_vector": {
+        "type": "keyword"
+      },
+      "cvss3": {
+        "type": "float"
+      },
+      "description": {
+        "fields": {
+          "keyword": {
+            "ignore_above": 256,
+            "type": "keyword"
+          }
+        },
+        "norms": false,
+        "type": "text"
+      },
+      "dns": {
+        "type": "keyword"
+      },
+      "exploitability": {
+        "fields": {
+          "keyword": {
+            "ignore_above": 256,
+            "type": "keyword"
+          }
+        },
+        "norms": false,
+        "type": "text"
+      },
+      "fqdn": {
+        "type": "keyword"
+      },
+      "geoip": {
+        "dynamic": true,
+        "type": "object",
+        "properties": {
+          "ip": {
+            "type": "ip"
+          },
+          "latitude": {
+            "type": "float"
+          },
+          "location": {
+            "type": "geo_point"
+          },
+          "longitude": {
+            "type": "float"
+          }
+        }
+      },
+      "history_id": {
+        "type": "keyword"
+      },
+      "host": {
+        "type": "keyword"
+      },
+      "host_end": {
+        "type": "date"
+      },
+      "host_start": {
+        "type": "date"
+      },
+      "impact": {
+        "fields": {
+          "keyword": {
+            "ignore_above": 256,
+            "type": "keyword"
+          }
+        },
+        "norms": false,
+        "type": "text"
+      },
+      "ip_status": {
+        "type": "keyword"
+      },
+      "ip": {
+        "type": "ip"
+      },
+      "last_updated": {
+        "type": "date"
+      },
+      "operating_system": {
+        "type": "keyword"
+      },
+      "path": {
+        "type": "keyword"
+      },
+      "pci_vuln": {
+        "type": "keyword"
+      },
+      "plugin_family": {
+        "type": "keyword"
+      },
+      "plugin_id": {
+        "type": "keyword"
+      },
+      "plugin_name": {
+        "type": "keyword"
+      },
+      "plugin_output": {
+        "fields": {
+          "keyword": {
+            "ignore_above": 256,
+            "type": "keyword"
+          }
+        },
+        "norms": false,
+        "type": "text"
+      },
+      "port": {
+        "type": "integer"
+      },
+      "protocol": {
+        "type": "keyword"
+      },
+      "results": {
+        "type": "text"
+      },
+      "risk_number": {
+        "type": "integer"
+      },
+      "risk_score_name": {
+        "type": "keyword"
+      },
+      "risk_score": {
+        "type": "float"
+      },
+      "risk": {
+        "type": "keyword"
+      },
+      "scan_id": {
+        "type": "keyword"
+      },
+      "scan_name": {
+        "type": "keyword"
+      },
+      "scan_reference": {
+        "type": "keyword"
+      },
+      "see_also": {
+        "type": "keyword"
+      },
+      "solution": {
+        "type": "keyword"
+      },
+      "source": {
+        "type": "keyword"
+      },
+      "ssl": {
+        "type": "keyword"
+      },
+      "synopsis": {
+        "type": "keyword"
+      },
+      "system_type": {
+        "type": "keyword"
+      },
+      "tags": {
+        "type": "keyword"
+      },
+      "threat": {
+        "type": "text"
+      },
+      "type": {
+        "type": "keyword"
+      },
+      "vendor_reference": {
+        "type": "keyword"
+      },
+      "vulnerability_state": {
+        "type": "keyword"
+      }
+    }
+  }
+}

resources/elk6/logstash.yml

@@ -0,0 +1,9 @@
+node.name: logstash
+path.config: /usr/share/logstash/pipeline/
+path.data: /tmp
+queue.drain: true
+queue.type: persisted
+xpack.monitoring.elasticsearch.password: changeme
+xpack.monitoring.elasticsearch.url: elasticsearch:9200
+xpack.monitoring.elasticsearch.username: elastic
+xpack.monitoring.enabled: false

resources/elk6/pipeline/1000_nessus_process_file.conf

@@ -0,0 +1,182 @@
+# Author: Austin Taylor and Justin Henderson
+# Email: email@austintaylor.io
+# Last Update: 12/20/2017
+# Version 0.3
+# Description: Take in nessus reports from vulnWhisperer and pumps into logstash
+
+
+input {
+  file {
+    path => "/opt/VulnWhisperer/data/nessus/**/*"
+    mode => "read"
+    start_position => "beginning"
+    file_completed_action => "delete"
+    tags => "nessus"
+  }
+  file {
+    path => "/opt/VulnWhisperer/data/tenable/*.csv"
+    mode => "read"
+    start_position => "beginning"
+    file_completed_action => "delete"
+    tags => "tenable"
+  }
+}
+
+filter {
+  if "nessus" in [tags] or "tenable" in [tags] {
+    # Drop the header column
+    if [message] =~ "^Plugin ID" { drop {} }
+
+    csv {
+      # columns => ["plugin_id", "cve", "cvss", "risk", "asset", "protocol", "port", "plugin_name", "synopsis", "description", "solution", "see_also", "plugin_output"]
+      columns => ["plugin_id", "cve", "cvss", "risk", "asset", "protocol", "port", "plugin_name", "synopsis", "description", "solution", "see_also", "plugin_output", "asset_uuid", "vulnerability_state", "ip", "fqdn", "netbios", "operating_system", "mac_address", "plugin_family", "cvss_base", "cvss_temporal", "cvss_temporal_vector", "cvss_vector", "cvss3_base", "cvss3_temporal", "cvss3_temporal_vector", "cvss3_vector", "system_type", "host_start", "host_end"]
+      separator => ","
+      source => "message"
+    }
+
+    ruby {
+        code => "if event.get('description')
+                    event.set('description', event.get('description').gsub(92.chr + 'n', 10.chr).gsub(92.chr + 'r', 13.chr))
+                end
+                 if event.get('synopsis')
+                    event.set('synopsis', event.get('synopsis').gsub(92.chr + 'n', 10.chr).gsub(92.chr + 'r', 13.chr))
+                end
+                 if event.get('solution')
+                    event.set('solution', event.get('solution').gsub(92.chr + 'n', 10.chr).gsub(92.chr + 'r', 13.chr))
+                end
+                 if event.get('see_also')
+                    event.set('see_also', event.get('see_also').gsub(92.chr + 'n', 10.chr).gsub(92.chr + 'r', 13.chr))
+                end
+                 if event.get('plugin_output')
+                    event.set('plugin_output', event.get('plugin_output').gsub(92.chr + 'n', 10.chr).gsub(92.chr + 'r', 13.chr))
+                end"
+    }
+
+    #If using filebeats as your source, you will need to replace the "path" field to "source"
+    # Remove when scan name is included in event (current method is error prone)
+    grok {
+      match => { "path" => "(?<scan_name>[a-zA-Z0-9_.\-]+)_%{INT:scan_id}_%{INT:history_id}_%{INT:last_updated}.csv$" }
+      tag_on_failure => []
+    }
+
+    # TODO remove when @timestamp is included in event
+    date {
+      match => [ "last_updated", "UNIX" ]
+      target => "@timestamp"
+      remove_field => ["last_updated"]
+    }
+
+    if [risk] == "None" {
+      mutate { add_field => { "risk_number" => 0 }}
+    }
+    if [risk] == "Low" {
+      mutate { add_field => { "risk_number" => 1 }}
+    }
+    if [risk] == "Medium" {
+      mutate { add_field => { "risk_number" => 2 }}
+    }
+    if [risk] == "High" {
+      mutate { add_field => { "risk_number" => 3 }}
+    }
+    if [risk] == "Critical" {
+      mutate { add_field => { "risk_number" => 4 }}
+    }
+
+    if ![cve] or [cve] == "nan" {
+      mutate { remove_field => [ "cve" ] }
+    }
+    if ![cvss] or [cvss] == "nan" {
+      mutate { remove_field => [ "cvss" ] }
+    }
+    if ![cvss_base] or [cvss_base] == "nan" {
+      mutate { remove_field => [ "cvss_base" ] }
+    }
+    if ![cvss_temporal] or [cvss_temporal] == "nan" {
+      mutate { remove_field => [ "cvss_temporal" ] }
+    }
+    if ![cvss_temporal_vector] or [cvss_temporal_vector] == "nan" {
+      mutate { remove_field => [ "cvss_temporal_vector" ] }
+    }
+    if ![cvss_vector] or [cvss_vector] == "nan" {
+      mutate { remove_field => [ "cvss_vector" ] }
+    }
+    if ![cvss3_base] or [cvss3_base] == "nan" {
+      mutate { remove_field => [ "cvss3_base" ] }
+    }
+    if ![cvss3_temporal] or [cvss3_temporal] == "nan" {
+      mutate { remove_field => [ "cvss3_temporal" ] }
+    }
+    if ![cvss3_temporal_vector] or [cvss3_temporal_vector] == "nan" {
+      mutate { remove_field => [ "cvss3_temporal_vector" ] }
+    }
+    if ![description] or [description] == "nan" {
+      mutate { remove_field => [ "description" ] }
+    }
+    if ![mac_address] or [mac_address] == "nan" {
+      mutate { remove_field => [ "mac_address" ] }
+    }
+    if ![netbios] or [netbios] == "nan" {
+      mutate { remove_field => [ "netbios" ] }
+    }
+    if ![operating_system] or [operating_system] == "nan" {
+      mutate { remove_field => [ "operating_system" ] }
+    }
+    if ![plugin_output] or [plugin_output] == "nan" {
+      mutate { remove_field => [ "plugin_output" ] }
+    }
+    if ![see_also] or [see_also] == "nan" {
+      mutate { remove_field => [ "see_also" ] }
+    }
+    if ![synopsis] or [synopsis] == "nan" {
+      mutate { remove_field => [ "synopsis" ] }
+    }
+    if ![system_type] or [system_type] == "nan" {
+      mutate { remove_field => [ "system_type" ] }
+    }
+
+    mutate {
+      remove_field => [ "message" ]
+      add_field => { "risk_score" => "%{cvss}" }
+    }
+    mutate {
+      convert => { "risk_score" => "float" }
+    }
+    if [risk_score] == 0 {
+      mutate {
+        add_field => { "risk_score_name" => "info" }
+      }
+    }
+    if [risk_score] > 0 and [risk_score] < 3 {
+      mutate {
+        add_field => { "risk_score_name" => "low" }
+      }
+    }
+    if [risk_score] >= 3 and [risk_score] < 6 {
+      mutate {
+        add_field => { "risk_score_name" => "medium" }
+      }
+    }
+    if [risk_score] >=6 and [risk_score] < 9 {
+      mutate {
+        add_field => { "risk_score_name" => "high" }
+      }
+    }
+    if [risk_score] >= 9 {
+      mutate {
+        add_field => { "risk_score_name" => "critical" }
+      }
+    }
+  }
+}
+
+output {
+  if "nessus" in [tags] or "tenable" in [tags]{
+    stdout {
+      codec => dots
+    }
+    elasticsearch {
+      hosts => [ "elasticsearch:9200" ]
+      index => "logstash-vulnwhisperer-%{+YYYY.MM}"
+    }
+  }
+}

resources/elk6/pipeline/2000_qualys_web_scans.conf

@@ -0,0 +1,160 @@
+# Author: Austin Taylor and Justin Henderson
+# Email: austin@hasecuritysolutions.com
+# Last Update: 12/30/2017
+# Version 0.3
+# Description: Take in qualys web scan reports from vulnWhisperer and pumps into logstash
+
+input {
+  file {
+    path => [ "/opt/VulnWhisperer/data/qualys/*.json" , "/opt/VulnWhisperer/data/qualys_web/*.json", "/opt/VulnWhisperer/data/qualys_vuln/*.json"] 
+    type => json
+    codec => json
+    start_position => "beginning"
+    tags => [ "qualys" ]
+    mode => "read"
+    start_position => "beginning"
+    file_completed_action => "delete"
+  }
+}
+
+filter {
+  if "qualys" in [tags] {
+    grok {
+      match => { "path" => [ "(?<tags>qualys_vuln)_scan_%{DATA}_%{INT:last_updated}.json$", "(?<tags>qualys_web)_%{INT:app_id}_%{INT:last_updated}.json$" ] }
+      tag_on_failure => []
+    }
+    
+    mutate {
+      replace => [ "message", "%{message}" ]
+      #gsub => [
+      #  "message", "\|\|\|", " ",
+      #  "message", "\t\t", " ",
+      #  "message", "    ", " ",
+      #  "message", "   ", " ",
+      #  "message", "  ", " ",
+      #  "message", "nan", " ",
+      #  "message",'\n',''
+      #]
+    }
+
+    if "qualys_web" in [tags] {
+    mutate {
+      add_field => { "asset" => "%{web_application_name}" }
+      add_field => { "risk_score" => "%{cvss}" }
+      }
+    } else if "qualys_vuln" in [tags] {
+      mutate {
+        add_field => { "asset" => "%{ip}" }
+        add_field => { "risk_score" => "%{cvss}" }
+      }
+    }
+
+    if [risk] == "1" {
+      mutate { add_field => { "risk_number" => 0 }}
+      mutate { replace => { "risk" => "info" }}
+    }
+    if [risk] == "2" {
+      mutate { add_field => { "risk_number" => 1 }}
+      mutate { replace => { "risk" => "low" }}
+    }
+    if [risk] == "3" {
+      mutate { add_field => { "risk_number" => 2 }}
+      mutate { replace => { "risk" => "medium" }}
+    }
+    if [risk] == "4" {
+      mutate { add_field => { "risk_number" => 3 }}
+      mutate { replace => { "risk" => "high" }}
+    }
+    if [risk] == "5" {
+      mutate { add_field => { "risk_number" => 4 }}
+      mutate { replace => { "risk" => "critical" }}
+    }
+
+    mutate {
+      remove_field => "message"
+    }
+
+    if [first_time_detected] {
+      date {
+        match => [ "first_time_detected", "dd MMM yyyy HH:mma 'GMT'ZZ", "dd MMM yyyy HH:mma 'GMT'" ]
+        target => "first_time_detected"
+      }
+    }
+    if [first_time_tested] {
+      date {
+        match => [ "first_time_tested", "dd MMM yyyy HH:mma 'GMT'ZZ", "dd MMM yyyy HH:mma 'GMT'" ]
+        target => "first_time_tested"
+      }
+    }
+    if [last_time_detected] {
+      date {
+        match => [ "last_time_detected", "dd MMM yyyy HH:mma 'GMT'ZZ", "dd MMM yyyy HH:mma 'GMT'" ]
+        target => "last_time_detected"
+      }
+    }
+    if [last_time_tested] {
+      date {
+        match => [ "last_time_tested", "dd MMM yyyy HH:mma 'GMT'ZZ", "dd MMM yyyy HH:mma 'GMT'" ]
+        target => "last_time_tested"
+      }
+    }
+
+    # TODO remove when @timestamp is included in event
+    date {
+      match => [ "last_updated", "UNIX" ]
+      target => "@timestamp"
+      remove_field => "last_updated"
+    }
+    mutate {
+      convert => { "plugin_id" => "integer"}
+      convert => { "id" => "integer"}
+      convert => { "risk_number" => "integer"}
+      convert => { "risk_score" => "float"}
+      convert => { "total_times_detected" => "integer"}
+      convert => { "cvss_temporal" => "float"}
+      convert => { "cvss" => "float"}
+    }
+    if [risk_score] == 0 {
+      mutate {
+        add_field => { "risk_score_name" => "info" }
+      }
+    }
+    if [risk_score] > 0 and [risk_score] < 3 {
+      mutate {
+        add_field => { "risk_score_name" => "low" }
+      }
+    }
+    if [risk_score] >= 3 and [risk_score] < 6 {
+      mutate {
+        add_field => { "risk_score_name" => "medium" }
+      }
+    }
+    if [risk_score] >=6 and [risk_score] < 9 {
+      mutate {
+        add_field => { "risk_score_name" => "high" }
+      }
+    }
+    if [risk_score] >= 9 {
+      mutate {
+        add_field => { "risk_score_name" => "critical" }
+      }
+    }
+
+    if [asset] =~ "\.yourdomain\.(com|net)$" {
+      mutate {
+        add_tag => [ "critical_asset" ]
+      }
+    }
+  }
+}
+output {
+  if "qualys" in [tags] {
+    stdout {
+      codec => dots
+    }
+    elasticsearch {
+      hosts => [ "elasticsearch:9200" ]
+      index => "logstash-vulnwhisperer-%{+YYYY.MM}"
+    }
+  }
+}

resources/elk6/pipeline/3000_openvas.conf

@@ -0,0 +1,154 @@
+# Author: Austin Taylor and Justin Henderson
+# Email: austin@hasecuritysolutions.com
+# Last Update: 03/04/2018
+# Version 0.3
+# Description: Take in qualys web scan reports from vulnWhisperer and pumps into logstash
+
+input {
+  file {
+    path => "/opt/VulnWhisperer/data/openvas/*.json"
+    type => json
+    codec => json
+    start_position => "beginning"
+    tags => [ "openvas_scan", "openvas" ]
+    mode => "read"
+    start_position => "beginning"
+    file_completed_action => "delete"
+
+  }
+}
+
+filter {
+  if "openvas_scan" in [tags] {
+    mutate {
+      replace => [ "message", "%{message}" ]
+      gsub => [
+        "message", "\|\|\|", " ",
+        "message", "\t\t", " ",
+        "message", "    ", " ",
+        "message", "   ", " ",
+        "message", "  ", " ",
+        "message", "nan", " ",
+        "message",'\n',''
+      ]
+    }
+
+
+    grok {
+        match => { "path" => "openvas_scan_%{DATA:scan_id}_%{INT:last_updated}.json$" }
+        tag_on_failure => []
+    }
+
+    mutate {
+      add_field => { "risk_score" => "%{cvss}" }
+    }
+
+    if [risk] == "1" {
+      mutate { add_field => { "risk_number" => 0 }}
+      mutate { replace => { "risk" => "info" }}
+    }
+    if [risk] == "2" {
+      mutate { add_field => { "risk_number" => 1 }}
+      mutate { replace => { "risk" => "low" }}
+    }
+    if [risk] == "3" {
+      mutate { add_field => { "risk_number" => 2 }}
+      mutate { replace => { "risk" => "medium" }}
+    }
+    if [risk] == "4" {
+      mutate { add_field => { "risk_number" => 3 }}
+      mutate { replace => { "risk" => "high" }}
+    }
+    if [risk] == "5" {
+      mutate { add_field => { "risk_number" => 4 }}
+      mutate { replace => { "risk" => "critical" }}
+    }
+
+    mutate {
+      remove_field => "message"
+    }
+
+    if [first_time_detected] {
+      date {
+        match => [ "first_time_detected", "dd MMM yyyy HH:mma 'GMT'ZZ", "dd MMM yyyy HH:mma 'GMT'" ]
+        target => "first_time_detected"
+      }
+    }
+    if [first_time_tested] {
+      date {
+        match => [ "first_time_tested", "dd MMM yyyy HH:mma 'GMT'ZZ", "dd MMM yyyy HH:mma 'GMT'" ]
+        target => "first_time_tested"
+      }
+    }
+    if [last_time_detected] {
+      date {
+        match => [ "last_time_detected", "dd MMM yyyy HH:mma 'GMT'ZZ", "dd MMM yyyy HH:mma 'GMT'" ]
+        target => "last_time_detected"
+      }
+    }
+    if [last_time_tested] {
+      date {
+        match => [ "last_time_tested", "dd MMM yyyy HH:mma 'GMT'ZZ", "dd MMM yyyy HH:mma 'GMT'" ]
+        target => "last_time_tested"
+      }
+    }
+
+    # TODO remove when @timestamp is included in event
+    date {
+      match => [ "last_updated", "UNIX" ]
+      target => "@timestamp"
+      remove_field => "last_updated"
+    }
+    mutate {
+      convert => { "plugin_id" => "integer"}
+      convert => { "id" => "integer"}
+      convert => { "risk_number" => "integer"}
+      convert => { "risk_score" => "float"}
+      convert => { "total_times_detected" => "integer"}
+      convert => { "cvss_temporal" => "float"}
+      convert => { "cvss" => "float"}
+    }
+    if [risk_score] == 0 {
+      mutate {
+        add_field => { "risk_score_name" => "info" }
+      }
+    }
+    if [risk_score] > 0 and [risk_score] < 3 {
+      mutate {
+        add_field => { "risk_score_name" => "low" }
+      }
+    }
+    if [risk_score] >= 3 and [risk_score] < 6 {
+      mutate {
+        add_field => { "risk_score_name" => "medium" }
+      }
+    }
+    if [risk_score] >=6 and [risk_score] < 9 {
+      mutate {
+        add_field => { "risk_score_name" => "high" }
+      }
+    }
+    if [risk_score] >= 9 {
+      mutate {
+        add_field => { "risk_score_name" => "critical" }
+      }
+    }
+    # Add your critical assets by subnet or by hostname. Comment this field out if you don't want to tag any, but the asset panel will break.
+    if [asset] =~ "^10\.0\.100\." {
+      mutate {
+        add_tag => [ "critical_asset" ]
+      }
+    }
+  }
+}
+output {
+  if "openvas" in [tags] {
+    stdout {
+      codec => dots
+    }
+    elasticsearch {
+      hosts => [ "elasticsearch:9200" ]
+      index => "logstash-vulnwhisperer-%{+YYYY.MM}"
+    }
+  }
+}

resources/elk6/pipeline/4000_jira.conf

@@ -0,0 +1,25 @@
+# Description: Take in jira tickets from vulnWhisperer and pumps into logstash
+
+input {
+  file {
+    path => "/opt/VulnWhisperer/data/jira/*.json"
+    type => json
+    codec => json
+    start_position => "beginning"
+    mode => "read"
+    start_position => "beginning"
+    file_completed_action => "delete"
+    
+    tags => [ "jira" ]
+  }
+}
+
+output {
+  if "jira" in [tags] {
+    stdout { codec => rubydebug }
+    elasticsearch {
+      hosts => [ "elasticsearch:9200" ]
+      index => "logstash-vulnwhisperer-%{+YYYY.MM}"
+    }
+  }
+}

resources/elk6/vulnwhisperer.ini

@@ -0,0 +1,109 @@
+[nessus]
+enabled=true
+hostname=localhost
+port=8834
+username=nessus_username
+password=nessus_password
+write_path=/opt/VulnWhisperer/data/nessus/
+db_path=/opt/VulnWhisperer/database
+trash=false
+verbose=true
+
+[tenable]
+enabled=true
+hostname=cloud.tenable.com
+port=443
+username=tenable.io_username
+password=tenable.io_password
+write_path=/opt/VulnWhisperer/data/tenable/
+db_path=/opt/VulnWhisperer/data/database
+trash=false
+verbose=true
+
+[qualys_web]
+#Reference https://www.qualys.com/docs/qualys-was-api-user-guide.pdf to find your API
+enabled = true
+hostname = qualysapi.qg2.apps.qualys.com
+username = exampleuser
+password = examplepass
+write_path=/opt/VulnWhisperer/data/qualys/
+db_path=/opt/VulnWhisperer/data/database
+verbose=true
+
+# Set the maximum number of retries each connection should attempt.
+#Note, this applies only to failed connections and timeouts, never to requests where the server returns a response.
+max_retries = 10
+# Template ID will need to be retrieved for each document. Please follow the reference guide above for instructions on how to get your template ID.
+template_id = 126024
+
+[qualys_vuln]
+#Reference https://www.qualys.com/docs/qualys-was-api-user-guide.pdf to find your API
+enabled = true
+hostname = qualysapi.qg2.apps.qualys.com
+username = exampleuser
+password = examplepass
+write_path=/opt/VulnWhisperer/data/qualys/
+db_path=/opt/VulnWhisperer/data/database
+verbose=true
+
+# Set the maximum number of retries each connection should attempt.
+#Note, this applies only to failed connections and timeouts, never to requests where the server returns a response.
+max_retries = 10
+# Template ID will need to be retrieved for each document. Please follow the reference guide above for instructions on how to get your template ID.
+template_id = 126024
+
+[detectify]
+#Reference https://developer.detectify.com/
+enabled = false
+hostname = api.detectify.com
+#username variable used as apiKey
+username = exampleuser
+#password variable used as secretKey
+password = examplepass
+write_path =/opt/VulnWhisperer/data/detectify/
+db_path = /opt/VulnWhisperer/data/database
+verbose = true
+
+[openvas]
+enabled = false
+hostname = localhost
+port = 4000
+username = exampleuser
+password = examplepass
+write_path=/opt/VulnWhisperer/data/openvas/
+db_path=/opt/VulnWhisperer/data/database
+verbose=true
+
+#[proxy]
+; This section is optional. Leave it out if you're not using a proxy.
+; You can use environmental variables as well: http://www.python-requests.org/en/latest/user/advanced/#proxies
+
+; proxy_protocol set to https, if not specified.
+#proxy_url = proxy.mycorp.com
+
+; proxy_port will override any port specified in proxy_url
+#proxy_port = 8080
+
+; proxy authentication
+#proxy_username = proxyuser
+#proxy_password = proxypass
+
+[jira]
+hostname = jira-host
+username = username
+password = password
+write_path = /opt/VulnWhisperer/data/jira/
+db_path = /opt/VulnWhisperer/data/database
+verbose = true
+dns_resolv = False
+
+#Sample jira report scan, will automatically be created for existent scans
+#[jira.qualys_vuln.test_scan]
+#source = qualys_vuln
+#scan_name = Test Scan
+#jira_project = PROJECT
+; if multiple components, separate by "," = None
+#components = 
+; minimum criticality to report (low, medium, high or critical) = None
+#min_critical_to_report = high
+

setup.py

@@ -4,7 +4,7 @@ from setuptools import setup, find_packages
 
 setup(
     name='VulnWhisperer',
-    version='1.5.0',
+    version='1.8',
     packages=find_packages(),
     url='https://github.com/austin-taylor/vulnwhisperer',
     license="""MIT License
@@ -26,7 +26,7 @@ setup(
     SOFTWARE.""",
     author='Austin Taylor',
     author_email='email@austintaylor.io',
-    description='Vulnerability assessment framework aggregator',
+    description='Vulnerability Assessment Framework Aggregator',
     scripts=['bin/vuln_whisperer']
 )
 

tests/test-docker.sh

@@ -0,0 +1,109 @@
+#!/usr/bin/env bash
+
+NORMAL=$(tput sgr0)
+GREEN=$(tput setaf 2)
+YELLOW=$(tput setaf 3)
+RED=$(tput setaf 1)
+
+function red() {
+    echo -e "$RED$*$NORMAL"
+}
+
+function green() {
+    echo -e "$GREEN$*$NORMAL"
+}
+
+function yellow() {
+    echo -e "$YELLOW$*$NORMAL"
+}
+
+return_code=0
+
+elasticsearch_url="localhost:9200"
+logstash_url="localhost:9600"
+
+until curl -s "$elasticsearch_url/_cluster/health?pretty" | grep '"status"' | grep -qE "green|yellow"; do
+    yellow "Waiting for Elasticsearch..."
+    sleep 5
+done
+green "✅ Elasticsearch status is green..."
+
+count=0
+until [[ $(curl -s "$logstash_url/_node/stats" | jq '.events.out') -ge 1236 ]]; do
+    yellow "Waiting for Logstash load to finish...  $(curl -s "$logstash_url/_node/stats" | jq '.events.out') of 1236 (attempt $count of 60)"
+    ((count++)) && ((count==60)) && break
+    sleep 5
+done
+
+if [[ count -le 60 && $(curl -s "$logstash_url/_node/stats" | jq '.events.out') -ge 1236 ]]; then
+    green "✅ Logstash load finished..."
+else
+    red "❌ Logstash load didn't complete... $(curl -s "$logstash_url/_node/stats" | jq '.events.out')"
+fi
+
+
+count=0
+until [[ $(curl -s "$elasticsearch_url/logstash-vulnwhisperer-2019.03/_count" | jq '.count') -ge 1232 ]] ; do
+    yellow "Waiting for Elasticsearch index to sync... $(curl -s "$elasticsearch_url/logstash-vulnwhisperer-2019.03/_count" | jq '.count') of 1232 logs loaded (attempt $count of 150)"
+    ((count++)) && ((count==150)) && break
+    sleep 2
+done
+if [[ count -le 50 && $(curl -s "$elasticsearch_url/logstash-vulnwhisperer-2019.03/_count" | jq '.count') -ge 1232 ]]; then
+    green "✅ logstash-vulnwhisperer-2019.03 document count >= 1232"
+else
+    red "❌ TIMED OUT waiting for logstash-vulnwhisperer-2019.03 document count: $(curl -s "$elasticsearch_url/logstash-vulnwhisperer-2019.03/_count" | jq) != 1232"
+fi
+
+# if [[ $(curl -s "$elasticsearch_url/logstash-vulnwhisperer-2019.03/_count" | jq '.count') == 1232 ]]; then
+#     green "✅ Passed: logstash-vulnwhisperer-2019.03 document count == 1232"
+# else
+#     red "❌ Failed: logstash-vulnwhisperer-2019.03 document count == 1232 was: $(curl -s "$elasticsearch_url/logstash-vulnwhisperer-2019.03/_count") instead"
+#     ((return_code = return_code + 1))
+# fi
+
+# Test Nessus plugin_name:Backported Security Patch Detection (FTP)
+nessus_doc=$(curl -s "$elasticsearch_url/logstash-vulnwhisperer-2019.03/_search?q=plugin_name:%22Backported%20Security%20Patch%20Detection%20(FTP)%22%20AND%20asset:176.28.50.164%20AND%20tags:nessus" | jq '.hits.hits[]._source')
+if echo $nessus_doc | jq '.risk' | grep -q "None"; then
+    green "✅ Passed: Nessus risk == None"
+else
+    red "❌ Failed: Nessus risk == None was: $(echo $nessus_doc | jq '.risk') instead"
+    ((return_code = return_code + 1))
+fi
+
+# Test Tenable plugin_name:Backported Security Patch Detection (FTP)
+tenable_doc=$(curl -s "$elasticsearch_url/logstash-vulnwhisperer-2019.03/_search?q=plugin_name:%22Backported%20Security%20Patch%20Detection%20(FTP)%22%20AND%20asset:176.28.50.164%20AND%20tags:tenable" | jq '.hits.hits[]._source')
+# Test asset
+if echo $tenable_doc | jq .asset | grep -q '176.28.50.164'; then
+    green "✅ Passed: Tenable asset == 176.28.50.164"
+else
+    red "❌ Failed: Tenable asset == 176.28.50.164 was: $(echo $tenable_doc | jq .asset) instead"
+    ((return_code = return_code + 1))
+fi
+
+# Test @timestamp
+if echo $tenable_doc | jq '.["@timestamp"]' | grep -q '2019-03-30T15:45:44.000Z'; then
+    green "✅ Passed: Tenable @timestamp == 2019-03-30T15:45:44.000Z"
+else
+    red "❌ Failed: Tenable @timestamp == 2019-03-30T15:45:44.000Z was: $(echo $tenable_doc | jq '.["@timestamp"]') instead"
+    ((return_code = return_code + 1))
+fi
+
+# Test Qualys plugin_name:OpenSSL Multiple Remote Security Vulnerabilities
+qualys_vuln_doc=$(curl -s "$elasticsearch_url/logstash-vulnwhisperer-2019.03/_search?q=tags:qualys_vuln%20AND%20ip:%22176.28.50.164%22%20AND%20plugin_name:%22OpenSSL%20Multiple%20Remote%20Security%20Vulnerabilities%22%20AND%20port:465" | jq '.hits.hits[]._source')
+# Test @timestamp
+if echo $qualys_vuln_doc | jq '.["@timestamp"]' | grep -q '2019-03-30T10:17:41.000Z'; then
+    green "✅ Passed: Qualys VM @timestamp == 2019-03-30T10:17:41.000Z"
+else
+    red "❌ Failed: Qualys VM @timestamp == 2019-03-30T10:17:41.000Z was: $(echo $qualys_vuln_doc | jq '.["@timestamp"]') instead"
+    ((return_code = return_code + 1))
+fi
+
+# Test @XXXX
+if echo $qualys_vuln_doc | jq '.cvss' | grep -q '6.8'; then
+    green "✅ Passed: Qualys VM cvss == 6.8"
+else
+    red "❌ Failed: Qualys VM cvss == 6.8 was: $(echo $qualys_vuln_doc | jq '.cvss') instead"
+    ((return_code = return_code + 1))
+fi
+
+exit $return_code

tests/test-vuln_whisperer.sh

@@ -0,0 +1,97 @@
+#!/usr/bin/env bash
+
+NORMAL=$(tput sgr0)
+GREEN=$(tput setaf 2)
+YELLOW=$(tput setaf 3)
+RED=$(tput setaf 1)
+
+function red() {
+    echo -e "$RED$*$NORMAL"
+}
+
+function green() {
+    echo -e "$GREEN$*$NORMAL"
+}
+
+function yellow() {
+    echo -e "$YELLOW$*$NORMAL"
+}
+
+return_code=0
+
+TEST_PATH=${TEST_PATH:-"tests/data"}
+
+yellow "\n*********************************************"
+yellow "* Test successful scan download and parsing *"
+yellow "*********************************************"
+rm -rf /opt/VulnWhisperer/*
+if vuln_whisperer -F -c configs/test.ini --mock --mock_dir "${TEST_PATH}"; then
+    green "\n✅ Passed: Test successful scan download and parsing"
+else
+    red "\n❌ Failed: Test successful scan download and parsing"
+    ((return_code = return_code + 1))
+fi
+
+yellow "\n*********************************************"
+yellow "*    Test run with no scans to import       *"
+yellow "*********************************************"
+if vuln_whisperer -F -c configs/test.ini --mock --mock_dir "${TEST_PATH}"; then
+    green "\n✅ Passed: Test run with no scans to import"
+else
+    red "\n❌ Failed: Test run with no scans to import"
+    ((return_code = return_code + 1))
+fi
+
+yellow "\n*********************************************"
+yellow "*           Test one failed scan            *"
+yellow "*********************************************"
+rm -rf /opt/VulnWhisperer/*
+yellow "Removing ${TEST_PATH}/nessus/GET_scans_exports_164_download"
+mv "${TEST_PATH}/nessus/GET_scans_exports_164_download"{,.bak}
+if vuln_whisperer -F -c configs/test.ini --mock --mock_dir "${TEST_PATH}"; [[ $? -eq 1 ]]; then
+    green "\n✅ Passed: Test one failed scan"
+else
+    red "\n❌ Failed: Test one failed scan"
+    ((return_code = return_code + 1))
+fi
+
+yellow "\n*********************************************"
+yellow "*           Test two failed scans           *"
+yellow "*********************************************"
+rm -rf /opt/VulnWhisperer/*
+yellow "Removing ${TEST_PATH}/qualys_vuln/scan_1553941061.87241"
+mv "${TEST_PATH}/qualys_vuln/scan_1553941061.87241"{,.bak}
+if vuln_whisperer -F -c configs/test.ini --mock --mock_dir "${TEST_PATH}"; [[ $? -eq 2 ]]; then
+    green "\n✅ Passed: Test two failed scans"
+else
+    red "\n❌ Failed: Test two failed scans"
+    ((return_code = return_code + 1))
+fi
+
+yellow "\n*********************************************"
+yellow "*   Test only nessus with one failed scan   *"
+yellow "*********************************************"
+rm -rf /opt/VulnWhisperer/*
+if vuln_whisperer -F -c configs/test.ini -s nessus --mock --mock_dir "${TEST_PATH}"; [[ $? -eq 1 ]]; then
+    green "\n✅ Passed: Test only nessus with one failed scan"
+else
+    red "\n❌ Failed: Test only nessus with one failed scan"
+    ((return_code = return_code + 1))
+fi
+
+yellow "*********************************************"
+yellow "* Test only Qualys VM with one failed scan  *"
+yellow "*********************************************"
+rm -rf /opt/VulnWhisperer/*
+if vuln_whisperer -F -c configs/test.ini -s qualys_vuln --mock --mock_dir "${TEST_PATH}"; [[ $? -eq 1 ]]; then
+    green "\n✅ Passed: Test only Qualys VM with one failed scan"
+else
+    red "\n❌ Failed: Test only Qualys VM with one failed scan"
+    ((return_code = return_code + 1))
+fi
+
+# Restore the removed files
+mv "${TEST_PATH}/qualys_vuln/scan_1553941061.87241.bak" "${TEST_PATH}/qualys_vuln/scan_1553941061.87241"
+mv "${TEST_PATH}/nessus/GET_scans_exports_164_download.bak" "${TEST_PATH}/nessus/GET_scans_exports_164_download"
+
+exit $return_code

vulnwhisp/__init__.py

@@ -1 +0,0 @@
-from utils.cli import bcolors

vulnwhisp/base/config.py

@@ -1,8 +1,8 @@
-import os
 import sys
+import logging
 
 # Support for python3
-if (sys.version_info > (3, 0)):
+if sys.version_info > (3, 0):
     import configparser as cp
 else:
     import ConfigParser as cp
@@ -14,17 +14,70 @@ class vwConfig(object):
         self.config_in = config_in
         self.config = cp.RawConfigParser()
         self.config.read(self.config_in)
+        self.logger = logging.getLogger('vwConfig')
 
     def get(self, section, option):
+        self.logger.debug('Calling get for {}:{}'.format(section, option))
         return self.config.get(section, option)
 
     def getbool(self, section, option):
+        self.logger.debug('Calling getbool for {}:{}'.format(section, option))
         return self.config.getboolean(section, option)
 
-    def get_enabled(self):
-	enabled = []
-	check = ["true", "True", "1"]
-	for section in self.config.sections():
-	    if self.get(section, "enabled") in check:
-		enabled.append(section)	
-	return enabled
+    def get_sections_with_attribute(self, attribute):
+        sections = []
+        # TODO: does this not also need the "yes" case?
+        check = ["true", "True", "1"]
+        for section in self.config.sections():
+            try:
+                if self.get(section, attribute) in check:
+                    sections.append(section)
+            except:
+                self.logger.warn("Section {} has no option '{}'".format(section, attribute))
+        return sections
+
+    def exists_jira_profiles(self, profiles):
+        # get list of profiles source_scanner.scan_name
+        for profile in profiles:
+            if not self.config.has_section(self.normalize_section(profile)):
+                self.logger.warn("JIRA Scan Profile missing")
+                return False
+        return True
+
+    def update_jira_profiles(self, profiles):
+        # create JIRA profiles in the ini config file
+        self.logger.debug('Updating Jira profiles: {}'.format(str(profiles)))
+
+        for profile in profiles:
+            #IMPORTANT profile scans/results will be normalized to lower and "_" instead of spaces for ini file section
+            section_name = self.normalize_section(profile)
+            try:
+                self.get(section_name, "source")
+                self.logger.info("Skipping creating of section '{}'; already exists".format(section_name))
+            except:
+                self.logger.warn("Creating config section for '{}'".format(section_name))
+                self.config.add_section(section_name)
+                self.config.set(section_name, 'source', profile.split('.')[0])
+                # in case any scan name contains '.' character
+                self.config.set(section_name, 'scan_name', '.'.join(profile.split('.')[1:]))
+                self.config.set(section_name, 'jira_project', '')
+                self.config.set(section_name, '; if multiple components, separate by ","')
+                self.config.set(section_name, 'components', '')
+                self.config.set(section_name, '; minimum criticality to report (low, medium, high or critical)')
+                self.config.set(section_name, 'min_critical_to_report', 'high')
+                self.config.set(section_name, '; automatically report, boolean value ')
+                self.config.set(section_name, 'autoreport', 'false')
+
+        # TODO: try/catch this
+        # writing changes back to file
+        with open(self.config_in, 'w') as configfile:
+            self.config.write(configfile)
+            self.logger.debug('Written configuration to {}'.format(self.config_in))
+
+        # FIXME: this is the same as return None, that is the default return for return-less functions
+        return
+
+    def normalize_section(self, profile):
+        profile = "jira.{}".format(profile.lower().replace(" ", "_"))
+        self.logger.debug('Normalized profile as: {}'.format(profile))
+        return profile

vulnwhisp/frameworks/nessus.py

@@ -1,224 +1,184 @@
-import requests
-from requests.packages.urllib3.exceptions import InsecureRequestWarning
-requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
-
-import pytz
-from datetime import datetime
-import json
-import sys
-import time
+import json
+import logging
+import sys
+import time
+from datetime import datetime
 
+import pytz
+import requests
 from requests.packages.urllib3.exceptions import InsecureRequestWarning
+
 requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
 
-
-
-class NessusAPI(object):
-    SESSION = '/session'
-    FOLDERS = '/folders'
-    SCANS = '/scans'
-    SCAN_ID = SCANS + '/{scan_id}'
-    HOST_VULN = SCAN_ID + '/hosts/{host_id}'
-    PLUGINS = HOST_VULN + '/plugins/{plugin_id}'
-    EXPORT = SCAN_ID + '/export'
-    EXPORT_TOKEN_DOWNLOAD = '/scans/exports/{token_id}/download'
-    EXPORT_FILE_DOWNLOAD = EXPORT + '/{file_id}/download'
-    EXPORT_STATUS = EXPORT + '/{file_id}/status'
-    EXPORT_HISTORY = EXPORT + '?history_id={history_id}'
-
-    def __init__(self, hostname=None, port=None, username=None, password=None, verbose=True):
-        if username is None or password is None:
-            raise Exception('ERROR: Missing username or password.')
-
-        self.user = username
-        self.password = password
-        self.base = 'https://{hostname}:{port}'.format(hostname=hostname, port=port)
-        self.verbose = verbose
-
-        self.headers = {
-            'Origin': self.base,
-            'Accept-Encoding': 'gzip, deflate, br',
-            'Accept-Language': 'en-US,en;q=0.8',
-            'User-Agent': 'VulnWhisperer for Nessus',
-            'Content-Type': 'application/json',
-            'Accept': 'application/json, text/javascript, */*; q=0.01',
-            'Referer': self.base,
-            'X-Requested-With': 'XMLHttpRequest',
-            'Connection': 'keep-alive',
-            'X-Cookie': None
-        }
-
-        self.login()
-        self.scan_ids = self.get_scan_ids()
-
-    def vprint(self, msg):
-        if self.verbose:
-            print(msg)
-
-    def login(self):
-        resp = self.get_token()
-        if resp.status_code is 200:
-            self.headers['X-Cookie'] = 'token={token}'.format(token=resp.json()['token'])
-        else:
-            raise Exception('[FAIL] Could not login to Nessus')
-
-    def request(self, url, data=None, headers=None, method='POST', download=False, json=False):
-        if headers is None:
-            headers = self.headers
-        timeout = 0
-        success = False
-
-        url = self.base + url
-        methods = {'GET': requests.get,
-                   'POST': requests.post,
-                   'DELETE': requests.delete}
-
-        while (timeout <= 10) and (not success):
-            data = methods[method](url, data=data, headers=self.headers, verify=False)
-            if data.status_code == 401:
-                if url == self.base + self.SESSION:
-                    break
-                try:
-                    self.login()
-                    timeout += 1
-                    self.vprint('[INFO] Token refreshed')
-                except Exception as e:
-                    self.vprint('[FAIL] Could not refresh token\nReason: %s' % e)
-            else:
-                success = True
-
-        if json:
-            data = data.json()
-        if download:
-            return data.content
-        return data
-
-    def get_token(self):
-        auth = '{"username":"%s", "password":"%s"}' % (self.user, self.password)
-        token = self.request(self.SESSION, data=auth, json=False)
-        return token
-
-    def logout(self):
-        self.request(self.SESSION, method='DELETE')
-
-    def get_folders(self):
-        folders = self.request(self.FOLDERS, method='GET', json=True)
-        return folders
-
-    def get_scans(self):
-        scans = self.request(self.SCANS, method='GET', json=True)
-        return scans
-
-    def get_scan_ids(self):
-        scans = self.get_scans()
-        scan_ids = [scan_id['id'] for scan_id in scans['scans']] if scans['scans'] else []
-        return scan_ids
-
-    def count_scan(self, scans, folder_id):
-        count = 0
-        for scan in scans:
-            if scan['folder_id'] == folder_id: count = count + 1
-        return count
-
-    def print_scans(self, data):
-        for folder in data['folders']:
-            print("\\{0} - ({1})\\".format(folder['name'], self.count_scan(data['scans'], folder['id'])))
-            for scan in data['scans']:
-                if scan['folder_id'] == folder['id']:
-                    print(
-                    "\t\"{0}\" - sid:{1} - uuid: {2}".format(scan['name'].encode('utf-8'), scan['id'], scan['uuid']))
-
-    def get_scan_details(self, scan_id):
-        data = self.request(self.SCAN_ID.format(scan_id=scan_id), method='GET', json=True)
-        return data
-
-    def get_scan_history(self, scan_id):
-        data = self.request(self.SCAN_ID.format(scan_id=scan_id), method='GET', json=True)
-        return data['history']
-
-    def get_scan_hosts(self, scan_id):
-        data = self.request(self.SCAN_ID.format(scan_id=scan_id), method='GET', json=True)
-        return data['hosts']
-
-    def get_host_vulnerabilities(self, scan_id, host_id):
-        query = self.HOST_VULN.format(scan_id=scan_id, host_id=host_id)
-        data = self.request(query, method='GET', json=True)
-        return data
-
-    def get_plugin_info(self, scan_id, host_id, plugin_id):
-        query = self.PLUGINS.format(scan_id=scan_id, host_id=host_id, plugin_id=plugin_id)
-        data = self.request(query, method='GET', json=True)
-        return data
-
-    def export_scan(self, scan_id, history_id):
-        data = {'format': 'csv'}
-        query = self.EXPORT_REPORT.format(scan_id=scan_id, history_id=history_id)
-        req = self.request(query, data=data, method='POST')
-        return req
-
-    def download_scan(self, scan_id=None, history=None, export_format="", chapters="", dbpasswd="", profile=""):
-        running = True
-        counter = 0
-
-        data = {'format': export_format}
-        if not history:
-            query = self.EXPORT.format(scan_id=scan_id)
-        else:
-            query = self.EXPORT_HISTORY.format(scan_id=scan_id, history_id=history)
-            scan_id = str(scan_id)
-        req = self.request(query, data=json.dumps(data), method='POST', json=True)
-        try:
-            file_id = req['file']
-            token_id = req['token'] if 'token' in req else req['temp_token']
-        except Exception as e:
-            print("[ERROR] %s" % e)
-        print('Download for file id ' + str(file_id) + '.')
-        while running:
-            time.sleep(2)
-            counter += 2
-            report_status = self.request(self.EXPORT_STATUS.format(scan_id=scan_id, file_id=file_id), method='GET',
-                                         json=True)
-            running = report_status['status'] != 'ready'
-            sys.stdout.write(".")
-            sys.stdout.flush()
-            if counter % 60 == 0:
-                print("")
-
-        print("")
-        if profile=='tenable':
-            content = self.request(self.EXPORT_FILE_DOWNLOAD.format(scan_id=scan_id, file_id=file_id), method='GET', download=True)
-        else:
-            content = self.request(self.EXPORT_TOKEN_DOWNLOAD.format(token_id=token_id), method='GET', download=True)
-        return content
-
-    @staticmethod
-    def merge_dicts(self, *dict_args):
-        """
-        Given any number of dicts, shallow copy and merge into a new dict,
-        precedence goes to key value pairs in latter dicts.
-        """
-        result = {}
-        for dictionary in dict_args:
-            result.update(dictionary)
-        return result
-
-    def get_utc_from_local(self, date_time, local_tz=None, epoch=True):
-        date_time = datetime.fromtimestamp(date_time)
-        if local_tz is None:
-            local_tz = pytz.timezone('US/Central')
-        else:
-            local_tz = pytz.timezone(local_tz)
-        # print date_time
-        local_time = local_tz.normalize(local_tz.localize(date_time))
-        local_time = local_time.astimezone(pytz.utc)
-        if epoch:
-            naive = local_time.replace(tzinfo=None)
-            local_time = int((naive - datetime(1970, 1, 1)).total_seconds())
-        return local_time
-
-    def tz_conv(self, tz):
-        time_map = {'Eastern Standard Time': 'US/Eastern',
-                    'Central Standard Time': 'US/Central',
-                    'Pacific Standard Time': 'US/Pacific',
-                    'None': 'US/Central'}
-        return time_map.get(tz, None)
+
+class NessusAPI(object):
+    SESSION = '/session'
+    FOLDERS = '/folders'
+    SCANS = '/scans'
+    SCAN_ID = SCANS + '/{scan_id}'
+    HOST_VULN = SCAN_ID + '/hosts/{host_id}'
+    PLUGINS = HOST_VULN + '/plugins/{plugin_id}'
+    EXPORT = SCAN_ID + '/export'
+    EXPORT_TOKEN_DOWNLOAD = '/scans/exports/{token_id}/download'
+    EXPORT_FILE_DOWNLOAD = EXPORT + '/{file_id}/download'
+    EXPORT_STATUS = EXPORT + '/{file_id}/status'
+    EXPORT_HISTORY = EXPORT + '?history_id={history_id}'
+
+    def __init__(self, hostname=None, port=None, username=None, password=None, verbose=True, profile=None, access_key=None, secret_key=None):
+        self.logger = logging.getLogger('NessusAPI')
+        if verbose:
+            self.logger.setLevel(logging.DEBUG)
+        if not all((username, password)) and not all((access_key, secret_key)):
+            raise Exception('ERROR: Missing username, password or API keys.')
+
+        self.profile = profile
+        self.user = username
+        self.password = password
+        self.api_keys = False
+        self.access_key = access_key
+        self.secret_key = secret_key
+        self.base = 'https://{hostname}:{port}'.format(hostname=hostname, port=port)
+        self.verbose = verbose
+
+        self.session = requests.Session()
+        self.session.verify = False
+        self.session.stream = True
+        self.session.headers = {
+            'Origin': self.base,
+            'Accept-Encoding': 'gzip, deflate, br',
+            'Accept-Language': 'en-US,en;q=0.8',
+            'User-Agent': 'VulnWhisperer for Nessus',
+            'Content-Type': 'application/json',
+            'Accept': 'application/json, text/javascript, */*; q=0.01',
+            'Referer': self.base,
+            'X-Requested-With': 'XMLHttpRequest',
+            'Connection': 'keep-alive',
+            'X-Cookie': None
+        }
+
+        if all((self.access_key, self.secret_key)):
+            self.logger.debug('Using {} API keys'.format(self.profile))
+            self.api_keys = True
+            self.session.headers['X-ApiKeys'] = 'accessKey={}; secretKey={}'.format(self.access_key, self.secret_key)
+        else:
+            self.login()
+
+        self.scans = self.get_scans()
+        self.scan_ids = self.get_scan_ids()
+
+    def login(self):
+        auth = '{"username":"%s", "password":"%s"}' % (self.user, self.password)
+        resp = self.request(self.SESSION, data=auth, json_output=False)
+        if resp.status_code == 200:
+            self.session.headers['X-Cookie'] = 'token={token}'.format(token=resp.json()['token'])
+        else:
+            raise Exception('[FAIL] Could not login to Nessus')
+
+    def request(self, url, data=None, headers=None, method='POST', download=False, json_output=False):
+        timeout = 0
+        success = False
+
+        method = method.lower()
+        url = self.base + url
+        self.logger.debug('Requesting to url {}'.format(url))
+
+        while (timeout <= 10) and (not success):
+            response = getattr(self.session, method)(url, data=data)
+            if response.status_code == 401:
+                if url == self.base + self.SESSION:
+                    break
+                try:
+                    timeout += 1
+                    if self.api_keys:
+                        continue
+                    self.login()
+                    self.logger.info('Token refreshed')
+                except Exception as e:
+                    self.logger.error('Could not refresh token\nReason: {}'.format(str(e)))
+            else:
+                success = True
+
+        if json_output:
+            return response.json()
+        if download:
+            self.logger.debug('Returning data.content')
+            response_data = ''
+            count = 0
+            for chunk in response.iter_content(chunk_size=8192):
+                count += 1
+                if chunk:
+                    response_data += chunk
+            self.logger.debug('Processed {} chunks'.format(count))
+            return response_data
+        return response
+
+    def get_scans(self):
+        scans = self.request(self.SCANS, method='GET', json_output=True)
+        return scans
+
+    def get_scan_ids(self):
+        scans = self.scans
+        scan_ids = [scan_id['id'] for scan_id in scans['scans']] if scans['scans'] else []
+        self.logger.debug('Found {} scan_ids'.format(len(scan_ids)))
+        return scan_ids
+
+    def get_scan_history(self, scan_id):
+        data = self.request(self.SCAN_ID.format(scan_id=scan_id), method='GET', json_output=True)
+        return data['history']
+
+    def download_scan(self, scan_id=None, history=None, export_format=""):
+        running = True
+        counter = 0
+
+        data = {'format': export_format}
+        if not history:
+            query = self.EXPORT.format(scan_id=scan_id)
+        else:
+            query = self.EXPORT_HISTORY.format(scan_id=scan_id, history_id=history)
+            scan_id = str(scan_id)
+        req = self.request(query, data=json.dumps(data), method='POST', json_output=True)
+        try:
+            file_id = req['file']
+            if self.profile == 'nessus':
+                token_id = req['token'] if 'token' in req else req['temp_token']
+        except Exception as e:
+            self.logger.error('{}'.format(str(e)))
+        self.logger.info('Download for file id {}'.format(str(file_id)))
+        while running:
+            time.sleep(2)
+            counter += 2
+            report_status = self.request(self.EXPORT_STATUS.format(scan_id=scan_id, file_id=file_id), method='GET',
+                                         json_output=True)
+            running = report_status['status'] != 'ready'
+            sys.stdout.write(".")
+            sys.stdout.flush()
+            # FIXME: why? can this be removed in favour of a counter?
+            if counter % 60 == 0:
+                self.logger.info("Completed: {}".format(counter))
+        self.logger.info("Done: {}".format(counter))
+        if self.profile == 'tenable' or self.api_keys:
+            content = self.request(self.EXPORT_FILE_DOWNLOAD.format(scan_id=scan_id, file_id=file_id), method='GET', download=True)
+        else:
+            content = self.request(self.EXPORT_TOKEN_DOWNLOAD.format(token_id=token_id), method='GET', download=True)
+        return content
+
+    def get_utc_from_local(self, date_time, local_tz=None, epoch=True):
+        date_time = datetime.fromtimestamp(date_time)
+        if local_tz is None:
+            local_tz = pytz.timezone('UTC')
+        else:
+            local_tz = pytz.timezone(local_tz)
+        local_time = local_tz.normalize(local_tz.localize(date_time))
+        local_time = local_time.astimezone(pytz.utc)
+        if epoch:
+            naive = local_time.replace(tzinfo=None)
+            local_time = int((naive - datetime(1970, 1, 1)).total_seconds())
+        self.logger.debug('Converted timestamp {} in datetime {}'.format(date_time, local_time))
+        return local_time
+
+    def tz_conv(self, tz):
+        time_map = {'Eastern Standard Time': 'US/Eastern',
+                    'Central Standard Time': 'US/Central',
+                    'Pacific Standard Time': 'US/Pacific',
+                    'None': 'US/Central'}
+        return time_map.get(tz, None)

vulnwhisp/frameworks/openvas.py

@@ -4,11 +4,11 @@ __author__ = 'Austin Taylor'
 
 import datetime as dt
 import io
+import logging
 
 import pandas as pd
 import requests
 from bs4 import BeautifulSoup
-from ..utils.cli import bcolors
 
 
 class OpenVAS_API(object):
@@ -21,6 +21,9 @@ class OpenVAS_API(object):
                  password=None,
                  report_format_id=None,
                  verbose=True):
+        self.logger = logging.getLogger('OpenVAS_API')
+        if verbose:
+            self.logger.setLevel(logging.DEBUG)
         if username is None or password is None:
             raise Exception('ERROR: Missing username or password.')
 
@@ -49,10 +52,6 @@ class OpenVAS_API(object):
         self.openvas_reports = self.get_reports()
         self.report_formats = self.get_report_formats()
 
-    def vprint(self, msg):
-        if self.verbose:
-            print(msg)
-
     def login(self):
         resp = self.get_token()
         if resp.status_code is 200:
@@ -90,9 +89,9 @@ class OpenVAS_API(object):
                 try:
                     self.login()
                     timeout += 1
-                    self.vprint('[INFO] Token refreshed')
+                    self.logger.info(' Token refreshed')
                 except Exception as e:
-                    self.vprint('[FAIL] Could not refresh token\nReason: %s' % e)
+                    self.logger.error('Could not refresh token\nReason: {}'.format(str(e)))
             else:
                 success = True
 
@@ -116,7 +115,7 @@ class OpenVAS_API(object):
             ('cmd', 'get_report_formats'),
             ('token', self.token)
         )
-        self.vprint('{info} Retrieving available report formats'.format(info=bcolors.INFO))
+        self.logger.info('Retrieving available report formats')
         data = self.request(url=self.OMP, method='GET', params=params)
 
         bs = BeautifulSoup(data.content, "lxml")
@@ -133,7 +132,7 @@ class OpenVAS_API(object):
         return format_mapping
 
     def get_reports(self, complete=True):
-        print('{info} Retreiving OpenVAS report data...'.format(info=bcolors.INFO))
+        self.logger.info('Retreiving OpenVAS report data...')
         params = (('cmd', 'get_reports'),
                   ('token', self.token),
                   ('max_results', 1),
@@ -184,7 +183,7 @@ class OpenVAS_API(object):
             ('report_format_id', '{report_format_id}'.format(report_format_id=self.report_formats['CSV Results'])),
             ('submit', 'Download'),
         )
-        print('Retrieving %s' % report_id)
+        self.logger.info('Retrieving {}'.format(report_id))
         req = self.request(self.OMP, params=params, method='GET')
         report_df = pd.read_csv(io.BytesIO(req.text.encode('utf-8')))
         report_df['report_ids'] = report_id

vulnwhisp/frameworks/qualys.py

@@ -1,837 +0,0 @@
-#!/usr/bin/python
-# -*- coding: utf-8 -*-
-__author__ = 'Austin Taylor'
-
-from lxml import objectify
-from lxml.builder import E
-import xml.etree.ElementTree as ET
-import pandas as pd
-import qualysapi
-import qualysapi.config as qcconf
-import requests
-import sys
-import os
-import csv
-import dateutil.parser as dp
-
-
-class qualysWhisperAPI(object):
-    COUNT_WEBAPP = '/count/was/webapp'
-    COUNT_WASSCAN = '/count/was/wasscan'
-    DELETE_REPORT = '/delete/was/report/{report_id}'
-    GET_WEBAPP_DETAILS = '/get/was/webapp/{was_id}'
-    QPS_REST_3 = '/qps/rest/3.0'
-    REPORT_DETAILS = '/get/was/report/{report_id}'
-    REPORT_STATUS = '/status/was/report/{report_id}'
-    REPORT_CREATE = '/create/was/report'
-    REPORT_DOWNLOAD = '/download/was/report/{report_id}'
-    SCAN_DETAILS = '/get/was/wasscan/{scan_id}'
-    SCAN_DOWNLOAD = '/download/was/wasscan/{scan_id}'
-    SEARCH_REPORTS = '/search/was/report'
-    SEARCH_WEB_APPS = '/search/was/webapp'
-    SEARCH_WAS_SCAN = '/search/was/wasscan'
-    VERSION = '/qps/rest/portal/version'
-
-    def __init__(self, config=None):
-        self.config = config
-        try:
-            self.qgc = qualysapi.connect(config)
-            print('[SUCCESS] - Connected to Qualys at %s' % self.qgc.server)
-        except Exception as e:
-            print('[ERROR] Could not connect to Qualys - %s' % e)
-        self.headers = {
-            "content-type": "text/xml"}
-        self.config_parse = qcconf.QualysConnectConfig(config)
-        try:
-            self.template_id = self.config_parse.get_template_id()
-        except:
-            print('ERROR - Could not retrieve template ID')
-
-    def request(self, path, method='get', data=None):
-        methods = {'get': requests.get,
-                   'post': requests.post}
-        base = 'https://' + self.qgc.server + path
-        req = methods[method](base, auth=self.qgc.auth, data=data, headers=self.headers).content
-        return req
-
-    def get_version(self):
-        return self.request(self.VERSION)
-
-    def get_scan_count(self, scan_name):
-        parameters = (
-            E.ServiceRequest(
-                E.filters(
-                    E.Criteria({'field': 'name', 'operator': 'CONTAINS'}, scan_name))))
-        xml_output = self.qgc.request(self.COUNT_WEBAPP, parameters)
-        root = objectify.fromstring(xml_output)
-        return root.count.text
-
-    def get_was_scan_count(self, status):
-        parameters = (
-            E.ServiceRequest(
-                E.filters(
-                    E.Criteria({'field': 'status', 'operator': 'EQUALS'}, status))))
-        xml_output = self.qgc.request(self.COUNT_WASSCAN, parameters)
-        root = objectify.fromstring(xml_output)
-        return root.count.text
-
-    def get_reports(self):
-        return self.qgc.request(self.SEARCH_REPORTS)
-
-    def xml_parser(self, xml, dupfield=None):
-        all_records = []
-        root = ET.XML(xml)
-        for i, child in enumerate(root):
-            for subchild in child:
-                record = {}
-                dup_tracker = 0
-                for p in subchild:
-                    record[p.tag] = p.text
-                    for o in p:
-                        if o.tag in record:
-                            dup_tracker += 1
-                            record[o.tag + '_%s' % dup_tracker] = o.text
-                        else:
-                            record[o.tag] = o.text
-                all_records.append(record)
-        return pd.DataFrame(all_records)
-
-    def get_report_list(self):
-        """Returns a dataframe of reports"""
-        return self.xml_parser(self.get_reports(), dupfield='user_id')
-
-    def get_web_apps(self):
-        """Returns webapps available for account"""
-        return self.qgc.request(self.SEARCH_WEB_APPS)
-
-    def get_web_app_list(self):
-        """Returns dataframe of webapps"""
-        return self.xml_parser(self.get_web_apps(), dupfield='user_id')
-
-    def get_web_app_details(self, was_id):
-        """Get webapp details - use to retrieve app ID tag"""
-        return self.qgc.request(self.GET_WEBAPP_DETAILS.format(was_id=was_id))
-
-    def get_scans_by_app_id(self, app_id):
-        data = self.generate_app_id_scan_XML(app_id)
-        return self.qgc.request(self.SEARCH_WAS_SCAN, data)
-
-    def get_scan_info(self, limit=1000, offset=1, status='FINISHED'):
-        """ Returns XML of ALL WAS Scans"""
-        data = self.generate_scan_result_XML(limit=limit, offset=offset, status=status)
-        return self.qgc.request(self.SEARCH_WAS_SCAN, data)
-
-    def get_all_scans(self, limit=1000, offset=1, status='FINISHED'):
-        qualys_api_limit = limit
-        dataframes = []
-        _records = []
-        total = int(self.get_was_scan_count(status=status))
-        print('Retrieving information for %s scans' % total)
-        for i in range(0, total):
-            if i % limit == 0:
-                if (total - i) < limit:
-                    qualys_api_limit = total - i
-                print('Making a request with a limit of %s at offset %s' % (str(qualys_api_limit), str(i + 1)))
-                scan_info = self.get_scan_info(limit=qualys_api_limit, offset=i + 1, status=status)
-                _records.append(scan_info)
-        print('Converting XML to DataFrame')
-        dataframes = [self.xml_parser(xml) for xml in _records]
-
-        return pd.concat(dataframes, axis=0).reset_index().drop('index', axis=1)
-
-    def get_scan_details(self, scan_id):
-        return self.qgc.request(self.SCAN_DETAILS.format(scan_id=scan_id))
-
-    def get_report_details(self, report_id):
-        return self.qgc.request(self.REPORT_DETAILS.format(report_id=report_id))
-
-    def get_report_status(self, report_id):
-        return self.qgc.request(self.REPORT_STATUS.format(report_id=report_id))
-
-    def download_report(self, report_id):
-        return self.qgc.request(self.REPORT_DOWNLOAD.format(report_id=report_id))
-
-    def download_scan_results(self, scan_id):
-        return self.qgc.request(self.SCAN_DOWNLOAD.format(scan_id=scan_id))
-
-    def generate_scan_result_XML(self, limit=1000, offset=1, status='FINISHED'):
-        report_xml = E.ServiceRequest(
-            E.filters(
-                E.Criteria({'field': 'status', 'operator': 'EQUALS'}, status
-                           ),
-            ),
-            E.preferences(
-                E.startFromOffset(str(offset)),
-                E.limitResults(str(limit))
-            ),
-        )
-        return report_xml
-
-    def generate_scan_report_XML(self, scan_id):
-        """Generates a CSV report for an asset based on template defined in .ini file"""
-        report_xml = E.ServiceRequest(
-            E.data(
-                E.Report(
-                    E.name('![CDATA[API Scan Report generated by VulnWhisperer]]>'),
-                    E.description('<![CDATA[CSV Scanning report for VulnWhisperer]]>'),
-                    E.format('CSV'),
-                    E.type('WAS_SCAN_REPORT'),
-                    E.template(
-                        E.id(self.template_id)
-                    ),
-                    E.config(
-                        E.scanReport(
-                            E.target(
-                                E.scans(
-                                    E.WasScan(
-                                        E.id(scan_id)
-                                    )
-                                ),
-                            ),
-                        ),
-                    )
-                )
-            )
-        )
-        return report_xml
-
-    def generate_webapp_report_XML(self, app_id):
-        """Generates a CSV report for an asset based on template defined in .ini file"""
-        report_xml = E.ServiceRequest(
-            E.data(
-                E.Report(
-                    E.name('![CDATA[API Web Application Report generated by VulnWhisperer]]>'),
-                    E.description('<![CDATA[CSV WebApp report for VulnWhisperer]]>'),
-                    E.format('CSV'),
-                    E.template(
-                        E.id(self.template_id)
-                    ),
-                    E.config(
-                        E.webAppReport(
-                            E.target(
-                                E.webapps(
-                                    E.WebApp(
-                                        E.id(app_id)
-                                    )
-                                ),
-                            ),
-                        ),
-                    )
-                )
-            )
-        )
-        return report_xml
-
-    def generate_app_id_scan_XML(self, app_id):
-        report_xml = E.ServiceRequest(
-            E.filters(
-                E.Criteria({'field': 'webApp.id', 'operator': 'EQUALS'}, app_id
-                           ),
-            ),
-        )
-        return report_xml
-
-    def create_report(self, report_id, kind='scan'):
-        mapper = {'scan': self.generate_scan_report_XML,
-                  'webapp': self.generate_webapp_report_XML}
-        try:
-            # print lxml.etree.tostring(mapper[kind](report_id), pretty_print=True)
-            data = mapper[kind](report_id)
-        except Exception as e:
-            print(e)
-
-        return self.qgc.request(self.REPORT_CREATE, data)
-
-    def delete_report(self, report_id):
-        return self.qgc.request(self.DELETE_REPORT.format(report_id=report_id))
-
-
-class qualysReportFields:
-    CATEGORIES = ['VULNERABILITY',
-                  'SENSITIVECONTENT',
-                  'INFORMATION_GATHERED']
-
-    # URL Vulnerability Information
-
-    VULN_BLOCK = [
-        CATEGORIES[0],
-        'ID',
-        'QID',
-        'Url',
-        'Param',
-        'Function',
-        'Form Entry Point',
-        'Access Path',
-        'Authentication',
-        'Ajax Request',
-        'Ajax Request ID',
-        'Ignored',
-        'Ignore Reason',
-        'Ignore Date',
-        'Ignore User',
-        'Ignore Comments',
-        'First Time Detected',
-        'Last Time Detected',
-        'Last Time Tested',
-        'Times Detected',
-        'Payload #1',
-        'Request Method #1',
-        'Request URL #1',
-        'Request Headers #1',
-        'Response #1',
-        'Evidence #1',
-    ]
-
-    INFO_HEADER = [
-        'Vulnerability Category',
-        'ID',
-        'QID',
-        'Response #1',
-        'Last Time Detected',
-    ]
-    INFO_BLOCK = [
-        CATEGORIES[2],
-        'ID',
-        'QID',
-        'Results',
-        'Detection Date',
-    ]
-
-    QID_HEADER = [
-        'QID',
-        'Id',
-        'Title',
-        'Category',
-        'Severity Level',
-        'Groups',
-        'OWASP',
-        'WASC',
-        'CWE',
-        'CVSS Base',
-        'CVSS Temporal',
-        'Description',
-        'Impact',
-        'Solution',
-    ]
-    GROUP_HEADER = ['GROUP', 'Name', 'Category']
-    OWASP_HEADER = ['OWASP', 'Code', 'Name']
-    WASC_HEADER = ['WASC', 'Code', 'Name']
-    SCAN_META = ['Web Application Name', 'URL', 'Owner', 'Scope', 'Operating System']
-    CATEGORY_HEADER = ['Category', 'Severity', 'Level', 'Description']
-
-
-class qualysUtils:
-    def __init__(self):
-        pass
-
-    def grab_section(
-            self,
-            report,
-            section,
-            end=[],
-            pop_last=False,
-    ):
-        temp_list = []
-        max_col_count = 0
-        with open(report, 'rb') as csvfile:
-            q_report = csv.reader(csvfile, delimiter=',', quotechar='"')
-            for line in q_report:
-                if set(line) == set(section):
-                    break
-
-            # Reads text until the end of the block:
-            for line in q_report:  # This keeps reading the file
-                temp_list.append(line)
-
-                if line in end:
-                    break
-            if pop_last and len(temp_list) > 1:
-                temp_list.pop(-1)
-        return temp_list
-
-    def iso_to_epoch(self, dt):
-        return dp.parse(dt).strftime('%s')
-
-    def cleanser(self, _data):
-        repls = (('\n', '|||'), ('\r', '|||'), (',', ';'), ('\t', '|||'))
-        if _data:
-            _data = reduce(lambda a, kv: a.replace(*kv), repls, str(_data))
-        return _data
-
-
-class qualysWebAppReport:
-    # URL Vulnerability Information
-    WEB_APP_VULN_BLOCK = list(qualysReportFields.VULN_BLOCK)
-    WEB_APP_VULN_BLOCK.insert(0, 'Web Application Name')
-    WEB_APP_VULN_BLOCK.insert(WEB_APP_VULN_BLOCK.index('Ignored'), 'Status')
-
-    WEB_APP_VULN_HEADER = list(WEB_APP_VULN_BLOCK)
-    WEB_APP_VULN_HEADER[WEB_APP_VULN_BLOCK.index(qualysReportFields.CATEGORIES[0])] = \
-        'Vulnerability Category'
-
-    WEB_APP_SENSITIVE_HEADER = list(WEB_APP_VULN_HEADER)
-    WEB_APP_SENSITIVE_HEADER.insert(WEB_APP_SENSITIVE_HEADER.index('Url'
-                                                                   ), 'Content')
-
-    WEB_APP_SENSITIVE_BLOCK = list(WEB_APP_SENSITIVE_HEADER)
-    WEB_APP_SENSITIVE_BLOCK[WEB_APP_SENSITIVE_BLOCK.index('Vulnerability Category'
-                                                          )] = qualysReportFields.CATEGORIES[1]
-
-    WEB_APP_INFO_HEADER = list(qualysReportFields.INFO_HEADER)
-    WEB_APP_INFO_HEADER.insert(0, 'Web Application Name')
-
-    WEB_APP_INFO_BLOCK = list(qualysReportFields.INFO_BLOCK)
-    WEB_APP_INFO_BLOCK.insert(0, 'Web Application Name')
-
-    QID_HEADER = list(qualysReportFields.QID_HEADER)
-    GROUP_HEADER = list(qualysReportFields.GROUP_HEADER)
-    OWASP_HEADER = list(qualysReportFields.OWASP_HEADER)
-    WASC_HEADER = list(qualysReportFields.WASC_HEADER)
-    SCAN_META = list(qualysReportFields.SCAN_META)
-    CATEGORY_HEADER = list(qualysReportFields.CATEGORY_HEADER)
-
-    def __init__(
-            self,
-            config=None,
-            file_in=None,
-            file_stream=False,
-            delimiter=',',
-            quotechar='"',
-    ):
-        self.file_in = file_in
-        self.file_stream = file_stream
-        self.report = None
-        self.utils = qualysUtils()
-
-        if config:
-            try:
-                self.qw = qualysWhisperAPI(config=config)
-            except Exception as e:
-                print('Could not load config! Please check settings for %s' \
-                      % e)
-
-        if file_stream:
-            self.open_file = file_in.splitlines()
-        elif file_in:
-
-            self.open_file = open(file_in, 'rb')
-
-        self.downloaded_file = None
-
-    def get_hostname(self, report):
-        host = ''
-        with open(report, 'rb') as csvfile:
-            q_report = csv.reader(csvfile, delimiter=',', quotechar='"')
-            for x in q_report:
-
-                if 'Web Application Name' in x[0]:
-                    host = q_report.next()[0]
-        return host
-
-    def get_scanreport_name(self, report):
-        scan_name = ''
-        with open(report, 'rb') as csvfile:
-            q_report = csv.reader(csvfile, delimiter=',', quotechar='"')
-            for x in q_report:
-
-                if 'Scans' in x[0]:
-                    scan_name = x[1]
-        return scan_name
-
-    def grab_sections(self, report):
-        all_dataframes = []
-        dict_tracker = {}
-        with open(report, 'rb') as csvfile:
-            dict_tracker['WEB_APP_VULN_BLOCK'] = pd.DataFrame(self.utils.grab_section(report,
-                                                                       self.WEB_APP_VULN_BLOCK,
-                                                                       end=[self.WEB_APP_SENSITIVE_BLOCK,
-                                                                            self.WEB_APP_INFO_BLOCK],
-                                                                       pop_last=True), columns=self.WEB_APP_VULN_HEADER)
-            dict_tracker['WEB_APP_SENSITIVE_BLOCK'] = pd.DataFrame(self.utils.grab_section(report,
-                                                                       self.WEB_APP_SENSITIVE_BLOCK,
-                                                                       end=[self.WEB_APP_INFO_BLOCK,
-                                                                            self.WEB_APP_SENSITIVE_BLOCK],
-                                                                       pop_last=True), columns=self.WEB_APP_SENSITIVE_HEADER)
-            dict_tracker['WEB_APP_INFO_BLOCK'] = pd.DataFrame(self.utils.grab_section(report,
-                                                                       self.WEB_APP_INFO_BLOCK,
-                                                                       end=[self.QID_HEADER],
-                                                                       pop_last=True), columns=self.WEB_APP_INFO_HEADER)
-            dict_tracker['QID_HEADER'] = pd.DataFrame(self.utils.grab_section(report,
-                                                                       self.QID_HEADER,
-                                                                       end=[self.GROUP_HEADER],
-                                                                       pop_last=True), columns=self.QID_HEADER)
-            dict_tracker['GROUP_HEADER'] = pd.DataFrame(self.utils.grab_section(report,
-                                                                       self.GROUP_HEADER,
-                                                                       end=[self.OWASP_HEADER],
-                                                                       pop_last=True), columns=self.GROUP_HEADER)
-            dict_tracker['OWASP_HEADER'] = pd.DataFrame(self.utils.grab_section(report,
-                                                                       self.OWASP_HEADER,
-                                                                       end=[self.WASC_HEADER],
-                                                                       pop_last=True), columns=self.OWASP_HEADER)
-            dict_tracker['WASC_HEADER'] = pd.DataFrame(self.utils.grab_section(report,
-                                                                       self.WASC_HEADER, end=[['APPENDIX']],
-                                                                       pop_last=True), columns=self.WASC_HEADER)
-            dict_tracker['CATEGORY_HEADER'] =pd.DataFrame(self.utils.grab_section(report,
-                                                                       self.CATEGORY_HEADER), columns=self.CATEGORY_HEADER)
-            all_dataframes.append(dict_tracker)
-
-        return all_dataframes
-
-    def data_normalizer(self, dataframes):
-        """
-        Merge and clean data
-        :param dataframes:
-        :return:
-        """
-        df_dict = dataframes[0]
-        merged_df = pd.concat([df_dict['WEB_APP_VULN_BLOCK'], df_dict['WEB_APP_SENSITIVE_BLOCK'],
-                               df_dict['WEB_APP_INFO_BLOCK']], axis=0,
-                              ignore_index=False)
-
-        merged_df = pd.merge(merged_df, df_dict['QID_HEADER'], left_on='QID',
-                             right_on='Id')
-
-        merged_df = pd.concat([dataframes[0], dataframes[1],
-                               dataframes[2]], axis=0,
-                              ignore_index=False)
-        merged_df = pd.merge(merged_df, dataframes[3], left_on='QID',
-                             right_on='Id')
-
-        if 'Content' not in merged_df:
-            merged_df['Content'] = ''
-
-        columns_to_cleanse = ['Payload #1', 'Request Method #1', 'Request URL #1',
-                              'Request Headers #1', 'Response #1', 'Evidence #1',
-                              'Description', 'Impact', 'Solution', 'Url', 'Content']
-
-        for col in columns_to_cleanse:
-            merged_df[col] = merged_df[col].astype(str).apply(self.utils.cleanser)
-
-        merged_df = pd.merge(merged_df, df_dict['CATEGORY_HEADER'])
-        merged_df = merged_df.drop(['QID_y', 'QID_x'], axis=1)
-        merged_df = merged_df.rename(columns={'Id': 'QID'})
-        merged_df = merged_df.replace('N/A','').fillna('')
-
-        try:
-            merged_df = \
-                merged_df[~merged_df.Title.str.contains('Links Crawled|External Links Discovered'
-                                                        )]
-        except Exception as e:
-            print(e)
-        return merged_df
-
-    def download_file(self, file_id):
-        report = self.qw.download_report(file_id)
-        filename = str(file_id) + '.csv'
-        file_out = open(filename, 'w')
-        for line in report.splitlines():
-            file_out.write(line + '\n')
-        file_out.close()
-        print('[ACTION] - File written to %s' % filename)
-        return filename
-
-    def remove_file(self, filename):
-        os.remove(filename)
-
-    def process_data(self, file_id, scan=True, cleanup=True):
-        """Downloads a file from qualys and normalizes it"""
-
-        download_file = self.download_file(file_id)
-        print('[ACTION] - Downloading file ID: %s' % file_id)
-        report_data = self.grab_sections(download_file)
-        merged_data = self.data_normalizer(report_data)
-        if scan:
-            scan_name = self.get_scanreport_name(download_file)
-            merged_data['ScanName'] = scan_name
-
-        # TODO cleanup old data (delete)
-
-        return merged_data
-
-    def whisper_reports(self, report_id, updated_date, cleanup=False):
-        """
-        report_id: App ID
-        updated_date: Last time scan was ran for app_id
-        """
-        vuln_ready = None
-        try:
-
-            if 'Z' in updated_date:
-                updated_date = self.utils.iso_to_epoch(updated_date)
-            report_name = 'qualys_web_' + str(report_id) \
-                          + '_{last_updated}'.format(last_updated=updated_date) \
-                          + '.csv'
-            if os.path.isfile(report_name):
-                print('[ACTION] - File already exist! Skipping...')
-                pass
-            else:
-                print('[ACTION] - Generating report for %s' % report_id)
-                status = self.qw.create_report(report_id)
-                root = objectify.fromstring(status)
-                if root.responseCode == 'SUCCESS':
-                    print('[INFO] - Successfully generated report for webapp: %s' \
-                          % report_id)
-                    generated_report_id = root.data.Report.id
-                    print ('[INFO] - New Report ID: %s' \
-                           % generated_report_id)
-                    vuln_ready = self.process_data(generated_report_id)
-
-                    vuln_ready.to_csv(report_name, index=False, header=True)  # add when timestamp occured
-                    print('[SUCCESS] - Report written to %s' \
-                          % report_name)
-                    if cleanup:
-                        print('[ACTION] - Removing report %s' \
-                              % generated_report_id)
-                        cleaning_up = \
-                            self.qw.delete_report(generated_report_id)
-                        self.remove_file(str(generated_report_id) + '.csv')
-                        print('[ACTION] - Deleted report: %s' \
-                              % generated_report_id)
-                else:
-                    print('Could not process report ID: %s' % status)
-        except Exception as e:
-            print('[ERROR] - Could not process %s - %s' % (report_id, e))
-        return vuln_ready
-
-
-class qualysScanReport:
-    # URL Vulnerability Information
-    WEB_SCAN_VULN_BLOCK = list(qualysReportFields.VULN_BLOCK)
-    WEB_SCAN_VULN_BLOCK.insert(WEB_SCAN_VULN_BLOCK.index('QID'), 'Detection ID')
-
-    WEB_SCAN_VULN_HEADER = list(WEB_SCAN_VULN_BLOCK)
-    WEB_SCAN_VULN_HEADER[WEB_SCAN_VULN_BLOCK.index(qualysReportFields.CATEGORIES[0])] = \
-        'Vulnerability Category'
-
-    WEB_SCAN_SENSITIVE_HEADER = list(WEB_SCAN_VULN_HEADER)
-    WEB_SCAN_SENSITIVE_HEADER.insert(WEB_SCAN_SENSITIVE_HEADER.index('Url'
-                                                                     ), 'Content')
-
-    WEB_SCAN_SENSITIVE_BLOCK = list(WEB_SCAN_SENSITIVE_HEADER)
-    WEB_SCAN_SENSITIVE_BLOCK.insert(WEB_SCAN_SENSITIVE_BLOCK.index('QID'), 'Detection ID')
-    WEB_SCAN_SENSITIVE_BLOCK[WEB_SCAN_SENSITIVE_BLOCK.index('Vulnerability Category'
-                                                            )] = qualysReportFields.CATEGORIES[1]
-
-    WEB_SCAN_INFO_HEADER = list(qualysReportFields.INFO_HEADER)
-    WEB_SCAN_INFO_HEADER.insert(WEB_SCAN_INFO_HEADER.index('QID'), 'Detection ID')
-
-    WEB_SCAN_INFO_BLOCK = list(qualysReportFields.INFO_BLOCK)
-    WEB_SCAN_INFO_BLOCK.insert(WEB_SCAN_INFO_BLOCK.index('QID'), 'Detection ID')
-
-    QID_HEADER = list(qualysReportFields.QID_HEADER)
-    GROUP_HEADER = list(qualysReportFields.GROUP_HEADER)
-    OWASP_HEADER = list(qualysReportFields.OWASP_HEADER)
-    WASC_HEADER = list(qualysReportFields.WASC_HEADER)
-    SCAN_META = list(qualysReportFields.SCAN_META)
-    CATEGORY_HEADER = list(qualysReportFields.CATEGORY_HEADER)
-
-    def __init__(
-            self,
-            config=None,
-            file_in=None,
-            file_stream=False,
-            delimiter=',',
-            quotechar='"',
-    ):
-        self.file_in = file_in
-        self.file_stream = file_stream
-        self.report = None
-        self.utils = qualysUtils()
-
-        if config:
-            try:
-                self.qw = qualysWhisperAPI(config=config)
-            except Exception as e:
-                print('Could not load config! Please check settings for %s' \
-                      % e)
-
-        if file_stream:
-            self.open_file = file_in.splitlines()
-        elif file_in:
-
-            self.open_file = open(file_in, 'rb')
-
-        self.downloaded_file = None
-
-    def grab_sections(self, report):
-        all_dataframes = []
-        dict_tracker = {}
-        with open(report, 'rb') as csvfile:
-            dict_tracker['WEB_SCAN_VULN_BLOCK'] = pd.DataFrame(self.utils.grab_section(report,
-                                                                                       self.WEB_SCAN_VULN_BLOCK,
-                                                                                       end=[
-                                                                                           self.WEB_SCAN_SENSITIVE_BLOCK,
-                                                                                           self.WEB_SCAN_INFO_BLOCK],
-                                                                                       pop_last=True),
-                                                               columns=self.WEB_SCAN_VULN_HEADER)
-            dict_tracker['WEB_SCAN_SENSITIVE_BLOCK'] = pd.DataFrame(self.utils.grab_section(report,
-                                                                                            self.WEB_SCAN_SENSITIVE_BLOCK,
-                                                                                            end=[
-                                                                                                self.WEB_SCAN_INFO_BLOCK,
-                                                                                                self.WEB_SCAN_SENSITIVE_BLOCK],
-                                                                                            pop_last=True),
-                                                                columns=self.WEB_SCAN_SENSITIVE_HEADER)
-            dict_tracker['WEB_SCAN_INFO_BLOCK'] = pd.DataFrame(self.utils.grab_section(report,
-                                                                                       self.WEB_SCAN_INFO_BLOCK,
-                                                                                       end=[self.QID_HEADER],
-                                                                                       pop_last=True),
-                                                                columns=self.WEB_SCAN_INFO_HEADER)
-            dict_tracker['QID_HEADER'] = pd.DataFrame(self.utils.grab_section(report,
-                                                                              self.QID_HEADER,
-                                                                              end=[self.GROUP_HEADER],
-                                                                              pop_last=True),
-                                                                columns=self.QID_HEADER)
-            dict_tracker['GROUP_HEADER'] = pd.DataFrame(self.utils.grab_section(report,
-                                                                                self.GROUP_HEADER,
-                                                                                end=[self.OWASP_HEADER],
-                                                                                pop_last=True),
-                                                                columns=self.GROUP_HEADER)
-            dict_tracker['OWASP_HEADER'] = pd.DataFrame(self.utils.grab_section(report,
-                                                                                self.OWASP_HEADER,
-                                                                                end=[self.WASC_HEADER],
-                                                                                pop_last=True),
-                                                                columns=self.OWASP_HEADER)
-            dict_tracker['WASC_HEADER'] = pd.DataFrame(self.utils.grab_section(report,
-                                                                               self.WASC_HEADER, end=[['APPENDIX']],
-                                                                               pop_last=True),
-                                                                columns=self.WASC_HEADER)
-
-            dict_tracker['SCAN_META'] = pd.DataFrame(self.utils.grab_section(report,
-                                                                             self.SCAN_META,
-                                                                             end=[self.CATEGORY_HEADER],
-                                                                             pop_last=True),
-                                                                columns=self.SCAN_META)
-
-            dict_tracker['CATEGORY_HEADER'] = pd.DataFrame(self.utils.grab_section(report,
-                                                                                   self.CATEGORY_HEADER),
-                                                                columns=self.CATEGORY_HEADER)
-            all_dataframes.append(dict_tracker)
-
-        return all_dataframes
-
-    def data_normalizer(self, dataframes):
-        """
-        Merge and clean data
-        :param dataframes:
-        :return:
-        """
-        df_dict = dataframes[0]
-        merged_df = pd.concat([df_dict['WEB_SCAN_VULN_BLOCK'], df_dict['WEB_SCAN_SENSITIVE_BLOCK'],
-                               df_dict['WEB_SCAN_INFO_BLOCK']], axis=0,
-                              ignore_index=False)
-        merged_df = pd.merge(merged_df, df_dict['QID_HEADER'], left_on='QID',
-                             right_on='Id')
-
-        if 'Content' not in merged_df:
-            merged_df['Content'] = ''
-
-        columns_to_cleanse = ['Payload #1', 'Request Method #1', 'Request URL #1',
-                              'Request Headers #1', 'Response #1', 'Evidence #1',
-                              'Description', 'Impact', 'Solution', 'Url', 'Content']
-
-        for col in columns_to_cleanse:
-            merged_df[col] = merged_df[col].apply(self.utils.cleanser)
-
-        merged_df = merged_df.drop(['QID_y', 'QID_x'], axis=1)
-        merged_df = merged_df.rename(columns={'Id': 'QID'})
-        merged_df = merged_df.assign(**df_dict['SCAN_META'].to_dict(orient='records')[0])
-
-        merged_df = pd.merge(merged_df, df_dict['CATEGORY_HEADER'], how='left', left_on=['Category', 'Severity Level'],
-                             right_on=['Category', 'Severity'], suffixes=('Severity', 'CatSev'))
-
-        merged_df = merged_df.replace('N/A', '').fillna('')
-
-        try:
-            merged_df = \
-                merged_df[~merged_df.Title.str.contains('Links Crawled|External Links Discovered'
-                                                        )]
-        except Exception as e:
-            print(e)
-        return merged_df
-
-    def download_file(self, path='', file_id=None):
-        report = self.qw.download_report(file_id)
-        filename = path + str(file_id) + '.csv'
-        file_out = open(filename, 'w')
-        for line in report.splitlines():
-            file_out.write(line + '\n')
-        file_out.close()
-        print('[ACTION] - File written to %s' % filename)
-        return filename
-
-    def remove_file(self, filename):
-        os.remove(filename)
-
-    def process_data(self, path='', file_id=None, cleanup=True):
-        """Downloads a file from qualys and normalizes it"""
-
-        download_file = self.download_file(path=path, file_id=file_id)
-        print('[ACTION] - Downloading file ID: %s' % file_id)
-        report_data = self.grab_sections(download_file)
-        merged_data = self.data_normalizer(report_data)
-        merged_data.sort_index(axis=1, inplace=True)
-        # TODO cleanup old data (delete)
-
-        return merged_data
-
-    def whisper_reports(self, report_id, updated_date, cleanup=False):
-        """
-        report_id: App ID
-        updated_date: Last time scan was ran for app_id
-        """
-        vuln_ready = None
-        try:
-
-            if 'Z' in updated_date:
-                updated_date = self.utils.iso_to_epoch(updated_date)
-            report_name = 'qualys_web_' + str(report_id) \
-                          + '_{last_updated}'.format(last_updated=updated_date) \
-                          + '.csv'
-            if os.path.isfile(report_name):
-                print('[ACTION] - File already exist! Skipping...')
-                pass
-            else:
-                print('[ACTION] - Generating report for %s' % report_id)
-                status = self.qw.create_report(report_id)
-                root = objectify.fromstring(status)
-                if root.responseCode == 'SUCCESS':
-                    print('[INFO] - Successfully generated report for webapp: %s' \
-                          % report_id)
-                    generated_report_id = root.data.Report.id
-                    print ('[INFO] - New Report ID: %s' \
-                           % generated_report_id)
-                    vuln_ready = self.process_data(generated_report_id)
-
-                    vuln_ready.to_csv(report_name, index=False, header=True)  # add when timestamp occured
-                    print('[SUCCESS] - Report written to %s' \
-                          % report_name)
-                    if cleanup:
-                        print('[ACTION] - Removing report %s from disk' \
-                              % generated_report_id)
-                        cleaning_up = \
-                            self.qw.delete_report(generated_report_id)
-                        self.remove_file(str(generated_report_id) + '.csv')
-                        print('[ACTION] - Deleted report from Qualys Database: %s' \
-                              % generated_report_id)
-                else:
-                    print('Could not process report ID: %s' % status)
-        except Exception as e:
-            print('[ERROR] - Could not process %s - %s' % (report_id, e))
-        return vuln_ready
-
-
-maxInt = int(4000000)
-maxSize = sys.maxsize
-
-if maxSize > maxInt and type(maxSize) == int:
-    maxInt = maxSize
-
-decrement = True
-
-while decrement:
-    decrement = False
-    try:
-        csv.field_size_limit(maxInt)
-    except OverflowError:
-        maxInt = int(maxInt/10)
-        decrement = True

vulnwhisp/frameworks/qualys_vuln.py

@@ -2,32 +2,33 @@
 # -*- coding: utf-8 -*-
 __author__ = 'Nathan Young'
 
+import logging
+import sys
 import xml.etree.ElementTree as ET
+
+import dateutil.parser as dp
 import pandas as pd
 import qualysapi
-import requests
-import sys
-import os
-import dateutil.parser as dp
 
 
 class qualysWhisperAPI(object):
     SCANS = 'api/2.0/fo/scan'
 
     def __init__(self, config=None):
+        self.logger = logging.getLogger('qualysWhisperAPI')
         self.config = config
         try:
-            self.qgc = qualysapi.connect(config)
+            self.qgc = qualysapi.connect(config, 'qualys_vuln')
             # Fail early if we can't make a request or auth is incorrect
             self.qgc.request('about.php')
-            print('[SUCCESS] - Connected to Qualys at %s' % self.qgc.server)
+            self.logger.info('Connected to Qualys at {}'.format(self.qgc.server))
         except Exception as e:
-            print('[ERROR] Could not connect to Qualys - %s' % e)
-            exit(1)
+            self.logger.error('Could not connect to Qualys: {}'.format(str(e)))
+            sys.exit(1)
 
     def scan_xml_parser(self, xml):
         all_records = []
-        root = ET.XML(xml)
+        root = ET.XML(xml.encode("utf-8"))
         for child in root.find('.//SCAN_LIST'):
             all_records.append({
                 'name': child.find('TITLE').text,
@@ -58,30 +59,34 @@ class qualysWhisperAPI(object):
             'scan_ref': scan_id
         }
         scan_json = self.qgc.request(self.SCANS, parameters)
-        
+
         # First two columns are metadata we already have
         # Last column corresponds to "target_distribution_across_scanner_appliances" element
-	# which doesn't follow the schema and breaks the pandas data manipulation
-	return pd.read_json(scan_json).iloc[2:-1]
+        # which doesn't follow the schema and breaks the pandas data manipulation
+        return pd.read_json(scan_json).iloc[2:-1]
+
 
 class qualysUtils:
     def __init__(self):
-        pass
+        self.logger = logging.getLogger('qualysUtils')
 
     def iso_to_epoch(self, dt):
-        return dp.parse(dt).strftime('%s')
+        out = dp.parse(dt).strftime('%s')
+        self.logger.info('Converted {} to {}'.format(dt, out))
+        return out
 
 
 class qualysVulnScan:
-   
+
     def __init__(
-            self,
-            config=None,
-            file_in=None,
-            file_stream=False,
-            delimiter=',',
-            quotechar='"',
-        ):
+        self,
+        config=None,
+        file_in=None,
+        file_stream=False,
+        delimiter=',',
+        quotechar='"',
+    ):
+        self.logger = logging.getLogger('qualysVulnScan')
         self.file_in = file_in
         self.file_stream = file_stream
         self.report = None
@@ -91,8 +96,7 @@ class qualysVulnScan:
             try:
                 self.qw = qualysWhisperAPI(config=config)
             except Exception as e:
-                print('Could not load config! Please check settings for %s' \
-                      % e)
+                self.logger.error('Could not load config! Please check settings. Error: {}'.format(str(e)))
 
         if file_stream:
             self.open_file = file_in.splitlines()
@@ -103,12 +107,18 @@ class qualysVulnScan:
 
     def process_data(self, scan_id=None):
         """Downloads a file from Qualys and normalizes it"""
-        
-        print('[ACTION] - Downloading scan ID: %s' % scan_id)
+        self.logger.info('Downloading scan ID: {}'.format(scan_id))
         scan_report = self.qw.get_scan_details(scan_id=scan_id)
-        keep_columns = ['category', 'cve_id', 'cvss3_base', 'cvss3_temporal', 'cvss_base', 'cvss_temporal', 'dns', 'exploitability', 'fqdn', 'impact', 'ip', 'ip_status', 'netbios', 'os', 'pci_vuln', 'port', 'protocol', 'qid', 'results', 'severity', 'solution', 'ssl', 'threat', 'title', 'type', 'vendor_reference']
-        scan_report = scan_report.filter(keep_columns)
-        scan_report['severity'] = scan_report['severity'].astype(int).astype(str)
-        scan_report['qid'] = scan_report['qid'].astype(int).astype(str)
+        if not scan_report.empty:
+            keep_columns = ['category', 'cve_id', 'cvss3_base', 'cvss3_temporal', 'cvss_base',
+                'cvss_temporal', 'dns', 'exploitability', 'fqdn', 'impact', 'ip', 'ip_status',
+                'netbios', 'os', 'pci_vuln', 'port', 'protocol', 'qid', 'results', 'severity',
+                'solution', 'ssl', 'threat', 'title', 'type', 'vendor_reference']
+            scan_report = scan_report.filter(keep_columns)
+            scan_report['severity'] = scan_report['severity'].astype(int).astype(str)
+            scan_report['qid'] = scan_report['qid'].astype(int).astype(str)
+        else:
+            self.logger.warn('Scan ID {} has no vulnerabilities, skipping.'.format(scan_id))
+            return scan_report
 
         return scan_report

vulnwhisp/frameworks/qualys_web.py

@@ -0,0 +1,465 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+__author__ = 'Austin Taylor'
+
+from lxml import objectify
+from lxml.builder import E
+import xml.etree.ElementTree as ET
+import pandas as pd
+import qualysapi
+import qualysapi.config as qcconf
+import requests
+import sys
+import os
+import csv
+import logging
+import dateutil.parser as dp
+
+
+class qualysWhisperAPI(object):
+    COUNT_WEBAPP = '/count/was/webapp'
+    COUNT_WASSCAN = '/count/was/wasscan'
+    DELETE_REPORT = '/delete/was/report/{report_id}'
+    GET_WEBAPP_DETAILS = '/get/was/webapp/{was_id}'
+    QPS_REST_3 = '/qps/rest/3.0'
+    REPORT_DETAILS = '/get/was/report/{report_id}'
+    REPORT_STATUS = '/status/was/report/{report_id}'
+    REPORT_CREATE = '/create/was/report'
+    REPORT_DOWNLOAD = '/download/was/report/{report_id}'
+    SCAN_DETAILS = '/get/was/wasscan/{scan_id}'
+    SCAN_DOWNLOAD = '/download/was/wasscan/{scan_id}'
+    SEARCH_REPORTS = '/search/was/report'
+    SEARCH_WEB_APPS = '/search/was/webapp'
+    SEARCH_WAS_SCAN = '/search/was/wasscan'
+    VERSION = '/qps/rest/portal/version'
+
+    def __init__(self, config=None):
+        self.logger = logging.getLogger('qualysWhisperAPI')
+        self.config = config
+        try:
+            self.qgc = qualysapi.connect(config, 'qualys_web')
+            self.logger.info('Connected to Qualys at {}'.format(self.qgc.server))
+        except Exception as e:
+            self.logger.error('Could not connect to Qualys: {}'.format(str(e)))
+        self.headers = {
+            #"content-type": "text/xml"}
+            "Accept" : "application/json",
+            "Content-Type": "application/json"}
+        self.config_parse = qcconf.QualysConnectConfig(config, 'qualys_web')
+        try:
+            self.template_id = self.config_parse.get_template_id()
+        except:
+            self.logger.error('Could not retrieve template ID')
+
+    ####
+    #### GET SCANS TO PROCESS
+    ####
+
+    def get_was_scan_count(self, status):
+        """
+        Checks number of scans, used to control the api limits
+        """
+        parameters = (
+            E.ServiceRequest(
+                E.filters(
+                    E.Criteria({'field': 'status', 'operator': 'EQUALS'}, status))))
+        xml_output = self.qgc.request(self.COUNT_WASSCAN, parameters)
+        root = objectify.fromstring(xml_output.encode('utf-8'))
+        return root.count.text
+
+    def generate_scan_result_XML(self, limit=1000, offset=1, status='FINISHED'):
+        report_xml = E.ServiceRequest(
+            E.filters(
+                E.Criteria({'field': 'status', 'operator': 'EQUALS'}, status
+                           ),
+            ),
+            E.preferences(
+                E.startFromOffset(str(offset)),
+                E.limitResults(str(limit))
+            ),
+        )
+        return report_xml
+
+    def get_scan_info(self, limit=1000, offset=1, status='FINISHED'):
+        """ Returns XML of ALL WAS Scans"""
+        data = self.generate_scan_result_XML(limit=limit, offset=offset, status=status)
+        return self.qgc.request(self.SEARCH_WAS_SCAN, data)
+
+    def xml_parser(self, xml, dupfield=None):
+        all_records = []
+        root = ET.XML(xml)
+        for i, child in enumerate(root):
+            for subchild in child:
+                record = {}
+                dup_tracker = 0
+                for p in subchild:
+                    record[p.tag] = p.text
+                    for o in p:
+                        if o.tag in record:
+                            dup_tracker += 1
+                            record[o.tag + '_%s' % dup_tracker] = o.text
+                        else:
+                            record[o.tag] = o.text
+                all_records.append(record)
+        return pd.DataFrame(all_records)
+
+    def get_all_scans(self, limit=1000, offset=1, status='FINISHED'):
+        qualys_api_limit = limit
+        dataframes = []
+        _records = []
+        try:
+            total = int(self.get_was_scan_count(status=status))
+            self.logger.error('Already have WAS scan count')
+            self.logger.info('Retrieving information for {} scans'.format(total))
+            for i in range(0, total):
+                if i % limit == 0:
+                    if (total - i) < limit:
+                        qualys_api_limit = total - i
+                    self.logger.info('Making a request with a limit of {} at offset {}'.format((str(qualys_api_limit)), str(i + 1)))
+                    scan_info = self.get_scan_info(limit=qualys_api_limit, offset=i + 1, status=status)
+                    _records.append(scan_info)
+            self.logger.debug('Converting XML to DataFrame')
+            dataframes = [self.xml_parser(xml) for xml in _records]
+        except Exception as e:
+            self.logger.error("Couldn't process all scans: {}".format(e))
+
+        return pd.concat(dataframes, axis=0).reset_index().drop('index', axis=1)
+
+    ####
+    #### CREATE VULNERABILITY REPORT AND DOWNLOAD IT
+    ####
+
+    def get_report_status(self, report_id):
+        return self.qgc.request(self.REPORT_STATUS.format(report_id=report_id))
+
+    def download_report(self, report_id):
+        return self.qgc.request(self.REPORT_DOWNLOAD.format(report_id=report_id))
+
+    def generate_scan_report_XML(self, scan_id):
+        """Generates a CSV report for an asset based on template defined in .ini file"""
+        report_xml = E.ServiceRequest(
+            E.data(
+                E.Report(
+                    E.name('<![CDATA[API Scan Report generated by VulnWhisperer]]>'),
+                    E.description('<![CDATA[CSV Scanning report for VulnWhisperer]]>'),
+                    E.format('CSV'),
+                    #type is not needed, as the template already has it
+                    E.type('WAS_SCAN_REPORT'),
+                    E.template(
+                        E.id(self.template_id)
+                    ),
+                    E.config(
+                        E.scanReport(
+                            E.target(
+                                E.scans(
+                                    E.WasScan(
+                                        E.id(scan_id)
+                                    )
+                                ),
+                            ),
+                        ),
+                    )
+                )
+            )
+        )
+        return report_xml
+
+    def create_report(self, report_id, kind='scan'):
+        mapper = {'scan': self.generate_scan_report_XML}
+        try:
+            data = mapper[kind](report_id)
+        except Exception as e:
+            self.logger.error('Error creating report: {}'.format(str(e)))
+        return self.qgc.request(self.REPORT_CREATE, data).encode('utf-8')
+
+    def delete_report(self, report_id):
+        return self.qgc.request(self.DELETE_REPORT.format(report_id=report_id))
+
+
+class qualysReportFields:
+    CATEGORIES = ['VULNERABILITY',
+                  'SENSITIVECONTENT',
+                  'INFORMATION_GATHERED']
+
+    # URL Vulnerability Information
+
+    VULN_BLOCK = [
+        CATEGORIES[0],
+        'ID',
+        'QID',
+        'Url',
+        'Param',
+        'Function',
+        'Form Entry Point',
+        'Access Path',
+        'Authentication',
+        'Ajax Request',
+        'Ajax Request ID',
+        'Ignored',
+        'Ignore Reason',
+        'Ignore Date',
+        'Ignore User',
+        'Ignore Comments',
+        'First Time Detected',
+        'Last Time Detected',
+        'Last Time Tested',
+        'Times Detected',
+        'Payload #1',
+        'Request Method #1',
+        'Request URL #1',
+        'Request Headers #1',
+        'Response #1',
+        'Evidence #1',
+    ]
+
+    INFO_HEADER = [
+        'Vulnerability Category',
+        'ID',
+        'QID',
+        'Response #1',
+        'Last Time Detected',
+    ]
+    INFO_BLOCK = [
+        CATEGORIES[2],
+        'ID',
+        'QID',
+        'Results',
+        'Detection Date',
+    ]
+
+    QID_HEADER = [
+        'QID',
+        'Id',
+        'Title',
+        'Category',
+        'Severity Level',
+        'Groups',
+        'OWASP',
+        'WASC',
+        'CWE',
+        'CVSS Base',
+        'CVSS Temporal',
+        'Description',
+        'Impact',
+        'Solution',
+    ]
+    GROUP_HEADER = ['GROUP', 'Name', 'Category']
+    OWASP_HEADER = ['OWASP', 'Code', 'Name']
+    WASC_HEADER = ['WASC', 'Code', 'Name']
+    SCAN_META = ['Web Application Name', 'URL', 'Owner', 'Scope', 'Operating System']
+    CATEGORY_HEADER = ['Category', 'Severity', 'Level', 'Description']
+
+
+class qualysUtils:
+    def __init__(self):
+        self.logger = logging.getLogger('qualysUtils')
+
+    def grab_section(
+            self,
+            report,
+            section,
+            end=[],
+            pop_last=False,
+    ):
+        temp_list = []
+        max_col_count = 0
+        with open(report, 'rb') as csvfile:
+            q_report = csv.reader(csvfile, delimiter=',', quotechar='"')
+            for line in q_report:
+                if set(line) == set(section):
+                    break
+
+            # Reads text until the end of the block:
+            for line in q_report:  # This keeps reading the file
+                temp_list.append(line)
+
+                if line in end:
+                    break
+            if pop_last and len(temp_list) > 1:
+                temp_list.pop(-1)
+        return temp_list
+
+    def iso_to_epoch(self, dt):
+        return dp.parse(dt).strftime('%s')
+
+    def cleanser(self, _data):
+        repls = (('\n', '|||'), ('\r', '|||'), (',', ';'), ('\t', '|||'))
+        if _data:
+            _data = reduce(lambda a, kv: a.replace(*kv), repls, str(_data))
+        return _data
+
+class qualysScanReport:
+    # URL Vulnerability Information
+    WEB_SCAN_VULN_BLOCK = list(qualysReportFields.VULN_BLOCK)
+    WEB_SCAN_VULN_BLOCK.insert(WEB_SCAN_VULN_BLOCK.index('QID'), 'Detection ID')
+
+    WEB_SCAN_VULN_HEADER = list(WEB_SCAN_VULN_BLOCK)
+    WEB_SCAN_VULN_HEADER[WEB_SCAN_VULN_BLOCK.index(qualysReportFields.CATEGORIES[0])] = \
+        'Vulnerability Category'
+
+    WEB_SCAN_SENSITIVE_HEADER = list(WEB_SCAN_VULN_HEADER)
+    WEB_SCAN_SENSITIVE_HEADER.insert(WEB_SCAN_SENSITIVE_HEADER.index('Url'
+                                                                     ), 'Content')
+
+    WEB_SCAN_SENSITIVE_BLOCK = list(WEB_SCAN_SENSITIVE_HEADER)
+    WEB_SCAN_SENSITIVE_BLOCK.insert(WEB_SCAN_SENSITIVE_BLOCK.index('QID'), 'Detection ID')
+    WEB_SCAN_SENSITIVE_BLOCK[WEB_SCAN_SENSITIVE_BLOCK.index('Vulnerability Category'
+                                                            )] = qualysReportFields.CATEGORIES[1]
+
+    WEB_SCAN_INFO_HEADER = list(qualysReportFields.INFO_HEADER)
+    WEB_SCAN_INFO_HEADER.insert(WEB_SCAN_INFO_HEADER.index('QID'), 'Detection ID')
+
+    WEB_SCAN_INFO_BLOCK = list(qualysReportFields.INFO_BLOCK)
+    WEB_SCAN_INFO_BLOCK.insert(WEB_SCAN_INFO_BLOCK.index('QID'), 'Detection ID')
+
+    QID_HEADER = list(qualysReportFields.QID_HEADER)
+    GROUP_HEADER = list(qualysReportFields.GROUP_HEADER)
+    OWASP_HEADER = list(qualysReportFields.OWASP_HEADER)
+    WASC_HEADER = list(qualysReportFields.WASC_HEADER)
+    SCAN_META = list(qualysReportFields.SCAN_META)
+    CATEGORY_HEADER = list(qualysReportFields.CATEGORY_HEADER)
+
+    def __init__(
+            self,
+            config=None,
+            file_in=None,
+            file_stream=False,
+            delimiter=',',
+            quotechar='"',
+    ):
+        self.logger = logging.getLogger('qualysScanReport')
+        self.file_in = file_in
+        self.file_stream = file_stream
+        self.report = None
+        self.utils = qualysUtils()
+
+        if config:
+            try:
+                self.qw = qualysWhisperAPI(config=config)
+            except Exception as e:
+                self.logger.error('Could not load config! Please check settings. Error: {}'.format(str(e)))
+
+        if file_stream:
+            self.open_file = file_in.splitlines()
+        elif file_in:
+
+            self.open_file = open(file_in, 'rb')
+
+        self.downloaded_file = None
+
+    def grab_sections(self, report):
+        all_dataframes = []
+        dict_tracker = {}
+        with open(report, 'rb') as csvfile:
+            dict_tracker['WEB_SCAN_VULN_BLOCK'] = pd.DataFrame(self.utils.grab_section(report,
+                                                                                       self.WEB_SCAN_VULN_BLOCK,
+                                                                                       end=[
+                                                                                           self.WEB_SCAN_SENSITIVE_BLOCK,
+                                                                                           self.WEB_SCAN_INFO_BLOCK],
+                                                                                       pop_last=True),
+                                                               columns=self.WEB_SCAN_VULN_HEADER)
+            dict_tracker['WEB_SCAN_SENSITIVE_BLOCK'] = pd.DataFrame(self.utils.grab_section(report,
+                                                                                            self.WEB_SCAN_SENSITIVE_BLOCK,
+                                                                                            end=[
+                                                                                                self.WEB_SCAN_INFO_BLOCK,
+                                                                                                self.WEB_SCAN_SENSITIVE_BLOCK],
+                                                                                            pop_last=True),
+                                                                columns=self.WEB_SCAN_SENSITIVE_HEADER)
+            dict_tracker['WEB_SCAN_INFO_BLOCK'] = pd.DataFrame(self.utils.grab_section(report,
+                                                                                       self.WEB_SCAN_INFO_BLOCK,
+                                                                                       end=[self.QID_HEADER],
+                                                                                       pop_last=True),
+                                                                columns=self.WEB_SCAN_INFO_HEADER)
+            dict_tracker['QID_HEADER'] = pd.DataFrame(self.utils.grab_section(report,
+                                                                              self.QID_HEADER,
+                                                                              end=[self.GROUP_HEADER],
+                                                                              pop_last=True),
+                                                                columns=self.QID_HEADER)
+            dict_tracker['GROUP_HEADER'] = pd.DataFrame(self.utils.grab_section(report,
+                                                                                self.GROUP_HEADER,
+                                                                                end=[self.OWASP_HEADER],
+                                                                                pop_last=True),
+                                                                columns=self.GROUP_HEADER)
+            dict_tracker['OWASP_HEADER'] = pd.DataFrame(self.utils.grab_section(report,
+                                                                                self.OWASP_HEADER,
+                                                                                end=[self.WASC_HEADER],
+                                                                                pop_last=True),
+                                                                columns=self.OWASP_HEADER)
+            dict_tracker['WASC_HEADER'] = pd.DataFrame(self.utils.grab_section(report,
+                                                                               self.WASC_HEADER, end=[['APPENDIX']],
+                                                                               pop_last=True),
+                                                                columns=self.WASC_HEADER)
+
+            dict_tracker['SCAN_META'] = pd.DataFrame(self.utils.grab_section(report,
+                                                                             self.SCAN_META,
+                                                                             end=[self.CATEGORY_HEADER],
+                                                                             pop_last=True),
+                                                                columns=self.SCAN_META)
+
+            dict_tracker['CATEGORY_HEADER'] = pd.DataFrame(self.utils.grab_section(report,
+                                                                                   self.CATEGORY_HEADER),
+                                                                columns=self.CATEGORY_HEADER)
+            all_dataframes.append(dict_tracker)
+
+        return all_dataframes
+
+    def data_normalizer(self, dataframes):
+        """
+        Merge and clean data
+        :param dataframes:
+        :return:
+        """
+        df_dict = dataframes[0]
+        merged_df = pd.concat([df_dict['WEB_SCAN_VULN_BLOCK'], df_dict['WEB_SCAN_SENSITIVE_BLOCK'],
+                               df_dict['WEB_SCAN_INFO_BLOCK']], axis=0,
+                              ignore_index=False)
+        merged_df = pd.merge(merged_df, df_dict['QID_HEADER'], left_on='QID',
+                             right_on='Id')
+
+        if 'Content' not in merged_df:
+            merged_df['Content'] = ''
+
+        columns_to_cleanse = ['Payload #1', 'Request Method #1', 'Request URL #1',
+                              'Request Headers #1', 'Response #1', 'Evidence #1',
+                              'Description', 'Impact', 'Solution', 'Url', 'Content']
+
+        for col in columns_to_cleanse:
+            merged_df[col] = merged_df[col].apply(self.utils.cleanser)
+
+        merged_df = merged_df.drop(['QID_y', 'QID_x'], axis=1)
+        merged_df = merged_df.rename(columns={'Id': 'QID'})
+
+        merged_df = merged_df.assign(**df_dict['SCAN_META'].to_dict(orient='records')[0])
+
+        merged_df = pd.merge(merged_df, df_dict['CATEGORY_HEADER'], how='left', left_on=['Category', 'Severity Level'],
+                             right_on=['Category', 'Severity'], suffixes=('Severity', 'CatSev'))
+
+        merged_df = merged_df.replace('N/A', '').fillna('')
+
+        try:
+            merged_df = \
+                merged_df[~merged_df.Title.str.contains('Links Crawled|External Links Discovered')]
+        except Exception as e:
+            self.logger.error('Error normalizing: {}'.format(str(e)))
+        return merged_df
+
+    def download_file(self, path='', file_id=None):
+        report = self.qw.download_report(file_id)
+        filename = path + str(file_id) + '.csv'
+        file_out = open(filename, 'w')
+        for line in report.splitlines():
+            file_out.write(line + '\n')
+        file_out.close()
+        self.logger.info('File written to {}'.format(filename))
+        return filename
+
+    def process_data(self, path='', file_id=None, cleanup=True):
+        """Downloads a file from qualys and normalizes it"""
+
+        download_file = self.download_file(path=path, file_id=file_id)
+        self.logger.info('Downloading file ID: {}'.format(file_id))
+        report_data = self.grab_sections(download_file)
+        merged_data = self.data_normalizer(report_data)
+        merged_data.sort_index(axis=1, inplace=True)
+
+        return merged_data

vulnwhisp/reporting/jira_api.py

@@ -0,0 +1,669 @@
+import json
+import os
+from datetime import datetime, date, timedelta
+
+from jira import JIRA
+import requests
+import logging
+from bottle import template
+import re
+
+class JiraAPI(object):
+    def __init__(self, hostname=None, username=None, password=None, path="", debug=False, clean_obsolete=True, max_time_window=12, decommission_time_window=3):
+        self.logger = logging.getLogger('JiraAPI')
+        if debug:
+            self.logger.setLevel(logging.DEBUG)
+
+        if "https://" not in hostname:
+            hostname = "https://{}".format(hostname)
+        self.username = username
+        self.password = password
+        self.jira = JIRA(options={'server': hostname}, basic_auth=(self.username, self.password))
+        self.logger.info("Created vjira service for {}".format(hostname))
+        self.all_tickets = []
+        self.excluded_tickets = []
+        self.JIRA_REOPEN_ISSUE = "Reopen Issue"
+        self.JIRA_CLOSE_ISSUE = "Close Issue"
+        self.JIRA_RESOLUTION_OBSOLETE = "Obsolete"
+        self.JIRA_RESOLUTION_FIXED = "Fixed"
+        self.template_path = 'vulnwhisp/reporting/resources/ticket.tpl'
+        self.max_ips_ticket = 30
+        self.attachment_filename = "vulnerable_assets.txt"
+        self.max_time_tracking = max_time_window #in months
+        if path:
+            self.download_tickets(path)
+        else:
+            self.logger.warn("No local path specified, skipping Jira ticket download.")
+        self.max_decommission_time = decommission_time_window #in months
+        # [HIGIENE] close tickets older than 12 months as obsolete (max_time_window defined)
+        if clean_obsolete:
+            self.close_obsolete_tickets()
+        # deletes the tag "server_decommission" from those tickets closed <=3 months ago
+        self.decommission_cleanup()
+        
+        self.jira_still_vulnerable_comment = '''This ticket has been reopened due to the vulnerability not having been fixed (if multiple assets are affected, all need to be fixed; if the server is down, lastest known vulnerability might be the one reported).
+        - In the case of the team accepting the risk and wanting to close the ticket, please add the label "*risk_accepted*" to the ticket before closing it.
+        - If server has been decommissioned, please add the label "*server_decommission*" to the ticket before closing it.
+        - If when checking the vulnerability it looks like a false positive, _+please elaborate in a comment+_ and add the label "*false_positive*" before closing it; we will review it and report it to the vendor.
+        
+        If you have further doubts, please contact the Security Team.'''
+
+    def create_ticket(self, title, desc, project="IS", components=[], tags=[], attachment_contents = []):
+        labels = ['vulnerability_management']
+        for tag in tags:
+            labels.append(str(tag))
+
+        self.logger.info("Creating ticket for project {} title: {}".format(project, title[:20]))
+        self.logger.debug("project {} has a component requirement: {}".format(project, components))
+        project_obj = self.jira.project(project)
+        components_ticket = []
+        for component in components:
+            exists = False
+            for c in project_obj.components:
+                if component == c.name:
+                    self.logger.debug("resolved component name {} to id {}".format(c.name, c.id))
+                    components_ticket.append({ "id": c.id })
+                    exists=True
+            if not exists:
+                self.logger.error("Error creating Ticket: component {} not found".format(component))
+                return 0
+        
+        try:
+            new_issue = self.jira.create_issue(project=project,
+                                               summary=title,
+                                               description=desc,
+                                               issuetype={'name': 'Bug'},
+                                               labels=labels,
+                                               components=components_ticket)
+            
+            self.logger.info("Ticket {} created successfully".format(new_issue))
+            
+            if attachment_contents:
+                self.add_content_as_attachment(new_issue, attachment_contents)
+        
+        except Exception as e:
+            self.logger.error("Failed to create ticket on Jira Project '{}'. Error: {}".format(project, e))
+            new_issue = False
+        
+        return new_issue
+    
+    #Basic JIRA Metrics
+    def metrics_open_tickets(self, project=None):
+        jql = "labels= vulnerability_management and resolution = Unresolved" 
+        if project:
+            jql += " and (project='{}')".format(project)
+        self.logger.debug('Executing: {}'.format(jql)) 
+        return len(self.jira.search_issues(jql, maxResults=0))
+
+    def metrics_closed_tickets(self, project=None):
+        jql = "labels= vulnerability_management and NOT resolution = Unresolved AND created >=startOfMonth(-{})".format(self.max_time_tracking) 
+        if project:
+            jql += " and (project='{}')".format(project)
+        return len(self.jira.search_issues(jql, maxResults=0))
+
+    def sync(self, vulnerabilities, project, components=[]):
+        #JIRA structure of each vulnerability: [source, scan_name, title, diagnosis, consequence, solution, ips, risk, references]
+        self.logger.info("JIRA Sync started")
+
+        for vuln in vulnerabilities:
+            # JIRA doesn't allow labels with spaces, so making sure that the scan_name doesn't have spaces
+            # if it has, they will be replaced by "_"
+            if " " in  vuln['scan_name']:
+                vuln['scan_name'] = "_".join(vuln['scan_name'].split(" "))
+            
+            # we exclude from the vulnerabilities to report those assets that already exist with *risk_accepted*/*server_decommission*
+            vuln = self.exclude_accepted_assets(vuln)
+            
+            # make sure after exclusion of risk_accepted assets there are still assets
+            if vuln['ips']:
+                exists = False
+                to_update = False
+                ticketid = ""
+                ticket_assets = []
+                exists, to_update, ticketid, ticket_assets = self.check_vuln_already_exists(vuln)
+
+                if exists:
+                    # If ticket "resolved" -> reopen, as vulnerability is still existent
+                    self.reopen_ticket(ticketid=ticketid, comment=self.jira_still_vulnerable_comment)
+                    self.add_label(ticketid, vuln['risk'])
+                    continue
+                elif to_update:
+                    self.ticket_update_assets(vuln, ticketid, ticket_assets)
+                    self.add_label(ticketid, vuln['risk'])
+                    continue
+                attachment_contents = []
+                # if assets >30, add as attachment
+                # create local text file with assets, attach it to ticket
+                if len(vuln['ips']) > self.max_ips_ticket:
+                    attachment_contents = vuln['ips']
+                    vuln['ips'] = ["Affected hosts ({assets}) exceed Jira's allowed character limit, added as an attachment.".format(assets = len(attachment_contents))]
+                try:
+                    tpl = template(self.template_path, vuln)
+                except Exception as e:
+                    self.logger.error('Exception templating: {}'.format(str(e)))
+                    return 0
+                self.create_ticket(title=vuln['title'], desc=tpl, project=project, components=components, tags=[vuln['source'], vuln['scan_name'], 'vulnerability', vuln['risk']], attachment_contents = attachment_contents)
+            else:
+                self.logger.info("Ignoring vulnerability as all assets are already reported in a risk_accepted ticket")
+        
+        self.close_fixed_tickets(vulnerabilities)
+        # we reinitialize so the next sync redoes the query with their specific variables
+        self.all_tickets = []
+        self.excluded_tickets = []
+        return True
+    
+    def exclude_accepted_assets(self, vuln):
+        # we want to check JIRA tickets with risk_accepted/server_decommission or false_positive labels sharing the same source
+        # will exclude tickets older than 12 months, old tickets will get closed for higiene and recreated if still vulnerable
+        labels = [vuln['source'], vuln['scan_name'], 'vulnerability_management', 'vulnerability'] 
+        
+        if not self.excluded_tickets:
+            jql = "{} AND labels in (risk_accepted,server_decommission, false_positive) AND NOT labels=advisory AND created >=startOfMonth(-{})".format(" AND ".join(["labels={}".format(label) for label in labels]), self.max_time_tracking)
+            self.excluded_tickets = self.jira.search_issues(jql, maxResults=0)
+
+        title = vuln['title']
+        #WARNING: function IGNORES DUPLICATES, after finding a "duplicate" will just return it exists
+        #it wont iterate over the rest of tickets looking for other possible duplicates/similar issues
+        self.logger.info("Comparing vulnerability to risk_accepted tickets")
+        assets_to_exclude = []
+        tickets_excluded_assets = []
+        for index in range(len(self.excluded_tickets)):
+            checking_ticketid, checking_title, checking_assets = self.ticket_get_unique_fields(self.excluded_tickets[index])
+            if title.encode('ascii') == checking_title.encode('ascii'):
+                if checking_assets:
+                    #checking_assets is a list, we add to our full list for later delete all assets
+                    assets_to_exclude+=checking_assets
+                    tickets_excluded_assets.append(checking_ticketid)
+       
+        if assets_to_exclude:
+            assets_to_remove = []
+            self.logger.warn("Vulnerable Assets seen on an already existing risk_accepted Jira ticket: {}".format(', '.join(tickets_excluded_assets)))
+            self.logger.debug("Original assets: {}".format(vuln['ips']))
+            #assets in vulnerability have the structure "ip - hostname - port", so we need to match by partial 
+            for exclusion in assets_to_exclude:
+                # for efficiency, we walk the backwards the array of ips from the scanners, as we will be popping out the matches 
+                # and we don't want it to affect the rest of the processing (otherwise, it would miss the asset right after the removed one)
+                for index in range(len(vuln['ips']))[::-1]:
+                    if exclusion == vuln['ips'][index].split(" - ")[0]:
+                        self.logger.debug("Deleting asset {} from vulnerability {}, seen in risk_accepted.".format(vuln['ips'][index], title))
+                        vuln['ips'].pop(index)
+            self.logger.debug("Modified assets: {}".format(vuln['ips']))
+
+        return vuln
+
+    def check_vuln_already_exists(self, vuln):
+        '''
+        This function compares a vulnerability with a collection of tickets.
+        Returns [exists (bool), is equal (bool), ticketid (str), assets (array)]
+        '''
+        # we need to return if the vulnerability has already been reported and the ID of the ticket for further processing
+        #function returns array [duplicated(bool), update(bool), ticketid, ticket_assets]
+        title = vuln['title']
+        labels = [vuln['source'], vuln['scan_name'], 'vulnerability_management', 'vulnerability'] 
+        #list(set()) to remove duplicates
+        assets = list(set(re.findall(r"\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b", ",".join(vuln['ips']))))
+        
+        if not self.all_tickets:
+            self.logger.info("Retrieving all JIRA tickets with the following tags {}".format(labels))
+            # we want to check all JIRA tickets, to include tickets moved to other queues
+            # will exclude tickets older than 12 months, old tickets will get closed for higiene and recreated if still vulnerable
+            jql = "{} AND NOT labels=advisory AND created >=startOfMonth(-{})".format(" AND ".join(["labels={}".format(label) for label in labels]), self.max_time_tracking)
+            
+            self.all_tickets = self.jira.search_issues(jql, maxResults=0)
+        
+        #WARNING: function IGNORES DUPLICATES, after finding a "duplicate" will just return it exists
+        #it wont iterate over the rest of tickets looking for other possible duplicates/similar issues
+        self.logger.info("Comparing Vulnerabilities to created tickets")
+        for index in range(len(self.all_tickets)):
+            checking_ticketid, checking_title, checking_assets = self.ticket_get_unique_fields(self.all_tickets[index])
+            # added "not risk_accepted", as if it is risk_accepted, we will create a new ticket excluding the accepted assets
+            if title.encode('ascii') == checking_title.encode('ascii') and not self.is_risk_accepted(self.jira.issue(checking_ticketid)): 
+                difference = list(set(assets).symmetric_difference(checking_assets))
+                #to check intersection - set(assets) & set(checking_assets)
+                if difference: 
+                    self.logger.info("Asset mismatch, ticket to update. Ticket ID: {}".format(checking_ticketid))
+                    return False, True, checking_ticketid, checking_assets #this will automatically validate
+                else:
+                    self.logger.info("Confirmed duplicated. TickedID: {}".format(checking_ticketid))
+                    return True, False, checking_ticketid, [] #this will automatically validate
+        return False, False, "", []
+
+    def ticket_get_unique_fields(self, ticket):
+        title = ticket.raw.get('fields', {}).get('summary').encode("ascii").strip()
+        ticketid = ticket.key.encode("ascii")
+
+        assets = self.get_assets_from_description(ticket)
+        if not assets:
+            #check if attachment, if so, get assets from attachment
+            assets = self.get_assets_from_attachment(ticket)
+                
+        return ticketid, title, assets
+
+    def get_assets_from_description(self, ticket, _raw = False):
+        # Get the assets as a string "host - protocol/port - hostname" separated by "\n"
+        # structure the text to have the same structure as the assets from the attachment
+        affected_assets = ""
+        try:
+            affected_assets = ticket.raw.get('fields', {}).get('description').encode("ascii").split("{panel:title=Affected Assets}")[1].split("{panel}")[0].replace('\n','').replace(' * ','\n').replace('\n', '', 1)
+        except Exception as e:
+            self.logger.error("Unable to process the Ticket's 'Affected Assets'. Ticket ID: {}. Reason: {}".format(ticket, e))
+
+        if affected_assets:
+            if _raw:
+                # from line 406 check if the text in the panel corresponds to having added an attachment
+                if "added as an attachment" in affected_assets:
+                    return False
+                return affected_assets
+
+            try:
+                # if _raw is not true, we return only the IPs of the affected assets
+                return list(set(re.findall(r"\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b", affected_assets)))
+            except Exception as e:
+                self.logger.error("Ticket IPs regex failed. Ticket ID: {}. Reason: {}".format(ticket, e))
+        return False
+
+    def get_assets_from_attachment(self, ticket, _raw = False):
+        # Get the assets as a string "host - protocol/port - hostname" separated by "\n"
+        affected_assets = []
+        try:
+            fields = self.jira.issue(ticket.key).raw.get('fields', {})
+            attachments = fields.get('attachment', {})
+            affected_assets = ""
+            #we will make sure we get the latest version of the file
+            latest = ''
+            attachment_id = ''
+            if attachments:
+                for item in attachments:
+                    if item.get('filename') == self.attachment_filename:
+                        if not latest:
+                            latest = item.get('created')
+                            attachment_id = item.get('id') 
+                        else:
+                            if latest < item.get('created'):
+                                latest = item.get('created')         
+                                attachment_id = item.get('id') 
+            affected_assets = self.jira.attachment(attachment_id).get()
+
+        except Exception as e:
+            self.logger.error("Failed to get assets from ticket attachment. Ticket ID: {}. Reason: {}".format(ticket, e))
+
+        if affected_assets:
+            if _raw:
+                return affected_assets
+
+            try:
+                # if _raw is not true, we return only the IPs of the affected assets
+                affected_assets = list(set(re.findall(r"\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b", affected_assets)))
+                return affected_assets
+            except Exception as e:
+                self.logger.error("Ticket IPs Attachment regex failed. Ticket ID: {}. Reason: {}".format(ticket, e))
+
+        return False
+
+    def parse_asset_to_json(self, asset):
+        hostname, protocol, port = "", "", ""
+        asset_info = asset.split(" - ")
+        ip = asset_info[0]
+        proto_port = asset_info[1]
+        # in case there is some case where hostname is not reported at all
+        if len(asset_info) == 3:
+            hostname = asset_info[2]
+        if proto_port != "N/A/N/A":
+            protocol, port = proto_port.split("/")
+            port = int(float(port))
+
+        asset_dict = {
+            "host": ip,
+            "protocol": protocol,
+            "port": port,
+            "hostname": hostname
+        }
+
+        return asset_dict
+
+    def clean_old_attachments(self, ticket):
+        fields = ticket.raw.get('fields')
+        attachments = fields.get('attachment')
+        if attachments:
+            for item in attachments:
+                if item.get('filename') == self.attachment_filename:
+                    self.jira.delete_attachment(item.get('id'))
+
+    def add_content_as_attachment(self, issue, contents):
+        try:
+            #Create the file locally with the data
+            attachment_file = open(self.attachment_filename, "w")
+            attachment_file.write("\n".join(contents))
+            attachment_file.close()
+            #Push the created file to the ticket
+            attachment_file = open(self.attachment_filename, "rb")
+            self.jira.add_attachment(issue, attachment_file, self.attachment_filename)
+            attachment_file.close()
+            #remove the temp file
+            os.remove(self.attachment_filename)
+            self.logger.info("Added attachment successfully.")
+        except:
+            self.logger.error("Error while attaching file to ticket.")
+            return False
+
+        return True
+
+    def get_ticket_reported_assets(self, ticket):
+        #[METRICS] return a list with all the affected assets for that vulnerability (including already resolved ones) 
+        return list(set(re.findall(r"\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b",str(self.jira.issue(ticket).raw))))
+
+    def get_resolution_time(self, ticket):
+        #get time a ticket took to be resolved
+        ticket_obj = self.jira.issue(ticket)
+        if self.is_ticket_resolved(ticket_obj):
+            ticket_data = ticket_obj.raw.get('fields')
+            #dates follow format '2018-11-06T10:36:13.849+0100'
+            created = [int(x) for x in ticket_data['created'].split('.')[0].replace('T', '-').replace(':','-').split('-')]
+            resolved =[int(x) for x in ticket_data['resolutiondate'].split('.')[0].replace('T', '-').replace(':','-').split('-')]
+            
+            start = datetime(created[0],created[1],created[2],created[3],created[4],created[5])
+            end = datetime(resolved[0],resolved[1],resolved[2],resolved[3],resolved[4],resolved[5])
+            return (end-start).days
+        else:
+            self.logger.error("Ticket {ticket} is not resolved, can't calculate resolution time".format(ticket=ticket))
+
+        return False
+
+    def ticket_update_assets(self, vuln, ticketid, ticket_assets):
+        # correct description will always be in the vulnerability to report, only needed to update description to new one
+        self.logger.info("Ticket {} exists, UPDATE requested".format(ticketid))
+        
+        #for now, if a vulnerability has been accepted ('accepted_risk'), ticket is completely ignored and not updated (no new assets)
+
+        #TODO when vulnerability accepted, create a new ticket with only the non-accepted vulnerable assets
+        #this would require go through the downloaded tickets, check duplicates/accepted ones, and if so,
+        #check on their assets to exclude them from the new ticket
+        risk_accepted = False
+        ticket_obj = self.jira.issue(ticketid)
+        if self.is_ticket_resolved(ticket_obj):
+            if self.is_risk_accepted(ticket_obj):
+                return 0
+            self.reopen_ticket(ticketid=ticketid, comment=self.jira_still_vulnerable_comment)
+        
+        #First will do the comparison of assets
+        ticket_obj.update()
+        assets = list(set(re.findall(r"\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b", ",".join(vuln['ips']))))
+        difference = list(set(assets).symmetric_difference(ticket_assets))
+        
+        comment = ''
+        added = ''
+        removed = ''
+        #put a comment with the assets that have been added/removed
+        for asset in difference:
+            if asset in assets:
+                if not added:
+                    added = '\nThe following assets *have been newly detected*:\n'
+                added += '* {}\n'.format(asset)
+            elif asset in ticket_assets:
+                if not removed:
+                    removed= '\nThe following assets *have been resolved*:\n'
+                removed += '* {}\n'.format(asset)
+
+        comment = added + removed
+        
+        #then will check if assets are too many that need to be added as an attachment
+        attachment_contents = []
+        if len(vuln['ips']) > self.max_ips_ticket:
+            attachment_contents = vuln['ips']
+            vuln['ips'] = ["Affected hosts ({assets}) exceed Jira's allowed character limit, added as an attachment.".format(assets = len(attachment_contents))]
+        
+        #fill the ticket description template
+        try:
+            tpl = template(self.template_path, vuln)
+        except Exception as e:
+            self.logger.error('Exception updating assets: {}'.format(str(e)))
+            return 0
+
+        #proceed checking if it requires adding as an attachment
+        try:
+            #update attachment with hosts and delete the old versions
+            if attachment_contents:
+                self.clean_old_attachments(ticket_obj)
+                self.add_content_as_attachment(ticket_obj, attachment_contents)
+                
+            ticket_obj.update(description=tpl, comment=comment, fields={"labels":ticket_obj.fields.labels})
+            self.logger.info("Ticket {} updated successfully".format(ticketid))
+            self.add_label(ticketid, 'updated')
+        except Exception as e:
+            self.logger.error("Error while trying up update ticket {ticketid}.\nReason: {e}".format(ticketid = ticketid, e=e))
+        return 0
+
+    def add_label(self, ticketid, label):
+        ticket_obj = self.jira.issue(ticketid)
+        
+        if label not in [x.encode('utf8') for x in ticket_obj.fields.labels]:
+            ticket_obj.fields.labels.append(label)
+        
+            try:
+                ticket_obj.update(fields={"labels":ticket_obj.fields.labels})
+                self.logger.info("Added label {label} to ticket {ticket}".format(label=label, ticket=ticketid))
+            except:
+                self.logger.error("Error while trying to add label {label} to ticket {ticket}".format(label=label, ticket=ticketid))
+        
+        return 0
+
+    def remove_label(self, ticketid, label):
+        ticket_obj = self.jira.issue(ticketid)
+        
+        if label in [x.encode('utf8') for x in ticket_obj.fields.labels]:
+            ticket_obj.fields.labels.remove(label)
+        
+            try:
+                ticket_obj.update(fields={"labels":ticket_obj.fields.labels})
+                self.logger.info("Removed label {label} from ticket {ticket}".format(label=label, ticket=ticketid))
+            except:
+                self.logger.error("Error while trying to remove label {label} to ticket {ticket}".format(label=label, ticket=ticketid))
+        else:
+            self.logger.error("Error: label {label} not in ticket {ticket}".format(label=label, ticket=ticketid))
+        
+        return 0
+
+    def close_fixed_tickets(self, vulnerabilities):
+        '''
+        Close tickets which vulnerabilities have been resolved and are still open.
+        Higiene clean up affects to all tickets created by the module, filters by label 'vulnerability_management'
+        '''
+        found_vulns = []
+        for vuln in vulnerabilities:
+            found_vulns.append(vuln['title'])
+
+        comment = '''This ticket is being closed as it appears that the vulnerability no longer exists.
+        If the vulnerability reappears, a new ticket will be opened.'''
+
+        for ticket in self.all_tickets:
+            if ticket.raw['fields']['summary'].strip() in found_vulns:
+                self.logger.info("Ticket {} is still vulnerable".format(ticket))
+                continue
+            self.logger.info("Ticket {} is no longer vulnerable".format(ticket))
+            self.close_ticket(ticket, self.JIRA_RESOLUTION_FIXED, comment) 
+        return 0
+
+
+    def is_ticket_reopenable(self, ticket_obj):
+        transitions = self.jira.transitions(ticket_obj)
+        for transition in transitions:
+            if transition.get('name') == self.JIRA_REOPEN_ISSUE:
+                self.logger.debug("Ticket is reopenable")
+                return True
+        self.logger.error("Ticket {} can't be opened. Check Jira transitions.".format(ticket_obj))
+        return False
+
+    def is_ticket_closeable(self, ticket_obj):
+        transitions = self.jira.transitions(ticket_obj)
+        for transition in transitions:
+            if transition.get('name') == self.JIRA_CLOSE_ISSUE:
+                return True
+        self.logger.error("Ticket {} can't closed. Check Jira transitions.".format(ticket_obj))
+        return False
+
+    def is_ticket_resolved(self, ticket_obj):
+        #Checks if a ticket is resolved or not
+        if ticket_obj is not None:
+            if ticket_obj.raw['fields'].get('resolution') is not None:
+                if ticket_obj.raw['fields'].get('resolution').get('name') != 'Unresolved':
+                    self.logger.debug("Checked ticket {} is already closed".format(ticket_obj))
+                    self.logger.info("Ticket {} is closed".format(ticket_obj))
+                    return True
+        self.logger.debug("Checked ticket {} is already open".format(ticket_obj))
+        return False
+
+
+    def is_risk_accepted(self, ticket_obj):
+        if ticket_obj is not None:
+            if ticket_obj.raw['fields'].get('labels') is not None:
+                labels = ticket_obj.raw['fields'].get('labels')
+                if "risk_accepted" in labels:
+                    self.logger.warn("Ticket {} accepted risk, will be ignored".format(ticket_obj))
+                    return True
+                elif "server_decommission" in labels:
+                    self.logger.warn("Ticket {} server decommissioned, will be ignored".format(ticket_obj))
+                    return True
+                elif "false_positive" in labels:
+                    self.logger.warn("Ticket {} flagged false positive, will be ignored".format(ticket_obj))
+                    return True
+        self.logger.info("Ticket {} risk has not been accepted".format(ticket_obj))
+        return False
+
+    def reopen_ticket(self, ticketid, ignore_labels=False, comment=""):
+        self.logger.debug("Ticket {} exists, REOPEN requested".format(ticketid))
+        # this will reopen a ticket by ticketid
+        ticket_obj = self.jira.issue(ticketid)
+        
+        if self.is_ticket_resolved(ticket_obj):
+            if (not self.is_risk_accepted(ticket_obj) or ignore_labels):
+                try:
+                    if self.is_ticket_reopenable(ticket_obj):
+                        error = self.jira.transition_issue(issue=ticketid, transition=self.JIRA_REOPEN_ISSUE, comment = comment)
+                        self.logger.info("Ticket {} reopened successfully".format(ticketid))
+                        if not ignore_labels:
+                            self.add_label(ticketid, 'reopened')
+                        return 1
+                except Exception as e:
+                    # continue with ticket data so that a new ticket is created in place of the "lost" one
+                    self.logger.error("error reopening ticket {}: {}".format(ticketid, e))
+                    return 0
+        return 0
+
+    def close_ticket(self, ticketid, resolution, comment):
+        # this will close a ticket by ticketid
+        self.logger.debug("Ticket {} exists, CLOSE requested".format(ticketid))
+        ticket_obj = self.jira.issue(ticketid)
+        if not self.is_ticket_resolved(ticket_obj):
+            try:
+                if self.is_ticket_closeable(ticket_obj):
+                    #need to add the label before closing the ticket
+                    self.add_label(ticketid, 'closed')
+                    error = self.jira.transition_issue(issue=ticketid, transition=self.JIRA_CLOSE_ISSUE, comment = comment, resolution = {"name": resolution })
+                    self.logger.info("Ticket {} closed successfully".format(ticketid))
+                    return 1
+            except Exception as e:
+                # continue with ticket data so that a new ticket is created in place of the "lost" one
+                self.logger.error("error closing ticket {}: {}".format(ticketid, e))
+                return 0
+                
+        return 0
+
+    def close_obsolete_tickets(self):
+        # Close tickets older than 12 months, vulnerabilities not solved will get created a new ticket 
+        self.logger.info("Closing obsolete tickets older than {} months".format(self.max_time_tracking))
+        jql = "labels=vulnerability_management AND NOT labels=advisory AND created <startOfMonth(-{}) and resolution=Unresolved".format(self.max_time_tracking)
+        tickets_to_close = self.jira.search_issues(jql, maxResults=0)
+        
+        comment = '''This ticket is being closed for hygiene, as it is more than {} months old.
+        If the vulnerability still exists, a new ticket will be opened.'''.format(self.max_time_tracking)
+        
+        for ticket in tickets_to_close:
+                self.close_ticket(ticket, self.JIRA_RESOLUTION_OBSOLETE, comment)
+        
+        return 0
+
+    def project_exists(self, project):
+        try:
+            self.jira.project(project)
+            return True
+        except:
+            return False
+        return False
+
+    def download_tickets(self, path):
+        '''
+        saves all tickets locally, local snapshot of vulnerability_management ticktes
+        '''
+        #check if file already exists
+        check_date = str(date.today())
+        fname = '{}jira_{}.json'.format(path, check_date) 
+        if os.path.isfile(fname):
+            self.logger.info("File {} already exists, skipping ticket download".format(fname))
+            return True
+        try:
+            self.logger.info("Saving locally tickets from the last {} months".format(self.max_time_tracking))
+            jql = "labels=vulnerability_management AND NOT labels=advisory AND created >=startOfMonth(-{})".format(self.max_time_tracking)
+            tickets_data = self.jira.search_issues(jql, maxResults=0)
+
+            #TODO process tickets, creating a new field called "_metadata" with all the affected assets well structured
+            # for future processing in ELK/Splunk; this includes downloading attachments with assets and processing them
+
+            processed_tickets = []
+
+            for ticket in tickets_data:
+                assets = self.get_assets_from_description(ticket, _raw=True)
+                if not assets:
+                    # check if attachment, if so, get assets from attachment
+                    assets = self.get_assets_from_attachment(ticket, _raw=True)
+                # process the affected assets to save them as json structure on a new field from the JSON
+                _metadata = {"affected_hosts": []}
+                if assets:
+                    if "\n" in assets:
+                        for asset in assets.split("\n"):
+                            assets_json = self.parse_asset_to_json(asset)
+                            _metadata["affected_hosts"].append(assets_json)
+                    else:
+                        assets_json = self.parse_asset_to_json(assets)
+                        _metadata["affected_hosts"].append(assets_json)
+
+
+                temp_ticket = ticket.raw.get('fields')
+                temp_ticket['_metadata'] = _metadata
+
+                processed_tickets.append(temp_ticket)
+            
+            #end of line needed, as writelines() doesn't add it automatically, otherwise one big line
+            to_save = [json.dumps(ticket.raw.get('fields'))+"\n" for ticket in tickets_data]
+            with open(fname, 'w') as outfile:
+                outfile.writelines(to_save)
+                self.logger.info("Tickets saved succesfully.")
+            
+            return True
+
+        except Exception as e:
+            self.logger.error("Tickets could not be saved locally: {}.".format(e))
+
+        return False 
+
+    def decommission_cleanup(self):
+        '''
+        deletes the server_decomission tag from those tickets that have been 
+        closed already for more than x months (default is 3 months) in order to clean solved issues
+        for statistics purposes
+        '''
+        self.logger.info("Deleting 'server_decommission' tag from tickets closed more than {} months ago".format(self.max_decommission_time))
+
+        jql = "labels=vulnerability_management AND labels=server_decommission and resolutiondate <=startOfMonth(-{})".format(self.max_decommission_time)
+        decommissioned_tickets = self.jira.search_issues(jql, maxResults=0)
+        
+        comment = '''This ticket is having deleted the *server_decommission* tag, as it is more than {} months old and is expected to already have been decommissioned.
+        If that is not the case and the vulnerability still exists, the vulnerability will be opened again.'''.format(self.max_decommission_time)
+        
+        for ticket in decommissioned_tickets:
+            #we open first the ticket, as we want to make sure the process is not blocked due to 
+            #an unexisting jira workflow or unallowed edit from closed tickets
+            self.reopen_ticket(ticketid=ticket, ignore_labels=True)
+            self.remove_label(ticket, 'server_decommission')
+            self.close_ticket(ticket, self.JIRA_RESOLUTION_FIXED, comment)
+        
+        return 0

vulnwhisp/reporting/resources/ticket.tpl

@@ -0,0 +1,34 @@
+{panel:title=Description}
+{{ !diagnosis}}
+{panel}
+
+
+{panel:title=Consequence}
+{{ !consequence}}
+{panel}
+
+{panel:title=Solution}
+{{ !solution}}
+{panel}
+
+{panel:title=Affected Assets}
+% for ip in ips:
+ * {{ip}}
+% end
+{panel}
+
+{panel:title=References}
+% for ref in references:
+ * {{ref}}
+% end
+{panel}
+
+
+
+Please do not delete or modify the ticket assigned tags or title, as they are used to be synced. If the ticket ceases to be recognised, a new ticket will raise.
+
+In the case of the team accepting the risk and wanting to close the ticket, please add the label "*risk_accepted*" to the ticket before closing it.
+
+If server has been decommissioned, please add the label "*server_decommission*" to the ticket before closing it.
+
+If when checking the vulnerability it looks like a false positive, _+please elaborate in a comment+_ and add the label "*false_positive*" before closing it; we will review it and report it to the vendor.

vulnwhisp/test/mock.py

@@ -0,0 +1,76 @@
+import os
+import logging
+import httpretty
+
+
+class mockAPI(object):
+    def __init__(self, mock_dir=None, debug=False):
+        self.mock_dir = mock_dir
+
+        if not self.mock_dir:
+            # Try to guess the mock_dir if python setup.py develop was used
+            self.mock_dir = '/'.join(__file__.split('/')[:-3]) + '/tests/data'
+
+        self.logger = logging.getLogger('mockAPI')
+        if debug:
+            self.logger.setLevel(logging.DEBUG)
+
+        self.logger.info('mockAPI initialised, API requests will be mocked')
+        self.logger.debug('Test path resolved as {}'.format(self.mock_dir))
+
+    def get_directories(self, path):
+        dir, subdirs, files = next(os.walk(path))
+        return subdirs
+
+    def get_files(self, path):
+        dir, subdirs, files = next(os.walk(path))
+        return files
+
+    def qualys_vuln_callback(self, request, uri, response_headers):
+        self.logger.debug('Simulating response for {} ({})'.format(uri, request.body))
+        if 'list' in request.parsed_body['action']:
+            return [200,
+                    response_headers,
+                    open('{}/{}'.format(self.qualys_vuln_path, 'scans')).read()]
+        elif 'fetch' in request.parsed_body['action']:
+            try:
+                response_body = open('{}/{}'.format(
+                                      self.qualys_vuln_path,
+                                      request.parsed_body['scan_ref'][0].replace('/', '_'))
+                                    ).read()
+            except:
+                # Can't find the file, just send an empty response
+                response_body = ''
+        return [200, response_headers, response_body]
+
+    def create_nessus_resource(self, framework):
+        for filename in self.get_files('{}/{}'.format(self.mock_dir, framework)):
+            method, resource = filename.split('_', 1)
+            resource = resource.replace('_', '/')
+            self.logger.debug('Adding mocked {} endpoint {} {}'.format(framework, method, resource))
+            httpretty.register_uri(
+                getattr(httpretty, method), 'https://{}:443/{}'.format(framework, resource),
+                body=open('{}/{}/{}'.format(self.mock_dir, framework, filename)).read()
+            )
+
+    def create_qualys_vuln_resource(self, framework):
+        # Create health check endpoint
+        self.logger.debug('Adding mocked {} endpoint {} {}'.format(framework, 'GET', 'msp/about.php'))
+        httpretty.register_uri(
+                httpretty.GET,
+                'https://{}:443/{}'.format(framework, 'msp/about.php'),
+                body='')
+
+        self.logger.debug('Adding mocked {} endpoint {} {}'.format(framework, 'POST', 'api/2.0/fo/scan'))
+        httpretty.register_uri(
+            httpretty.POST, 'https://{}:443/{}'.format(framework, 'api/2.0/fo/scan/'),
+            body=self.qualys_vuln_callback)
+
+    def mock_endpoints(self):
+        for framework in self.get_directories(self.mock_dir):
+            if framework in ['nessus', 'tenable']:
+                self.create_nessus_resource(framework)
+            elif framework == 'qualys_vuln':
+                self.qualys_vuln_path = self.mock_dir + '/' + framework
+                self.create_qualys_vuln_resource(framework)
+        httpretty.enable()

vulnwhisp/utils/cli.py

@@ -1,17 +0,0 @@
-class bcolors:
-    """
-    Utility to add colors to shell for scripts
-    """
-    HEADERS = '\033[95m'
-    OKBLUE = '\033[94m'
-    OKGREEN = '\033[92m'
-    WARNING = '\033[93m'
-    FAIL = '\033[91m'
-    ENDC = '\033[0m'
-    BOLD = '\033[1m'
-    UNDERLINE = '\033[4m'
-
-    INFO = '{info}[INFO]{endc}'.format(info=OKBLUE, endc=ENDC)
-    ACTION = '{info}[ACTION]{endc}'.format(info=OKBLUE, endc=ENDC)
-    SUCCESS = '{green}[SUCCESS]{endc}'.format(green=OKGREEN, endc=ENDC)
-    FAIL = '{red}[FAIL]{endc}'.format(red=FAIL, endc=ENDC)