Compare commits
12 Commits
2to3
...
a9289d8e47
Author | SHA1 | Date | |
---|---|---|---|
a9289d8e47 | |||
67ec8af3ae | |||
691f45a1dc | |||
80197454a3 | |||
841cd09f2d | |||
e7183864d0 | |||
12ac3dbf62 | |||
e41ec93058 | |||
8a86e3142a | |||
9d003d12b4 | |||
63c638751b | |||
a3e85b7207 |
53
.github/ISSUE_TEMPLATE/bug_report.md
vendored
53
.github/ISSUE_TEMPLATE/bug_report.md
vendored
@ -1,42 +1,41 @@
|
|||||||
---
|
---
|
||||||
name: Bug report
|
name: Rapport de bug
|
||||||
about: Create a report to help us improve
|
about: Créez un rapport pour nous aider à nous améliorer
|
||||||
title: ''
|
title: ''
|
||||||
labels: ''
|
labels: ''
|
||||||
assignees: ''
|
assignees: ''
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
**Describe the bug**
|
**Décrivez le bug**
|
||||||
A clear and concise description of what the bug is.
|
Une description claire et concise de ce qu'est le bug.
|
||||||
|
|
||||||
**Affected module**
|
**Module affecté**
|
||||||
Which one is the module that is not working as expected, e.g. Nessus, Qualys WAS, Qualys VM, OpenVAS, ELK, Jira...).
|
Lequel des modules ne fonctionne pas comme prévu, par exemple, Nessus, Qualys WAS, Qualys VM, OpenVAS, ELK, Jira...
|
||||||
|
|
||||||
**VulnWhisperer debug trail**
|
**Trace de débogage de VulnWhisperer**
|
||||||
If applicable, paste the VulnWhisperer debug trail of the execution for further detail (execute with '-d' flag).
|
Si possible, veuillez joindre la trace de débogage de l'exécution pour une enquête plus approfondie (exécuter avec l'option `-d`).
|
||||||
|
|
||||||
**To Reproduce**
|
**Pour reproduire**
|
||||||
Steps to reproduce the behavior:
|
Étapes pour reproduire le comportement :
|
||||||
1. Go to '...'
|
1. Allez à '...'
|
||||||
2. Click on '....'
|
2. Cliquez sur '....'
|
||||||
3. Scroll down to '....'
|
3. Voir l'erreur
|
||||||
4. See error
|
|
||||||
|
|
||||||
**Expected behavior**
|
**Comportement attendu**
|
||||||
A clear and concise description of what you expected to happen.
|
Une description claire et concise de ce à quoi vous vous attendiez.
|
||||||
|
|
||||||
**Screenshots**
|
**Captures d'écran**
|
||||||
If applicable, add screenshots to help explain your problem.
|
Si applicable, ajoutez des captures d'écran pour aider à expliquer votre problème.
|
||||||
|
|
||||||
**System in which VulnWhisperer runs (please complete the following information):**
|
**Système sur lequel VulnWhisperer s'exécute (veuillez compléter les informations suivantes) :**
|
||||||
- OS: [e.g. Ubuntu Server]
|
- OS : [ex. Ubuntu Server]
|
||||||
- Version: [e.g. 18.04.2 LTS]
|
- Version : [ex. 18.04.2 LTS]
|
||||||
- VulnWhisperer Version: [e.g. 1.7.1]
|
- Version de VulnWhisperer : [ex. 1.7.1]
|
||||||
|
|
||||||
**Additional context**
|
**Contexte additionnel**
|
||||||
Add any other context about the problem here.
|
Ajoutez tout autre contexte sur le problème ici.
|
||||||
|
|
||||||
## Important Note
|
**Note importante**
|
||||||
As VulnWhisperer relies on ELK for the data aggregation, it is expected that you already have an ELK instance or the knowledge to deploy one.
|
Comme VulnWhisperer s'appuie sur ELK pour l'agrégation de données, il est attendu que vous ayez déjà une instance ELK ou les connaissances pour en déployer une.
|
||||||
In order to speed up deployment, we provide an updated and tested docker-compose file which deploys all the needed infrastructure and we will support its deployment, but we will not be giving support to ELK instances.
|
Pour accélérer le déploiement, nous fournissons un fichier docker-compose à jour et testé qui déploie toute l'infrastructure nécessaire et nous supporterons son déploiement, mais nous ne donnerons pas de support pour les instances ELK.
|
186
README.md
186
README.md
@ -1,19 +1,15 @@
|
|||||||
<p align="center"><img src="https://github.com/austin-taylor/vulnwhisperer/blob/master/docs/source/vuln_whisperer_logo_s.png" width="400px"></p>
|
<p align="center"><img src="https://git.gudita.com/Cyberdefense/VulnWhisperer/raw/branch/master/docs/source/vuln_whisperer_logo_s.png" width="400px"></p>
|
||||||
<p align="center"> <i>Create <u><b>actionable data</b></u> from your vulnerability scans </i> </p>
|
<p align="center"> <i>Créez des <u><b>données exploitables</b></u> à partir de vos scans de vulnérabilités</i> </p>
|
||||||
|
|
||||||
<p align="center" style="width:400px"><img src="https://github.com/austin-taylor/vulnwhisperer/blob/master/docs/source/vulnWhispererWebApplications.png" style="width:400px"></p>
|
<p align="center" style="width:400px"><img src="https://git.gudita.com/Cyberdefense/VulnWhisperer/raw/branch/master/docs/source/vulnWhispererWebApplications.png" style="width:400px"></p>
|
||||||
|
|
||||||
|
|
||||||
VulnWhisperer is a vulnerability management tool and report aggregator. VulnWhisperer will pull all the reports from the different Vulnerability scanners and create a file with a unique filename for each one, using that data later to sync with Jira and feed Logstash. Jira does a closed cycle full Sync with the data provided by the Scanners, while Logstash indexes and tags all of the information inside the report (see logstash files at /resources/elk6/pipeline/). Data is then shipped to ElasticSearch to be indexed, and ends up in a visual and searchable format in Kibana with already defined dashboards.
|
VulnWhisperer est un outil de gestion des vulnérabilités et un agrégateur de rapports. VulnWhisperer récupère tous les rapports des différents scanners de vulnérabilités et crée un fichier avec un nom unique pour chacun, utilisant ensuite ces données pour se synchroniser avec Jira et alimenter Logstash. Jira effectue une synchronisation complète en cycle fermé avec les données fournies par les scanners, tandis que Logstash indexe et étiquette toutes les informations contenues dans le rapport (voir les fichiers logstash dans `/resources/elk6/pipeline/`). Les données sont ensuite envoyées à ElasticSearch pour être indexées, et finissent dans un format visuel et consultable dans Kibana avec des tableaux de bord déjà définis.
|
||||||
|
|
||||||
[](https://travis-ci.org/HASecuritySolutions/VulnWhisperer)
|
VulnWhisperer est un projet open-source financé par la communauté. VulnWhisperer est actuellement fonctionnel mais nécessite une refonte de la documentation et une revue de code. Si vous souhaitez de l'aide, si vous êtes intéressé par de nouvelles fonctionnalités, ou si vous recherchez un support payant, veuillez nous contacter à **info@sahelcyber.com**.
|
||||||
[](http://choosealicense.com/licenses/mit/)
|
|
||||||
[](https://twitter.com/VulnWhisperer)
|
|
||||||
|
|
||||||
Currently Supports
|
|
||||||
-----------------
|
|
||||||
|
|
||||||
### Vulnerability Frameworks
|
### Scanners de Vulnérabilités Supportés
|
||||||
|
|
||||||
- [X] [Nessus (**v6**/**v7**/**v8**)](https://www.tenable.com/products/nessus/nessus-professional)
|
- [X] [Nessus (**v6**/**v7**/**v8**)](https://www.tenable.com/products/nessus/nessus-professional)
|
||||||
- [X] [Qualys Web Applications](https://www.qualys.com/apps/web-app-scanning/)
|
- [X] [Qualys Web Applications](https://www.qualys.com/apps/web-app-scanning/)
|
||||||
@ -26,143 +22,117 @@ Currently Supports
|
|||||||
- [ ] [NMAP](https://nmap.org/)
|
- [ ] [NMAP](https://nmap.org/)
|
||||||
- [ ] [Burp Suite](https://portswigger.net/burp)
|
- [ ] [Burp Suite](https://portswigger.net/burp)
|
||||||
- [ ] [OWASP ZAP](https://www.zaproxy.org/)
|
- [ ] [OWASP ZAP](https://www.zaproxy.org/)
|
||||||
- [ ] More to come
|
- [ ] Et d'autres à venir
|
||||||
|
|
||||||
### Reporting Frameworks
|
### Plateformes de Reporting Supportées
|
||||||
|
|
||||||
- [X] [ELK (**v6**/**v7**)](https://www.elastic.co/elk-stack)
|
- [X] [Elastic Stack (**v6**/**v7**)](https://www.elastic.co/elk-stack)
|
||||||
|
- [ ] [OpenSearch - Envisagé pour la prochaine mise à jour](https://opensearch.org/)
|
||||||
- [X] [Jira](https://www.atlassian.com/software/jira)
|
- [X] [Jira](https://www.atlassian.com/software/jira)
|
||||||
- [ ] [Splunk](https://www.splunk.com/)
|
- [ ] [Splunk](https://www.splunk.com/)
|
||||||
|
|
||||||
Getting Started
|
## Démarrage
|
||||||
===============
|
|
||||||
|
|
||||||
1) Follow the [install requirements](#installreq)
|
1) Suivez les [prérequis d'installation](#installreq)
|
||||||
2) Fill out the section you want to process in <a href="https://github.com/HASecuritySolutions/VulnWhisperer/blob/master/configs/frameworks_example.ini">frameworks_example.ini file</a>
|
2) Remplissez la section que vous souhaitez traiter dans le fichier <a href="https://git.gudita.com/Cyberdefense/VulnWhisperer/src/branch/master/configs/frameworks_example.ini">frameworks_example.ini</a>
|
||||||
3) [JIRA] If using Jira, fill Jira config in the config file mentioned above.
|
3) [JIRA] Si vous utilisez Jira, remplissez la configuration Jira dans le fichier de configuration mentionné ci-dessus.
|
||||||
3) [ELK] Modify the IP settings in the <a href="https://github.com/HASecuritySolutions/VulnWhisperer/tree/master/resources/elk6/pipeline">Logstash files to accommodate your environment</a> and import them to your logstash conf directory (default is /etc/logstash/conf.d/)
|
3) [ELK] Modifiez les paramètres IP dans les <a href="https://git.gudita.com/Cyberdefense/VulnWhisperer/src/branch/master/resources/elk6/pipeline">fichiers Logstash pour correspondre à votre environnement</a> et importez-les dans votre répertoire de configuration logstash (par défaut `/etc/logstash/conf.d/`)
|
||||||
4) [ELK] Import the <a href="https://github.com/HASecuritySolutions/VulnWhisperer/blob/master/resources/elk6/kibana.json">Kibana visualizations</a>
|
4) [ELK] Importez les <a href="https://git.gudita.com/Cyberdefense/VulnWhisperer/src/branch/master/resources/elk6/kibana.json">visualisations Kibana</a>
|
||||||
5) [Run Vulnwhisperer](#run)
|
5) [Exécutez Vulnwhisperer](#run)
|
||||||
|
|
||||||
Need assistance or just want to chat? Join our [slack channel](https://join.slack.com/t/vulnwhisperer/shared_invite/enQtNDQ5MzE4OTIyODU0LWQxZTcxYTY0MWUwYzA4MTlmMWZlYWY2Y2ZmM2EzNDFmNWVlOTM4MzNjYzI0YzdkMDA0YmQyYWRhZGI2NGUxNGI)
|
> **Note importante concernant les liens du Wiki :** La migration de Gitea ne transfère pas toujours le Wiki d'un projet GitHub (qui est techniquement un dépôt séparé). Si les liens vers le Wiki (comme le guide de déploiement ELK) ne fonctionnent pas, vous devrez peut-être recréer ces pages manuellement dans l'onglet "Wiki" de votre dépôt sur Gitea.
|
||||||
|
|
||||||
Requirements
|
Besoin d'aide ou juste envie de discuter ? Rejoignez notre [canal Slack](https://join.slack.com/t/vulnwhisperer/shared_invite/enQtNDQ5MzE4OTIyODU0LWQxZTcxYTY0MWUwYzA4MTlmMWZlYWY2Y2ZmM2EzNDFmNWVlOTM4MzNjYzI0YzdkMDA0YmQyYWRhZGI2NGUxNGI)
|
||||||
-------------
|
|
||||||
####
|
## Prérequis
|
||||||
* Python 2.7
|
* Python 2.7
|
||||||
* Vulnerability Scanner
|
* Un Scanner de Vulnérabilités
|
||||||
* Reporting System: Jira / ElasticStack 6.6
|
* Un Système de Reporting : Jira / ElasticStack 6.6
|
||||||
|
|
||||||
<a id="installreq">Install Requirements-VulnWhisperer(may require sudo)</a>
|
<a id="installreq"></a>
|
||||||
--------------------
|
## Prérequis d'Installation - VulnWhisperer (peut nécessiter sudo)
|
||||||
**Install OS packages requirement dependencies** (Debian-based distros, CentOS don't need it)
|
**Installez les dépendances des paquets du système d'exploitation** (pour les distributions basées sur Debian, CentOS n'en a pas besoin)
|
||||||
```shell
|
```shell
|
||||||
|
|
||||||
sudo apt-get install zlib1g-dev libxml2-dev libxslt1-dev
|
sudo apt-get install zlib1g-dev libxml2-dev libxslt1-dev
|
||||||
```
|
|
||||||
|
|
||||||
**(Optional) Use a python virtualenv to not mess with host python libraries**
|
(Optionnel) Utilisez un environnement virtuel python pour ne pas perturber les bibliothèques python de l'hôte
|
||||||
```shell
|
|
||||||
virtualenv venv (will create the python 2.7 virtualenv)
|
|
||||||
source venv/bin/activate (start the virtualenv, now pip will run there and should install libraries without sudo)
|
|
||||||
|
|
||||||
deactivate (for quitting the virtualenv once you are done)
|
virtualenv venv # créera l'environnement virtuel python 2.7
|
||||||
```
|
|
||||||
|
|
||||||
**Install python libraries requirements**
|
source venv/bin/activate # démarre l'environnement, pip s'exécutera ici et devrait installer les bibliothèques sans sudo
|
||||||
|
|
||||||
```python
|
deactivate # pour quitter l'environnement virtuel une fois que vous avez terminé
|
||||||
pip install -r /path/to/VulnWhisperer/requirements.txt
|
|
||||||
cd /path/to/VulnWhisperer
|
Installez les dépendances des bibliothèques python
|
||||||
|
|
||||||
|
pip install -r /chemin/vers/VulnWhisperer/requirements.txt
|
||||||
|
cd /chemin/vers/VulnWhisperer
|
||||||
python setup.py install
|
python setup.py install
|
||||||
```
|
|
||||||
|
|
||||||
**(Optional) If using a proxy, add proxy URL as environment variable to PATH**
|
(Optionnel) Si vous utilisez un proxy, ajoutez l'URL du proxy comme variable d'environnement au PATH
|
||||||
```shell
|
|
||||||
export HTTP_PROXY=http://example.com:8080
|
|
||||||
export HTTPS_PROXY=http://example.com:8080
|
|
||||||
```
|
|
||||||
|
|
||||||
Now you're ready to pull down scans. (see <a href="#run">run section</a>)
|
export HTTP_PROXY=[http://exemple.com:8080](http://exemple.com:8080)
|
||||||
|
export HTTPS_PROXY=[http://exemple.com:8080](http://exemple.com:8080)
|
||||||
|
|
||||||
|
Vous êtes maintenant prêt à télécharger les scans.
|
||||||
|
|
||||||
Configuration
|
Configuration
|
||||||
-----
|
Il y a quelques étapes de configuration pour mettre en place VulnWhisperer :
|
||||||
|
|
||||||
There are a few configuration steps to setting up VulnWhisperer:
|
Configurer le fichier Ini
|
||||||
* Configure Ini file
|
|
||||||
* Setup Logstash File
|
|
||||||
* Import ElasticSearch Templates
|
|
||||||
* Import Kibana Dashboards
|
|
||||||
|
|
||||||
<a href="https://github.com/austin-taylor/VulnWhisperer/blob/master/configs/frameworks_example.ini">frameworks_example.ini file</a>
|
Configurer le fichier Logstash
|
||||||
<p align="left" style="width:200px"><img src="https://github.com/austin-taylor/vulnwhisperer/blob/master/docs/source/config_example.png" style="width:200px"></p>
|
|
||||||
|
|
||||||
|
Importer les modèles ElasticSearch
|
||||||
|
|
||||||
<a id="run">Run</a>
|
Importer les tableaux de bord Kibana
|
||||||
-----
|
|
||||||
To run, fill out the configuration file with your vulnerability scanner settings. Then you can execute from the command line.
|
Exécution
|
||||||
```python
|
Pour exécuter, remplissez le fichier de configuration avec les paramètres de votre scanner de vulnérabilités. Ensuite, vous pouvez l'exécuter depuis la ligne de commande.
|
||||||
(optional flag: -F -> provides "Fancy" log colouring, good for comprehension when manually executing VulnWhisperer)
|
|
||||||
|
# (optionnel : -F -> fournit une coloration "Fantaisie" des logs, utile pour la compréhension lors de l'exécution manuelle de VulnWhisperer)
|
||||||
vuln_whisperer -c configs/frameworks_example.ini -s nessus
|
vuln_whisperer -c configs/frameworks_example.ini -s nessus
|
||||||
or
|
# ou
|
||||||
vuln_whisperer -c configs/frameworks_example.ini -s qualys
|
vuln_whisperer -c configs/frameworks_example.ini -s qualys
|
||||||
|
|
||||||
```
|
Si aucune section n'est spécifiée (ex. -s nessus), vulnwhisperer vérifiera dans le fichier de configuration les modules ayant la propriété enabled=true et les exécutera séquentiellement.
|
||||||
If no section is specified (e.g. -s nessus), vulnwhisperer will check on the config file for the modules that have the property `enabled=true` and run them sequentially.
|
|
||||||
|
|
||||||
<p align="center" style="width:300px"><img src="https://github.com/austin-taylor/vulnwhisperer/blob/master/docs/source/running_vuln_whisperer.png" style="width:400px"></p>
|
|
||||||
Next you'll need to import the visualizations into Kibana and setup your logstash config. You can either follow the sample setup instructions [here](https://github.com/HASecuritySolutions/VulnWhisperer/wiki/Sample-Guide-ELK-Deployment) or go for the `docker-compose` solution we offer.
|
|
||||||
|
|
||||||
|
|
||||||
Docker-compose
|
Docker-compose
|
||||||
-----
|
ELK est un monde en soi, et pour les nouveaux venus sur la plateforme, cela nécessite des compétences de base sous Linux et généralement un peu de dépannage jusqu'à ce qu'il soit déployé et fonctionne comme prévu. Comme nous ne sommes pas en mesure de fournir un support pour les problèmes ELK de chaque utilisateur, nous avons mis en place un docker-compose qui inclut :
|
||||||
ELK is a whole world by itself, and for newcomers to the platform, it requires basic Linux skills and usually a bit of troubleshooting until it is deployed and working as expected. As we are not able to provide support for each users ELK problems, we put together a docker-compose which includes:
|
|
||||||
|
|
||||||
- VulnWhisperer
|
VulnWhisperer
|
||||||
- Logstash 6.6
|
|
||||||
- ElasticSearch 6.6
|
|
||||||
- Kibana 6.6
|
|
||||||
|
|
||||||
The docker-compose just requires specifying the paths where the VulnWhisperer data will be saved, and where the config files reside. If ran directly after `git clone`, with just adding the Scanner config to the VulnWhisperer config file ([/resources/elk6/vulnwhisperer.ini](https://github.com/HASecuritySolutions/VulnWhisperer/blob/master/resources/elk6/vulnwhisperer.ini)), it will work out of the box.
|
Logstash 6.6
|
||||||
|
|
||||||
It also takes care to load the Kibana Dashboards and Visualizations automatically through the API, which needs to be done manually otherwise at Kibana's startup.
|
ElasticSearch 6.6
|
||||||
|
|
||||||
For more info about the docker-compose, check on the [docker-compose wiki](https://github.com/HASecuritySolutions/VulnWhisperer/wiki/docker-compose-Instructions) or the [FAQ](https://github.com/HASecuritySolutions/VulnWhisperer/wiki).
|
Kibana 6.6
|
||||||
|
|
||||||
Getting Started
|
Le docker-compose nécessite simplement de spécifier les chemins où les données de VulnWhisperer seront sauvegardées, et où se trouvent les fichiers de configuration. S'il est exécuté directement après un git clone, en ajoutant simplement la configuration du scanner au fichier de configuration de VulnWhisperer (/resources/elk6/vulnwhisperer.ini), il fonctionnera immédiatement.
|
||||||
===============
|
|
||||||
|
|
||||||
Our current Roadmap is as follows:
|
Il se charge également de charger automatiquement les tableaux de bord et les visualisations Kibana via l'API, ce qui doit être fait manuellement autrement au démarrage de Kibana.
|
||||||
- [ ] Create a Vulnerability Standard
|
|
||||||
- [ ] Map every scanner results to the standard
|
|
||||||
- [ ] Create Scanner module guidelines for easy integration of new scanners (consistency will allow #14)
|
|
||||||
- [ ] Refactor the code to reuse functions and enable full compatibility among modules
|
|
||||||
- [ ] Change Nessus CSV to JSON (Consistency and Fix #82)
|
|
||||||
- [ ] Adapt single Logstash to standard and Kibana Dashboards
|
|
||||||
- [ ] Implement Detectify Scanner
|
|
||||||
- [ ] Implement Splunk Reporting/Dashboards
|
|
||||||
|
|
||||||
On top of this, we try to focus on fixing bugs as soon as possible, which might delay the development. We also very welcome PR's, and once we have the new standard implemented, it will be very easy to add compatibility with new scanners.
|
Pour plus d'informations sur le docker-compose, consultez le wiki docker-compose ou la FAQ.
|
||||||
|
|
||||||
The Vulnerability Standard will initially be a new simple one level JSON with all the information that matches from the different scanners having standardized variable names, while maintaining the rest of the variables as they are. In the future, once everything is implemented, we will evaluate moving to an existing standard like ECS or AWS Vulnerability Schema; we prioritize functionality over perfection.
|
Feuille de route
|
||||||
|
Notre feuille de route actuelle est la suivante :
|
||||||
|
|
||||||
Video Walkthrough -- Featured on ElasticWebinar
|
[ ] Créer un standard de vulnérabilité
|
||||||
----------------------------------------------
|
|
||||||
<a href="http://www.youtube.com/watch?feature=player_embedded&v=zrEuTtRUfNw?start=30
|
|
||||||
" target="_blank"><img src="https://github.com/austin-taylor/vulnwhisperer/blob/master/docs/source/elastic_webinar.png"
|
|
||||||
alt="Elastic presentation on VulnWhisperer" border="10" /></a>
|
|
||||||
|
|
||||||
Authors
|
[ ] Mapper les résultats de chaque scanner au standard
|
||||||
------
|
|
||||||
- [Austin Taylor (@HuntOperator)](https://github.com/austin-taylor)
|
|
||||||
- [Justin Henderson (@smapper)](https://github.com/SMAPPER)
|
|
||||||
|
|
||||||
Contributors
|
[ ] Créer des directives de module de scanner pour une intégration facile de nouveaux scanners
|
||||||
------------
|
|
||||||
- [Quim Montal (@qmontal)](https://github.com/qmontal)
|
|
||||||
- [@pemontto](https://github.com/pemontto)
|
|
||||||
- [@cybergoof](https://github.com/cybergoof)
|
|
||||||
|
|
||||||
AS SEEN ON TV
|
[ ] Refactoriser le code pour réutiliser les fonctions et permettre une compatibilité totale entre les modules
|
||||||
-------------
|
|
||||||
<p align="center" style="width:400px"><a href="https://twitter.com/MalwareJake/status/935654519471353856"><img src="https://github.com/austin-taylor/vulnwhisperer/blob/master/docs/source/as_seen_on_tv.png" style="width:400px"></a></p>
|
[ ] Changer Nessus CSV en JSON
|
||||||
|
|
||||||
|
[ ] Adapter le Logstash unique au standard et aux tableaux de bord Kibana
|
||||||
|
|
||||||
|
[ ] Implémenter le scanner Detectify
|
||||||
|
|
||||||
|
[ ] Implémenter le reporting/tableaux de bord Splunk
|
||||||
|
|
||||||
|
En plus de cela, nous essayons de nous concentrer sur la correction des bugs dès que possible, ce qui peut retarder le développement. Nous accueillons également très volontiers les PR (Pull Requests), et une fois que nous aurons implémenté le nouveau standard, il sera très facile d'ajouter la compatibilité avec de nouveaux scanners.
|
||||||
|
|
||||||
|
Le standard de vulnérabilité sera initialement un nouveau JSON simple à un niveau avec toutes les informations correspondantes des différents scanners ayant des noms de variables standardisés, tout en conservant le reste des variables telles quelles.
|
||||||
|
@ -93,7 +93,7 @@ def main():
|
|||||||
scanname=args.scanname)
|
scanname=args.scanname)
|
||||||
exit_code += vw.whisper_vulnerabilities()
|
exit_code += vw.whisper_vulnerabilities()
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error("VulnWhisperer was unable to perform the processing on '{}'".format(section))
|
logger.error("VulnWhisperer was unable to perform the processing on '{}'".format(args.source))
|
||||||
else:
|
else:
|
||||||
logger.info('Running vulnwhisperer for section {}'.format(args.section))
|
logger.info('Running vulnwhisperer for section {}'.format(args.section))
|
||||||
vw = vulnWhisperer(config=args.config,
|
vw = vulnWhisperer(config=args.config,
|
||||||
|
@ -6,8 +6,8 @@ access_key=
|
|||||||
secret_key=
|
secret_key=
|
||||||
username=nessus_username
|
username=nessus_username
|
||||||
password=nessus_password
|
password=nessus_password
|
||||||
write_path=/tmp/VulnWhisperer/data/nessus/
|
write_path=/opt/VulnWhisperer/data/nessus/
|
||||||
db_path=/tmp/VulnWhisperer/data/database
|
db_path=/opt/VulnWhisperer/data/database
|
||||||
trash=false
|
trash=false
|
||||||
verbose=true
|
verbose=true
|
||||||
|
|
||||||
@ -19,8 +19,8 @@ access_key=
|
|||||||
secret_key=
|
secret_key=
|
||||||
username=tenable.io_username
|
username=tenable.io_username
|
||||||
password=tenable.io_password
|
password=tenable.io_password
|
||||||
write_path=/tmp/VulnWhisperer/data/tenable/
|
write_path=/opt/VulnWhisperer/data/tenable/
|
||||||
db_path=/tmp/VulnWhisperer/data/database
|
db_path=/opt/VulnWhisperer/data/database
|
||||||
trash=false
|
trash=false
|
||||||
verbose=true
|
verbose=true
|
||||||
|
|
||||||
@ -30,8 +30,8 @@ enabled = false
|
|||||||
hostname = qualys_web
|
hostname = qualys_web
|
||||||
username = exampleuser
|
username = exampleuser
|
||||||
password = examplepass
|
password = examplepass
|
||||||
write_path=/tmp/VulnWhisperer/data/qualys_web/
|
write_path=/opt/VulnWhisperer/data/qualys_web/
|
||||||
db_path=/tmp/VulnWhisperer/data/database
|
db_path=/opt/VulnWhisperer/data/database
|
||||||
verbose=true
|
verbose=true
|
||||||
|
|
||||||
# Set the maximum number of retries each connection should attempt.
|
# Set the maximum number of retries each connection should attempt.
|
||||||
@ -46,8 +46,8 @@ enabled = true
|
|||||||
hostname = qualys_vuln
|
hostname = qualys_vuln
|
||||||
username = exampleuser
|
username = exampleuser
|
||||||
password = examplepass
|
password = examplepass
|
||||||
write_path=/tmp/VulnWhisperer/data/qualys_vuln/
|
write_path=/opt/VulnWhisperer/data/qualys_vuln/
|
||||||
db_path=/tmp/VulnWhisperer/data/database
|
db_path=/opt/VulnWhisperer/data/database
|
||||||
verbose=true
|
verbose=true
|
||||||
|
|
||||||
[detectify]
|
[detectify]
|
||||||
@ -58,8 +58,8 @@ hostname = detectify
|
|||||||
username = exampleuser
|
username = exampleuser
|
||||||
#password variable used as secretKey
|
#password variable used as secretKey
|
||||||
password = examplepass
|
password = examplepass
|
||||||
write_path =/tmp/VulnWhisperer/data/detectify/
|
write_path =/opt/VulnWhisperer/data/detectify/
|
||||||
db_path = /tmp/VulnWhisperer/data/database
|
db_path = /opt/VulnWhisperer/data/database
|
||||||
verbose = true
|
verbose = true
|
||||||
|
|
||||||
[openvas]
|
[openvas]
|
||||||
@ -68,8 +68,8 @@ hostname = openvas
|
|||||||
port = 4000
|
port = 4000
|
||||||
username = exampleuser
|
username = exampleuser
|
||||||
password = examplepass
|
password = examplepass
|
||||||
write_path=/tmp/VulnWhisperer/data/openvas/
|
write_path=/opt/VulnWhisperer/data/openvas/
|
||||||
db_path=/tmp/VulnWhisperer/data/database
|
db_path=/opt/VulnWhisperer/data/database
|
||||||
verbose=true
|
verbose=true
|
||||||
|
|
||||||
[jira]
|
[jira]
|
||||||
@ -77,8 +77,8 @@ enabled = false
|
|||||||
hostname = jira-host
|
hostname = jira-host
|
||||||
username = username
|
username = username
|
||||||
password = password
|
password = password
|
||||||
write_path = /tmp/VulnWhisperer/data/jira/
|
write_path = /opt/VulnWhisperer/data/jira/
|
||||||
db_path = /tmp/VulnWhisperer/data/database
|
db_path = /opt/VulnWhisperer/data/database
|
||||||
verbose = true
|
verbose = true
|
||||||
dns_resolv = False
|
dns_resolv = False
|
||||||
|
|
||||||
|
@ -2,7 +2,7 @@ pandas==0.20.3
|
|||||||
setuptools==40.4.3
|
setuptools==40.4.3
|
||||||
pytz==2017.2
|
pytz==2017.2
|
||||||
Requests==2.20.0
|
Requests==2.20.0
|
||||||
lxml==4.1.1
|
lxml==4.6.5
|
||||||
future-fstrings
|
future-fstrings
|
||||||
bs4
|
bs4
|
||||||
jira
|
jira
|
||||||
|
@ -2,7 +2,7 @@
|
|||||||
# Email: austin@hasecuritysolutions.com
|
# Email: austin@hasecuritysolutions.com
|
||||||
# Last Update: 03/04/2018
|
# Last Update: 03/04/2018
|
||||||
# Version 0.3
|
# Version 0.3
|
||||||
# Description: Take in qualys web scan reports from vulnWhisperer and pumps into logstash
|
# Description: Take in Openvas web scan reports from vulnWhisperer and pumps into logstash
|
||||||
|
|
||||||
input {
|
input {
|
||||||
file {
|
file {
|
||||||
|
1
setup.py
1
setup.py
@ -1,6 +1,5 @@
|
|||||||
#!/usr/bin/env python
|
#!/usr/bin/env python
|
||||||
|
|
||||||
from __future__ import absolute_import
|
|
||||||
from setuptools import setup, find_packages
|
from setuptools import setup, find_packages
|
||||||
|
|
||||||
setup(
|
setup(
|
||||||
|
@ -1,4 +1,3 @@
|
|||||||
from __future__ import absolute_import
|
|
||||||
import sys
|
import sys
|
||||||
import logging
|
import logging
|
||||||
|
|
||||||
@ -6,7 +5,7 @@ import logging
|
|||||||
if sys.version_info > (3, 0):
|
if sys.version_info > (3, 0):
|
||||||
import configparser as cp
|
import configparser as cp
|
||||||
else:
|
else:
|
||||||
import six.moves.configparser as cp
|
import ConfigParser as cp
|
||||||
|
|
||||||
|
|
||||||
class vwConfig(object):
|
class vwConfig(object):
|
||||||
|
@ -1,4 +1,3 @@
|
|||||||
from __future__ import absolute_import
|
|
||||||
import json
|
import json
|
||||||
import logging
|
import logging
|
||||||
import sys
|
import sys
|
||||||
|
@ -1,6 +1,5 @@
|
|||||||
#!/usr/bin/python
|
#!/usr/bin/python
|
||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
from __future__ import absolute_import
|
|
||||||
__author__ = 'Austin Taylor'
|
__author__ = 'Austin Taylor'
|
||||||
|
|
||||||
import datetime as dt
|
import datetime as dt
|
||||||
|
@ -1,6 +1,5 @@
|
|||||||
#!/usr/bin/python
|
#!/usr/bin/python
|
||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
from __future__ import absolute_import
|
|
||||||
__author__ = 'Nathan Young'
|
__author__ = 'Nathan Young'
|
||||||
|
|
||||||
import logging
|
import logging
|
||||||
@ -19,9 +18,9 @@ class qualysWhisperAPI(object):
|
|||||||
self.logger = logging.getLogger('qualysWhisperAPI')
|
self.logger = logging.getLogger('qualysWhisperAPI')
|
||||||
self.config = config
|
self.config = config
|
||||||
try:
|
try:
|
||||||
self.qgc = qualysapi.connect(config_file=config, section='qualys_vuln')
|
self.qgc = qualysapi.connect(config, 'qualys_vuln')
|
||||||
# Fail early if we can't make a request or auth is incorrect
|
# Fail early if we can't make a request or auth is incorrect
|
||||||
# self.qgc.request('about.php')
|
self.qgc.request('about.php')
|
||||||
self.logger.info('Connected to Qualys at {}'.format(self.qgc.server))
|
self.logger.info('Connected to Qualys at {}'.format(self.qgc.server))
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
self.logger.error('Could not connect to Qualys: {}'.format(str(e)))
|
self.logger.error('Could not connect to Qualys: {}'.format(str(e)))
|
||||||
|
@ -1,8 +1,5 @@
|
|||||||
#!/usr/bin/python
|
#!/usr/bin/python
|
||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
from __future__ import absolute_import
|
|
||||||
from six.moves import range
|
|
||||||
from functools import reduce
|
|
||||||
__author__ = 'Austin Taylor'
|
__author__ = 'Austin Taylor'
|
||||||
|
|
||||||
from lxml import objectify
|
from lxml import objectify
|
||||||
@ -17,16 +14,24 @@ import os
|
|||||||
import csv
|
import csv
|
||||||
import logging
|
import logging
|
||||||
import dateutil.parser as dp
|
import dateutil.parser as dp
|
||||||
csv.field_size_limit(sys.maxsize)
|
|
||||||
|
|
||||||
|
|
||||||
class qualysWhisperAPI(object):
|
class qualysWhisperAPI(object):
|
||||||
|
COUNT_WEBAPP = '/count/was/webapp'
|
||||||
COUNT_WASSCAN = '/count/was/wasscan'
|
COUNT_WASSCAN = '/count/was/wasscan'
|
||||||
DELETE_REPORT = '/delete/was/report/{report_id}'
|
DELETE_REPORT = '/delete/was/report/{report_id}'
|
||||||
|
GET_WEBAPP_DETAILS = '/get/was/webapp/{was_id}'
|
||||||
|
QPS_REST_3 = '/qps/rest/3.0'
|
||||||
|
REPORT_DETAILS = '/get/was/report/{report_id}'
|
||||||
REPORT_STATUS = '/status/was/report/{report_id}'
|
REPORT_STATUS = '/status/was/report/{report_id}'
|
||||||
REPORT_CREATE = '/create/was/report'
|
REPORT_CREATE = '/create/was/report'
|
||||||
REPORT_DOWNLOAD = '/download/was/report/{report_id}'
|
REPORT_DOWNLOAD = '/download/was/report/{report_id}'
|
||||||
|
SCAN_DETAILS = '/get/was/wasscan/{scan_id}'
|
||||||
|
SCAN_DOWNLOAD = '/download/was/wasscan/{scan_id}'
|
||||||
|
SEARCH_REPORTS = '/search/was/report'
|
||||||
|
SEARCH_WEB_APPS = '/search/was/webapp'
|
||||||
SEARCH_WAS_SCAN = '/search/was/wasscan'
|
SEARCH_WAS_SCAN = '/search/was/wasscan'
|
||||||
|
VERSION = '/qps/rest/portal/version'
|
||||||
|
|
||||||
def __init__(self, config=None):
|
def __init__(self, config=None):
|
||||||
self.logger = logging.getLogger('qualysWhisperAPI')
|
self.logger = logging.getLogger('qualysWhisperAPI')
|
||||||
@ -36,6 +41,10 @@ class qualysWhisperAPI(object):
|
|||||||
self.logger.info('Connected to Qualys at {}'.format(self.qgc.server))
|
self.logger.info('Connected to Qualys at {}'.format(self.qgc.server))
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
self.logger.error('Could not connect to Qualys: {}'.format(str(e)))
|
self.logger.error('Could not connect to Qualys: {}'.format(str(e)))
|
||||||
|
self.headers = {
|
||||||
|
#"content-type": "text/xml"}
|
||||||
|
"Accept" : "application/json",
|
||||||
|
"Content-Type": "application/json"}
|
||||||
self.config_parse = qcconf.QualysConnectConfig(config, 'qualys_web')
|
self.config_parse = qcconf.QualysConnectConfig(config, 'qualys_web')
|
||||||
try:
|
try:
|
||||||
self.template_id = self.config_parse.get_template_id()
|
self.template_id = self.config_parse.get_template_id()
|
||||||
@ -60,8 +69,14 @@ class qualysWhisperAPI(object):
|
|||||||
|
|
||||||
def generate_scan_result_XML(self, limit=1000, offset=1, status='FINISHED'):
|
def generate_scan_result_XML(self, limit=1000, offset=1, status='FINISHED'):
|
||||||
report_xml = E.ServiceRequest(
|
report_xml = E.ServiceRequest(
|
||||||
E.filters(E.Criteria({'field': 'status', 'operator': 'EQUALS'}, status)),
|
E.filters(
|
||||||
E.preferences(E.startFromOffset(str(offset)), E.limitResults(str(limit))),
|
E.Criteria({'field': 'status', 'operator': 'EQUALS'}, status
|
||||||
|
),
|
||||||
|
),
|
||||||
|
E.preferences(
|
||||||
|
E.startFromOffset(str(offset)),
|
||||||
|
E.limitResults(str(limit))
|
||||||
|
),
|
||||||
)
|
)
|
||||||
return report_xml
|
return report_xml
|
||||||
|
|
||||||
@ -100,10 +115,8 @@ class qualysWhisperAPI(object):
|
|||||||
if i % limit == 0:
|
if i % limit == 0:
|
||||||
if (total - i) < limit:
|
if (total - i) < limit:
|
||||||
qualys_api_limit = total - i
|
qualys_api_limit = total - i
|
||||||
self.logger.info('Making a request with a limit of {} at offset {}'
|
self.logger.info('Making a request with a limit of {} at offset {}'.format((str(qualys_api_limit)), str(i + 1)))
|
||||||
.format((str(qualys_api_limit)), str(i + 1)))
|
scan_info = self.get_scan_info(limit=qualys_api_limit, offset=i + 1, status=status)
|
||||||
scan_info = self.get_scan_info(
|
|
||||||
limit=qualys_api_limit, offset=i + 1, status=status)
|
|
||||||
_records.append(scan_info)
|
_records.append(scan_info)
|
||||||
self.logger.debug('Converting XML to DataFrame')
|
self.logger.debug('Converting XML to DataFrame')
|
||||||
dataframes = [self.xml_parser(xml) for xml in _records]
|
dataframes = [self.xml_parser(xml) for xml in _records]
|
||||||
@ -120,8 +133,7 @@ class qualysWhisperAPI(object):
|
|||||||
return self.qgc.request(self.REPORT_STATUS.format(report_id=report_id))
|
return self.qgc.request(self.REPORT_STATUS.format(report_id=report_id))
|
||||||
|
|
||||||
def download_report(self, report_id):
|
def download_report(self, report_id):
|
||||||
return self.qgc.request(
|
return self.qgc.request(self.REPORT_DOWNLOAD.format(report_id=report_id))
|
||||||
self.REPORT_DOWNLOAD.format(report_id=report_id), http_method='get')
|
|
||||||
|
|
||||||
def generate_scan_report_XML(self, scan_id):
|
def generate_scan_report_XML(self, scan_id):
|
||||||
"""Generates a CSV report for an asset based on template defined in .ini file"""
|
"""Generates a CSV report for an asset based on template defined in .ini file"""
|
||||||
@ -133,8 +145,20 @@ class qualysWhisperAPI(object):
|
|||||||
E.format('CSV'),
|
E.format('CSV'),
|
||||||
#type is not needed, as the template already has it
|
#type is not needed, as the template already has it
|
||||||
E.type('WAS_SCAN_REPORT'),
|
E.type('WAS_SCAN_REPORT'),
|
||||||
E.template(E.id(self.template_id)),
|
E.template(
|
||||||
E.config(E.scanReport(E.target(E.scans(E.WasScan(E.id(scan_id))))))
|
E.id(self.template_id)
|
||||||
|
),
|
||||||
|
E.config(
|
||||||
|
E.scanReport(
|
||||||
|
E.target(
|
||||||
|
E.scans(
|
||||||
|
E.WasScan(
|
||||||
|
E.id(scan_id)
|
||||||
|
)
|
||||||
|
),
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
@ -151,14 +175,95 @@ class qualysWhisperAPI(object):
|
|||||||
def delete_report(self, report_id):
|
def delete_report(self, report_id):
|
||||||
return self.qgc.request(self.DELETE_REPORT.format(report_id=report_id))
|
return self.qgc.request(self.DELETE_REPORT.format(report_id=report_id))
|
||||||
|
|
||||||
|
|
||||||
|
class qualysReportFields:
|
||||||
|
CATEGORIES = ['VULNERABILITY',
|
||||||
|
'SENSITIVECONTENT',
|
||||||
|
'INFORMATION_GATHERED']
|
||||||
|
|
||||||
|
# URL Vulnerability Information
|
||||||
|
|
||||||
|
VULN_BLOCK = [
|
||||||
|
CATEGORIES[0],
|
||||||
|
'ID',
|
||||||
|
'QID',
|
||||||
|
'Url',
|
||||||
|
'Param',
|
||||||
|
'Function',
|
||||||
|
'Form Entry Point',
|
||||||
|
'Access Path',
|
||||||
|
'Authentication',
|
||||||
|
'Ajax Request',
|
||||||
|
'Ajax Request ID',
|
||||||
|
'Ignored',
|
||||||
|
'Ignore Reason',
|
||||||
|
'Ignore Date',
|
||||||
|
'Ignore User',
|
||||||
|
'Ignore Comments',
|
||||||
|
'First Time Detected',
|
||||||
|
'Last Time Detected',
|
||||||
|
'Last Time Tested',
|
||||||
|
'Times Detected',
|
||||||
|
'Payload #1',
|
||||||
|
'Request Method #1',
|
||||||
|
'Request URL #1',
|
||||||
|
'Request Headers #1',
|
||||||
|
'Response #1',
|
||||||
|
'Evidence #1',
|
||||||
|
]
|
||||||
|
|
||||||
|
INFO_HEADER = [
|
||||||
|
'Vulnerability Category',
|
||||||
|
'ID',
|
||||||
|
'QID',
|
||||||
|
'Response #1',
|
||||||
|
'Last Time Detected',
|
||||||
|
]
|
||||||
|
INFO_BLOCK = [
|
||||||
|
CATEGORIES[2],
|
||||||
|
'ID',
|
||||||
|
'QID',
|
||||||
|
'Results',
|
||||||
|
'Detection Date',
|
||||||
|
]
|
||||||
|
|
||||||
|
QID_HEADER = [
|
||||||
|
'QID',
|
||||||
|
'Id',
|
||||||
|
'Title',
|
||||||
|
'Category',
|
||||||
|
'Severity Level',
|
||||||
|
'Groups',
|
||||||
|
'OWASP',
|
||||||
|
'WASC',
|
||||||
|
'CWE',
|
||||||
|
'CVSS Base',
|
||||||
|
'CVSS Temporal',
|
||||||
|
'Description',
|
||||||
|
'Impact',
|
||||||
|
'Solution',
|
||||||
|
]
|
||||||
|
GROUP_HEADER = ['GROUP', 'Name', 'Category']
|
||||||
|
OWASP_HEADER = ['OWASP', 'Code', 'Name']
|
||||||
|
WASC_HEADER = ['WASC', 'Code', 'Name']
|
||||||
|
SCAN_META = ['Web Application Name', 'URL', 'Owner', 'Scope', 'Operating System']
|
||||||
|
CATEGORY_HEADER = ['Category', 'Severity', 'Level', 'Description']
|
||||||
|
|
||||||
|
|
||||||
class qualysUtils:
|
class qualysUtils:
|
||||||
def __init__(self):
|
def __init__(self):
|
||||||
self.logger = logging.getLogger('qualysUtils')
|
self.logger = logging.getLogger('qualysUtils')
|
||||||
|
|
||||||
def grab_section(self, report, section, end=[], pop_last=False):
|
def grab_section(
|
||||||
|
self,
|
||||||
|
report,
|
||||||
|
section,
|
||||||
|
end=[],
|
||||||
|
pop_last=False,
|
||||||
|
):
|
||||||
temp_list = []
|
temp_list = []
|
||||||
max_col_count = 0
|
max_col_count = 0
|
||||||
with open(report, 'rt') as csvfile:
|
with open(report, 'rb') as csvfile:
|
||||||
q_report = csv.reader(csvfile, delimiter=',', quotechar='"')
|
q_report = csv.reader(csvfile, delimiter=',', quotechar='"')
|
||||||
for line in q_report:
|
for line in q_report:
|
||||||
if set(line) == set(section):
|
if set(line) == set(section):
|
||||||
@ -184,53 +289,44 @@ class qualysUtils:
|
|||||||
return _data
|
return _data
|
||||||
|
|
||||||
class qualysScanReport:
|
class qualysScanReport:
|
||||||
CATEGORIES = ['VULNERABILITY', 'SENSITIVECONTENT', 'INFORMATION_GATHERED']
|
# URL Vulnerability Information
|
||||||
|
WEB_SCAN_VULN_BLOCK = list(qualysReportFields.VULN_BLOCK)
|
||||||
|
WEB_SCAN_VULN_BLOCK.insert(WEB_SCAN_VULN_BLOCK.index('QID'), 'Detection ID')
|
||||||
|
|
||||||
WEB_SCAN_BLOCK = [
|
WEB_SCAN_VULN_HEADER = list(WEB_SCAN_VULN_BLOCK)
|
||||||
"ID", "Detection ID", "QID", "Url", "Param/Cookie", "Function",
|
WEB_SCAN_VULN_HEADER[WEB_SCAN_VULN_BLOCK.index(qualysReportFields.CATEGORIES[0])] = \
|
||||||
"Form Entry Point", "Access Path", "Authentication", "Ajax Request",
|
'Vulnerability Category'
|
||||||
"Ajax Request ID", "Ignored", "Ignore Reason", "Ignore Date", "Ignore User",
|
|
||||||
"Ignore Comments", "Detection Date", "Payload #1", "Request Method #1",
|
|
||||||
"Request URL #1", "Request Headers #1", "Response #1", "Evidence #1",
|
|
||||||
"Unique ID", "Flags", "Protocol", "Virtual Host", "IP", "Port", "Result",
|
|
||||||
"Info#1", "CVSS V3 Base", "CVSS V3 Temporal", "CVSS V3 Attack Vector",
|
|
||||||
"Request Body #1"
|
|
||||||
]
|
|
||||||
WEB_SCAN_VULN_BLOCK = [CATEGORIES[0]] + WEB_SCAN_BLOCK
|
|
||||||
WEB_SCAN_SENSITIVE_BLOCK = [CATEGORIES[1]] + WEB_SCAN_BLOCK
|
|
||||||
|
|
||||||
WEB_SCAN_HEADER = ["Vulnerability Category"] + WEB_SCAN_BLOCK
|
WEB_SCAN_SENSITIVE_HEADER = list(WEB_SCAN_VULN_HEADER)
|
||||||
WEB_SCAN_HEADER[WEB_SCAN_HEADER.index("Detection Date")] = "Last Time Detected"
|
WEB_SCAN_SENSITIVE_HEADER.insert(WEB_SCAN_SENSITIVE_HEADER.index('Url'
|
||||||
|
), 'Content')
|
||||||
|
|
||||||
|
WEB_SCAN_SENSITIVE_BLOCK = list(WEB_SCAN_SENSITIVE_HEADER)
|
||||||
|
WEB_SCAN_SENSITIVE_BLOCK.insert(WEB_SCAN_SENSITIVE_BLOCK.index('QID'), 'Detection ID')
|
||||||
|
WEB_SCAN_SENSITIVE_BLOCK[WEB_SCAN_SENSITIVE_BLOCK.index('Vulnerability Category'
|
||||||
|
)] = qualysReportFields.CATEGORIES[1]
|
||||||
|
|
||||||
WEB_SCAN_INFO_BLOCK = [
|
WEB_SCAN_INFO_HEADER = list(qualysReportFields.INFO_HEADER)
|
||||||
"INFORMATION_GATHERED", "ID", "Detection ID", "QID", "Results", "Detection Date",
|
WEB_SCAN_INFO_HEADER.insert(WEB_SCAN_INFO_HEADER.index('QID'), 'Detection ID')
|
||||||
"Unique ID", "Flags", "Protocol", "Virtual Host", "IP", "Port", "Result",
|
|
||||||
"Info#1"
|
|
||||||
]
|
|
||||||
|
|
||||||
WEB_SCAN_INFO_HEADER = [
|
WEB_SCAN_INFO_BLOCK = list(qualysReportFields.INFO_BLOCK)
|
||||||
"Vulnerability Category", "ID", "Detection ID", "QID", "Results", "Last Time Detected",
|
WEB_SCAN_INFO_BLOCK.insert(WEB_SCAN_INFO_BLOCK.index('QID'), 'Detection ID')
|
||||||
"Unique ID", "Flags", "Protocol", "Virtual Host", "IP", "Port", "Result",
|
|
||||||
"Info#1"
|
|
||||||
]
|
|
||||||
|
|
||||||
QID_HEADER = [
|
QID_HEADER = list(qualysReportFields.QID_HEADER)
|
||||||
"QID", "Id", "Title", "Category", "Severity Level", "Groups", "OWASP", "WASC",
|
GROUP_HEADER = list(qualysReportFields.GROUP_HEADER)
|
||||||
"CWE", "CVSS Base", "CVSS Temporal", "Description", "Impact", "Solution",
|
OWASP_HEADER = list(qualysReportFields.OWASP_HEADER)
|
||||||
"CVSS V3 Base", "CVSS V3 Temporal", "CVSS V3 Attack Vector"
|
WASC_HEADER = list(qualysReportFields.WASC_HEADER)
|
||||||
]
|
SCAN_META = list(qualysReportFields.SCAN_META)
|
||||||
GROUP_HEADER = ['GROUP', 'Name', 'Category']
|
CATEGORY_HEADER = list(qualysReportFields.CATEGORY_HEADER)
|
||||||
OWASP_HEADER = ['OWASP', 'Code', 'Name']
|
|
||||||
WASC_HEADER = ['WASC', 'Code', 'Name']
|
|
||||||
SCAN_META = [
|
|
||||||
"Web Application Name", "URL", "Owner", "Scope", "ID", "Tags",
|
|
||||||
"Custom Attributes"
|
|
||||||
]
|
|
||||||
CATEGORY_HEADER = ['Category', 'Severity', 'Level', 'Description']
|
|
||||||
|
|
||||||
def __init__(self, config=None, file_in=None,
|
def __init__(
|
||||||
file_stream=False, delimiter=',', quotechar='"'):
|
self,
|
||||||
|
config=None,
|
||||||
|
file_in=None,
|
||||||
|
file_stream=False,
|
||||||
|
delimiter=',',
|
||||||
|
quotechar='"',
|
||||||
|
):
|
||||||
self.logger = logging.getLogger('qualysScanReport')
|
self.logger = logging.getLogger('qualysScanReport')
|
||||||
self.file_in = file_in
|
self.file_in = file_in
|
||||||
self.file_stream = file_stream
|
self.file_stream = file_stream
|
||||||
@ -241,79 +337,71 @@ class qualysScanReport:
|
|||||||
try:
|
try:
|
||||||
self.qw = qualysWhisperAPI(config=config)
|
self.qw = qualysWhisperAPI(config=config)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
self.logger.error(
|
self.logger.error('Could not load config! Please check settings. Error: {}'.format(str(e)))
|
||||||
'Could not load config! Please check settings. Error: {}'.format(
|
|
||||||
str(e)))
|
|
||||||
|
|
||||||
if file_stream:
|
if file_stream:
|
||||||
self.open_file = file_in.splitlines()
|
self.open_file = file_in.splitlines()
|
||||||
elif file_in:
|
elif file_in:
|
||||||
|
|
||||||
self.open_file = open(file_in, 'rb')
|
self.open_file = open(file_in, 'rb')
|
||||||
|
|
||||||
self.downloaded_file = None
|
self.downloaded_file = None
|
||||||
|
|
||||||
def grab_sections(self, report):
|
def grab_sections(self, report):
|
||||||
return {
|
all_dataframes = []
|
||||||
'WEB_SCAN_VULN_BLOCK': pd.DataFrame(
|
dict_tracker = {}
|
||||||
self.utils.grab_section(
|
with open(report, 'rb') as csvfile:
|
||||||
report,
|
dict_tracker['WEB_SCAN_VULN_BLOCK'] = pd.DataFrame(self.utils.grab_section(report,
|
||||||
self.WEB_SCAN_VULN_BLOCK,
|
self.WEB_SCAN_VULN_BLOCK,
|
||||||
end=[self.WEB_SCAN_SENSITIVE_BLOCK, self.WEB_SCAN_INFO_BLOCK],
|
end=[
|
||||||
pop_last=True),
|
|
||||||
columns=self.WEB_SCAN_HEADER),
|
|
||||||
'WEB_SCAN_SENSITIVE_BLOCK': pd.DataFrame(
|
|
||||||
self.utils.grab_section(report,
|
|
||||||
self.WEB_SCAN_SENSITIVE_BLOCK,
|
self.WEB_SCAN_SENSITIVE_BLOCK,
|
||||||
end=[self.WEB_SCAN_INFO_BLOCK, self.WEB_SCAN_SENSITIVE_BLOCK],
|
self.WEB_SCAN_INFO_BLOCK],
|
||||||
pop_last=True),
|
pop_last=True),
|
||||||
columns=self.WEB_SCAN_HEADER),
|
columns=self.WEB_SCAN_VULN_HEADER)
|
||||||
'WEB_SCAN_INFO_BLOCK': pd.DataFrame(
|
dict_tracker['WEB_SCAN_SENSITIVE_BLOCK'] = pd.DataFrame(self.utils.grab_section(report,
|
||||||
self.utils.grab_section(
|
self.WEB_SCAN_SENSITIVE_BLOCK,
|
||||||
report,
|
end=[
|
||||||
|
self.WEB_SCAN_INFO_BLOCK,
|
||||||
|
self.WEB_SCAN_SENSITIVE_BLOCK],
|
||||||
|
pop_last=True),
|
||||||
|
columns=self.WEB_SCAN_SENSITIVE_HEADER)
|
||||||
|
dict_tracker['WEB_SCAN_INFO_BLOCK'] = pd.DataFrame(self.utils.grab_section(report,
|
||||||
self.WEB_SCAN_INFO_BLOCK,
|
self.WEB_SCAN_INFO_BLOCK,
|
||||||
end=[self.QID_HEADER],
|
end=[self.QID_HEADER],
|
||||||
pop_last=True),
|
pop_last=True),
|
||||||
columns=self.WEB_SCAN_INFO_HEADER),
|
columns=self.WEB_SCAN_INFO_HEADER)
|
||||||
|
dict_tracker['QID_HEADER'] = pd.DataFrame(self.utils.grab_section(report,
|
||||||
'QID_HEADER': pd.DataFrame(
|
|
||||||
self.utils.grab_section(
|
|
||||||
report,
|
|
||||||
self.QID_HEADER,
|
self.QID_HEADER,
|
||||||
end=[self.GROUP_HEADER],
|
end=[self.GROUP_HEADER],
|
||||||
pop_last=True),
|
pop_last=True),
|
||||||
columns=self.QID_HEADER),
|
columns=self.QID_HEADER)
|
||||||
'GROUP_HEADER': pd.DataFrame(
|
dict_tracker['GROUP_HEADER'] = pd.DataFrame(self.utils.grab_section(report,
|
||||||
self.utils.grab_section(
|
|
||||||
report,
|
|
||||||
self.GROUP_HEADER,
|
self.GROUP_HEADER,
|
||||||
end=[self.OWASP_HEADER],
|
end=[self.OWASP_HEADER],
|
||||||
pop_last=True),
|
pop_last=True),
|
||||||
columns=self.GROUP_HEADER),
|
columns=self.GROUP_HEADER)
|
||||||
'OWASP_HEADER': pd.DataFrame(
|
dict_tracker['OWASP_HEADER'] = pd.DataFrame(self.utils.grab_section(report,
|
||||||
self.utils.grab_section(
|
|
||||||
report,
|
|
||||||
self.OWASP_HEADER,
|
self.OWASP_HEADER,
|
||||||
end=[self.WASC_HEADER],
|
end=[self.WASC_HEADER],
|
||||||
pop_last=True),
|
pop_last=True),
|
||||||
columns=self.OWASP_HEADER),
|
columns=self.OWASP_HEADER)
|
||||||
'WASC_HEADER': pd.DataFrame(
|
dict_tracker['WASC_HEADER'] = pd.DataFrame(self.utils.grab_section(report,
|
||||||
self.utils.grab_section(
|
self.WASC_HEADER, end=[['APPENDIX']],
|
||||||
report,
|
|
||||||
self.WASC_HEADER,
|
|
||||||
end=[['APPENDIX']],
|
|
||||||
pop_last=True),
|
pop_last=True),
|
||||||
columns=self.WASC_HEADER),
|
columns=self.WASC_HEADER)
|
||||||
'SCAN_META': pd.DataFrame(
|
|
||||||
self.utils.grab_section(report,
|
dict_tracker['SCAN_META'] = pd.DataFrame(self.utils.grab_section(report,
|
||||||
self.SCAN_META,
|
self.SCAN_META,
|
||||||
end=[self.CATEGORY_HEADER],
|
end=[self.CATEGORY_HEADER],
|
||||||
pop_last=True),
|
pop_last=True),
|
||||||
columns=self.SCAN_META),
|
columns=self.SCAN_META)
|
||||||
'CATEGORY_HEADER': pd.DataFrame(
|
|
||||||
self.utils.grab_section(report,
|
dict_tracker['CATEGORY_HEADER'] = pd.DataFrame(self.utils.grab_section(report,
|
||||||
self.CATEGORY_HEADER),
|
self.CATEGORY_HEADER),
|
||||||
columns=self.CATEGORY_HEADER)
|
columns=self.CATEGORY_HEADER)
|
||||||
}
|
all_dataframes.append(dict_tracker)
|
||||||
|
|
||||||
|
return all_dataframes
|
||||||
|
|
||||||
def data_normalizer(self, dataframes):
|
def data_normalizer(self, dataframes):
|
||||||
"""
|
"""
|
||||||
@ -321,21 +409,12 @@ class qualysScanReport:
|
|||||||
:param dataframes:
|
:param dataframes:
|
||||||
:return:
|
:return:
|
||||||
"""
|
"""
|
||||||
df_dict = dataframes
|
df_dict = dataframes[0]
|
||||||
merged_df = pd.concat([
|
merged_df = pd.concat([df_dict['WEB_SCAN_VULN_BLOCK'], df_dict['WEB_SCAN_SENSITIVE_BLOCK'],
|
||||||
df_dict['WEB_SCAN_VULN_BLOCK'],
|
df_dict['WEB_SCAN_INFO_BLOCK']], axis=0,
|
||||||
df_dict['WEB_SCAN_SENSITIVE_BLOCK'],
|
ignore_index=False)
|
||||||
df_dict['WEB_SCAN_INFO_BLOCK']
|
merged_df = pd.merge(merged_df, df_dict['QID_HEADER'], left_on='QID',
|
||||||
], axis=0, ignore_index=False)
|
right_on='Id')
|
||||||
|
|
||||||
merged_df = pd.merge(
|
|
||||||
merged_df,
|
|
||||||
df_dict['QID_HEADER'].drop(
|
|
||||||
#these columns always seem to be the same as what we're merging into
|
|
||||||
['CVSS V3 Attack Vector', 'CVSS V3 Base', 'CVSS V3 Temporal'],
|
|
||||||
axis=1),
|
|
||||||
left_on='QID', right_on='Id'
|
|
||||||
)
|
|
||||||
|
|
||||||
if 'Content' not in merged_df:
|
if 'Content' not in merged_df:
|
||||||
merged_df['Content'] = ''
|
merged_df['Content'] = ''
|
||||||
@ -352,11 +431,8 @@ class qualysScanReport:
|
|||||||
|
|
||||||
merged_df = merged_df.assign(**df_dict['SCAN_META'].to_dict(orient='records')[0])
|
merged_df = merged_df.assign(**df_dict['SCAN_META'].to_dict(orient='records')[0])
|
||||||
|
|
||||||
merged_df = pd.merge(
|
merged_df = pd.merge(merged_df, df_dict['CATEGORY_HEADER'], how='left', left_on=['Category', 'Severity Level'],
|
||||||
merged_df, df_dict['CATEGORY_HEADER'],
|
right_on=['Category', 'Severity'], suffixes=('Severity', 'CatSev'))
|
||||||
how='left', left_on=['Category', 'Severity Level'],
|
|
||||||
right_on=['Category', 'Severity'], suffixes=('Severity', 'CatSev')
|
|
||||||
)
|
|
||||||
|
|
||||||
merged_df = merged_df.replace('N/A', '').fillna('')
|
merged_df = merged_df.replace('N/A', '').fillna('')
|
||||||
|
|
||||||
|
@ -1,18 +1,15 @@
|
|||||||
from __future__ import absolute_import
|
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
from datetime import datetime, date
|
from datetime import datetime, date, timedelta
|
||||||
|
|
||||||
from jira import JIRA
|
from jira import JIRA
|
||||||
|
import requests
|
||||||
import logging
|
import logging
|
||||||
from bottle import template
|
from bottle import template
|
||||||
import re
|
import re
|
||||||
from six.moves import range
|
|
||||||
|
|
||||||
|
|
||||||
class JiraAPI(object):
|
class JiraAPI(object):
|
||||||
def __init__(self, hostname=None, username=None, password=None, path="", debug=False, clean_obsolete=True,
|
def __init__(self, hostname=None, username=None, password=None, path="", debug=False, clean_obsolete=True, max_time_window=12, decommission_time_window=3):
|
||||||
max_time_window=12, decommission_time_window=3):
|
|
||||||
self.logger = logging.getLogger('JiraAPI')
|
self.logger = logging.getLogger('JiraAPI')
|
||||||
if debug:
|
if debug:
|
||||||
self.logger.setLevel(logging.DEBUG)
|
self.logger.setLevel(logging.DEBUG)
|
||||||
@ -44,15 +41,10 @@ class JiraAPI(object):
|
|||||||
# deletes the tag "server_decommission" from those tickets closed <=3 months ago
|
# deletes the tag "server_decommission" from those tickets closed <=3 months ago
|
||||||
self.decommission_cleanup()
|
self.decommission_cleanup()
|
||||||
|
|
||||||
self.jira_still_vulnerable_comment = '''This ticket has been reopened due to the vulnerability not having been \
|
self.jira_still_vulnerable_comment = '''This ticket has been reopened due to the vulnerability not having been fixed (if multiple assets are affected, all need to be fixed; if the server is down, lastest known vulnerability might be the one reported).
|
||||||
fixed (if multiple assets are affected, all need to be fixed; if the server is down, lastest known \
|
- In the case of the team accepting the risk and wanting to close the ticket, please add the label "*risk_accepted*" to the ticket before closing it.
|
||||||
vulnerability might be the one reported).
|
- If server has been decommissioned, please add the label "*server_decommission*" to the ticket before closing it.
|
||||||
- In the case of the team accepting the risk and wanting to close the ticket, please add the label \
|
- If when checking the vulnerability it looks like a false positive, _+please elaborate in a comment+_ and add the label "*false_positive*" before closing it; we will review it and report it to the vendor.
|
||||||
"*risk_accepted*" to the ticket before closing it.
|
|
||||||
- If server has been decommissioned, please add the label "*server_decommission*" to the ticket before closing \
|
|
||||||
it.
|
|
||||||
- If when checking the vulnerability it looks like a false positive, _+please elaborate in a comment+_ and add \
|
|
||||||
the label "*false_positive*" before closing it; we will review it and report it to the vendor.
|
|
||||||
|
|
||||||
If you have further doubts, please contact the Security Team.'''
|
If you have further doubts, please contact the Security Team.'''
|
||||||
|
|
||||||
@ -76,6 +68,7 @@ class JiraAPI(object):
|
|||||||
self.logger.error("Error creating Ticket: component {} not found".format(component))
|
self.logger.error("Error creating Ticket: component {} not found".format(component))
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
try:
|
||||||
new_issue = self.jira.create_issue(project=project,
|
new_issue = self.jira.create_issue(project=project,
|
||||||
summary=title,
|
summary=title,
|
||||||
description=desc,
|
description=desc,
|
||||||
@ -88,6 +81,10 @@ class JiraAPI(object):
|
|||||||
if attachment_contents:
|
if attachment_contents:
|
||||||
self.add_content_as_attachment(new_issue, attachment_contents)
|
self.add_content_as_attachment(new_issue, attachment_contents)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
self.logger.error("Failed to create ticket on Jira Project '{}'. Error: {}".format(project, e))
|
||||||
|
new_issue = False
|
||||||
|
|
||||||
return new_issue
|
return new_issue
|
||||||
|
|
||||||
#Basic JIRA Metrics
|
#Basic JIRA Metrics
|
||||||
@ -99,15 +96,13 @@ class JiraAPI(object):
|
|||||||
return len(self.jira.search_issues(jql, maxResults=0))
|
return len(self.jira.search_issues(jql, maxResults=0))
|
||||||
|
|
||||||
def metrics_closed_tickets(self, project=None):
|
def metrics_closed_tickets(self, project=None):
|
||||||
jql = "labels= vulnerability_management and NOT resolution = Unresolved AND created >=startOfMonth(-{})".format(
|
jql = "labels= vulnerability_management and NOT resolution = Unresolved AND created >=startOfMonth(-{})".format(self.max_time_tracking)
|
||||||
self.max_time_tracking)
|
|
||||||
if project:
|
if project:
|
||||||
jql += " and (project='{}')".format(project)
|
jql += " and (project='{}')".format(project)
|
||||||
return len(self.jira.search_issues(jql, maxResults=0))
|
return len(self.jira.search_issues(jql, maxResults=0))
|
||||||
|
|
||||||
def sync(self, vulnerabilities, project, components=[]):
|
def sync(self, vulnerabilities, project, components=[]):
|
||||||
# JIRA structure of each vulnerability: [source, scan_name, title, diagnosis, consequence, solution,
|
#JIRA structure of each vulnerability: [source, scan_name, title, diagnosis, consequence, solution, ips, risk, references]
|
||||||
# ips, risk, references]
|
|
||||||
self.logger.info("JIRA Sync started")
|
self.logger.info("JIRA Sync started")
|
||||||
|
|
||||||
for vuln in vulnerabilities:
|
for vuln in vulnerabilities:
|
||||||
@ -116,8 +111,7 @@ class JiraAPI(object):
|
|||||||
if " " in vuln['scan_name']:
|
if " " in vuln['scan_name']:
|
||||||
vuln['scan_name'] = "_".join(vuln['scan_name'].split(" "))
|
vuln['scan_name'] = "_".join(vuln['scan_name'].split(" "))
|
||||||
|
|
||||||
# we exclude from the vulnerabilities to report those assets that already exist
|
# we exclude from the vulnerabilities to report those assets that already exist with *risk_accepted*/*server_decommission*
|
||||||
# with *risk_accepted*/*server_decommission*
|
|
||||||
vuln = self.exclude_accepted_assets(vuln)
|
vuln = self.exclude_accepted_assets(vuln)
|
||||||
|
|
||||||
# make sure after exclusion of risk_accepted assets there are still assets
|
# make sure after exclusion of risk_accepted assets there are still assets
|
||||||
@ -142,17 +136,13 @@ class JiraAPI(object):
|
|||||||
# create local text file with assets, attach it to ticket
|
# create local text file with assets, attach it to ticket
|
||||||
if len(vuln['ips']) > self.max_ips_ticket:
|
if len(vuln['ips']) > self.max_ips_ticket:
|
||||||
attachment_contents = vuln['ips']
|
attachment_contents = vuln['ips']
|
||||||
vuln['ips'] = [
|
vuln['ips'] = ["Affected hosts ({assets}) exceed Jira's allowed character limit, added as an attachment.".format(assets = len(attachment_contents))]
|
||||||
"Affected hosts ({assets}) exceed Jira's allowed character limit, added as an attachment.".format(
|
|
||||||
assets=len(attachment_contents))]
|
|
||||||
try:
|
try:
|
||||||
tpl = template(self.template_path, vuln)
|
tpl = template(self.template_path, vuln)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
self.logger.error('Exception templating: {}'.format(str(e)))
|
self.logger.error('Exception templating: {}'.format(str(e)))
|
||||||
return 0
|
return 0
|
||||||
self.create_ticket(title=vuln['title'], desc=tpl, project=project, components=components,
|
self.create_ticket(title=vuln['title'], desc=tpl, project=project, components=components, tags=[vuln['source'], vuln['scan_name'], 'vulnerability', vuln['risk']], attachment_contents = attachment_contents)
|
||||||
tags=[vuln['source'], vuln['scan_name'], 'vulnerability', vuln['risk']],
|
|
||||||
attachment_contents=attachment_contents)
|
|
||||||
else:
|
else:
|
||||||
self.logger.info("Ignoring vulnerability as all assets are already reported in a risk_accepted ticket")
|
self.logger.info("Ignoring vulnerability as all assets are already reported in a risk_accepted ticket")
|
||||||
|
|
||||||
@ -168,8 +158,7 @@ class JiraAPI(object):
|
|||||||
labels = [vuln['source'], vuln['scan_name'], 'vulnerability_management', 'vulnerability']
|
labels = [vuln['source'], vuln['scan_name'], 'vulnerability_management', 'vulnerability']
|
||||||
|
|
||||||
if not self.excluded_tickets:
|
if not self.excluded_tickets:
|
||||||
jql = "{} AND labels in (risk_accepted,server_decommission, false_positive) AND NOT labels=advisory AND created >=startOfMonth(-{})".format(
|
jql = "{} AND labels in (risk_accepted,server_decommission, false_positive) AND NOT labels=advisory AND created >=startOfMonth(-{})".format(" AND ".join(["labels={}".format(label) for label in labels]), self.max_time_tracking)
|
||||||
" AND ".join(["labels={}".format(label) for label in labels]), self.max_time_tracking)
|
|
||||||
self.excluded_tickets = self.jira.search_issues(jql, maxResults=0)
|
self.excluded_tickets = self.jira.search_issues(jql, maxResults=0)
|
||||||
|
|
||||||
title = vuln['title']
|
title = vuln['title']
|
||||||
@ -179,8 +168,7 @@ class JiraAPI(object):
|
|||||||
assets_to_exclude = []
|
assets_to_exclude = []
|
||||||
tickets_excluded_assets = []
|
tickets_excluded_assets = []
|
||||||
for index in range(len(self.excluded_tickets)):
|
for index in range(len(self.excluded_tickets)):
|
||||||
checking_ticketid, checking_title, checking_assets = self.ticket_get_unique_fields(
|
checking_ticketid, checking_title, checking_assets = self.ticket_get_unique_fields(self.excluded_tickets[index])
|
||||||
self.excluded_tickets[index])
|
|
||||||
if title.encode('ascii') == checking_title.encode('ascii'):
|
if title.encode('ascii') == checking_title.encode('ascii'):
|
||||||
if checking_assets:
|
if checking_assets:
|
||||||
#checking_assets is a list, we add to our full list for later delete all assets
|
#checking_assets is a list, we add to our full list for later delete all assets
|
||||||
@ -189,8 +177,7 @@ class JiraAPI(object):
|
|||||||
|
|
||||||
if assets_to_exclude:
|
if assets_to_exclude:
|
||||||
assets_to_remove = []
|
assets_to_remove = []
|
||||||
self.logger.warn("Vulnerable Assets seen on an already existing risk_accepted Jira ticket: {}".format(
|
self.logger.warn("Vulnerable Assets seen on an already existing risk_accepted Jira ticket: {}".format(', '.join(tickets_excluded_assets)))
|
||||||
', '.join(tickets_excluded_assets)))
|
|
||||||
self.logger.debug("Original assets: {}".format(vuln['ips']))
|
self.logger.debug("Original assets: {}".format(vuln['ips']))
|
||||||
#assets in vulnerability have the structure "ip - hostname - port", so we need to match by partial
|
#assets in vulnerability have the structure "ip - hostname - port", so we need to match by partial
|
||||||
for exclusion in assets_to_exclude:
|
for exclusion in assets_to_exclude:
|
||||||
@ -198,9 +185,7 @@ class JiraAPI(object):
|
|||||||
# and we don't want it to affect the rest of the processing (otherwise, it would miss the asset right after the removed one)
|
# and we don't want it to affect the rest of the processing (otherwise, it would miss the asset right after the removed one)
|
||||||
for index in range(len(vuln['ips']))[::-1]:
|
for index in range(len(vuln['ips']))[::-1]:
|
||||||
if exclusion == vuln['ips'][index].split(" - ")[0]:
|
if exclusion == vuln['ips'][index].split(" - ")[0]:
|
||||||
self.logger.debug(
|
self.logger.debug("Deleting asset {} from vulnerability {}, seen in risk_accepted.".format(vuln['ips'][index], title))
|
||||||
"Deleting asset {} from vulnerability {}, seen in risk_accepted.".format(vuln['ips'][index],
|
|
||||||
title))
|
|
||||||
vuln['ips'].pop(index)
|
vuln['ips'].pop(index)
|
||||||
self.logger.debug("Modified assets: {}".format(vuln['ips']))
|
self.logger.debug("Modified assets: {}".format(vuln['ips']))
|
||||||
|
|
||||||
@ -222,8 +207,7 @@ class JiraAPI(object):
|
|||||||
self.logger.info("Retrieving all JIRA tickets with the following tags {}".format(labels))
|
self.logger.info("Retrieving all JIRA tickets with the following tags {}".format(labels))
|
||||||
# we want to check all JIRA tickets, to include tickets moved to other queues
|
# we want to check all JIRA tickets, to include tickets moved to other queues
|
||||||
# will exclude tickets older than 12 months, old tickets will get closed for higiene and recreated if still vulnerable
|
# will exclude tickets older than 12 months, old tickets will get closed for higiene and recreated if still vulnerable
|
||||||
jql = "{} AND NOT labels=advisory AND created >=startOfMonth(-{})".format(
|
jql = "{} AND NOT labels=advisory AND created >=startOfMonth(-{})".format(" AND ".join(["labels={}".format(label) for label in labels]), self.max_time_tracking)
|
||||||
" AND ".join(["labels={}".format(label) for label in labels]), self.max_time_tracking)
|
|
||||||
|
|
||||||
self.all_tickets = self.jira.search_issues(jql, maxResults=0)
|
self.all_tickets = self.jira.search_issues(jql, maxResults=0)
|
||||||
|
|
||||||
@ -233,8 +217,7 @@ class JiraAPI(object):
|
|||||||
for index in range(len(self.all_tickets)):
|
for index in range(len(self.all_tickets)):
|
||||||
checking_ticketid, checking_title, checking_assets = self.ticket_get_unique_fields(self.all_tickets[index])
|
checking_ticketid, checking_title, checking_assets = self.ticket_get_unique_fields(self.all_tickets[index])
|
||||||
# added "not risk_accepted", as if it is risk_accepted, we will create a new ticket excluding the accepted assets
|
# added "not risk_accepted", as if it is risk_accepted, we will create a new ticket excluding the accepted assets
|
||||||
if title.encode('ascii') == checking_title.encode('ascii') and not self.is_risk_accepted(
|
if title.encode('ascii') == checking_title.encode('ascii') and not self.is_risk_accepted(self.jira.issue(checking_ticketid)):
|
||||||
self.jira.issue(checking_ticketid)):
|
|
||||||
difference = list(set(assets).symmetric_difference(checking_assets))
|
difference = list(set(assets).symmetric_difference(checking_assets))
|
||||||
#to check intersection - set(assets) & set(checking_assets)
|
#to check intersection - set(assets) & set(checking_assets)
|
||||||
if difference:
|
if difference:
|
||||||
@ -261,12 +244,9 @@ class JiraAPI(object):
|
|||||||
# structure the text to have the same structure as the assets from the attachment
|
# structure the text to have the same structure as the assets from the attachment
|
||||||
affected_assets = ""
|
affected_assets = ""
|
||||||
try:
|
try:
|
||||||
affected_assets = \
|
affected_assets = ticket.raw.get('fields', {}).get('description').encode("ascii").split("{panel:title=Affected Assets}")[1].split("{panel}")[0].replace('\n','').replace(' * ','\n').replace('\n', '', 1)
|
||||||
ticket.raw.get('fields', {}).get('description').encode("ascii").split("{panel:title=Affected Assets}")[
|
|
||||||
1].split("{panel}")[0].replace('\n', '').replace(' * ', '\n').replace('\n', '', 1)
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
self.logger.error(
|
self.logger.error("Unable to process the Ticket's 'Affected Assets'. Ticket ID: {}. Reason: {}".format(ticket, e))
|
||||||
"Unable to process the Ticket's 'Affected Assets'. Ticket ID: {}. Reason: {}".format(ticket, e))
|
|
||||||
|
|
||||||
if affected_assets:
|
if affected_assets:
|
||||||
if _raw:
|
if _raw:
|
||||||
@ -305,8 +285,7 @@ class JiraAPI(object):
|
|||||||
affected_assets = self.jira.attachment(attachment_id).get()
|
affected_assets = self.jira.attachment(attachment_id).get()
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
self.logger.error(
|
self.logger.error("Failed to get assets from ticket attachment. Ticket ID: {}. Reason: {}".format(ticket, e))
|
||||||
"Failed to get assets from ticket attachment. Ticket ID: {}. Reason: {}".format(ticket, e))
|
|
||||||
|
|
||||||
if affected_assets:
|
if affected_assets:
|
||||||
if _raw:
|
if _raw:
|
||||||
@ -379,10 +358,8 @@ class JiraAPI(object):
|
|||||||
if self.is_ticket_resolved(ticket_obj):
|
if self.is_ticket_resolved(ticket_obj):
|
||||||
ticket_data = ticket_obj.raw.get('fields')
|
ticket_data = ticket_obj.raw.get('fields')
|
||||||
#dates follow format '2018-11-06T10:36:13.849+0100'
|
#dates follow format '2018-11-06T10:36:13.849+0100'
|
||||||
created = [int(x) for x in
|
created = [int(x) for x in ticket_data['created'].split('.')[0].replace('T', '-').replace(':','-').split('-')]
|
||||||
ticket_data['created'].split('.')[0].replace('T', '-').replace(':', '-').split('-')]
|
resolved =[int(x) for x in ticket_data['resolutiondate'].split('.')[0].replace('T', '-').replace(':','-').split('-')]
|
||||||
resolved = [int(x) for x in
|
|
||||||
ticket_data['resolutiondate'].split('.')[0].replace('T', '-').replace(':', '-').split('-')]
|
|
||||||
|
|
||||||
start = datetime(created[0],created[1],created[2],created[3],created[4],created[5])
|
start = datetime(created[0],created[1],created[2],created[3],created[4],created[5])
|
||||||
end = datetime(resolved[0],resolved[1],resolved[2],resolved[3],resolved[4],resolved[5])
|
end = datetime(resolved[0],resolved[1],resolved[2],resolved[3],resolved[4],resolved[5])
|
||||||
@ -433,9 +410,7 @@ class JiraAPI(object):
|
|||||||
attachment_contents = []
|
attachment_contents = []
|
||||||
if len(vuln['ips']) > self.max_ips_ticket:
|
if len(vuln['ips']) > self.max_ips_ticket:
|
||||||
attachment_contents = vuln['ips']
|
attachment_contents = vuln['ips']
|
||||||
vuln['ips'] = [
|
vuln['ips'] = ["Affected hosts ({assets}) exceed Jira's allowed character limit, added as an attachment.".format(assets = len(attachment_contents))]
|
||||||
"Affected hosts ({assets}) exceed Jira's allowed character limit, added as an attachment.".format(
|
|
||||||
assets=len(attachment_contents))]
|
|
||||||
|
|
||||||
#fill the ticket description template
|
#fill the ticket description template
|
||||||
try:
|
try:
|
||||||
@ -455,8 +430,7 @@ class JiraAPI(object):
|
|||||||
self.logger.info("Ticket {} updated successfully".format(ticketid))
|
self.logger.info("Ticket {} updated successfully".format(ticketid))
|
||||||
self.add_label(ticketid, 'updated')
|
self.add_label(ticketid, 'updated')
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
self.logger.error(
|
self.logger.error("Error while trying up update ticket {ticketid}.\nReason: {e}".format(ticketid = ticketid, e=e))
|
||||||
"Error while trying up update ticket {ticketid}.\nReason: {e}".format(ticketid=ticketid, e=e))
|
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
def add_label(self, ticketid, label):
|
def add_label(self, ticketid, label):
|
||||||
@ -468,9 +442,8 @@ class JiraAPI(object):
|
|||||||
try:
|
try:
|
||||||
ticket_obj.update(fields={"labels":ticket_obj.fields.labels})
|
ticket_obj.update(fields={"labels":ticket_obj.fields.labels})
|
||||||
self.logger.info("Added label {label} to ticket {ticket}".format(label=label, ticket=ticketid))
|
self.logger.info("Added label {label} to ticket {ticket}".format(label=label, ticket=ticketid))
|
||||||
except Exception as e:
|
except:
|
||||||
self.logger.error(
|
self.logger.error("Error while trying to add label {label} to ticket {ticket}".format(label=label, ticket=ticketid))
|
||||||
"Error while trying to add label {label} to ticket {ticket}".format(label=label, ticket=ticketid))
|
|
||||||
|
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
@ -483,9 +456,8 @@ class JiraAPI(object):
|
|||||||
try:
|
try:
|
||||||
ticket_obj.update(fields={"labels":ticket_obj.fields.labels})
|
ticket_obj.update(fields={"labels":ticket_obj.fields.labels})
|
||||||
self.logger.info("Removed label {label} from ticket {ticket}".format(label=label, ticket=ticketid))
|
self.logger.info("Removed label {label} from ticket {ticket}".format(label=label, ticket=ticketid))
|
||||||
except Exception as e:
|
except:
|
||||||
self.logger.error("Error while trying to remove label {label} to ticket {ticket}".format(label=label,
|
self.logger.error("Error while trying to remove label {label} to ticket {ticket}".format(label=label, ticket=ticketid))
|
||||||
ticket=ticketid))
|
|
||||||
else:
|
else:
|
||||||
self.logger.error("Error: label {label} not in ticket {ticket}".format(label=label, ticket=ticketid))
|
self.logger.error("Error: label {label} not in ticket {ticket}".format(label=label, ticket=ticketid))
|
||||||
|
|
||||||
@ -511,13 +483,14 @@ class JiraAPI(object):
|
|||||||
self.close_ticket(ticket, self.JIRA_RESOLUTION_FIXED, comment)
|
self.close_ticket(ticket, self.JIRA_RESOLUTION_FIXED, comment)
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|
||||||
def is_ticket_reopenable(self, ticket_obj):
|
def is_ticket_reopenable(self, ticket_obj):
|
||||||
transitions = self.jira.transitions(ticket_obj)
|
transitions = self.jira.transitions(ticket_obj)
|
||||||
for transition in transitions:
|
for transition in transitions:
|
||||||
if transition.get('name') == self.JIRA_REOPEN_ISSUE:
|
if transition.get('name') == self.JIRA_REOPEN_ISSUE:
|
||||||
self.logger.debug("Ticket is reopenable")
|
self.logger.debug("Ticket is reopenable")
|
||||||
return True
|
return True
|
||||||
self.logger.warn("Ticket can't be opened. Check Jira transitions.")
|
self.logger.error("Ticket {} can't be opened. Check Jira transitions.".format(ticket_obj))
|
||||||
return False
|
return False
|
||||||
|
|
||||||
def is_ticket_closeable(self, ticket_obj):
|
def is_ticket_closeable(self, ticket_obj):
|
||||||
@ -525,7 +498,7 @@ class JiraAPI(object):
|
|||||||
for transition in transitions:
|
for transition in transitions:
|
||||||
if transition.get('name') == self.JIRA_CLOSE_ISSUE:
|
if transition.get('name') == self.JIRA_CLOSE_ISSUE:
|
||||||
return True
|
return True
|
||||||
self.logger.warn("Ticket can't closed. Check Jira transitions.")
|
self.logger.error("Ticket {} can't closed. Check Jira transitions.".format(ticket_obj))
|
||||||
return False
|
return False
|
||||||
|
|
||||||
def is_ticket_resolved(self, ticket_obj):
|
def is_ticket_resolved(self, ticket_obj):
|
||||||
@ -539,6 +512,7 @@ class JiraAPI(object):
|
|||||||
self.logger.debug("Checked ticket {} is already open".format(ticket_obj))
|
self.logger.debug("Checked ticket {} is already open".format(ticket_obj))
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
|
||||||
def is_risk_accepted(self, ticket_obj):
|
def is_risk_accepted(self, ticket_obj):
|
||||||
if ticket_obj is not None:
|
if ticket_obj is not None:
|
||||||
if ticket_obj.raw['fields'].get('labels') is not None:
|
if ticket_obj.raw['fields'].get('labels') is not None:
|
||||||
@ -564,8 +538,7 @@ class JiraAPI(object):
|
|||||||
if (not self.is_risk_accepted(ticket_obj) or ignore_labels):
|
if (not self.is_risk_accepted(ticket_obj) or ignore_labels):
|
||||||
try:
|
try:
|
||||||
if self.is_ticket_reopenable(ticket_obj):
|
if self.is_ticket_reopenable(ticket_obj):
|
||||||
error = self.jira.transition_issue(issue=ticketid, transition=self.JIRA_REOPEN_ISSUE,
|
error = self.jira.transition_issue(issue=ticketid, transition=self.JIRA_REOPEN_ISSUE, comment = comment)
|
||||||
comment=comment)
|
|
||||||
self.logger.info("Ticket {} reopened successfully".format(ticketid))
|
self.logger.info("Ticket {} reopened successfully".format(ticketid))
|
||||||
if not ignore_labels:
|
if not ignore_labels:
|
||||||
self.add_label(ticketid, 'reopened')
|
self.add_label(ticketid, 'reopened')
|
||||||
@ -585,8 +558,7 @@ class JiraAPI(object):
|
|||||||
if self.is_ticket_closeable(ticket_obj):
|
if self.is_ticket_closeable(ticket_obj):
|
||||||
#need to add the label before closing the ticket
|
#need to add the label before closing the ticket
|
||||||
self.add_label(ticketid, 'closed')
|
self.add_label(ticketid, 'closed')
|
||||||
error = self.jira.transition_issue(issue=ticketid, transition=self.JIRA_CLOSE_ISSUE,
|
error = self.jira.transition_issue(issue=ticketid, transition=self.JIRA_CLOSE_ISSUE, comment = comment, resolution = {"name": resolution })
|
||||||
comment=comment, resolution={"name": resolution})
|
|
||||||
self.logger.info("Ticket {} closed successfully".format(ticketid))
|
self.logger.info("Ticket {} closed successfully".format(ticketid))
|
||||||
return 1
|
return 1
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
@ -599,8 +571,7 @@ class JiraAPI(object):
|
|||||||
def close_obsolete_tickets(self):
|
def close_obsolete_tickets(self):
|
||||||
# Close tickets older than 12 months, vulnerabilities not solved will get created a new ticket
|
# Close tickets older than 12 months, vulnerabilities not solved will get created a new ticket
|
||||||
self.logger.info("Closing obsolete tickets older than {} months".format(self.max_time_tracking))
|
self.logger.info("Closing obsolete tickets older than {} months".format(self.max_time_tracking))
|
||||||
jql = "labels=vulnerability_management AND NOT labels=advisory AND created <startOfMonth(-{}) and resolution=Unresolved".format(
|
jql = "labels=vulnerability_management AND NOT labels=advisory AND created <startOfMonth(-{}) and resolution=Unresolved".format(self.max_time_tracking)
|
||||||
self.max_time_tracking)
|
|
||||||
tickets_to_close = self.jira.search_issues(jql, maxResults=0)
|
tickets_to_close = self.jira.search_issues(jql, maxResults=0)
|
||||||
|
|
||||||
comment = '''This ticket is being closed for hygiene, as it is more than {} months old.
|
comment = '''This ticket is being closed for hygiene, as it is more than {} months old.
|
||||||
@ -631,8 +602,7 @@ class JiraAPI(object):
|
|||||||
return True
|
return True
|
||||||
try:
|
try:
|
||||||
self.logger.info("Saving locally tickets from the last {} months".format(self.max_time_tracking))
|
self.logger.info("Saving locally tickets from the last {} months".format(self.max_time_tracking))
|
||||||
jql = "labels=vulnerability_management AND NOT labels=advisory AND created >=startOfMonth(-{})".format(
|
jql = "labels=vulnerability_management AND NOT labels=advisory AND created >=startOfMonth(-{})".format(self.max_time_tracking)
|
||||||
self.max_time_tracking)
|
|
||||||
tickets_data = self.jira.search_issues(jql, maxResults=0)
|
tickets_data = self.jira.search_issues(jql, maxResults=0)
|
||||||
|
|
||||||
#TODO process tickets, creating a new field called "_metadata" with all the affected assets well structured
|
#TODO process tickets, creating a new field called "_metadata" with all the affected assets well structured
|
||||||
@ -656,6 +626,7 @@ class JiraAPI(object):
|
|||||||
assets_json = self.parse_asset_to_json(assets)
|
assets_json = self.parse_asset_to_json(assets)
|
||||||
_metadata["affected_hosts"].append(assets_json)
|
_metadata["affected_hosts"].append(assets_json)
|
||||||
|
|
||||||
|
|
||||||
temp_ticket = ticket.raw.get('fields')
|
temp_ticket = ticket.raw.get('fields')
|
||||||
temp_ticket['_metadata'] = _metadata
|
temp_ticket['_metadata'] = _metadata
|
||||||
|
|
||||||
@ -680,16 +651,13 @@ class JiraAPI(object):
|
|||||||
closed already for more than x months (default is 3 months) in order to clean solved issues
|
closed already for more than x months (default is 3 months) in order to clean solved issues
|
||||||
for statistics purposes
|
for statistics purposes
|
||||||
'''
|
'''
|
||||||
self.logger.info("Deleting 'server_decommission' tag from tickets closed more than {} months ago".format(
|
self.logger.info("Deleting 'server_decommission' tag from tickets closed more than {} months ago".format(self.max_decommission_time))
|
||||||
self.max_decommission_time))
|
|
||||||
|
|
||||||
jql = "labels=vulnerability_management AND labels=server_decommission and resolutiondate <=startOfMonth(-{})".format(
|
jql = "labels=vulnerability_management AND labels=server_decommission and resolutiondate <=startOfMonth(-{})".format(self.max_decommission_time)
|
||||||
self.max_decommission_time)
|
|
||||||
decommissioned_tickets = self.jira.search_issues(jql, maxResults=0)
|
decommissioned_tickets = self.jira.search_issues(jql, maxResults=0)
|
||||||
|
|
||||||
comment = '''This ticket is having deleted the *server_decommission* tag, as it is more than {} months old and is expected to already have been decommissioned.
|
comment = '''This ticket is having deleted the *server_decommission* tag, as it is more than {} months old and is expected to already have been decommissioned.
|
||||||
If that is not the case and the vulnerability still exists, the vulnerability will be opened again.'''.format(
|
If that is not the case and the vulnerability still exists, the vulnerability will be opened again.'''.format(self.max_decommission_time)
|
||||||
self.max_decommission_time)
|
|
||||||
|
|
||||||
for ticket in decommissioned_tickets:
|
for ticket in decommissioned_tickets:
|
||||||
#we open first the ticket, as we want to make sure the process is not blocked due to
|
#we open first the ticket, as we want to make sure the process is not blocked due to
|
||||||
|
@ -1,4 +1,3 @@
|
|||||||
from __future__ import absolute_import
|
|
||||||
import os
|
import os
|
||||||
import logging
|
import logging
|
||||||
import httpretty
|
import httpretty
|
||||||
@ -21,12 +20,10 @@ class mockAPI(object):
|
|||||||
|
|
||||||
def get_directories(self, path):
|
def get_directories(self, path):
|
||||||
dir, subdirs, files = next(os.walk(path))
|
dir, subdirs, files = next(os.walk(path))
|
||||||
self.logger.debug('Subdirectories found: {}'.format(subdirs))
|
|
||||||
return subdirs
|
return subdirs
|
||||||
|
|
||||||
def get_files(self, path):
|
def get_files(self, path):
|
||||||
dir, subdirs, files = next(os.walk(path))
|
dir, subdirs, files = next(os.walk(path))
|
||||||
self.logger.debug('Files found: {}'.format(files))
|
|
||||||
return files
|
return files
|
||||||
|
|
||||||
def qualys_vuln_callback(self, request, uri, response_headers):
|
def qualys_vuln_callback(self, request, uri, response_headers):
|
||||||
|
@ -1,17 +1,13 @@
|
|||||||
#!/usr/bin/python
|
#!/usr/bin/python
|
||||||
# -*- coding: utf-8 -*-
|
# -*- coding: utf-8 -*-
|
||||||
from __future__ import absolute_import
|
|
||||||
from six.moves import range
|
|
||||||
from functools import reduce
|
|
||||||
|
|
||||||
__author__ = 'Austin Taylor'
|
__author__ = 'Austin Taylor'
|
||||||
|
|
||||||
from .base.config import vwConfig
|
from base.config import vwConfig
|
||||||
from .frameworks.nessus import NessusAPI
|
from frameworks.nessus import NessusAPI
|
||||||
from .frameworks.qualys_web import qualysScanReport
|
from frameworks.qualys_web import qualysScanReport
|
||||||
from .frameworks.qualys_vuln import qualysVulnScan
|
from frameworks.qualys_vuln import qualysVulnScan
|
||||||
from .frameworks.openvas import OpenVAS_API
|
from frameworks.openvas import OpenVAS_API
|
||||||
from .reporting.jira_api import JiraAPI
|
from reporting.jira_api import JiraAPI
|
||||||
import pandas as pd
|
import pandas as pd
|
||||||
from lxml import objectify
|
from lxml import objectify
|
||||||
import sys
|
import sys
|
||||||
@ -25,6 +21,7 @@ import socket
|
|||||||
|
|
||||||
|
|
||||||
class vulnWhispererBase(object):
|
class vulnWhispererBase(object):
|
||||||
|
|
||||||
CONFIG_SECTION = None
|
CONFIG_SECTION = None
|
||||||
|
|
||||||
def __init__(
|
def __init__(
|
||||||
@ -68,6 +65,8 @@ class vulnWhispererBase(object):
|
|||||||
self.db_path = self.config.get(self.CONFIG_SECTION, 'db_path')
|
self.db_path = self.config.get(self.CONFIG_SECTION, 'db_path')
|
||||||
self.verbose = self.config.getbool(self.CONFIG_SECTION, 'verbose')
|
self.verbose = self.config.getbool(self.CONFIG_SECTION, 'verbose')
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
if self.db_name is not None:
|
if self.db_name is not None:
|
||||||
if self.db_path:
|
if self.db_path:
|
||||||
self.database = os.path.join(self.db_path,
|
self.database = os.path.join(self.db_path,
|
||||||
@ -89,8 +88,7 @@ class vulnWhispererBase(object):
|
|||||||
self.cur = self.conn.cursor()
|
self.cur = self.conn.cursor()
|
||||||
self.logger.info('Connected to database at {loc}'.format(loc=self.database))
|
self.logger.info('Connected to database at {loc}'.format(loc=self.database))
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
self.logger.error(
|
self.logger.error('Could not connect to database at {loc}\nReason: {e} - Please ensure the path exist'.format(
|
||||||
'Could not connect to database at {loc}\nReason: {e} - Please ensure the path exist'.format(
|
|
||||||
e=e,
|
e=e,
|
||||||
loc=self.database))
|
loc=self.database))
|
||||||
else:
|
else:
|
||||||
@ -187,8 +185,7 @@ class vulnWhispererBase(object):
|
|||||||
"""
|
"""
|
||||||
try:
|
try:
|
||||||
self.conn.text_factory = str
|
self.conn.text_factory = str
|
||||||
self.cur.execute('SELECT uuid FROM scan_history where source = "{config_section}"'.format(
|
self.cur.execute('SELECT uuid FROM scan_history where source = "{config_section}"'.format(config_section=self.CONFIG_SECTION))
|
||||||
config_section=self.CONFIG_SECTION))
|
|
||||||
results = frozenset([r[0] for r in self.cur.fetchall()])
|
results = frozenset([r[0] for r in self.cur.fetchall()])
|
||||||
except:
|
except:
|
||||||
results = []
|
results = []
|
||||||
@ -211,9 +208,7 @@ class vulnWhispererBase(object):
|
|||||||
|
|
||||||
try:
|
try:
|
||||||
self.conn.text_factory = str
|
self.conn.text_factory = str
|
||||||
self.cur.execute(
|
self.cur.execute('SELECT filename FROM scan_history WHERE source="{}" AND scan_name="{}" ORDER BY last_modified DESC LIMIT 1;'.format(source, scan_name))
|
||||||
'SELECT filename FROM scan_history WHERE source="{}" AND scan_name="{}" ORDER BY last_modified DESC LIMIT 1;'.format(
|
|
||||||
source, scan_name))
|
|
||||||
#should always return just one filename
|
#should always return just one filename
|
||||||
results = [r[0] for r in self.cur.fetchall()][0]
|
results = [r[0] for r in self.cur.fetchall()][0]
|
||||||
|
|
||||||
@ -221,13 +216,10 @@ class vulnWhispererBase(object):
|
|||||||
#TODO delete backward compatibility check after some versions
|
#TODO delete backward compatibility check after some versions
|
||||||
last_column_table = self.cur.execute('PRAGMA table_info(scan_history)').fetchall()[-1][1]
|
last_column_table = self.cur.execute('PRAGMA table_info(scan_history)').fetchall()[-1][1]
|
||||||
if results and last_column_table == self.table_columns[-1]:
|
if results and last_column_table == self.table_columns[-1]:
|
||||||
reported = self.cur.execute(
|
reported = self.cur.execute('SELECT reported FROM scan_history WHERE filename="{}"'.format(results)).fetchall()
|
||||||
'SELECT reported FROM scan_history WHERE filename="{}"'.format(results)).fetchall()
|
|
||||||
reported = reported[0][0]
|
reported = reported[0][0]
|
||||||
if reported:
|
if reported:
|
||||||
self.logger.debug(
|
self.logger.debug("Last downloaded scan from source {source} scan_name {scan_name} has already been reported".format(source=source, scan_name=scan_name))
|
||||||
"Last downloaded scan from source {source} scan_name {scan_name} has already been reported".format(
|
|
||||||
source=source, scan_name=scan_name))
|
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
self.logger.error("Error when getting latest results from {}.{} : {}".format(source, scan_name, e))
|
self.logger.error("Error when getting latest results from {}.{} : {}".format(source, scan_name, e))
|
||||||
@ -262,8 +254,8 @@ class vulnWhispererBase(object):
|
|||||||
|
|
||||||
return results
|
return results
|
||||||
|
|
||||||
|
|
||||||
class vulnWhispererNessus(vulnWhispererBase):
|
class vulnWhispererNessus(vulnWhispererBase):
|
||||||
|
|
||||||
CONFIG_SECTION = None
|
CONFIG_SECTION = None
|
||||||
|
|
||||||
def __init__(
|
def __init__(
|
||||||
@ -330,6 +322,8 @@ class vulnWhispererNessus(vulnWhispererBase):
|
|||||||
return False
|
return False
|
||||||
#sys.exit(1)
|
#sys.exit(1)
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
def scan_count(self, scans, completed=False):
|
def scan_count(self, scans, completed=False):
|
||||||
"""
|
"""
|
||||||
|
|
||||||
@ -371,6 +365,7 @@ class vulnWhispererNessus(vulnWhispererBase):
|
|||||||
scan_records = [s for s in scan_records if s['status'] == 'completed']
|
scan_records = [s for s in scan_records if s['status'] == 'completed']
|
||||||
return scan_records
|
return scan_records
|
||||||
|
|
||||||
|
|
||||||
def whisper_nessus(self):
|
def whisper_nessus(self):
|
||||||
if self.nessus_connect:
|
if self.nessus_connect:
|
||||||
scan_data = self.nessus.scans
|
scan_data = self.nessus.scans
|
||||||
@ -425,8 +420,7 @@ class vulnWhispererNessus(vulnWhispererBase):
|
|||||||
s['uuid'],
|
s['uuid'],
|
||||||
)
|
)
|
||||||
|
|
||||||
# TODO Create directory sync function which scans the directory for files that exist already and
|
# TODO Create directory sync function which scans the directory for files that exist already and populates the database
|
||||||
# populates the database
|
|
||||||
|
|
||||||
folder_id = s['folder_id']
|
folder_id = s['folder_id']
|
||||||
if self.CONFIG_SECTION == 'tenable':
|
if self.CONFIG_SECTION == 'tenable':
|
||||||
@ -456,26 +450,22 @@ class vulnWhispererNessus(vulnWhispererBase):
|
|||||||
0,
|
0,
|
||||||
)
|
)
|
||||||
self.record_insert(record_meta)
|
self.record_insert(record_meta)
|
||||||
self.logger.info(
|
self.logger.info('File {filename} already exist! Updating database'.format(filename=relative_path_name))
|
||||||
'File {filename} already exist! Updating database'.format(filename=relative_path_name))
|
|
||||||
else:
|
else:
|
||||||
try:
|
try:
|
||||||
file_req = \
|
file_req = \
|
||||||
self.nessus.download_scan(scan_id=scan_id, history=history_id,
|
self.nessus.download_scan(scan_id=scan_id, history=history_id,
|
||||||
export_format='csv')
|
export_format='csv')
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
self.logger.error(
|
self.logger.error('Could not download {} scan {}: {}'.format(self.CONFIG_SECTION, scan_id, str(e)))
|
||||||
'Could not download {} scan {}: {}'.format(self.CONFIG_SECTION, scan_id, str(e)))
|
|
||||||
self.exit_code += 1
|
self.exit_code += 1
|
||||||
continue
|
continue
|
||||||
|
|
||||||
clean_csv = \
|
clean_csv = \
|
||||||
pd.read_csv(io.StringIO(file_req.decode('utf-8')))
|
pd.read_csv(io.StringIO(file_req.decode('utf-8')))
|
||||||
if len(clean_csv) > 2:
|
if len(clean_csv) > 2:
|
||||||
self.logger.info('Processing {}/{} for scan: {}'.format(scan_count, len(scan_list),
|
self.logger.info('Processing {}/{} for scan: {}'.format(scan_count, len(scan_list), scan_name.encode('utf8')))
|
||||||
scan_name.encode('utf8')))
|
columns_to_cleanse = ['CVSS','CVE','Description','Synopsis','Solution','See Also','Plugin Output', 'MAC Address']
|
||||||
columns_to_cleanse = ['CVSS', 'CVE', 'Description', 'Synopsis', 'Solution', 'See Also',
|
|
||||||
'Plugin Output', 'MAC Address']
|
|
||||||
|
|
||||||
for col in columns_to_cleanse:
|
for col in columns_to_cleanse:
|
||||||
if col in clean_csv:
|
if col in clean_csv:
|
||||||
@ -496,8 +486,7 @@ class vulnWhispererNessus(vulnWhispererBase):
|
|||||||
)
|
)
|
||||||
self.record_insert(record_meta)
|
self.record_insert(record_meta)
|
||||||
self.logger.info('{filename} records written to {path} '.format(filename=clean_csv.shape[0],
|
self.logger.info('{filename} records written to {path} '.format(filename=clean_csv.shape[0],
|
||||||
path=file_name.encode(
|
path=file_name.encode('utf8')))
|
||||||
'utf8')))
|
|
||||||
else:
|
else:
|
||||||
record_meta = (
|
record_meta = (
|
||||||
scan_name,
|
scan_name,
|
||||||
@ -512,32 +501,27 @@ class vulnWhispererNessus(vulnWhispererBase):
|
|||||||
0,
|
0,
|
||||||
)
|
)
|
||||||
self.record_insert(record_meta)
|
self.record_insert(record_meta)
|
||||||
self.logger.warn(
|
self.logger.warn('{} has no host available... Updating database and skipping!'.format(file_name))
|
||||||
'{} has no host available... Updating database and skipping!'.format(file_name))
|
|
||||||
self.conn.close()
|
self.conn.close()
|
||||||
self.logger.info('Scan aggregation complete! Connection to database closed.')
|
self.logger.info('Scan aggregation complete! Connection to database closed.')
|
||||||
else:
|
else:
|
||||||
self.logger.error(
|
self.logger.error('Failed to use scanner at {host}:{port}'.format(host=self.hostname, port=self.nessus_port))
|
||||||
'Failed to use scanner at {host}:{port}'.format(host=self.hostname, port=self.nessus_port))
|
|
||||||
self.exit_code += 1
|
self.exit_code += 1
|
||||||
return self.exit_code
|
return self.exit_code
|
||||||
|
|
||||||
|
|
||||||
class vulnWhispererQualys(vulnWhispererBase):
|
class vulnWhispererQualys(vulnWhispererBase):
|
||||||
|
|
||||||
CONFIG_SECTION = 'qualys_web'
|
CONFIG_SECTION = 'qualys_web'
|
||||||
COLUMN_MAPPING = {'Access Path': 'access_path',
|
COLUMN_MAPPING = {'Access Path': 'access_path',
|
||||||
'Ajax Request': 'ajax_request',
|
'Ajax Request': 'ajax_request',
|
||||||
'Ajax Request ID': 'ajax_request_id',
|
'Ajax Request ID': 'ajax_request_id',
|
||||||
'Authentication': 'authentication',
|
'Authentication': 'authentication',
|
||||||
'CVSS Base': 'cvss',
|
'CVSS Base': 'cvss',
|
||||||
'CVSS V3 Attack Vector': 'cvss_v3_attack_vector',
|
|
||||||
'CVSS V3 Base': 'cvss_v3_base',
|
|
||||||
'CVSS V3 Temporal': 'cvss_v3_temporal',
|
|
||||||
'CVSS Temporal': 'cvss_temporal',
|
'CVSS Temporal': 'cvss_temporal',
|
||||||
'CWE': 'cwe',
|
'CWE': 'cwe',
|
||||||
'Category': 'category',
|
'Category': 'category',
|
||||||
'Content': 'content',
|
'Content': 'content',
|
||||||
'Custom Attributes': 'custom_attributes',
|
|
||||||
'DescriptionSeverity': 'severity_description',
|
'DescriptionSeverity': 'severity_description',
|
||||||
'DescriptionCatSev': 'category_description',
|
'DescriptionCatSev': 'category_description',
|
||||||
'Detection ID': 'detection_id',
|
'Detection ID': 'detection_id',
|
||||||
@ -553,19 +537,15 @@ class vulnWhispererQualys(vulnWhispererBase):
|
|||||||
'Ignore User': 'ignore_user',
|
'Ignore User': 'ignore_user',
|
||||||
'Ignored': 'ignored',
|
'Ignored': 'ignored',
|
||||||
'Impact': 'impact',
|
'Impact': 'impact',
|
||||||
'Info#1': 'info_1',
|
|
||||||
'Last Time Detected': 'last_time_detected',
|
'Last Time Detected': 'last_time_detected',
|
||||||
'Last Time Tested': 'last_time_tested',
|
'Last Time Tested': 'last_time_tested',
|
||||||
'Level': 'level',
|
'Level': 'level',
|
||||||
'OWASP': 'owasp',
|
'OWASP': 'owasp',
|
||||||
'Operating System': 'operating_system',
|
'Operating System': 'operating_system',
|
||||||
'Owner': 'owner',
|
'Owner': 'owner',
|
||||||
'Param/Cookie': 'param',
|
'Param': 'param',
|
||||||
'Payload #1': 'payload_1',
|
'Payload #1': 'payload_1',
|
||||||
'Port': 'port',
|
|
||||||
'Protocol': 'protocol',
|
|
||||||
'QID': 'plugin_id',
|
'QID': 'plugin_id',
|
||||||
'Request Body #1': 'request_body_1',
|
|
||||||
'Request Headers #1': 'request_headers_1',
|
'Request Headers #1': 'request_headers_1',
|
||||||
'Request Method #1': 'request_method_1',
|
'Request Method #1': 'request_method_1',
|
||||||
'Request URL #1': 'request_url_1',
|
'Request URL #1': 'request_url_1',
|
||||||
@ -574,17 +554,13 @@ class vulnWhispererQualys(vulnWhispererBase):
|
|||||||
'Severity': 'risk',
|
'Severity': 'risk',
|
||||||
'Severity Level': 'security_level',
|
'Severity Level': 'security_level',
|
||||||
'Solution': 'solution',
|
'Solution': 'solution',
|
||||||
'Tags': 'tags',
|
|
||||||
'Times Detected': 'times_detected',
|
'Times Detected': 'times_detected',
|
||||||
'Title': 'plugin_name',
|
'Title': 'plugin_name',
|
||||||
'URL': 'url',
|
'URL': 'url',
|
||||||
'Unique ID': 'unique_id',
|
|
||||||
'Url': 'uri',
|
'Url': 'uri',
|
||||||
'Vulnerability Category': 'vulnerability_category',
|
'Vulnerability Category': 'vulnerability_category',
|
||||||
'Virtual Host': 'virutal_host',
|
|
||||||
'WASC': 'wasc',
|
'WASC': 'wasc',
|
||||||
'Web Application Name': 'web_application_name'}
|
'Web Application Name': 'web_application_name'}
|
||||||
|
|
||||||
def __init__(
|
def __init__(
|
||||||
self,
|
self,
|
||||||
config=None,
|
config=None,
|
||||||
@ -692,8 +668,7 @@ class vulnWhispererQualys(vulnWhispererBase):
|
|||||||
self.logger.info('Removing report {} from Qualys Database'.format(generated_report_id))
|
self.logger.info('Removing report {} from Qualys Database'.format(generated_report_id))
|
||||||
cleaning_up = self.qualys_scan.qw.delete_report(generated_report_id)
|
cleaning_up = self.qualys_scan.qw.delete_report(generated_report_id)
|
||||||
os.remove(self.path_check(str(generated_report_id) + '.csv'))
|
os.remove(self.path_check(str(generated_report_id) + '.csv'))
|
||||||
self.logger.info(
|
self.logger.info('Deleted report from local disk: {}'.format(self.path_check(str(generated_report_id))))
|
||||||
'Deleted report from local disk: {}'.format(self.path_check(str(generated_report_id))))
|
|
||||||
else:
|
else:
|
||||||
self.logger.error('Could not process report ID: {}'.format(status))
|
self.logger.error('Could not process report ID: {}'.format(status))
|
||||||
|
|
||||||
@ -701,6 +676,7 @@ class vulnWhispererQualys(vulnWhispererBase):
|
|||||||
self.logger.error('Could not process {}: {}'.format(report_id, str(e)))
|
self.logger.error('Could not process {}: {}'.format(report_id, str(e)))
|
||||||
return vuln_ready
|
return vuln_ready
|
||||||
|
|
||||||
|
|
||||||
def identify_scans_to_process(self):
|
def identify_scans_to_process(self):
|
||||||
if self.uuids:
|
if self.uuids:
|
||||||
self.scans_to_process = self.latest_scans[~self.latest_scans['id'].isin(self.uuids)]
|
self.scans_to_process = self.latest_scans[~self.latest_scans['id'].isin(self.uuids)]
|
||||||
@ -708,6 +684,7 @@ class vulnWhispererQualys(vulnWhispererBase):
|
|||||||
self.scans_to_process = self.latest_scans
|
self.scans_to_process = self.latest_scans
|
||||||
self.logger.info('Identified {new} scans to be processed'.format(new=len(self.scans_to_process)))
|
self.logger.info('Identified {new} scans to be processed'.format(new=len(self.scans_to_process)))
|
||||||
|
|
||||||
|
|
||||||
def process_web_assets(self):
|
def process_web_assets(self):
|
||||||
counter = 0
|
counter = 0
|
||||||
self.identify_scans_to_process()
|
self.identify_scans_to_process()
|
||||||
@ -788,6 +765,7 @@ class vulnWhispererOpenVAS(vulnWhispererBase):
|
|||||||
if report_id:
|
if report_id:
|
||||||
self.logger.info('Processing report ID: {}'.format(report_id))
|
self.logger.info('Processing report ID: {}'.format(report_id))
|
||||||
|
|
||||||
|
|
||||||
scan_name = report_id.replace('-', '')
|
scan_name = report_id.replace('-', '')
|
||||||
report_name = 'openvas_scan_{scan_name}_{last_updated}.{extension}'.format(scan_name=scan_name,
|
report_name = 'openvas_scan_{scan_name}_{last_updated}.{extension}'.format(scan_name=scan_name,
|
||||||
last_updated=launched_date,
|
last_updated=launched_date,
|
||||||
@ -855,8 +833,7 @@ class vulnWhispererOpenVAS(vulnWhispererBase):
|
|||||||
for scan in self.scans_to_process.iterrows():
|
for scan in self.scans_to_process.iterrows():
|
||||||
counter += 1
|
counter += 1
|
||||||
info = scan[1]
|
info = scan[1]
|
||||||
self.logger.info(
|
self.logger.info('Processing {}/{} - Report ID: {}'.format(counter, len(self.scans_to_process), info['report_ids']))
|
||||||
'Processing {}/{} - Report ID: {}'.format(counter, len(self.scans_to_process), info['report_ids']))
|
|
||||||
self.whisper_reports(report_id=info['report_ids'],
|
self.whisper_reports(report_id=info['report_ids'],
|
||||||
launched_date=info['epoch'])
|
launched_date=info['epoch'])
|
||||||
self.logger.info('Processing complete')
|
self.logger.info('Processing complete')
|
||||||
@ -867,6 +844,7 @@ class vulnWhispererOpenVAS(vulnWhispererBase):
|
|||||||
|
|
||||||
|
|
||||||
class vulnWhispererQualysVuln(vulnWhispererBase):
|
class vulnWhispererQualysVuln(vulnWhispererBase):
|
||||||
|
|
||||||
CONFIG_SECTION = 'qualys_vuln'
|
CONFIG_SECTION = 'qualys_vuln'
|
||||||
COLUMN_MAPPING = {'cvss_base': 'cvss',
|
COLUMN_MAPPING = {'cvss_base': 'cvss',
|
||||||
'cvss3_base': 'cvss3',
|
'cvss3_base': 'cvss3',
|
||||||
@ -967,6 +945,7 @@ class vulnWhispererQualysVuln(vulnWhispererBase):
|
|||||||
self.logger.info('Report written to {}'.format(report_name))
|
self.logger.info('Report written to {}'.format(report_name))
|
||||||
return self.exit_code
|
return self.exit_code
|
||||||
|
|
||||||
|
|
||||||
def identify_scans_to_process(self):
|
def identify_scans_to_process(self):
|
||||||
self.latest_scans = self.qualys_scan.qw.get_all_scans()
|
self.latest_scans = self.qualys_scan.qw.get_all_scans()
|
||||||
if self.uuids:
|
if self.uuids:
|
||||||
@ -977,6 +956,7 @@ class vulnWhispererQualysVuln(vulnWhispererBase):
|
|||||||
self.scans_to_process = self.latest_scans
|
self.scans_to_process = self.latest_scans
|
||||||
self.logger.info('Identified {new} scans to be processed'.format(new=len(self.scans_to_process)))
|
self.logger.info('Identified {new} scans to be processed'.format(new=len(self.scans_to_process)))
|
||||||
|
|
||||||
|
|
||||||
def process_vuln_scans(self):
|
def process_vuln_scans(self):
|
||||||
counter = 0
|
counter = 0
|
||||||
self.identify_scans_to_process()
|
self.identify_scans_to_process()
|
||||||
@ -996,6 +976,7 @@ class vulnWhispererQualysVuln(vulnWhispererBase):
|
|||||||
|
|
||||||
|
|
||||||
class vulnWhispererJIRA(vulnWhispererBase):
|
class vulnWhispererJIRA(vulnWhispererBase):
|
||||||
|
|
||||||
CONFIG_SECTION = 'jira'
|
CONFIG_SECTION = 'jira'
|
||||||
|
|
||||||
def __init__(
|
def __init__(
|
||||||
@ -1047,11 +1028,10 @@ class vulnWhispererJIRA(vulnWhispererBase):
|
|||||||
|
|
||||||
if not self.config.exists_jira_profiles(profiles):
|
if not self.config.exists_jira_profiles(profiles):
|
||||||
self.config.update_jira_profiles(profiles)
|
self.config.update_jira_profiles(profiles)
|
||||||
self.logger.info(
|
self.logger.info("Jira profiles have been created in {config}, please fill the variables before rerunning the module.".format(config=self.config_path))
|
||||||
"Jira profiles have been created in {config}, please fill the variables before rerunning the module.".format(
|
|
||||||
config=self.config_path))
|
|
||||||
sys.exit(0)
|
sys.exit(0)
|
||||||
|
|
||||||
|
|
||||||
def get_env_variables(self, source, scan_name):
|
def get_env_variables(self, source, scan_name):
|
||||||
# function returns an array with [jira_project, jira_components, datafile_path]
|
# function returns an array with [jira_project, jira_components, datafile_path]
|
||||||
|
|
||||||
@ -1089,15 +1069,11 @@ class vulnWhispererJIRA(vulnWhispererBase):
|
|||||||
fullpath = "{}/{}".format(root,filename)
|
fullpath = "{}/{}".format(root,filename)
|
||||||
|
|
||||||
if reported:
|
if reported:
|
||||||
self.logger.warn(
|
self.logger.warn('Last Scan of "{scan_name}" for source "{source}" has already been reported; will be skipped.'.format(scan_name=scan_name, source=source))
|
||||||
'Last Scan of "{scan_name}" for source "{source}" has already been reported; will be skipped.'.format(
|
|
||||||
scan_name=scan_name, source=source))
|
|
||||||
return [False] * 5
|
return [False] * 5
|
||||||
|
|
||||||
if not fullpath:
|
if not fullpath:
|
||||||
self.logger.error(
|
self.logger.error('Scan of "{scan_name}" for source "{source}" has not been found. Please check that the scanner data files are in place.'.format(scan_name=scan_name, source=source))
|
||||||
'Scan of "{scan_name}" for source "{source}" has not been found. Please check that the scanner data files are in place.'.format(
|
|
||||||
scan_name=scan_name, source=source))
|
|
||||||
sys.exit(1)
|
sys.exit(1)
|
||||||
|
|
||||||
dns_resolv = self.config.get('jira','dns_resolv')
|
dns_resolv = self.config.get('jira','dns_resolv')
|
||||||
@ -1111,6 +1087,7 @@ class vulnWhispererJIRA(vulnWhispererBase):
|
|||||||
|
|
||||||
return project, components, fullpath, min_critical, dns_resolv
|
return project, components, fullpath, min_critical, dns_resolv
|
||||||
|
|
||||||
|
|
||||||
def parse_nessus_vulnerabilities(self, fullpath, source, scan_name, min_critical):
|
def parse_nessus_vulnerabilities(self, fullpath, source, scan_name, min_critical):
|
||||||
|
|
||||||
vulnerabilities = []
|
vulnerabilities = []
|
||||||
@ -1139,8 +1116,7 @@ class vulnWhispererJIRA(vulnWhispererBase):
|
|||||||
vuln['consequence'] = df.loc[index]['Description'].replace('\\n',' ')
|
vuln['consequence'] = df.loc[index]['Description'].replace('\\n',' ')
|
||||||
vuln['solution'] = df.loc[index]['Solution'].replace('\\n',' ')
|
vuln['solution'] = df.loc[index]['Solution'].replace('\\n',' ')
|
||||||
vuln['ips'] = []
|
vuln['ips'] = []
|
||||||
vuln['ips'].append(
|
vuln['ips'].append("{} - {}/{}".format(df.loc[index]['Host'], df.loc[index]['Protocol'], df.loc[index]['Port']))
|
||||||
"{} - {}/{}".format(df.loc[index]['Host'], df.loc[index]['Protocol'], df.loc[index]['Port']))
|
|
||||||
vuln['risk'] = df.loc[index]['Risk'].lower()
|
vuln['risk'] = df.loc[index]['Risk'].lower()
|
||||||
|
|
||||||
# Nessus "nan" value gets automatically casted to float by python
|
# Nessus "nan" value gets automatically casted to float by python
|
||||||
@ -1154,8 +1130,7 @@ class vulnWhispererJIRA(vulnWhispererBase):
|
|||||||
# grouping assets by vulnerability to open on single ticket, as each asset has its own nessus entry
|
# grouping assets by vulnerability to open on single ticket, as each asset has its own nessus entry
|
||||||
for vuln in vulnerabilities:
|
for vuln in vulnerabilities:
|
||||||
if vuln['title'] == df.loc[index]['Name']:
|
if vuln['title'] == df.loc[index]['Name']:
|
||||||
vuln['ips'].append("{} - {}/{}".format(df.loc[index]['Host'], df.loc[index]['Protocol'],
|
vuln['ips'].append("{} - {}/{}".format(df.loc[index]['Host'], df.loc[index]['Protocol'], df.loc[index]['Port']))
|
||||||
df.loc[index]['Port']))
|
|
||||||
|
|
||||||
return vulnerabilities
|
return vulnerabilities
|
||||||
|
|
||||||
@ -1180,8 +1155,7 @@ class vulnWhispererJIRA(vulnWhispererBase):
|
|||||||
continue
|
continue
|
||||||
|
|
||||||
elif data[index]['type'] == 'Practice' or data[index]['type'] == 'Ig':
|
elif data[index]['type'] == 'Practice' or data[index]['type'] == 'Ig':
|
||||||
self.logger.debug("Vulnerability '{vuln}' ignored, as it is 'Practice/Potential', not verified.".format(
|
self.logger.debug("Vulnerability '{vuln}' ignored, as it is 'Practice/Potential', not verified.".format(vuln=data[index]['plugin_name']))
|
||||||
vuln=data[index]['plugin_name']))
|
|
||||||
continue
|
continue
|
||||||
|
|
||||||
if not vulnerabilities or data[index]['plugin_name'] not in [entry['title'] for entry in vulnerabilities]:
|
if not vulnerabilities or data[index]['plugin_name'] not in [entry['title'] for entry in vulnerabilities]:
|
||||||
@ -1197,8 +1171,7 @@ class vulnWhispererJIRA(vulnWhispererBase):
|
|||||||
vuln['ips'] = []
|
vuln['ips'] = []
|
||||||
#TODO ADDED DNS RESOLUTION FROM QUALYS! \n SEPARATORS INSTEAD OF \\n!
|
#TODO ADDED DNS RESOLUTION FROM QUALYS! \n SEPARATORS INSTEAD OF \\n!
|
||||||
|
|
||||||
vuln['ips'].append(
|
vuln['ips'].append("{ip} - {protocol}/{port} - {dns}".format(**self.get_asset_fields(data[index], dns_resolv)))
|
||||||
"{ip} - {protocol}/{port} - {dns}".format(**self.get_asset_fields(data[index], dns_resolv)))
|
|
||||||
|
|
||||||
#different risk system than Nessus!
|
#different risk system than Nessus!
|
||||||
vuln['risk'] = risks[int(data[index]['risk'])-1]
|
vuln['risk'] = risks[int(data[index]['risk'])-1]
|
||||||
@ -1213,8 +1186,7 @@ class vulnWhispererJIRA(vulnWhispererBase):
|
|||||||
# grouping assets by vulnerability to open on single ticket, as each asset has its own nessus entry
|
# grouping assets by vulnerability to open on single ticket, as each asset has its own nessus entry
|
||||||
for vuln in vulnerabilities:
|
for vuln in vulnerabilities:
|
||||||
if vuln['title'] == data[index]['plugin_name']:
|
if vuln['title'] == data[index]['plugin_name']:
|
||||||
vuln['ips'].append(
|
vuln['ips'].append("{ip} - {protocol}/{port} - {dns}".format(**self.get_asset_fields(data[index], dns_resolv)))
|
||||||
"{ip} - {protocol}/{port} - {dns}".format(**self.get_asset_fields(data[index], dns_resolv)))
|
|
||||||
|
|
||||||
return vulnerabilities
|
return vulnerabilities
|
||||||
|
|
||||||
@ -1228,7 +1200,7 @@ class vulnWhispererJIRA(vulnWhispererBase):
|
|||||||
if vuln['dns']:
|
if vuln['dns']:
|
||||||
values['dns'] = vuln['dns']
|
values['dns'] = vuln['dns']
|
||||||
else:
|
else:
|
||||||
if values['ip'] in list(self.host_resolv_cache.keys()):
|
if values['ip'] in self.host_resolv_cache.keys():
|
||||||
self.logger.debug("Hostname from {ip} cached, retrieving from cache.".format(ip=values['ip']))
|
self.logger.debug("Hostname from {ip} cached, retrieving from cache.".format(ip=values['ip']))
|
||||||
values['dns'] = self.host_resolv_cache[values['ip']]
|
values['dns'] = self.host_resolv_cache[values['ip']]
|
||||||
else:
|
else:
|
||||||
@ -1254,16 +1226,14 @@ class vulnWhispererJIRA(vulnWhispererBase):
|
|||||||
|
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|
||||||
def jira_sync(self, source, scan_name):
|
def jira_sync(self, source, scan_name):
|
||||||
self.logger.info("Jira Sync triggered for source '{source}' and scan '{scan_name}'".format(source=source,
|
self.logger.info("Jira Sync triggered for source '{source}' and scan '{scan_name}'".format(source=source, scan_name=scan_name))
|
||||||
scan_name=scan_name))
|
|
||||||
|
|
||||||
project, components, fullpath, min_critical, dns_resolv = self.get_env_variables(source, scan_name)
|
project, components, fullpath, min_critical, dns_resolv = self.get_env_variables(source, scan_name)
|
||||||
|
|
||||||
if not project:
|
if not project:
|
||||||
self.logger.debug(
|
self.logger.debug("Skipping scan for source '{source}' and scan '{scan_name}': vulnerabilities have already been reported.".format(source=source, scan_name=scan_name))
|
||||||
"Skipping scan for source '{source}' and scan '{scan_name}': vulnerabilities have already been reported.".format(
|
|
||||||
source=source, scan_name=scan_name))
|
|
||||||
return False
|
return False
|
||||||
|
|
||||||
vulnerabilities = []
|
vulnerabilities = []
|
||||||
@ -1274,21 +1244,23 @@ class vulnWhispererJIRA(vulnWhispererBase):
|
|||||||
|
|
||||||
#***Qualys VM parsing***
|
#***Qualys VM parsing***
|
||||||
if source == "qualys_vuln":
|
if source == "qualys_vuln":
|
||||||
vulnerabilities = self.parse_qualys_vuln_vulnerabilities(fullpath, source, scan_name, min_critical,
|
vulnerabilities = self.parse_qualys_vuln_vulnerabilities(fullpath, source, scan_name, min_critical, dns_resolv)
|
||||||
dns_resolv)
|
|
||||||
|
|
||||||
#***JIRA sync***
|
#***JIRA sync***
|
||||||
|
try:
|
||||||
if vulnerabilities:
|
if vulnerabilities:
|
||||||
self.logger.info('{source} data has been successfuly parsed'.format(source=source.upper()))
|
self.logger.info('{source} data has been successfuly parsed'.format(source=source.upper()))
|
||||||
self.logger.info('Starting JIRA sync')
|
self.logger.info('Starting JIRA sync')
|
||||||
|
|
||||||
self.jira.sync(vulnerabilities, project, components)
|
self.jira.sync(vulnerabilities, project, components)
|
||||||
else:
|
else:
|
||||||
self.logger.info(
|
self.logger.info("[{source}.{scan_name}] No vulnerabilities or vulnerabilities not parsed.".format(source=source, scan_name=scan_name))
|
||||||
"[{source}.{scan_name}] No vulnerabilities or vulnerabilities not parsed.".format(source=source,
|
|
||||||
scan_name=scan_name))
|
|
||||||
self.set_latest_scan_reported(fullpath.split("/")[-1])
|
self.set_latest_scan_reported(fullpath.split("/")[-1])
|
||||||
return False
|
return False
|
||||||
|
except Exception as e:
|
||||||
|
self.logger.error("Error: {}".format(e))
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
#writing to file those assets without DNS resolution
|
#writing to file those assets without DNS resolution
|
||||||
#if its not empty
|
#if its not empty
|
||||||
@ -1310,12 +1282,12 @@ class vulnWhispererJIRA(vulnWhispererBase):
|
|||||||
self.jira_sync(self.config.get(scan, 'source'), self.config.get(scan, 'scan_name'))
|
self.jira_sync(self.config.get(scan, 'source'), self.config.get(scan, 'scan_name'))
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
self.logger.error(
|
self.logger.error(
|
||||||
"VulnWhisperer wasn't able to report the vulnerabilities from the '{}'s source".format(
|
"VulnWhisperer wasn't able to report the vulnerabilities from the '{}'s source, section {}.\
|
||||||
self.config.get(scan, 'source')))
|
\nError: {}".format(
|
||||||
|
self.config.get(scan, 'source'), self.config.get(scan, 'scan_name'), e))
|
||||||
return True
|
return True
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
|
||||||
class vulnWhisperer(object):
|
class vulnWhisperer(object):
|
||||||
|
|
||||||
def __init__(self,
|
def __init__(self,
|
||||||
@ -1339,6 +1311,7 @@ class vulnWhisperer(object):
|
|||||||
self.scanname = scanname
|
self.scanname = scanname
|
||||||
self.exit_code = 0
|
self.exit_code = 0
|
||||||
|
|
||||||
|
|
||||||
def whisper_vulnerabilities(self):
|
def whisper_vulnerabilities(self):
|
||||||
|
|
||||||
if self.profile == 'nessus':
|
if self.profile == 'nessus':
|
||||||
@ -1353,9 +1326,9 @@ class vulnWhisperer(object):
|
|||||||
self.exit_code += vw.process_web_assets()
|
self.exit_code += vw.process_web_assets()
|
||||||
|
|
||||||
elif self.profile == 'openvas':
|
elif self.profile == 'openvas':
|
||||||
vw_openvas = vulnWhispererOpenVAS(config=self.config)
|
vw = vulnWhispererOpenVAS(config=self.config)
|
||||||
if vw:
|
if vw:
|
||||||
self.exit_code += vw_openvas.process_openvas_scans()
|
self.exit_code += vw.process_openvas_scans()
|
||||||
|
|
||||||
elif self.profile == 'tenable':
|
elif self.profile == 'tenable':
|
||||||
vw = vulnWhispererNessus(config=self.config,
|
vw = vulnWhispererNessus(config=self.config,
|
||||||
|
Reference in New Issue
Block a user