diff --git a/README.md b/README.md index 0315a55..0973e56 100644 --- a/README.md +++ b/README.md @@ -4,8 +4,7 @@

-VulnWhisperer is a vulnerability data and report aggregator. VulnWhisperer will pull all the reports - and create a file with a unique filename which is then fed into logstash. Logstash extracts data from the filename and tags all of the information inside the report (see logstash_vulnwhisp.conf file). Data is then shipped to elasticsearch to be indexed. +VulnWhisperer is a vulnerability management tool and report aggregator. VulnWhisperer will pull all the reports from the different Vulnerability scanners and create a file with a unique filename for each one, using that data later to sync with Jira and feed Logstash. Jira does a closed cycle full Sync with the data provided by the Scanenrs, while Logstash indexes and tags all of the information inside the report (see logstash files at /resources/elk6/pipeline/). Data is then shipped to ElasticSearch to be indexed, and ends up in a visual and searchable format in Kibana with already defined dashboards. [![Build Status](https://travis-ci.org/HASecuritySolutions/VulnWhisperer.svg?branch=master)](https://travis-ci.org/HASecuritySolutions/VulnWhisperer) [![MIT License](https://img.shields.io/badge/license-MIT-blue.svg?style=flat)](http://choosealicense.com/licenses/mit/) @@ -39,9 +38,10 @@ Getting Started =============== 1) Follow the [install requirements](#installreq) -2) Fill out the section you want to process in example.ini file -3) Modify the IP settings in the logstash files to accomodate your environment and import them to your logstash conf directory (default is /etc/logstash/conf.d/) -4) Import the kibana visualizations +2) Fill out the section you want to process in frameworks_example.ini file +3) [JIRA] If using Jira, fill Jira config in the config file mentioned above. +3) [ELK] Modify the IP settings in the Logstash files to accommodate your environment and import them to your logstash conf directory (default is /etc/logstash/conf.d/) +4) [ELK] Import the Kibana visualizations 5) [Run Vulnwhisperer](#run) Need assistance or just want to chat? Join our [slack channel](https://join.slack.com/t/vulnwhisperer/shared_invite/enQtNDQ5MzE4OTIyODU0LWQxZTcxYTY0MWUwYzA4MTlmMWZlYWY2Y2ZmM2EzNDFmNWVlOTM4MzNjYzI0YzdkMDA0YmQyYWRhZGI2NGUxNGI) @@ -49,20 +49,27 @@ Need assistance or just want to chat? Join our [slack channel](https://join.slac Requirements ------------- #### -* ElasticStack 5.x * Python 2.7 * Vulnerability Scanner -* Optional: Message broker such as Kafka or RabbitMQ +* Reporting System: Jira / ElasticStack 6.6 Install Requirements-VulnWhisperer(may require sudo) -------------------- -**First install requirement dependencies** +**Install OS packages requirement dependencies** (Debian-based distros, CentOS don't need it) ```shell sudo apt-get install zlib1g-dev libxml2-dev libxslt1-dev ``` -**Then install requirements** +**(Optional) Use a python virtualenv to not mess with host python libraries** +```shell +virtualenv venv (will create the python 2.7 virtualenv) +source venv/bin/activate (start the virtualenv, now pip will run there and should install libraries without sudo) + +deactivate (for quitting the virtualenv once you are done) +``` + +**Install python libraries requirements** ```python pip install -r /path/to/VulnWhisperer/requirements.txt @@ -72,89 +79,6 @@ python setup.py install Now you're ready to pull down scans. (see run section) - -Install Requirements-ELK Node **\*SAMPLE\*** --------------------- -The following instructions should be utilized as a **Sample Guide** in the absence of an existing ELK Cluster/Node. This will cover a Debian example install guide of a stand-alone node of Elasticsearch & Kibana. - -While Logstash is included in this install guide, it it recommended that a seperate host pulling the VulnWhisperer data is utilized with Logstash to ship the data to the Elasticsearch node. - -*Please note there is a docker-compose.yml available as well.* - -**Debian:** *(https://www.elastic.co/guide/en/elasticsearch/reference/5.6/deb.html)* -```shell -sudo apt-get install -y default-jre -wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - -sudo apt-get install apt-transport-https -echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list -sudo apt-get update && sudo apt-get install elasticsearch kibana logstash -sudo /bin/systemctl daemon-reload -sudo /bin/systemctl enable elasticsearch.service -sudo /bin/systemctl enable kibana.service -sudo /bin/systemctl enable logstash.service -``` - -**Elasticsearch & Kibana Sample Config Notes** - -Utilizing your favorite text editor: -* Grab your host IP and change the IP of your /etc/elasticsearch/elasticsearch.yml file. (This defaults to 'localhost') -* Validate Elasticsearch is set to run on port 9200 (Default) -* Grab your host IP and change the IP of your /etc/kibana/kibana.yml file. (This defaults to 'localhost') *Validate that Kibana is pointing to the correct Elasticsearch IP (This was set in the previous step)* -* Validate Kibana is set to run on port 5601 (Default) - -*Start elasticsearch and validate they are running/communicating with one another:* -```shell -sudo service elasticsearch start -sudo service kibana start -``` -OR -```shell -sudo systemctl start elasticsearch.service -sudo systemctl start kibana.service -``` - -**Logstash Sample Config Notes** - -* Copy/Move the Logstash .conf files from */VulnWhisperer/logstash/* to */etc/logstash/conf.d/* -* Validate the Logstash.conf files *input* contains the correct location of VulnWhisper Scans in the *input.file.path* directory identified below: -``` -input { - file { - path => "/opt/vulnwhisperer/nessus/**/*" - start_position => "beginning" - tags => "nessus" - type => "nessus" - } -} -``` -* Validate the Logstash.conf files *output* contains the correct Elasticsearch IP set during the previous step above: (This will default to localhost) -``` -output { - if "nessus" in [tags] or [type] == "nessus" { - #stdout { codec => rubydebug } - elasticsearch { - hosts => [ "localhost:9200" ] - index => "logstash-vulnwhisperer-%{+YYYY.MM}" - } - } -``` -* Validate logstash has the correct file permissions to read the location of the VulnWhisperer Scans - -Once configured run Logstash: (Running Logstash as a service will pick up all the files in */etc/logstash/conf.d/* If you would like to run only one logstash file please reference the command below): - -Logstash as a service: -```shell -sudo service logstash start -``` -*OR* -```shell -sudo systemctl start logstash.service -``` -Single Logstash file: -```shell -sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/1000_nessus_process_file.conf -``` - Configuration ----- @@ -178,64 +102,43 @@ or vuln_whisperer -c configs/frameworks_example.ini -s qualys ``` -If no section is specified (e.g. -s nessus), vulnwhisperer will check on the config file for the modules that have the property enabled=true and run them sequentially. +If no section is specified (e.g. -s nessus), vulnwhisperer will check on the config file for the modules that have the property `enabled=true` and run them sequentially.

-Next you'll need to import the visualizations into Kibana and setup your logstash config. A more thorough README is underway with setup instructions. +Next you'll need to import the visualizations into Kibana and setup your logstash config. You can either follow the sample setup instructions [here](https://github.com/HASecuritySolutions/VulnWhisperer/wiki/Sample-Guide-ELK-Deployment) or go for the `docker-compose` solution we offer. Docker-compose ----- -The docker-compose file has been tested and running on a Ubuntu 18.04 environment, with docker-ce v.18.06. The structure's purpose is to store locally the data from the scanners, letting vulnwhisperer update the records and Logstash feed them to ElasticSearch, so it requires a local storage folder. +ELK is a whole world by itself, and for newcomers to the platform, it requires basic Linux skills and usually a bit of troubleshooting until it is deployed and working as expected. As we are not able to provide support for each users ELK problems, we put together a docker-compose which includes: -- It will run out of the box if you create on the root directory of VulnWhisperer a folder named "data", which needs permissions for other users to read/write/execute in order to sync: -```shell - mkdir data && chmod -R 666 data #data/database/report_tracker.db will need 777 to use with local vulnwhisperer -``` -otherwise the users running inside the docker containers will not be able to work with it properly. If you don't apply chmod recursively, it will still work to sync the data, but only root use in localhost will have access to the created data (if you run local vulnwhisperer with the same data will break). -- docker/logstash.yml file will need other read/write permissions in order for logstash container to use the configuration file; youll need to run: -```shell -chmod 666 docker/logstash.yml +- VulnWhisperer +- Logstash 6.6 +- ElasticSearch 6.6 +- Kibana 6.6 -- The vulnwhisperer container inside of docker-compose is using network_mode=host instead of the bridge mode by default; this is due to issues encountered when the container is trying to pull data from your scanners from a different VLAN than the one you currently are. The host network mode uses the DNS and interface from the host itself, fixing those issues, but it breaks the network isolation from the container (this is due to docker creating bridge interfaces to route the traffic, blocking both container's and host's network). If you change this to bridge, you might need to add your DNS to the config in order to resolve internal hostnames. -- ElasticSearch requires having the value vm.max_map_count with a minimum of 262144; otherwise, it will probably break at launch. Please check https://elk-docker.readthedocs.io/#prerequisites to solve that. -- If you want to change the "data" folder for storing the results, remember to change it from both the docker-compose.yml file and the logstash files that are in the root "docker/" folder. -- Hostnames do NOT allow _ (underscores) on it, if you change the hostname configuration from the docker-compose file and add underscores, config files from logstash will fail. -- If you are having issues with the connection between hosts, to troubleshoot them you can spawn a shell in said host doing the following: -```shell -docker ps #check the container is running -docker exec -i -t vulnwhisp-vulnwhisperer /bin/bash #where vulnwhisp-vulnwhisperer is the container name you want to troubleshoot -``` -You can also make sure that all ELK components are working by doing "curl -i host:9200 (elastic)/ host:5601 (kibana) /host:9600 (logstash). WARNING! It is possible that logstash is not exposing to the external network the port but it does to its internal docker network "esnet". -- If Kibana is not showing the results, check that you are searching on the whole ES range, as by default it shows logs for the last 15 minutes (you can choose up to last 5 years) -- X-Pack has been disabled by default due to the noise, plus being a trial version. You can enable it modifying the docker-compose.yml and docker/logstash.conf files. Logstash.conf contains the default credentials for the X-Pack enabled ES. -- On Logstash container, "/usr/share/logstash/pipeline/" is the default path for pipelines and "/usr/share/logstash/config/" for logstash.yml file, instead of "/etc/logstash/conf.d/" and "/etc/logstash/". -- In order to make vulnwhisperer run periodically, and only the vulnwhisperer code, add to crontab the following: +The docker-compose just requires specifying the paths where the VulnWhisperer data will be saved, and where the config files reside. If ran directly after `git clone`, with just adding the Scanner config to the VulnWhisperer config file (/resources/elk6/vulnwhisperer.ini), it will work out of the box. +It also takes care to load the Kibana Dashboards and Visualizations automatically through the API, which needs to be done manually otherwise at Kibana's startup. -```shell -0 8 * * * /usr/bin/docker-compose up vulnwhisp-vulnwhisperer -``` +For more info about the docker-compose, check on the [docker-compose wiki](https://github.com/HASecuritySolutions/VulnWhisperer/wiki/docker-compose-Instructions) or the [FAQ](https://github.com/HASecuritySolutions/VulnWhisperer/wiki). -To launch docker-compose, do: -```shell -docker-compose -f docker-compose.yml up -``` +Getting Started +=============== +Our current Roadmap is as follows: +- [ ] Create a Vulnerability Standard +- [ ] Map every scanner results to the standard +- [ ] Create Scanner module guidelines for easy integration of new scanners (consistency will allow #14) +- [ ] Refactor the code to reuse functions and enable full compatibility among modules +- [ ] Change Nessus CSV to JSON (Consistency and Fix #82) +- [ ] Adapt single Logstash to standard and Kibana Dashboards +- [ ] Implement Detectify Scanner +- [ ] Implement Splunk Reporting/Dashboards -Running Nightly ---------------- -If you're running linux, be sure to setup a cronjob to remove old files that get stored in the database. Be sure to change .csv if you're using json. +On top of this, we try to focus on fixing bugs as soon as possible, which might delay the development. We also very welcome PR's, and once we have the new standard implemented, it will be very easy to add compatibility with new scanners. -Setup crontab -e with the following config (modify to your environment) - this will run vulnwhisperer each night at 0130: - -`00 1 * * * /usr/bin/find /opt/vulnwhisp/ -type f -name '*.csv' -ctime +3 -exec rm {} \;` - -`30 1 * * * /usr/local/bin/vuln_whisperer -c /opt/vulnwhisp/configs/example.ini` - -Another option is to tell logstash to delete files after they have been processed. - -_For windows, you may need to type the full path of the binary in vulnWhisperer located in the bin directory._ +The Vulnerability Standard will initially be a new simple one level JSON with all the information that matches from the different scanners having standardized variable names, while maintaining the rest of the variables as they are. In the future, once everything is implemented, we will evaluate moving to an existing standard like ECS or AWS Vulnerability Schema; we prioritize functionality over perfection. Video Walkthrough -- Featured on ElasticWebinar ---------------------------------------------- @@ -250,9 +153,9 @@ Authors Contributors ------------ - - [@pemontto](https://github.com/pemontto) - [Quim Montal (@qmontal)](https://github.com/qmontal) - - [Andrea Lusuardi (@uovobw)](https://github.com/uovobw) + - [@pemontto](https://github.com/pemontto) + - [@cybergoof](https://github.com/cybergoof) AS SEEN ON TV -------------