Bump lxml from 4.6.5 to 4.9.1

Bumps [lxml](https://github.com/lxml/lxml) from 4.6.5 to 4.9.1.
- [Release notes](https://github.com/lxml/lxml/releases)
- [Changelog](https://github.com/lxml/lxml/blob/master/CHANGES.txt)
- [Commits](https://github.com/lxml/lxml/compare/lxml-4.6.5...lxml-4.9.1)

---
updated-dependencies:
- dependency-name: lxml
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Dockerfile

@@ -1,4 +1,4 @@
-FROM centos:latest
+FROM centos:7
 
 MAINTAINER Justin Henderson justin@hasecuritysolutions.com
 

README.md

@@ -6,8 +6,10 @@
 
 VulnWhisperer is a vulnerability management tool and report aggregator. VulnWhisperer will pull all the reports from the different Vulnerability scanners and create a file with a unique filename for each one, using that data later to sync with Jira and feed Logstash. Jira does a closed cycle full Sync with the data provided by the Scanners, while Logstash indexes and tags all of the information inside the report (see logstash files at /resources/elk6/pipeline/). Data is then shipped to ElasticSearch to be indexed, and ends up in a visual and searchable format in Kibana with already defined dashboards.
 
+VulnWhisperer is an open-source community funded project. VulnWhisperer currently works but is due for a documentation overhaul and code review. This is on the roadmap for the next month or two (February or March of 2022 - hopefully). Please note, crowd funding is an option. If you would like help getting VulnWhisperer up and running, are interested in new features, or are looking for paid support (for those of you that require commercial support contracts to implement open-source solutions), please reach out to **info@hasecuritysolutions.com**.
+
 [![Build Status](https://travis-ci.org/HASecuritySolutions/VulnWhisperer.svg?branch=master)](https://travis-ci.org/HASecuritySolutions/VulnWhisperer)
-[![MIT License](https://img.shields.io/badge/license-MIT-blue.svg?style=flat)](http://choosealicense.com/licenses/mit/)
+[![GitHub license](https://img.shields.io/github/license/HASecuritySolutions/VulnWhisperer)](https://github.com/HASecuritySolutions/VulnWhisperer/blob/master/LICENSE)
 [![Twitter](https://img.shields.io/twitter/follow/VulnWhisperer.svg?style=social&label=Follow)](https://twitter.com/VulnWhisperer)
 
 Currently Supports
@@ -30,7 +32,8 @@ Currently Supports
 
 ### Reporting Frameworks
 
-- [X] [ELK](https://www.elastic.co/elk-stack)
+- [X] [Elastic Stack (**v6**/**v7**)](https://www.elastic.co/elk-stack)
+- [ ] [OpenSearch - Being considered for next update](https://opensearch.org/)
 - [X] [Jira](https://www.atlassian.com/software/jira)
 - [ ] [Splunk](https://www.splunk.com/)
 

bin/vuln_whisperer

@@ -83,14 +83,17 @@ def main():
             enabled_sections = config.get_sections_with_attribute('enabled')
 
             for section in enabled_sections:
-                vw = vulnWhisperer(config=args.config,
-                                   profile=section,
-                                   verbose=args.verbose,
-                                   username=args.username,
-                                   password=args.password,
-                                   source=args.source,
-                                   scanname=args.scanname)
-                exit_code += vw.whisper_vulnerabilities()
+                try:
+                    vw = vulnWhisperer(config=args.config,
+                                       profile=section,
+                                       verbose=args.verbose,
+                                       username=args.username,
+                                       password=args.password,
+                                       source=args.source,
+                                       scanname=args.scanname)
+                    exit_code += vw.whisper_vulnerabilities()
+                except Exception as e:
+                    logger.error("VulnWhisperer was unable to perform the processing on '{}'".format(args.source))
         else:
             logger.info('Running vulnwhisperer for section {}'.format(args.section))
             vw = vulnWhisperer(config=args.config,

configs/frameworks_example.ini

@@ -2,6 +2,8 @@
 enabled=true
 hostname=localhost
 port=8834
+access_key=
+secret_key=
 username=nessus_username
 password=nessus_password
 write_path=/opt/VulnWhisperer/data/nessus/
@@ -13,6 +15,8 @@ verbose=true
 enabled=true
 hostname=cloud.tenable.com
 port=443
+access_key=
+secret_key=
 username=tenable.io_username
 password=tenable.io_password
 write_path=/opt/VulnWhisperer/data/tenable/
@@ -37,7 +41,7 @@ max_retries = 10
 template_id = 126024
 
 [qualys_vuln]
-#Reference https://www.qualys.com/docs/qualys-was-api-user-guide.pdf to find your API
+#Reference https://www.qualys.com/docs/qualys-api-vmpc-user-guide.pdf to find your API
 enabled = true
 hostname = qualysapi.qg2.apps.qualys.com
 username = exampleuser

configs/test.ini

@@ -2,6 +2,8 @@
 enabled=true
 hostname=nessus
 port=443
+access_key=
+secret_key=
 username=nessus_username
 password=nessus_password
 write_path=/opt/VulnWhisperer/data/nessus/
@@ -13,6 +15,8 @@ verbose=true
 enabled=true
 hostname=tenable
 port=443
+access_key=
+secret_key=
 username=tenable.io_username
 password=tenable.io_password
 write_path=/opt/VulnWhisperer/data/tenable/

requirements.txt

@@ -1,12 +1,12 @@
 pandas==0.20.3
 setuptools==40.4.3
 pytz==2017.2
-Requests==2.18.3
-lxml==4.1.1
+Requests==2.20.0
+lxml==4.9.1
 future-fstrings
 bs4
 jira
 bottle
 coloredlogs
-qualysapi>=5.1.0
-httpretty
+qualysapi==6.0.0
+httpretty

resources/elk5-old_compatibility/docker/3000_openvas.conf

@@ -2,7 +2,7 @@
 # Email: austin@hasecuritysolutions.com
 # Last Update: 03/04/2018
 # Version 0.3
-# Description: Take in qualys web scan reports from vulnWhisperer and pumps into logstash
+# Description: Take in Openvas web scan reports from vulnWhisperer and pumps into logstash
 
 input {
   file {

resources/elk6/logstash-vulnwhisperer-template_elk7.json

@@ -0,0 +1,231 @@
+{
+  "index_patterns": "logstash-vulnwhisperer-*",
+  "mappings": {
+    "properties": {
+      "@timestamp": {
+        "type": "date"
+      },
+      "@version": {
+        "type": "keyword"
+      },
+      "asset": {
+        "type": "text",
+        "norms": false,
+        "fields": {
+          "keyword": {
+            "type": "keyword",
+            "ignore_above": 256
+          }
+        }
+      },
+      "asset_uuid": {
+        "type": "keyword"
+      },
+      "assign_ip": {
+        "type": "ip"
+      },
+      "category": {
+        "type": "keyword"
+      },
+      "cve": {
+        "type": "keyword"
+      },
+      "cvss_base": {
+        "type": "float"
+      },
+      "cvss_temporal_vector": {
+        "type": "keyword"
+      },
+      "cvss_temporal": {
+        "type": "float"
+      },
+      "cvss_vector": {
+        "type": "keyword"
+      },
+      "cvss": {
+        "type": "float"
+      },
+      "cvss3_base": {
+        "type": "float"
+      },
+      "cvss3_temporal_vector": {
+        "type": "keyword"
+      },
+      "cvss3_temporal": {
+        "type": "float"
+      },
+      "cvss3_vector": {
+        "type": "keyword"
+      },
+      "cvss3": {
+        "type": "float"
+      },
+      "description": {
+        "fields": {
+          "keyword": {
+            "ignore_above": 256,
+            "type": "keyword"
+          }
+        },
+        "norms": false,
+        "type": "text"
+      },
+      "dns": {
+        "type": "keyword"
+      },
+      "exploitability": {
+        "fields": {
+          "keyword": {
+            "ignore_above": 256,
+            "type": "keyword"
+          }
+        },
+        "norms": false,
+        "type": "text"
+      },
+      "fqdn": {
+        "type": "keyword"
+      },
+      "geoip": {
+        "dynamic": true,
+        "type": "object",
+        "properties": {
+          "ip": {
+            "type": "ip"
+          },
+          "latitude": {
+            "type": "float"
+          },
+          "location": {
+            "type": "geo_point"
+          },
+          "longitude": {
+            "type": "float"
+          }
+        }
+      },
+      "history_id": {
+        "type": "keyword"
+      },
+      "host": {
+        "type": "keyword"
+      },
+      "host_end": {
+        "type": "date"
+      },
+      "host_start": {
+        "type": "date"
+      },
+      "impact": {
+        "fields": {
+          "keyword": {
+            "ignore_above": 256,
+            "type": "keyword"
+          }
+        },
+        "norms": false,
+        "type": "text"
+      },
+      "ip_status": {
+        "type": "keyword"
+      },
+      "ip": {
+        "type": "ip"
+      },
+      "last_updated": {
+        "type": "date"
+      },
+      "operating_system": {
+        "type": "keyword"
+      },
+      "path": {
+        "type": "keyword"
+      },
+      "pci_vuln": {
+        "type": "keyword"
+      },
+      "plugin_family": {
+        "type": "keyword"
+      },
+      "plugin_id": {
+        "type": "keyword"
+      },
+      "plugin_name": {
+        "type": "keyword"
+      },
+      "plugin_output": {
+        "fields": {
+          "keyword": {
+            "ignore_above": 256,
+            "type": "keyword"
+          }
+        },
+        "norms": false,
+        "type": "text"
+      },
+      "port": {
+        "type": "integer"
+      },
+      "protocol": {
+        "type": "keyword"
+      },
+      "results": {
+        "type": "text"
+      },
+      "risk_number": {
+        "type": "integer"
+      },
+      "risk_score_name": {
+        "type": "keyword"
+      },
+      "risk_score": {
+        "type": "float"
+      },
+      "risk": {
+        "type": "keyword"
+      },
+      "scan_id": {
+        "type": "keyword"
+      },
+      "scan_name": {
+        "type": "keyword"
+      },
+      "scan_reference": {
+        "type": "keyword"
+      },
+      "see_also": {
+        "type": "keyword"
+      },
+      "solution": {
+        "type": "keyword"
+      },
+      "source": {
+        "type": "keyword"
+      },
+      "ssl": {
+        "type": "keyword"
+      },
+      "synopsis": {
+        "type": "keyword"
+      },
+      "system_type": {
+        "type": "keyword"
+      },
+      "tags": {
+        "type": "keyword"
+      },
+      "threat": {
+        "type": "text"
+      },
+      "type": {
+        "type": "keyword"
+      },
+      "vendor_reference": {
+        "type": "keyword"
+      },
+      "vulnerability_state": {
+        "type": "keyword"
+      }
+    }
+  }
+}

vulnwhisp/base/config.py

@@ -31,7 +31,7 @@ class vwConfig(object):
         for section in self.config.sections():
             try:
                 if self.get(section, attribute) in check:
-                    sections.append(section)	
+                    sections.append(section)
             except:
                 self.logger.warn("Section {} has no option '{}'".format(section, attribute))
         return sections
@@ -45,7 +45,7 @@ class vwConfig(object):
         return True
 
     def update_jira_profiles(self, profiles):
-        # create JIRA profiles in the ini config file 
+        # create JIRA profiles in the ini config file
         self.logger.debug('Updating Jira profiles: {}'.format(str(profiles)))
 
         for profile in profiles:
@@ -67,7 +67,7 @@ class vwConfig(object):
                 self.config.set(section_name, 'min_critical_to_report', 'high')
                 self.config.set(section_name, '; automatically report, boolean value ')
                 self.config.set(section_name, 'autoreport', 'false')
-        
+
         # TODO: try/catch this
         # writing changes back to file
         with open(self.config_in, 'w') as configfile:

vulnwhisp/frameworks/nessus.py

@@ -24,15 +24,19 @@ class NessusAPI(object):
     EXPORT_STATUS = EXPORT + '/{file_id}/status'
     EXPORT_HISTORY = EXPORT + '?history_id={history_id}'
 
-    def __init__(self, hostname=None, port=None, username=None, password=None, verbose=True):
+    def __init__(self, hostname=None, port=None, username=None, password=None, verbose=True, profile=None, access_key=None, secret_key=None):
         self.logger = logging.getLogger('NessusAPI')
         if verbose:
             self.logger.setLevel(logging.DEBUG)
-        if username is None or password is None:
-            raise Exception('ERROR: Missing username or password.')
+        if not all((username, password)) and not all((access_key, secret_key)):
+            raise Exception('ERROR: Missing username, password or API keys.')
 
+        self.profile = profile
         self.user = username
         self.password = password
+        self.api_keys = False
+        self.access_key = access_key
+        self.secret_key = secret_key
         self.base = 'https://{hostname}:{port}'.format(hostname=hostname, port=port)
         self.verbose = verbose
 
@@ -52,7 +56,13 @@ class NessusAPI(object):
             'X-Cookie': None
         }
 
-        self.login()
+        if all((self.access_key, self.secret_key)):
+            self.logger.debug('Using {} API keys'.format(self.profile))
+            self.api_keys = True
+            self.session.headers['X-ApiKeys'] = 'accessKey={}; secretKey={}'.format(self.access_key, self.secret_key)
+        else:
+            self.login()
+
         self.scans = self.get_scans()
         self.scan_ids = self.get_scan_ids()
 
@@ -67,7 +77,7 @@ class NessusAPI(object):
     def request(self, url, data=None, headers=None, method='POST', download=False, json_output=False):
         timeout = 0
         success = False
-        
+
         method = method.lower()
         url = self.base + url
         self.logger.debug('Requesting to url {}'.format(url))
@@ -78,8 +88,10 @@ class NessusAPI(object):
                 if url == self.base + self.SESSION:
                     break
                 try:
-                    self.login()
                     timeout += 1
+                    if self.api_keys:
+                        continue
+                    self.login()
                     self.logger.info('Token refreshed')
                 except Exception as e:
                     self.logger.error('Could not refresh token\nReason: {}'.format(str(e)))
@@ -114,7 +126,7 @@ class NessusAPI(object):
         data = self.request(self.SCAN_ID.format(scan_id=scan_id), method='GET', json_output=True)
         return data['history']
 
-    def download_scan(self, scan_id=None, history=None, export_format="", profile=""):
+    def download_scan(self, scan_id=None, history=None, export_format=""):
         running = True
         counter = 0
 
@@ -127,7 +139,8 @@ class NessusAPI(object):
         req = self.request(query, data=json.dumps(data), method='POST', json_output=True)
         try:
             file_id = req['file']
-            token_id = req['token'] if 'token' in req else req['temp_token']
+            if self.profile == 'nessus':
+                token_id = req['token'] if 'token' in req else req['temp_token']
         except Exception as e:
             self.logger.error('{}'.format(str(e)))
         self.logger.info('Download for file id {}'.format(str(file_id)))
@@ -143,7 +156,7 @@ class NessusAPI(object):
             if counter % 60 == 0:
                 self.logger.info("Completed: {}".format(counter))
         self.logger.info("Done: {}".format(counter))
-        if profile == 'tenable':
+        if self.profile == 'tenable' or self.api_keys:
             content = self.request(self.EXPORT_FILE_DOWNLOAD.format(scan_id=scan_id, file_id=file_id), method='GET', download=True)
         else:
             content = self.request(self.EXPORT_TOKEN_DOWNLOAD.format(token_id=token_id), method='GET', download=True)
@@ -152,7 +165,7 @@ class NessusAPI(object):
     def get_utc_from_local(self, date_time, local_tz=None, epoch=True):
         date_time = datetime.fromtimestamp(date_time)
         if local_tz is None:
-            local_tz = pytz.timezone('US/Central')
+            local_tz = pytz.timezone('UTC')
         else:
             local_tz = pytz.timezone(local_tz)
         local_time = local_tz.normalize(local_tz.localize(date_time))

vulnwhisp/frameworks/qualys_web.py

@@ -428,7 +428,7 @@ class qualysScanReport:
 
         merged_df = merged_df.drop(['QID_y', 'QID_x'], axis=1)
         merged_df = merged_df.rename(columns={'Id': 'QID'})
-        
+
         merged_df = merged_df.assign(**df_dict['SCAN_META'].to_dict(orient='records')[0])
 
         merged_df = pd.merge(merged_df, df_dict['CATEGORY_HEADER'], how='left', left_on=['Category', 'Severity Level'],

vulnwhisp/reporting/jira_api.py

@@ -67,18 +67,23 @@ class JiraAPI(object):
             if not exists:
                 self.logger.error("Error creating Ticket: component {} not found".format(component))
                 return 0
-                    
-        new_issue = self.jira.create_issue(project=project,
-                                           summary=title,
-                                           description=desc,
-                                           issuetype={'name': 'Bug'},
-                                           labels=labels,
-                                           components=components_ticket)
         
-        self.logger.info("Ticket {} created successfully".format(new_issue))
+        try:
+            new_issue = self.jira.create_issue(project=project,
+                                               summary=title,
+                                               description=desc,
+                                               issuetype={'name': 'Bug'},
+                                               labels=labels,
+                                               components=components_ticket)
+            
+            self.logger.info("Ticket {} created successfully".format(new_issue))
+            
+            if attachment_contents:
+                self.add_content_as_attachment(new_issue, attachment_contents)
         
-        if attachment_contents:
-            self.add_content_as_attachment(new_issue, attachment_contents)
+        except Exception as e:
+            self.logger.error("Failed to create ticket on Jira Project '{}'. Error: {}".format(project, e))
+            new_issue = False
         
         return new_issue
     
@@ -226,32 +231,44 @@ class JiraAPI(object):
     def ticket_get_unique_fields(self, ticket):
         title = ticket.raw.get('fields', {}).get('summary').encode("ascii").strip()
         ticketid = ticket.key.encode("ascii")
-        assets = []
-        try:
-            affected_assets_section = ticket.raw.get('fields', {}).get('description').encode("ascii").split("{panel:title=Affected Assets}")[1].split("{panel}")[0]
-            assets = list(set(re.findall(r"\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b", affected_assets_section)))
-        
-        except Exception as e:
-            self.logger.error("Ticket IPs regex failed. Ticket ID: {}. Reason: {}".format(ticketid, e))
-            assets = []
-        
-        try:     
-            if not assets:
-                #check if attachment, if so, get assets from attachment
-                affected_assets_section = self.check_ips_attachment(ticket)
-                if affected_assets_section:
-                    assets = list(set(re.findall(r"\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b", affected_assets_section)))
-        except Exception as e:
-            self.logger.error("Ticket IPs Attachment regex failed. Ticket ID: {}. Reason: {}".format(ticketid, e))
+
+        assets = self.get_assets_from_description(ticket)
+        if not assets:
+            #check if attachment, if so, get assets from attachment
+            assets = self.get_assets_from_attachment(ticket)
                 
         return ticketid, title, assets
 
-    def check_ips_attachment(self, ticket):
-        affected_assets_section = []
+    def get_assets_from_description(self, ticket, _raw = False):
+        # Get the assets as a string "host - protocol/port - hostname" separated by "\n"
+        # structure the text to have the same structure as the assets from the attachment
+        affected_assets = ""
+        try:
+            affected_assets = ticket.raw.get('fields', {}).get('description').encode("ascii").split("{panel:title=Affected Assets}")[1].split("{panel}")[0].replace('\n','').replace(' * ','\n').replace('\n', '', 1)
+        except Exception as e:
+            self.logger.error("Unable to process the Ticket's 'Affected Assets'. Ticket ID: {}. Reason: {}".format(ticket, e))
+
+        if affected_assets:
+            if _raw:
+                # from line 406 check if the text in the panel corresponds to having added an attachment
+                if "added as an attachment" in affected_assets:
+                    return False
+                return affected_assets
+
+            try:
+                # if _raw is not true, we return only the IPs of the affected assets
+                return list(set(re.findall(r"\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b", affected_assets)))
+            except Exception as e:
+                self.logger.error("Ticket IPs regex failed. Ticket ID: {}. Reason: {}".format(ticket, e))
+        return False
+
+    def get_assets_from_attachment(self, ticket, _raw = False):
+        # Get the assets as a string "host - protocol/port - hostname" separated by "\n"
+        affected_assets = []
         try:
             fields = self.jira.issue(ticket.key).raw.get('fields', {})
             attachments = fields.get('attachment', {})
-            affected_assets_section = ""
+            affected_assets = ""
             #we will make sure we get the latest version of the file
             latest = ''
             attachment_id = ''
@@ -265,12 +282,44 @@ class JiraAPI(object):
                             if latest < item.get('created'):
                                 latest = item.get('created')         
                                 attachment_id = item.get('id') 
-            affected_assets_section = self.jira.attachment(attachment_id).get()
+            affected_assets = self.jira.attachment(attachment_id).get()
 
         except Exception as e:
             self.logger.error("Failed to get assets from ticket attachment. Ticket ID: {}. Reason: {}".format(ticket, e))
 
-        return affected_assets_section
+        if affected_assets:
+            if _raw:
+                return affected_assets
+
+            try:
+                # if _raw is not true, we return only the IPs of the affected assets
+                affected_assets = list(set(re.findall(r"\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b", affected_assets)))
+                return affected_assets
+            except Exception as e:
+                self.logger.error("Ticket IPs Attachment regex failed. Ticket ID: {}. Reason: {}".format(ticket, e))
+
+        return False
+
+    def parse_asset_to_json(self, asset):
+        hostname, protocol, port = "", "", ""
+        asset_info = asset.split(" - ")
+        ip = asset_info[0]
+        proto_port = asset_info[1]
+        # in case there is some case where hostname is not reported at all
+        if len(asset_info) == 3:
+            hostname = asset_info[2]
+        if proto_port != "N/A/N/A":
+            protocol, port = proto_port.split("/")
+            port = int(float(port))
+
+        asset_dict = {
+            "host": ip,
+            "protocol": protocol,
+            "port": port,
+            "hostname": hostname
+        }
+
+        return asset_dict
 
     def clean_old_attachments(self, ticket):
         fields = ticket.raw.get('fields')
@@ -441,7 +490,7 @@ class JiraAPI(object):
             if transition.get('name') == self.JIRA_REOPEN_ISSUE:
                 self.logger.debug("Ticket is reopenable")
                 return True
-        self.logger.warn("Ticket can't be opened. Check Jira transitions.")
+        self.logger.error("Ticket {} can't be opened. Check Jira transitions.".format(ticket_obj))
         return False
 
     def is_ticket_closeable(self, ticket_obj):
@@ -449,7 +498,7 @@ class JiraAPI(object):
         for transition in transitions:
             if transition.get('name') == self.JIRA_CLOSE_ISSUE:
                 return True
-        self.logger.warn("Ticket can't closed. Check Jira transitions.")
+        self.logger.error("Ticket {} can't closed. Check Jira transitions.".format(ticket_obj))
         return False
 
     def is_ticket_resolved(self, ticket_obj):
@@ -522,7 +571,7 @@ class JiraAPI(object):
     def close_obsolete_tickets(self):
         # Close tickets older than 12 months, vulnerabilities not solved will get created a new ticket 
         self.logger.info("Closing obsolete tickets older than {} months".format(self.max_time_tracking))
-        jql = "labels=vulnerability_management AND created <startOfMonth(-{}) and resolution=Unresolved".format(self.max_time_tracking)
+        jql = "labels=vulnerability_management AND NOT labels=advisory AND created <startOfMonth(-{}) and resolution=Unresolved".format(self.max_time_tracking)
         tickets_to_close = self.jira.search_issues(jql, maxResults=0)
         
         comment = '''This ticket is being closed for hygiene, as it is more than {} months old.
@@ -553,8 +602,35 @@ class JiraAPI(object):
             return True
         try:
             self.logger.info("Saving locally tickets from the last {} months".format(self.max_time_tracking))
-            jql = "labels=vulnerability_management AND created >=startOfMonth(-{})".format(self.max_time_tracking)
+            jql = "labels=vulnerability_management AND NOT labels=advisory AND created >=startOfMonth(-{})".format(self.max_time_tracking)
             tickets_data = self.jira.search_issues(jql, maxResults=0)
+
+            #TODO process tickets, creating a new field called "_metadata" with all the affected assets well structured
+            # for future processing in ELK/Splunk; this includes downloading attachments with assets and processing them
+
+            processed_tickets = []
+
+            for ticket in tickets_data:
+                assets = self.get_assets_from_description(ticket, _raw=True)
+                if not assets:
+                    # check if attachment, if so, get assets from attachment
+                    assets = self.get_assets_from_attachment(ticket, _raw=True)
+                # process the affected assets to save them as json structure on a new field from the JSON
+                _metadata = {"affected_hosts": []}
+                if assets:
+                    if "\n" in assets:
+                        for asset in assets.split("\n"):
+                            assets_json = self.parse_asset_to_json(asset)
+                            _metadata["affected_hosts"].append(assets_json)
+                    else:
+                        assets_json = self.parse_asset_to_json(assets)
+                        _metadata["affected_hosts"].append(assets_json)
+
+
+                temp_ticket = ticket.raw.get('fields')
+                temp_ticket['_metadata'] = _metadata
+
+                processed_tickets.append(temp_ticket)
             
             #end of line needed, as writelines() doesn't add it automatically, otherwise one big line
             to_save = [json.dumps(ticket.raw.get('fields'))+"\n" for ticket in tickets_data]
@@ -566,7 +642,7 @@ class JiraAPI(object):
 
         except Exception as e:
             self.logger.error("Tickets could not be saved locally: {}.".format(e))
-        
+
         return False 
 
     def decommission_cleanup(self):

vulnwhisp/test/mock.py

@@ -35,13 +35,13 @@ class mockAPI(object):
         elif 'fetch' in request.parsed_body['action']:
             try:
                 response_body = open('{}/{}'.format(
-                                      self.qualys_vuln_path, 
+                                      self.qualys_vuln_path,
                                       request.parsed_body['scan_ref'][0].replace('/', '_'))
                                     ).read()
             except:
                 # Can't find the file, just send an empty response
                 response_body = ''
-        return [200, response_headers, response_body] 
+        return [200, response_headers, response_body]
 
     def create_nessus_resource(self, framework):
         for filename in self.get_files('{}/{}'.format(self.mock_dir, framework)):
@@ -60,7 +60,7 @@ class mockAPI(object):
                 httpretty.GET,
                 'https://{}:443/{}'.format(framework, 'msp/about.php'),
                 body='')
-        
+
         self.logger.debug('Adding mocked {} endpoint {} {}'.format(framework, 'POST', 'api/2.0/fo/scan'))
         httpretty.register_uri(
             httpretty.POST, 'https://{}:443/{}'.format(framework, 'api/2.0/fo/scan/'),

vulnwhisp/vulnwhisp.py

@@ -55,8 +55,12 @@ class vulnWhispererBase(object):
             except:
                 self.enabled = False
             self.hostname = self.config.get(self.CONFIG_SECTION, 'hostname')
-            self.username = self.config.get(self.CONFIG_SECTION, 'username')
-            self.password = self.config.get(self.CONFIG_SECTION, 'password')
+            try:
+                self.username = self.config.get(self.CONFIG_SECTION, 'username')
+                self.password = self.config.get(self.CONFIG_SECTION, 'password')
+            except:
+                self.username = None
+                self.password = None
             self.write_path = self.config.get(self.CONFIG_SECTION, 'write_path')
             self.db_path = self.config.get(self.CONFIG_SECTION, 'db_path')
             self.verbose = self.config.getbool(self.CONFIG_SECTION, 'verbose')
@@ -144,7 +148,7 @@ class vulnWhispererBase(object):
 
     def record_insert(self, record):
         #for backwards compatibility with older versions without "reported" field
-        
+
         try:
             #-1 to get the latest column, 1 to get the column name (old version would be "processed", new "reported")
             #TODO delete backward compatibility check after some versions
@@ -171,7 +175,7 @@ class vulnWhispererBase(object):
             return True
         except Exception as e:
             self.logger.error('Failed while setting scan with file {} as processed'.format(filename))
-            
+
         return False
 
     def retrieve_uuids(self):
@@ -200,7 +204,8 @@ class vulnWhispererBase(object):
     def get_latest_results(self, source, scan_name):
         processed = 0
         results = []
-        
+        reported = ""
+
         try:
             self.conn.text_factory = str
             self.cur.execute('SELECT filename FROM scan_history WHERE source="{}" AND scan_name="{}" ORDER BY last_modified DESC LIMIT 1;'.format(source, scan_name))
@@ -218,11 +223,12 @@ class vulnWhispererBase(object):
 
         except Exception as e:
             self.logger.error("Error when getting latest results from {}.{} : {}".format(source, scan_name, e))
+
         return results, reported
-        
+
     def get_scan_profiles(self):
         # Returns a list of source.scan_name elements from the database
-        
+
         # we get the list of sources
         try:
             self.conn.text_factory = str
@@ -231,7 +237,7 @@ class vulnWhispererBase(object):
         except:
             sources = []
             self.logger.error("Process failed at executing 'SELECT DISTINCT source FROM scan_history;'")
-        
+
         results = []
 
         # we get the list of scans within each source
@@ -274,6 +280,8 @@ class vulnWhispererNessus(vulnWhispererBase):
 
         self.develop = True
         self.purge = purge
+        self.access_key = None
+        self.secret_key = None
 
         if config is not None:
             try:
@@ -283,24 +291,36 @@ class vulnWhispererNessus(vulnWhispererBase):
                                                         'trash')
 
                 try:
-                    self.logger.info('Attempting to connect to nessus...')
+                    self.access_key = self.config.get(self.CONFIG_SECTION,'access_key')
+                    self.secret_key = self.config.get(self.CONFIG_SECTION,'secret_key')
+                except:
+                    pass
+
+                try:
+                    self.logger.info('Attempting to connect to {}...'.format(self.CONFIG_SECTION))
                     self.nessus = \
                         NessusAPI(hostname=self.hostname,
                                   port=self.nessus_port,
                                   username=self.username,
-                                  password=self.password)
+                                  password=self.password,
+                                  profile=self.CONFIG_SECTION,
+                                  access_key=self.access_key,
+                                  secret_key=self.secret_key
+                                  )
                     self.nessus_connect = True
-                    self.logger.info('Connected to nessus on {host}:{port}'.format(host=self.hostname,
+                    self.logger.info('Connected to {} on {host}:{port}'.format(self.CONFIG_SECTION, host=self.hostname,
                                                                                    port=str(self.nessus_port)))
                 except Exception as e:
                     self.logger.error('Exception: {}'.format(str(e)))
                     raise Exception(
-                        'Could not connect to nessus -- Please verify your settings in {config} are correct and try again.\nReason: {e}'.format(
+                        'Could not connect to {} -- Please verify your settings in {config} are correct and try again.\nReason: {e}'.format(
+                            self.CONFIG_SECTION,
                             config=self.config.config_in,
                             e=e))
             except Exception as e:
                 self.logger.error('Could not properly load your config!\nReason: {e}'.format(e=e))
-                sys.exit(1)
+                return False
+                #sys.exit(1)
 
 
 
@@ -435,20 +455,21 @@ class vulnWhispererNessus(vulnWhispererBase):
                         try:
                             file_req = \
                                 self.nessus.download_scan(scan_id=scan_id, history=history_id,
-                                                        export_format='csv', profile=self.CONFIG_SECTION)
+                                                        export_format='csv')
                         except Exception as e:
                             self.logger.error('Could not download {} scan {}: {}'.format(self.CONFIG_SECTION, scan_id, str(e)))
                             self.exit_code += 1
                             continue
-                            
+
                         clean_csv = \
                             pd.read_csv(io.StringIO(file_req.decode('utf-8')))
                         if len(clean_csv) > 2:
                             self.logger.info('Processing {}/{} for scan: {}'.format(scan_count, len(scan_list), scan_name.encode('utf8')))
-                            columns_to_cleanse = ['CVSS','CVE','Description','Synopsis','Solution','See Also','Plugin Output']
+                            columns_to_cleanse = ['CVSS','CVE','Description','Synopsis','Solution','See Also','Plugin Output', 'MAC Address']
 
                             for col in columns_to_cleanse:
-                                clean_csv[col] = clean_csv[col].astype(str).apply(self.cleanser)
+                                if col in clean_csv:
+                                    clean_csv[col] = clean_csv[col].astype(str).apply(self.cleanser)
 
                             clean_csv.to_csv(relative_path_name, index=False)
                             record_meta = (
@@ -555,8 +576,11 @@ class vulnWhispererQualys(vulnWhispererBase):
         self.logger = logging.getLogger('vulnWhispererQualys')
         if debug:
             self.logger.setLevel(logging.DEBUG)
-        
-        self.qualys_scan = qualysScanReport(config=config)
+        try:
+            self.qualys_scan = qualysScanReport(config=config)
+        except Exception as e:
+            self.logger.error("Unable to establish connection with Qualys scanner. Reason: {}".format(e))
+            return False
         self.latest_scans = self.qualys_scan.qw.get_all_scans()
         self.directory_check()
         self.scans_to_process = None
@@ -642,8 +666,7 @@ class vulnWhispererQualys(vulnWhispererBase):
 
                     if cleanup:
                         self.logger.info('Removing report {} from Qualys Database'.format(generated_report_id))
-                        cleaning_up = \
-                            self.qualys_scan.qw.delete_report(generated_report_id)
+                        cleaning_up = self.qualys_scan.qw.delete_report(generated_report_id)
                         os.remove(self.path_check(str(generated_report_id) + '.csv'))
                         self.logger.info('Deleted report from local disk: {}'.format(self.path_check(str(generated_report_id))))
                 else:
@@ -728,10 +751,14 @@ class vulnWhispererOpenVAS(vulnWhispererBase):
         self.develop = True
         self.purge = purge
         self.scans_to_process = None
-        self.openvas_api = OpenVAS_API(hostname=self.hostname,
-                                       port=self.port,
-                                       username=self.username,
-                                       password=self.password)
+        try:
+            self.openvas_api = OpenVAS_API(hostname=self.hostname,
+                                           port=self.port,
+                                           username=self.username,
+                                           password=self.password)
+        except Exception as e:
+            self.logger.error("Unable to establish connection with OpenVAS scanner. Reason: {}".format(e))
+            return False
 
     def whisper_reports(self, output_format='json', launched_date=None, report_id=None, cleanup=True):
         report = None
@@ -837,13 +864,16 @@ class vulnWhispererQualysVuln(vulnWhispererBase):
             username=None,
             password=None,
         ):
-        
+
         super(vulnWhispererQualysVuln, self).__init__(config=config)
         self.logger = logging.getLogger('vulnWhispererQualysVuln')
         if debug:
             self.logger.setLevel(logging.DEBUG)
-
-        self.qualys_scan = qualysVulnScan(config=config)
+        try:
+            self.qualys_scan = qualysVulnScan(config=config)
+        except Exception as e:
+            self.logger.error("Unable to create connection with Qualys. Reason: {}".format(e))
+            return False
         self.directory_check()
         self.scans_to_process = None
 
@@ -854,7 +884,7 @@ class vulnWhispererQualysVuln(vulnWhispererBase):
                         scan_reference=None,
                         output_format='json',
                         cleanup=True):
-            launched_date
+
             if 'Z' in launched_date:
                 launched_date = self.qualys_scan.utils.iso_to_epoch(launched_date)
             report_name = 'qualys_vuln_' + report_id.replace('/','_') \
@@ -966,8 +996,15 @@ class vulnWhispererJIRA(vulnWhispererBase):
         self.config_path = config
         self.config = vwConfig(config)
         self.host_resolv_cache = {}
-        self.directory_check()   
-                 
+        self.host_no_resolv = []
+        self.no_resolv_by_team_dict = {}
+        #Save locally those assets without DNS entry for flag to system owners
+        self.no_resolv_fname="no_resolv.txt"
+        if os.path.isfile(self.no_resolv_fname):
+            with open(self.no_resolv_fname, "r") as json_file:
+                self.no_resolv_by_team_dict = json.load(json_file)
+        self.directory_check()
+
         if config is not None:
             try:
                 self.logger.info('Attempting to connect to jira...')
@@ -983,17 +1020,18 @@ class vulnWhispererJIRA(vulnWhispererBase):
                 raise Exception(
                     'Could not connect to nessus -- Please verify your settings in {config} are correct and try again.\nReason: {e}'.format(
                         config=self.config.config_in, e=e))
-                sys.exit(1)
-   
+                return False
+                #sys.exit(1)
+
         profiles = []
         profiles = self.get_scan_profiles()
-        
+
         if not self.config.exists_jira_profiles(profiles):
             self.config.update_jira_profiles(profiles)
             self.logger.info("Jira profiles have been created in {config}, please fill the variables before rerunning the module.".format(config=self.config_path))
             sys.exit(0)
-    
-   
+
+
     def get_env_variables(self, source, scan_name):
         # function returns an array with [jira_project, jira_components, datafile_path]
 
@@ -1004,32 +1042,32 @@ class vulnWhispererJIRA(vulnWhispererBase):
         if project == "":
             self.logger.error('JIRA project is missing on the configuration file!')
             sys.exit(0)
-        
+
         # check that project actually exists
         if not self.jira.project_exists(project):
             self.logger.error("JIRA project '{project}' doesn't exist!".format(project=project))
             sys.exit(0)
-         
+
         components = self.config.get(jira_section,'components').split(',')
-        
+
         #cleaning empty array from ''
         if not components[0]:
             components = []
-        
+
         min_critical = self.config.get(jira_section,'min_critical_to_report')
         if not min_critical:
             self.logger.error('"min_critical_to_report" variable on config file is empty.')
             sys.exit(0)
-            
+
         #datafile path
         filename, reported = self.get_latest_results(source, scan_name)
         fullpath = ""
-        
+
         # search data files under user specified directory
         for root, dirnames, filenames in os.walk(vwConfig(self.config_path).get(source,'write_path')):
             if filename in filenames:
                 fullpath = "{}/{}".format(root,filename)
-        
+
         if reported:
             self.logger.warn('Last Scan of "{scan_name}" for source "{source}" has already been reported; will be skipped.'.format(scan_name=scan_name, source=source))
             return [False] * 5
@@ -1037,7 +1075,7 @@ class vulnWhispererJIRA(vulnWhispererBase):
         if not fullpath:
             self.logger.error('Scan of "{scan_name}" for source "{source}" has not been found. Please check that the scanner data files are in place.'.format(scan_name=scan_name, source=source))
             sys.exit(1)
-    
+
         dns_resolv = self.config.get('jira','dns_resolv')
         if dns_resolv in ('False', 'false', ''):
             dns_resolv = False
@@ -1051,22 +1089,22 @@ class vulnWhispererJIRA(vulnWhispererBase):
 
 
     def parse_nessus_vulnerabilities(self, fullpath, source, scan_name, min_critical):
-        
+
         vulnerabilities = []
 
         # we need to parse the CSV
-        risks = ['none', 'low', 'medium', 'high', 'critical'] 
+        risks = ['none', 'low', 'medium', 'high', 'critical']
         min_risk = int([i for i,x in enumerate(risks) if x == min_critical][0])
 
         df = pd.read_csv(fullpath, delimiter=',')
-        
+
         #nessus fields we want - ['Host','Protocol','Port', 'Name', 'Synopsis', 'Description', 'Solution', 'See Also']
         for index in range(len(df)):
             # filtering vulnerabilities by criticality, discarding low risk
             to_report = int([i for i,x in enumerate(risks) if x == df.loc[index]['Risk'].lower()][0])
             if to_report < min_risk:
                 continue
-            
+
             if not vulnerabilities or df.loc[index]['Name'] not in [entry['title'] for entry in vulnerabilities]:
                 vuln = {}
                 #vulnerabilities should have all the info for creating all JIRA labels
@@ -1080,7 +1118,7 @@ class vulnWhispererJIRA(vulnWhispererBase):
                 vuln['ips'] = []
                 vuln['ips'].append("{} - {}/{}".format(df.loc[index]['Host'], df.loc[index]['Protocol'], df.loc[index]['Port']))
                 vuln['risk'] = df.loc[index]['Risk'].lower()
-                
+
                 # Nessus "nan" value gets automatically casted to float by python
                 if not (type(df.loc[index]['See Also']) is float):
                     vuln['references'] = df.loc[index]['See Also'].split("\\n")
@@ -1093,24 +1131,24 @@ class vulnWhispererJIRA(vulnWhispererBase):
                 for vuln in vulnerabilities:
                     if vuln['title'] == df.loc[index]['Name']:
                         vuln['ips'].append("{} - {}/{}".format(df.loc[index]['Host'], df.loc[index]['Protocol'], df.loc[index]['Port']))
-        
+
         return vulnerabilities
-    
+
     def parse_qualys_vuln_vulnerabilities(self, fullpath, source, scan_name, min_critical, dns_resolv = False):
         #parsing of the qualys vulnerabilities schema
         #parse json
         vulnerabilities = []
 
-        risks = ['info', 'low', 'medium', 'high', 'critical'] 
+        risks = ['info', 'low', 'medium', 'high', 'critical']
         # +1 as array is 0-4, but score is 1-5
         min_risk = int([i for i,x in enumerate(risks) if x == min_critical][0])+1
-        
+
         try:
-            data=[json.loads(line) for line in open(fullpath).readlines()] 
+            data=[json.loads(line) for line in open(fullpath).readlines()]
         except Exception as e:
             self.logger.warn("Scan has no vulnerabilities, skipping.")
             return vulnerabilities
-       
+
         #qualys fields we want - []
         for index in range(len(data)):
             if int(data[index]['risk']) < min_risk:
@@ -1119,7 +1157,7 @@ class vulnWhispererJIRA(vulnWhispererBase):
             elif data[index]['type'] == 'Practice' or data[index]['type'] == 'Ig':
                 self.logger.debug("Vulnerability '{vuln}' ignored, as it is 'Practice/Potential', not verified.".format(vuln=data[index]['plugin_name']))
                 continue
-            
+
             if not vulnerabilities or data[index]['plugin_name'] not in [entry['title'] for entry in vulnerabilities]:
                 vuln = {}
                 #vulnerabilities should have all the info for creating all JIRA labels
@@ -1132,12 +1170,12 @@ class vulnWhispererJIRA(vulnWhispererBase):
                 vuln['solution'] = data[index]['solution'].replace('\\n',' ')
                 vuln['ips'] = []
                 #TODO ADDED DNS RESOLUTION FROM QUALYS! \n SEPARATORS INSTEAD OF \\n!
-                
+
                 vuln['ips'].append("{ip} - {protocol}/{port} - {dns}".format(**self.get_asset_fields(data[index], dns_resolv)))
 
                 #different risk system than Nessus!
                 vuln['risk'] = risks[int(data[index]['risk'])-1]
-                
+
                 # Nessus "nan" value gets automatically casted to float by python
                 if not (type(data[index]['vendor_reference']) is float or data[index]['vendor_reference'] == None):
                     vuln['references'] = data[index]['vendor_reference'].split("\\n")
@@ -1155,8 +1193,8 @@ class vulnWhispererJIRA(vulnWhispererBase):
     def get_asset_fields(self, vuln, dns_resolv):
         values = {}
         values['ip'] = vuln['ip']
-        values['protocol'] = vuln['protocol'] 
-        values['port'] = vuln['port'] 
+        values['protocol'] = vuln['protocol']
+        values['port'] = vuln['port']
         values['dns'] = ''
         if dns_resolv:
             if vuln['dns']:
@@ -1173,6 +1211,7 @@ class vulnWhispererJIRA(vulnWhispererBase):
                         self.logger.debug("Hostname found: {hostname}.".format(hostname=values['dns']))
                     except:
                         self.host_resolv_cache[values['ip']] = ''
+                        self.host_no_resolv.append(values['ip'])
                         self.logger.debug("Hostname not found for: {ip}.".format(ip=values['ip']))
 
         for key in values.keys():
@@ -1206,18 +1245,31 @@ class vulnWhispererJIRA(vulnWhispererBase):
         #***Qualys VM parsing***
         if source == "qualys_vuln":
             vulnerabilities = self.parse_qualys_vuln_vulnerabilities(fullpath, source, scan_name, min_critical, dns_resolv)
-        
+
         #***JIRA sync***
-        if vulnerabilities:
-            self.logger.info('{source} data has been successfuly parsed'.format(source=source.upper()))
-            self.logger.info('Starting JIRA sync')
-            
-            self.jira.sync(vulnerabilities, project, components)
-        else:
-            self.logger.info("[{source}.{scan_name}] No vulnerabilities or vulnerabilities not parsed.".format(source=source, scan_name=scan_name))
-            self.set_latest_scan_reported(fullpath.split("/")[-1])
+        try:
+            if vulnerabilities:
+                self.logger.info('{source} data has been successfuly parsed'.format(source=source.upper()))
+                self.logger.info('Starting JIRA sync')
+
+                self.jira.sync(vulnerabilities, project, components)
+            else:
+                self.logger.info("[{source}.{scan_name}] No vulnerabilities or vulnerabilities not parsed.".format(source=source, scan_name=scan_name))
+                self.set_latest_scan_reported(fullpath.split("/")[-1])
+                return False
+        except Exception as e:
+            self.logger.error("Error: {}".format(e))
             return False
 
+
+        #writing to file those assets without DNS resolution
+        #if its not empty
+        if self.host_no_resolv:
+            #we will replace old list of non resolved for the new one or create if it doesn't exist already
+            self.no_resolv_by_team_dict[scan_name] = self.host_no_resolv
+            with open(self.no_resolv_fname, 'w') as outfile:  
+                    json.dump(self.no_resolv_by_team_dict, outfile)
+        
         self.set_latest_scan_reported(fullpath.split("/")[-1])
         return True
 
@@ -1226,7 +1278,13 @@ class vulnWhispererJIRA(vulnWhispererBase):
 
         if autoreport_sections:
             for scan in autoreport_sections:
-                self.jira_sync(self.config.get(scan, 'source'), self.config.get(scan, 'scan_name'))
+                try:
+                    self.jira_sync(self.config.get(scan, 'source'), self.config.get(scan, 'scan_name'))
+                except Exception as e:
+                    self.logger.error(
+                        "VulnWhisperer wasn't able to report the vulnerabilities from the '{}'s source, section {}.\
+                         \nError: {}".format(
+                            self.config.get(scan, 'source'), self.config.get(scan, 'scan_name'), e))
             return True
         return False
 
@@ -1258,43 +1316,43 @@ class vulnWhisperer(object):
 
         if self.profile == 'nessus':
             vw = vulnWhispererNessus(config=self.config,
-                                     username=self.username,
-                                     password=self.password,
-                                     verbose=self.verbose,
                                      profile=self.profile)
-            self.exit_code += vw.whisper_nessus()
+            if vw:
+                self.exit_code += vw.whisper_nessus()
 
         elif self.profile == 'qualys_web':
             vw = vulnWhispererQualys(config=self.config)
-            self.exit_code += vw.process_web_assets()
+            if vw:
+                self.exit_code += vw.process_web_assets()
 
         elif self.profile == 'openvas':
-            vw_openvas = vulnWhispererOpenVAS(config=self.config)
-            self.exit_code += vw_openvas.process_openvas_scans()
+            vw = vulnWhispererOpenVAS(config=self.config)
+            if vw:
+                self.exit_code += vw.process_openvas_scans()
 
         elif self.profile == 'tenable':
             vw = vulnWhispererNessus(config=self.config,
-                                     username=self.username,
-                                     password=self.password,
-                                     verbose=self.verbose,
                                      profile=self.profile)
-            self.exit_code += vw.whisper_nessus()
+            if vw:
+                self.exit_code += vw.whisper_nessus()
 
         elif self.profile == 'qualys_vuln':
             vw = vulnWhispererQualysVuln(config=self.config)
-            self.exit_code += vw.process_vuln_scans()
-        
+            if vw:
+                self.exit_code += vw.process_vuln_scans()
+
         elif self.profile == 'jira':
             #first we check config fields are created, otherwise we create them
             vw = vulnWhispererJIRA(config=self.config)
-            if not (self.source and self.scanname):
-                self.logger.info('No source/scan_name selected, all enabled scans will be synced')
-                success = vw.sync_all()
-                if not success:
-                    self.logger.error('All scans sync failed!')
-                    self.logger.error('Source scanner and scan name needed!')
-                    return 0
-            else:
-                vw.jira_sync(self.source, self.scanname)
+            if vw:
+                if not (self.source and self.scanname):
+                    self.logger.info('No source/scan_name selected, all enabled scans will be synced')
+                    success = vw.sync_all()
+                    if not success:
+                        self.logger.error('All scans sync failed!')
+                        self.logger.error('Source scanner and scan name needed!')
+                        return 0
+                else:
+                    vw.jira_sync(self.source, self.scanname)
 
         return self.exit_code