×
Recorded Future
Recorded Future

Overview

The Security Flow Recorded Future Plugin adds the ability to gather threat intelligence on IoCs (URLs, domains, IP addresses, file hashes, and vulnerabilities), trigger flows based on alerts, and update alerts. The Plugin provides Incident enrichment with threat intelligence using either cached risk list data or up-to-date information from the Recorded Future API. Additionally, cached IoC or CVE risk lists are searchable.

Functionality

The Security Flow Recorded Future plugin provides the ability to

  • gather threat intelligence on domains, hashes, IP addresses, URLs, and CVEs
  • search cached risk lists from matching IoCs
  • trigger flows periodically based on searching for alerts
  • lookup alert details
  • update alerts

When risk lists are enabled, changes to a plugin instance’s risk list configuration will trigger a pull. The following rules apply for risk list updates:

  • Changing any setting will not interrupt an existing risk list pull.
  • Enabling the risk list pull will trigger the queuing of a pull request.
  • Changing the risk list name will trigger the queuing of a pull request.
  • The risk list cache entries are updated based on the following the selected refresh cycle. The defaults are shown here.
    • Domains – Every Two (2) hours
    • Hashes – Once a day
    • IP Addresses – Hourly
    • URLs – Every Two (2) hours
    • Vulnerabilities – Once a day
  • The risk list cache entries expire, i.e. are removed from the cache, based on aging out according to the refresh cycles.
  • Until cached, a risk list check will fail to find the threat intelligence.
  • The risk list cache is periodically checked once every five (5) minutes.

Instance Configuration Parameters

Property
Description

Instance Name

Name for the Recorded Future instance.


Unique ID

A system-wide unique identifier for this plugin instance used to locate the service.


API Key

API key used to access the Recorded Future services.


Fallback to Connect API if not Found in Risk List

Boolean value determining if the Recorded Future API should be queried about a specific Indicator of Compromise (IoC) when it is not found in the risk list cache. This setting determines the behavior of the NL Recorded Future node and the NL Broadcast Gather Threat Intelligence node.


Enable Domain Risk List Caching

Enables caching a Domain Risk List. This selected Domain Risk List is refreshed approximately on the selected refresh interval.


Domain Risk List

The name of the Domain Risk List to cache. The drop down is seeded with the available lists.


Domain Risk List Pull Interval

The time interval between Domain Risk List pulls. Defaults to the recommended ‘Once Every Two Hours’ interval if not specified.


Enable Hash Risk List Caching

Enables caching a Hash Risk List. This selected Hash Risk List is refreshed approximately on the selected refresh interval.


Hash Risk List

The name of the Hash Risk List to cache. The drop down is seeded with the available lists.


Hash Risk List Pull Interval

The time interval between Hash Risk List pulls. Defaults to the recommended ‘Daily’ interval if not specified.


Enable IP Address Risk List Caching

Enables caching an IP Address Risk List. This selected IP Address Risk List is refreshed approximately on the selected refresh interval.


IP Address Risk List

The name of the IP Address Risk List to cache. The drop down is seeded with the available lists.


IP Address Risk List Pull Interval

The time interval between IP Address Risk List pulls. Defaults to the recommended ‘Hourly’ interval if not specified.


Enable URL Risk List Caching

Enables caching a URL Risk List. This selected URL Risk List is refreshed approximately on the selected refresh interval.


URL Risk List

The name of the URL Risk List to cache. The drop down is seeded with the available lists.


URL Risk List Pull Interval

The time interval between URL Risk List pulls. Defaults to the recommended ‘Once Every Two Hours’ interval if not specified.


Enable Vulnerability Risk List Caching

Enables caching a Vulnerability risk list. This selected Vulnerability Risk List is refreshed approximately on the selected refresh interval.


Vulnerability Risk List

The name of the Vulnerability Risk List to cache. The drop down is seeded with the available lists.


Vulnerability Risk List Pull Interval

The time interval between Vulnerability Risk List pulls. Defaults to the recommended ‘Daily’ interval if not specified.

Flow Node

Communication node which controls the Nevelex Labs Recorded Future Plugin Instance by directly querying the Recorded Future Connect API or by using the cached risk lists. The risk lists are used by default if they are enabled. If the plugin instance is configured to fallback to the Connect API via the Fallback to Connect API if not Found in Risk List setting, the node will make a Recorded Future API request if the IoC is not found in the risk list.
Property
Description

Name

The display name of the node within the flows.


Unique ID

System-wide unique ID of the plugin instance.


Report Type

Configuration option determining the type of Recorded Future API call to make or risk list cache to search:

  • Search Within: dynamically search in the message and perform operations based on the IoC type found
  • Domain Report: Request a report on the specified Domain
  • File Hash Report: Request a report on the specified hash
  • IP Address Report: Request a report on the specified IP Address
  • URL Report: Request a report on the specified URL
  • Vulnerability (CVE) Report: Request a report on the specified CVE name

Search Within / Domain / File Hash / IP Address / URL / CVE

This field defines the location from the message, flow, global, or JSONata expression to use as the data source for the IoC or CVE Threat Intelligence report. The following contexts are supported:

  • msg: This selects part of the incoming message as the source of the data. This is typical choice.
  • flow: This selects part of the flow context’s saved data as the source. This information is shared with only the nodes on a given tab.
  • global: This selects part of the global context’s saved data as the source. This information is shared by all nodes regardless of tab.
  • J: expression: JSONata expression language to perform query and transform operations on the payload.
This node has been deprecated, but remains usable in existing flows. Use the NL-Message-Analysis node instead.

Helper node used to classify an Indicator of Compromise (IoC) or a CVE from a Recorded Future report as trusted or malicious based on the response's overall score. The field is found within the response at data.risk.score. This node takes as input the successful response from the NL Recorded Future node or the NL Broadcast Gather Threat Intelligence node.

Report analysis nodes have five (5) outputs.

  1. Malicious: The Malicious Risk Score matched the Indicator of Compromise's risk score.
  2. Trusted: The Trusted Risk Score matched the Indicator of Compromise's risk score.
  3. No Match: The rules did not match the IoC as either trusted or malicious.
  4. Report Missing: The incoming message did not include a report for Recorded Future.
  5. Report Error: A report exists with an error message of some type.

Property
Description

Name

The display name of the node within the flows.


Malicious Risk Score

Any risk score matching the defined Malicious Risk Score level is considered a malicious IoC. The following values are supported:

  • Unusual (>= 5)
  • >= 10
  • >= 15
  • >= 20
  • Suspicious (>= 25)
  • >= 30
  • >= 35
  • >= 40
  • >= 45
  • >= 50
  • >= 55
  • >= 60
  • Malicious (>= 65)
  • >= 70
  • >= 75
  • >= 80
  • >= 85
  • Very Malicious (>= 90)
  • >= 95

Trusted Risk Score

Any risk score matching the defined Trusted Risk Score level is considered a trusted IoC. This field can be set to Never to never consider an IoC trusted. The following values are supported:

  • Never
  • None (== 0)
  • Unusual (< 5)
  • < 10
  • < 15
  • < 20
  • Suspicious (< 25)
  • < 30
  • < 35
  • < 40
  • < 45
  • < 50
  • < 55
  • < 60

Audit Missing Report

If checked, a missing report audit message will be added to the Incident’s timeline.

Recorded Future risk list search node to find matching Indicators of Compromise (IoCs) from the cached risked lists. This node is useful for writing flows to propagating IoCs from Recorded Future to other services within your infrastructure. All filter criteria are logically AND'ed together.

Property
Description

Name

The display name of the node within the flows.


Unique Id

System-wide unique ID of the plugin instance.


Risk List

Use to determine which Recorded Future risk list cache to search through.


Limit Matches by Risk Score

Only those IoCs or CVEs matching the designated Risk Score rule will be allowed through.


Risk Score

Recorded Future risk score filter enabled when Limit Matches by Risk Score is checked. The following values are supported:

  • Unusual (>= 5)
  • Suspicious (>= 25)
  • >= 30
  • >= 35
  • >= 40
  • >= 45
  • >= 50
  • >= 55
  • >= 60
  • Malicious (>= 65)
  • >= 70
  • >= 75
  • >= 80
  • >= 85
  • Very Malicious (>= 90)
  • >= 95

Limit IoCs by Search Pattern

Only those IoCs matching the designated Search rule will be allowed through.


Search

Recorded Future IoC name filter enabled when Limit IoCs by Search Pattern is checked. The value may be one of the following:

  • Static string: Searches the IoC (domain, URL, hash, or IP address) for containment of the hard-coded string.
  • Dynamic string: Searches the IoC (domain, URL, hash, or IP address) for containment.
  • Regular expression: Searches the IoC (domain, URL, hash, or IP address) for a regular express match.

Limit to Undefined, Unknown, or Trusted IoCs

If the IoC is already considered malicious, it will be filtered out. A malicious Security Flow trust level is a value between 0 <= tl < 5, where tl is the numeric trust level.


Matches Per Incident Limit

To limit the amount of data in any particular incident, this condition is used to limit the total number of matches produced by any search performed by this node. The following values are supported:

  • 100 Matches
  • 200 Matches
  • 300 Matches
  • 400 Matches
  • 500 Matches
  • 1000 Matches

Matches Per Msg Limit

To limit the amount of data in any particular msg, this condition is used to limit the number of matches included in an individual msg sent through the flow. The following values are supported:

  • 10 Matches
  • 25 Matches
  • 50 Matches
  • 75 Matches
  • 100 Matches

Delay Between Msg's

To rate limit the number of msg'es pumped into the flow, this condition adds a delay between msg'es. The following values are supported:

  • 1 Minute
  • 2 Minutes
  • 3 Minutes
  • 4 Minutes
  • 5 Minutes

Immediately Associate IoCs with Incident

Enables setting of the Security Flow Trust Level for the IoC. The Default Trust Level element is enabled and shown when this option is enabled. If disabled, the IoCs should be associated with the incident elsewhere in the flow.


Default Trust Level

Defines the Security Flow Trust Level for the IoC. The value may be one of the following:

  • Calculated from Risk Score: Uses the following equation:
    tl = ( 99 - rs ) / 10, where tl is the trust level and
    rs is the risk score.
  • #: The trust level is statically assigned based on the selected number.

Remember Processed Vulnerability

This setting enables saving Vulnerability information to prevent future searches from sending the same Vulnerability (CVE) through the flows. This information is saved on a per node basis, so deleting the node from a flow and then deploying removes the saved information. Once deleted and deployed, the previously saved information is not recoverable.

Recorded Future Alert search node to find matching alerts to use as triggering events within a flow.

There are a number of configuration parameters which define the behavior of this node. The API calls, which are used to search for alerts rules and search for matching alerts, do not use API credits as described in the API Credits section of the Recorded Future Alert API page.

Property
Description

Name

The display name of the node within the flows.


Unique ID

System-wide unique ID of the plugin instance.


Repeat

The time interval between alert searches.


Status

Limit matches to alerts with the selected status. If -- Ignored -- is selected, this filter is not used in the search.


Search

Free form text search of the alert’s content. Leave this field blank to ignore this filter during the search.


Assignee

Email address of the assignee. Leave this field blank to ignore this filter during the search.


Triggered

Time relative to now when the alert was triggered. This criteria is always used to limit the search results.


Alert Rule

Prior to searching for alerts, this filter may be used to limit the alerts to only those triggered by specific alert rules. If -- Any Alert Rule -- is selected, this filter is not used in the search.


Alert Name

When Alert Rule is set to Find By Name, this filter searches the alert rule names for matches. If the supplied text finds multiple matching alert rules, each alert rule is individually used as a filter in a search.


Alert ID

When Alert Rule is set to Find By ID, this filter limits the search to the specified alert rule.


Aggregation

This node supports five modes for aggregating incidents.

  • None: Never aggregate any incidents (default).
  • Field Match: For a given source field of a message (or jsonata query of the message), aggregate together all messages whose field value exactly matches a previously checked message.
  • Exact Match: For a given source field of the message (or jsonata query of the message), aggregate together all messages whose field exactly matches a given value. One can specify multiple values. Each value specifies a separate grouping of incidents. Useful to aggregate messages with known content.
  • Keyword Match: For a given source field of the message (or jsonata query of the message), aggregate together all messages with a given keyword appearing as a word within the field. One can specify multiple keywords. Each keyword specifies a separate grouping of incidents. An incident is grouped with the first keyword it matches. Useful to aggregate messages with somewhat known content.
  • Fuzzy Match: For a given source field of the message (or jsonata query of the message), aggregate together all messages for which this field is sufficiently similar to the message which started the incident. With this method, one can specify a similarity threshold. Messages matched with the fuzzy matcher get a similarity attribute added to the message which can be inspected to assist when establishing a threshold.
The Recorded Future Alerts node is used to load an alert's detailed information or update an alert given its ID.
Property
Description

Name

The display name of the node within the flows.


Unique ID

System-wide unique ID of the plugin instance.


Action

Configuration option determining the type of operation to perform:

  • Lookup Alert: Retrieves the details of an alert given its Alert ID. The API call uses an API credit as described in the API Credits section of the Recorded Future Alert API page.
  • Update Alert: Updates a subset of alert fields given its Alert ID. Per Recorded Future support, this API call does not use an API credit.

Successful results for an action are placed in msg.payload.recordedfuture.response.


Alert ID

The ID of the alert to be loaded or updated. When the Alert ID value is set to Default Payload Alert ID Path, the node will look for the alert ID in payload.alert.id, the default path of the alert ID from the output of NL-Recorded-Future-Alert-Query node.


Assignee

The optional Assignee accepts an ID, username, user hash or email address.


Status

The alert’s new Status. The Status may be set to Not Specified to avoid changing the current status. Valid Status values are unassigned, assigned, pending, dismiss, no-action, actionable, and tuning. If the evaluated Status is invalid, Not Specified is assumed.


Note

The optional Note field uses variable substitution from the incoming message using a mustache format. Visit the Template Engine and Formatters page to learn more.

Learn More

JSON Message Format

The following samples show the JSON content added to the message payload, which conform to the Node Messaging Format. The content exists within the recordedfuture object.

Success

The italicized, green text is inserted into the message payload upon a successful request from a cached risk list. There is more information in the API request response, but it is not shown here. The response has been trimmed for readability.

"payload": {
  "recordedfuture": {  
    "topic": "/nevelexlabs/event/3ef51ea7.b9c29a/domainreportreply",
    "response": {
      "data": {
        "risk": {
          "rules": 2,
          "score": 91,
          "riskString": "2/34",
          "criticality": 4,
          "riskSummary": "2 of 34 Risk Rules currently observed.",
          "evidenceDetails": [ ... ],
          "criticalityLabel": "Very Malicious"
        },
        "entity": {
          "name": "IOC"
          ...
        },
        "intelCard": "https://app.recordedfuture.com/live/sc/entity/idn%3A..."
      },
      "risklistCache": true
    }
  }
}

Error

The italicized, maroon text is inserted into the message payload upon a failed request.

"payload": {
    "recordedfuture": {
        "error": {
            "error_code": 5,
            "error_message": "Error text"
        }
    }
}
Nevelex Labs, Main Office

Metro Office Park
2950 Metro Drive, Suite 104
Bloomington, MN 55425
Phone: +1 952-500-8921

©Nevelex Labs, LLC. 2018-2021, All Rights Reserved.

EULA