Overview
The Security Flow Recorded Future Plugin adds the ability to gather threat intelligence on IoCs (URLs, domains, IP addresses, file hashes, and vulnerabilities), trigger flows based on alerts, update alerts, retrieve entities, and manage lists. The Plugin provides Incident enrichment with threat intelligence using either cached risk list data or up-to-date information from the Recorded Future API. Additionally, cached IoC or CVE risk lists are searchable.
Functionality
The Security Flow Recorded Future plugin provides the ability to
- gather threat intelligence on domains, hashes, IP addresses, URLs, and CVEs
- search cached risk lists from matching IoCs
- trigger flows periodically based on searching for alerts
- lookup alert details
- update alerts
- find entities
- search for user lists
- create a user list
- retrieve metadata for a user list
- retrieve the status of a user list
- retrieve the entities within a user list
- add an entity to a user list
- remove an entity from a user list
When risk lists are enabled, changes to a plugin instance’s risk list configuration will trigger a pull. The following rules apply for risk list updates:
- Changing any setting will not interrupt an existing risk list pull.
- Enabling the risk list pull will trigger the queuing of a pull request.
- Changing the risk list name will trigger the queuing of a pull request.
- The risk list cache entries are updated based on the following the selected refresh cycle. The defaults are shown here.
- Domains – Every Two (2) hours
- Hashes – Once a day
- IP Addresses – Hourly
- URLs – Every Two (2) hours
- Vulnerabilities – Once a day
- The risk list cache entries expire, i.e. are removed from the cache, based on aging out according to the refresh cycles.
- Until cached, a risk list check will fail to find the threat intelligence.
- The risk list cache is periodically checked once every five (5) minutes.
Instance Configuration Parameters
Name for the Recorded Future instance.
A system-wide unique identifier for this plugin instance used to locate the service.
API key used to access the Recorded Future services.
Boolean value determining if the Recorded Future API should be queried about a specific Indicator of Compromise (IoC) when it is not found in the risk list cache. This setting determines the behavior of the NL Recorded Future node and the NL Broadcast Gather Threat Intelligence node.
Enables caching a Domain Risk List. This selected Domain Risk List is refreshed approximately on the selected refresh interval.
The name of the Domain Risk List to cache. The drop down is seeded with the available lists.
The time interval between Domain Risk List pulls. Defaults to the recommended ‘Once Every Two Hours’ interval if not specified.
Enables caching a Hash Risk List. This selected Hash Risk List is refreshed approximately on the selected refresh interval.
The name of the Hash Risk List to cache. The drop down is seeded with the available lists.
The time interval between Hash Risk List pulls. Defaults to the recommended ‘Daily’ interval if not specified.
Enables caching an IP Address Risk List. This selected IP Address Risk List is refreshed approximately on the selected refresh interval.
The name of the IP Address Risk List to cache. The drop down is seeded with the available lists.
The time interval between IP Address Risk List pulls. Defaults to the recommended ‘Hourly’ interval if not specified.
Enables caching a URL Risk List. This selected URL Risk List is refreshed approximately on the selected refresh interval.
The name of the URL Risk List to cache. The drop down is seeded with the available lists.
The time interval between URL Risk List pulls. Defaults to the recommended ‘Once Every Two Hours’ interval if not specified.
Enables caching a Vulnerability risk list. This selected Vulnerability Risk List is refreshed approximately on the selected refresh interval.
The name of the Vulnerability Risk List to cache. The drop down is seeded with the available lists.
The time interval between Vulnerability Risk List pulls. Defaults to the recommended ‘Daily’ interval if not specified.
Flow Node
Fallback to Connect API if not Found in Risk List setting, the node will make a Recorded Future API request if the IoC is not found in the risk list.The display name of the node within the flows.
System-wide unique ID of the plugin instance.
Configuration option determining the type of Recorded Future API call to make or risk list cache to search:
- Search Within: dynamically search in the message and perform operations based on the IoC type found
- Domain Report: Request a report on the specified Domain
- File Hash Report: Request a report on the specified hash
- IP Address Report: Request a report on the specified IP Address
- URL Report: Request a report on the specified URL
- Vulnerability (CVE) Report: Request a report on the specified CVE name
This field defines the location from the message, flow, global, or JSONata expression to use as the data source for the IoC or CVE Threat Intelligence report. The following contexts are supported:
- msg: This selects part of the incoming message as the source of the data. This is typical choice.
- flow: This selects part of the flow context’s saved data as the source. This information is shared with only the nodes on a given tab.
- global: This selects part of the global context’s saved data as the source. This information is shared by all nodes regardless of tab.
- J: expression: JSONata expression language to perform query and transform operations on the payload.
Recorded Future Alert search node to find matching alerts to use as triggering events within a flow.
There are a number of configuration parameters which define the behavior of this node. The API calls, which are used to search for alerts rules and search for matching alerts, do not use API credits as described in the API Credits section of the Recorded Future Alert API page.
The display name of the node within the flows.
System-wide unique ID of the plugin instance.
The time interval between alert searches.
Limit matches to alerts with the selected status. If -- Ignored -- is selected, this filter is not used in the search.
Free form text search of the alert’s content. Leave this field blank to ignore this filter during the search.
Email address of the assignee. Leave this field blank to ignore this filter during the search.
Time relative to now when the alert was triggered. This criteria is always used to limit the search results.
Prior to searching for alerts, this filter may be used to limit the alerts to only those triggered by specific alert rules. If -- Any Alert Rule -- is selected, this filter is not used in the search.
When Alert Rule is set to Find By Name, this filter searches the alert rule names for matches. If the supplied text finds multiple matching alert rules, each alert rule is individually used as a filter in a search.
When Alert Rule is set to Find By ID, this filter limits the search to the specified alert rule.
This node supports five modes for aggregating incidents.
- None: Never aggregate any incidents (default).
- Field Match: For a given source field of a message (or jsonata query of the message), aggregate together all messages whose field value exactly matches a previously checked message.
- Exact Match: For a given source field of the message (or jsonata query of the message), aggregate together all messages whose field exactly matches a given value. One can specify multiple values. Each value specifies a separate grouping of incidents. Useful to aggregate messages with known content.
- Keyword Match: For a given source field of the message (or jsonata query of the message), aggregate together all messages with a given keyword appearing as a word within the field. One can specify multiple keywords. Each keyword specifies a separate grouping of incidents. An incident is grouped with the first keyword it matches. Useful to aggregate messages with somewhat known content.
- Fuzzy Match: For a given source field of the message (or jsonata query of the message), aggregate together all messages for which this field is sufficiently similar to the message which started the incident. With this method, one can specify a similarity threshold. Messages matched with the fuzzy matcher get a similarity attribute added to the message which can be inspected to assist when establishing a threshold.
The display name of the node within the flows.
System-wide unique ID of the plugin instance.
Configuration option determining the type of operation to perform:
- Lookup Alert: Retrieves the details of an alert given its Alert ID. The API call uses an API credit as described in the API Credits section of the Recorded Future Alert API page.
- Update Alert: Updates a subset of alert fields given its Alert ID. Per Recorded Future support, this API call does not use an API credit.
Successful results for an action are placed in msg.payload.recordedfuture.response.
The ID of the alert to be loaded or updated. When the Alert ID value is set to Default Payload Alert ID Path, the node will look for the alert ID in payload.alert.id, the default path of the alert ID from the output of NL-Recorded-Future-Alert-Query node.
The optional Assignee accepts an ID, username, user hash or email address.
The alert’s new Status. The Status may be set to Not Specified to avoid changing the current status. Valid Status values are unassigned, assigned, pending, dismiss, no-action, actionable, and tuning. If the evaluated Status is invalid, Not Specified is assumed.
The optional Note field uses variable substitution from the incoming message using a mustache format. Visit the Template Engine and Formatters page to learn more.
entities. The entities element is an array of EntityMatchResponse objects. See the Entity Match API for more information. The first found entity is placed in the standard response location at entity.The display name of the node within the flows.
System-wide unique ID of the plugin instance.
Keyword search term used to locate matching entities, such as people, usernames, IP addresses, hashes, etc…
If specified, Entity Types is an array of entity types or a comma separated list of types. Some possible values for entity types are Person, Username, IndustryTerm, IpAddress, InternetDomainName, Hash, and CyberVulnerability.
The maximum number of returned entities is limited to the Size Limit.
The display name of the node within the flows.
A system-wide unique identifier for this plugin instance used to locate the service.
Configuration option determining the type of operation to perform:
- Add List Entity by ID: Adds an entity to a list using the entity’s ID.
- Add List Entity by Name & Type: Adds an entity to a list using the entity’s name and type.
- Create List: Creates a new new list.
- Get List Entities: Retrieves a list’s current set of entities.
- Get List Info: Retrieves metadata about a list.
- Get List Status: Retrieves the status and number of entities in a list.
- Remove List Entity by ID: Removes an entity from a list using the entity’s ID.
- Remove List Entity by Name & Type: Removes an entity from a list using the entity’s name and type.
- Search for Lists: Searches for existing lists.
Successful results for an action are placed in msg.payload.recordedfuture.[uniqueId].response.
Unique ID string for the list. If the List ID is set to Standard List ID Location, the list ID is obtained from list.id in the standard response location. The list ID is typically set by the Search for Lists action.
Unique ID strong for the entity. If the Entity ID is set to Standard Entity ID Location, the entity ID is obtained from entity.id in the standard response location. The entity ID is set by the NL-Recorded-Future-Entities node.
The full name of an entity. Used when adding or removing entities by name and type. The Entity Name is a domain name, hash, IP address, or vulnerability.
The type of the Entity Name being added or removed from a list. The valid Entity Type values are IpAddress, InternetDomainName, Hash, or CyberVulnerability.
The query used when searching for existing lists.
The name of the list being created.
The type of list being created or searched for. If not specified, it defaults to entity. The valid List Type values are entity, source, text, custom, ip, domain, tech_stack, industry, brand, partner, industry_peer, location, supplier, vulnerability, company, hash, operation, attacker, target, or method.
The maximum number of returned lists is limited to the Size Limit.
The display name of the node within the flows.
System-wide unique ID of the plugin instance.
The time interval between playbook alert searches. The left inject button on this node is used to immediately trigger the retrieval of playbook alerts. If the repeat interval is set to zero, querying will not occur unless manually triggered using the left inject button.
Limit matches to alerts with the selected statuses. If no statuses are selected, this filter is ignored.
Limit matches to alerts with the selected priorities. If no priorities are selected, this filter is ignored.
Limit matches to alerts with the selected categories. If no categories are selected, this filter is ignored.
The user IDs, in uhash format, of the assignees. Leave this field blank to ignore this filter during the search. A comma separated list of assignees is allowed.
Time relative to now when the playbook alert was created. If the value is set to zero, it is ignored during the search.
Time relative to now when the playbook alert was updated. If the value is set to zero, it is ignored during the search.
When enabled, this node loads the panel details for each playbook alert prior to triggering the flow. If this API call fails, the payload.alert.errorLoadingPanels is set to an error message.
When set to a value other than -- Do Not Update --, this node updates the status for each playbook alert prior to triggering the flow. If this API call is successful, the payload.alert.newStatus is set to the new status and payload.alert.status contains the original status. If this API call fails, the payload.alert.errorUpdatingStatus is set to an error message.
The maximum number of returned entities is limited to the Size Limit.
Sort the results list by the created or modified date.
Sort results in ascending or descending order.
This node supports five modes for aggregating incidents.
- None: Never aggregate any incidents (default).
- Field Match: For a given source field of a message (or jsonata query of the message), aggregate together all messages whose field value exactly matches a previously checked message.
- Exact Match: For a given source field of the message (or jsonata query of the message), aggregate together all messages whose field exactly matches a given value. One can specify multiple values. Each value specifies a separate grouping of incidents. Useful to aggregate messages with known content.
- Keyword Match: For a given source field of the message (or jsonata query of the message), aggregate together all messages with a given keyword appearing as a word within the field. One can specify multiple keywords. Each keyword specifies a separate grouping of incidents. An incident is grouped with the first keyword it matches. Useful to aggregate messages with somewhat known content.
- Fuzzy Match: For a given source field of the message (or jsonata query of the message), aggregate together all messages for which this field is sufficiently similar to the message which started the incident. With this method, one can specify a similarity threshold. Messages matched with the fuzzy matcher get a similarity attribute added to the message which can be inspected to assist when establishing a threshold.
The display name of the node within the flows.
System-wide unique ID of the plugin instance.
Configuration option determining the type of operation to perform:
- Find Matching Alerts: Finds playbook alerts matching the configuration.
- Load Alert Panels: Loads the panels for a playbook alert.
- Update Alert: Updates a playbook alert.
Successful results for an action are placed in msg.payload.recordedfuture.[uniqueId].response.
Limit matches to alerts with the selected statuses. Possible values are Dismissed, InProgress, New, and Resolved. This value may be a comma separated status list or an array of statuses.
Limit matches to alerts with the selected priorities. Possible values are Informational, Moderate, and High. This value may be a comma separated priority list or an array of priorities.
Limit matches to alerts with the selected categories. Possible values are domain_abuse and malware_sandbox. This value may be a comma separated category list or an array of categories.
The user IDs, in uhash format, of the assignees. Leave this field blank to ignore this filter during the search. This value may be a comma separated uhash list or an array of uhashes.
Allows for filtering by Created or Updated timestamps. To generate the start timestamp and end timestamp, mustache template substitution is applied to generate the final values. The example templates for both the Created or Updated filters search for playbook alerts affected within the last 30 days. Visit the Template Engine and Formatters page to learn more.
Filters for playbook alerts created after the specified start time. The example template filters for creation dates starting 30 days ago relative to the current time.
Filters for playbook alerts created before the specified end time. The example template filters for creation dates relative to the current time.
Filters for playbook alerts updated after the specified start time. The example template filters for update dates starting 30 days ago relative to the current time.
Filters for playbook alerts updated before the specified end time. The example template filters for update dates relative to the current time.
If this value evaluates to a JavaScript truthy value, the panels for the playbook alert are automatically loaded. The Load Alert Panels action may also be used to perform this API call.
The Alert ID specifies the playbook alert’s ID to load or update. If Alert ID is set to Standard Alert ID Location, the alert ID is obtain within the standard response location at alert.playbook_alert_id. The standard location is set when an alert is found by either the NL-Recorded-Future-Playbook-Alert-Query node or this node.
The Category specifies the playbook alert’s category. Possible values are domain_abuse and malware_sandbox. If Category is set to Standard Alert Category Location, the category is obtain within the standard response location at alert.category. The standard location is set when an alert is found by either the NL-Recorded-Future-Playbook-Alert-Query node or this node.
The new playbook alert status. The status must evaluate to one of Dismissed, InProgress, New, or Resolved.
The new playbook alert priority. The priority must evaluate to one of Informational, Moderate, or High.
The user ID, in uhash format, of the assignee.
The log entry for the update operation uses mustache template substitution to generate the message. Visit the Template Engine and Formatters page to learn more.
Recorded Future risk list search node to find matching Indicators of Compromise (IoCs) from the cached risked lists. This node is useful for writing flows to propagating IoCs from Recorded Future to other services within your infrastructure. All filter criteria are logically AND'ed together.
The display name of the node within the flows.
System-wide unique ID of the plugin instance.
Use to determine which Recorded Future risk list cache to search through.
Only those IoCs or CVEs matching the designated Risk Score rule will be allowed through.
Recorded Future risk score filter enabled when Limit Matches by Risk Score is checked. The following values are supported:
- Unusual (>= 5)
- Suspicious (>= 25)
- >= 30
- >= 35
- >= 40
- >= 45
- >= 50
- >= 55
- >= 60
- Malicious (>= 65)
- >= 70
- >= 75
- >= 80
- >= 85
- Very Malicious (>= 90)
- >= 95
Only those IoCs matching the designated Search rule will be allowed through.
Recorded Future IoC name filter enabled when Limit IoCs by Search Pattern is checked. The value may be one of the following:
- Static string: Searches the IoC (domain, URL, hash, or IP address) for containment of the hard-coded string.
- Dynamic string: Searches the IoC (domain, URL, hash, or IP address) for containment.
- Regular expression: Searches the IoC (domain, URL, hash, or IP address) for a regular express match.
If the IoC is already considered malicious, it will be filtered out. A malicious Security Flow trust level is a value between 0 <= tl < 5, where tl is the numeric trust level.
To limit the amount of data in any particular incident, this condition is used to limit the total number of matches produced by any search performed by this node. The following values are supported:
- 100 Matches
- 200 Matches
- 300 Matches
- 400 Matches
- 500 Matches
- 1000 Matches
To limit the amount of data in any particular msg, this condition is used to limit the number of matches included in an individual msg sent through the flow. The following values are supported:
- 10 Matches
- 25 Matches
- 50 Matches
- 75 Matches
- 100 Matches
To rate limit the number of msg'es pumped into the flow, this condition adds a delay between msg'es. The following values are supported:
- 1 Minute
- 2 Minutes
- 3 Minutes
- 4 Minutes
- 5 Minutes
Enables setting of the Security Flow Trust Level for the IoC. The Default Trust Level element is enabled and shown when this option is enabled. If disabled, the IoCs should be associated with the incident elsewhere in the flow.
Defines the Security Flow Trust Level for the IoC. The value may be one of the following:
- Calculated from Risk Score: Uses the following equation:
tl = ( 99 - rs ) / 10, wheretlis the trust level and
rsis the risk score. - #: The trust level is statically assigned based on the selected number.
This setting enables saving Vulnerability information to prevent future searches from sending the same Vulnerability (CVE) through the flows. This information is saved on a per node basis, so deleting the node from a flow and then deploying removes the saved information. Once deleted and deployed, the previously saved information is not recoverable.
NL-Message-Analysis node instead.
Helper node used to classify an Indicator of Compromise (IoC) or a CVE
from a Recorded Future report as trusted or malicious based on the
response's overall score. The field is found within
the response at data.risk.score.
This node takes as input the successful response from
the NL Recorded Future node or the
NL Broadcast Gather Threat Intelligence node.
Report analysis nodes have five (5) outputs.
- Malicious: The
Malicious Risk Scorematched the Indicator of Compromise's risk score. - Trusted: The
Trusted Risk Scorematched the Indicator of Compromise's risk score. - No Match: The rules did not match the IoC as either trusted or malicious.
- Report Missing: The incoming message did not include a report for Recorded Future.
- Report Error: A report exists with an error message of some type.
The display name of the node within the flows.
Any risk score matching the defined Malicious Risk Score level is considered a malicious IoC. The following values are supported:
- Unusual (>= 5)
- >= 10
- >= 15
- >= 20
- Suspicious (>= 25)
- >= 30
- >= 35
- >= 40
- >= 45
- >= 50
- >= 55
- >= 60
- Malicious (>= 65)
- >= 70
- >= 75
- >= 80
- >= 85
- Very Malicious (>= 90)
- >= 95
Any risk score matching the defined Trusted Risk Score level is considered a trusted IoC. This field can be set to Never to never consider an IoC trusted. The following values are supported:
- Never
- None (== 0)
- Unusual (< 5)
- < 10
- < 15
- < 20
- Suspicious (< 25)
- < 30
- < 35
- < 40
- < 45
- < 50
- < 55
- < 60
If checked, a missing report audit message will be added to the Incident’s timeline.
Learn More
JSON Message Format
The following samples show the JSON content added to the message payload, which conform to the Node Messaging Format. The content exists within the recordedfuture object.
Success
The italicized, green text is inserted into the message payload upon a successful request from a cached risk list. There is more information in the API request response, but it is not shown here. The response has been trimmed for readability.
"payload": {
"recordedfuture": {
"topic": "/nevelexlabs/event/3ef51ea7.b9c29a/domainreportreply",
"response": {
"data": {
"risk": {
"rules": 2,
"score": 91,
"riskString": "2/34",
"criticality": 4,
"riskSummary": "2 of 34 Risk Rules currently observed.",
"evidenceDetails": [ ... ],
"criticalityLabel": "Very Malicious"
},
"entity": {
"name": "IOC"
...
},
"intelCard": "https://app.recordedfuture.com/live/sc/entity/idn%3A..."
},
"risklistCache": true
}
}
}
Error
The italicized, maroon text is inserted into the message payload upon a failed request.
"payload": {
"recordedfuture": {
"error": {
"error_code": 5,
"error_message": "Error text"
}
}
}
Metro Office Park
2950 Metro Drive, Suite 104
Bloomington, MN 55425
Phone: +1 952-500-8921
©Nevelex Labs, LLC. 2018-2024, All Rights Reserved.
EULA