urlscan.io

Overview

The Security Flow URL Scan Plugin provides Incident enrichment with threat intelligence from the urlscan.io API. This plugin enriches domain, file hash, IP address, and URL Indicators of Compromise (IoCs).

Functionality

The URL Scan Plugin provides the functionality to gather threat intelligence on domains, hashes, IP addresses, and URLs.

When sending URLs to the plugin instance, those requests trigger a URL scan within the urlscan.io API. Those requests require an API Key and are throttled to one request every two seconds. Domains are treated as URLs and thus trigger a URL scan.

Instance Configuration Parameters

Property

Description

Instance Name

Name for the URL Scan instance.

Unique ID

A system-wide unique identifier for this plugin instance used to locate the service.

API Key

API Key necessary to perform scans of URLs submitted to this plugin instance.

Scan Visibility

Sets the visibility on a scan to either Public, Unlisted or Private.

Public: Scan is visible on the front page, in the public search results, and info pages on the urlscan.io website.
Unlisted: Scan is not visible on the public page or search results, but is visible to vetted security researchers and security companies in urlscan.io Pro platform. Use this to submit malicious websites that might contain PII or non-public information.
Private: Scan is only visible to urlscan.io user in users personalized search or if sharing the scan ID with third parties. Scans will be deleted from urlscan system after a certain retention period. Use this to prevent others seeing the URLs submitted.

Flow Nodes

Communication node which controls the Security Flow URL Scan Plugin Instance according to the specified configuration.

Property

Description

Name

The display name of the node within the flows.

Unique ID

System-wide unique ID of the plugin instance.

Action

Configuration option determining the type of urlscan.io API call to make:

Search Within: Dynamically search in the message and perform operations based on the IoC type found
Domain Report: Request a report for the specified Domain
File Hash Report: Request a report for the specified hash
IP Address Report: Request a report for the specified IP Address
URL Report: Request a report for the specified URL

Search Within / URL / Domain / IP / Hash

This field defines the location from the message, flow, global, or JSONata expression to use as the data source for the IoC Threat Intelligence report. The following contexts are supported:

msg: This selects part of the incoming message as the source of the data. This is typical choice.
flow: This selects part of the flow context’s saved data as the source. This information is shared with only the nodes on a given tab.
global: This selects part of the global context’s saved data as the source. This information is shared by all nodes regardless of tab.
J: expression: JSONata expression language to perform query and transform operations on the payload.

This node has been deprecated, but remains usable in existing flows. Use the NL-Message-Analysis node instead.

Helper node used to classify an Indicator of Compromise (IoC) from an urlscan.io report as trusted or malicious based on the response's overall verdict score. The field is found within the response at verdicts.overall.score. This node takes as input the successful response from the NL URLScan node or the NL Broadcast Gather Threat Intelligence node.

Report analysis nodes have five (5) outputs.

Malicious: The Malicious Score matched the Indicator of Compromise's score.
Trusted: The Trusted Score matched the Indicator of Compromise's score.
No Match: The heuristics did not detected a potential Indicator of Compromise or the score was zero (0).
Report Missing: The incoming message did not include a report for urlscan.io.
Report Error: A report exists with an error message of some type.

Property

Description

Name

The display name of the node within the flows.

Malicious Score

Any score matching the defined Malicious Score level is considered a malicious IoC. Malicious score checks are done against one of the selected ranges.

>= 1
>= 5
>= 10
>= 15
>= 20
>= 25
>= 30
>= 35
>= 40
>= 45
>= 50
>= 55
>= 60
>= 65
>= 70
>= 75
>= 80
>= 85
>= 90
>= 95
== 100

Trusted Score

Any score matching the defined Trusted Score level is considered a trusted IoC. This field can be set to Never to never consider an IoC trusted. A score of zero (0) indicates the IoC is unknown in urlscan.io, so that will always go out the No Match output. Trusted score checks are done against one of the selected ranges.

Never
< 2
< 3
< 4
< 5
< 10
< 15
< 20
< 25
< 30
< 35
< 40
< 45
< 50

Audit Missing Report

If checked and no urlscan.io report exists in the incoming message, a missing report audit message will be added to the Incident’s timeline.

Learn More

JSON Message Format

The following samples show the JSON content added to the message payload, which conform to Node Messaging Format. The content exists within the urlscan object.

Success

The italicized, green text is inserted into the message payload upon a successful request. The node in the following sample used “google.com” as input for a domain report:

"payload": {
   urlscan: {
  "topic": "/nevelexlabs/service/urlscan/domain/report",
  "response": {
    "data": {
      "requests": [
        {
          "request": {
            "requestId": "72686AEDD6E483502C4E5A2F6C1C39BB",
            "loaderId": "72686AEDD6E483502C4E5A2F6C1C39BB",
            "documentURL": "https://docs.google.com/static/document/client/css/926478166-homescreen_quantum_css_webkit_ltr.css",
            "request": {
              "url": "https://docs.google.com/static/document/client/css/926478166-homescreen_quantum_css_webkit_ltr.css",
              "method": "GET",
              "headers": {
                "Upgrade-Insecure-Requests": "1",
                "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36",
                "Sec-Fetch-Dest": "document"
              },
              "mixedContentType": "none",
              "initialPriority": "VeryHigh",
              "referrerPolicy": "no-referrer-when-downgrade"
            },
            "timestamp": 20048118.872147,
            "wallTime": 1581550943.121858,
            "initiator": {
              "type": "other"
            },
            "type": "Document",
            "frameId": "012302576BC41B83EF32E7CB4ADF5905",
            "hasUserGesture": false
          },
          "response": {
            "encodedDataLength": 9939,
            "dataLength": 87613,
            "requestId": "72686AEDD6E483502C4E5A2F6C1C39BB",
            "type": "Document",
            "response": {
              "url": "https://docs.google.com/static/document/client/css/926478166-homescreen_quantum_css_webkit_ltr.css",
              "status": 200,
              "statusText": "",
              "headers": {
                "status": "200",
                "accept-ranges": "bytes",
                "vary": "Accept-Encoding, Origin",
                "content-encoding": "gzip",
                "content-type": "text/css",
                "content-length": "9583",
                "date": "Tue, 11 Feb 2020 10:53:47 GMT",
                "expires": "Wed, 10 Feb 2021 10:53:47 GMT",
                "last-modified": "Fri, 10 Aug 2018 17:48:17 GMT",
                "x-content-type-options": "nosniff",
                "server": "sffe",
                "x-xss-protection": "0",
                "cache-control": "public, max-age=31536000",
                "age": "132516",
                "alt-svc": "quic=\":443\"; ma=2592000; v=\"46,43\",h3-Q050=\":443\"; ma=2592000,h3-Q049=\":443\"; ma=2592000,h3-Q048=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000"
              },
              "mimeType": "text/css",
              "requestHeaders": {
                ":method": "GET",
                ":authority": "docs.google.com",
                ":scheme": "https",
                ":path": "/static/document/client/css/926478166-homescreen_quantum_css_webkit_ltr.css",
                "pragma": "no-cache",
                "cache-control": "no-cache",
                "upgrade-insecure-requests": "1",
                "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36",
                "sec-fetch-dest": "document",
                "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
                "sec-fetch-site": "none",
                "sec-fetch-mode": "navigate",
                "sec-fetch-user": "?1",
                "accept-encoding": "gzip, deflate, br",
                "accept-language": "en-US"
              },
              "remoteIPAddress": "[2a00:1450:4001:81f::200e]",
              "remotePort": 443,
              "fromPrefetchCache": false,
              "encodedDataLength": 347,
              "timing": {
                "requestTime": 20048118.872313,
                "proxyStart": -1,
                "proxyEnd": -1,
                "dnsStart": 0.32,
                "dnsEnd": 1.074,
                "connectStart": 1.074,
                "connectEnd": 20.327,
                "sslStart": 6.115,
                "sslEnd": 20.319,
                "workerStart": -1,
                "workerReady": -1,
                "sendStart": 20.4,
                "sendEnd": 20.459,
                "pushStart": 0,
                "pushEnd": 0,
                "receiveHeadersEnd": 26.862
              },
              "protocol": "h2",
              "securityState": "secure",
              "securityDetails": {
                "protocol": "TLS 1.3",
                "keyExchange": "",
                "keyExchangeGroup": "X25519",
                "cipher": "AES_128_GCM",
                "certificateId": 0,
                "subjectName": "*.google.com",
                "sanList": [
                  "*.google.com",
                  "*.android.com",
                  "*.appengine.google.com",
                  "*.cloud.google.com",
                  "*.crowdsource.google.com",
                  "*.g.co",
                  "*.gcp.gvt2.com",
                  "*.gcpcdn.gvt1.com",
                  "*.ggpht.cn",
                  "*.gkecnapps.cn",
                  "*.google-analytics.com",
                  "*.google.ca",
                  "*.google.cl",
                  "*.google.co.in",
                  "*.google.co.jp",
                  "*.google.co.uk",
                  "*.google.com.ar",
                  "*.google.com.au",
                  "*.google.com.br",
                  "*.google.com.co",
                  "*.google.com.mx",
                  "*.google.com.tr",
                  "*.google.com.vn",
                  "*.google.de",
                  "*.google.es",
                  "*.google.fr",
                  "*.google.hu",
                  "*.google.it",
                  "*.google.nl",
                  "*.google.pl",
                  "*.google.pt",
                  "*.googleadapis.com",
                  "*.googleapis.cn",
                  "*.googlecnapps.cn",
                  "*.googlecommerce.com",
                  "*.googlevideo.com",
                  "*.gstatic.cn",
                  "*.gstatic.com",
                  "*.gstaticcnapps.cn",
                  "*.gvt1.com",
                  "*.gvt2.com",
                  "*.metric.gstatic.com",
                  "*.urchin.com",
                  "*.url.google.com",
                  "*.wear.gkecnapps.cn",
                  "*.youtube-nocookie.com",
                  "*.youtube.com",
                  "*.youtubeeducation.com",
                  "*.youtubekids.com",
                  "*.yt.be",
                  "*.ytimg.com",
                  "android.clients.google.com",
                  "android.com",
                  "developer.android.google.cn",
                  "developers.android.google.cn",
                  "g.co",
                  "ggpht.cn",
                  "gkecnapps.cn",
                  "goo.gl",
                  "google-analytics.com",
                  "google.com",
                  "googlecnapps.cn",
                  "googlecommerce.com",
                  "source.android.google.cn",
                  "urchin.com",
                  "www.goo.gl",
                  "youtu.be",
                  "youtube.com",
                  "youtubeeducation.com",
                  "youtubekids.com",
                  "yt.be"
                ],
                "issuer": "GTS CA 1O1",
                "validFrom": 1579594566,
                "validTo": 1586852166,
                "signedCertificateTimestampList": [],
                "certificateTransparencyCompliance": "unknown"
              },
              "securityHeaders": [
                {
                  "name": "X-Content-Type-Options",
                  "value": "nosniff"
                },
                {
                  "name": "X-Xss-Protection",
                  "value": "0"
                }
              ]
            },
            "hash": "2132a3ab5c2b1b064edbb0bd0dbf0528e1981086f2d8cf046530cc036a2724fd",
            "size": 87613,
            "asn": {
              "ip": "2a00:1450:4001:81f::200e",
              "asn": "15169",
              "country": "US",
              "registrar": "arin",
              "date": "2000-03-30",
              "description": "GOOGLE, US",
              "route": "2a00:1450:4001::/48",
              "name": "GOOGLE"
            },
            "geoip": {
              "range": "",
              "country": "DE",
              "region": "HE",
              "city": "Frankfurt am Main",
              "ll": [
                50.1188,
                8.6843
              ],
              "metro": 0,
              "area": 100,
              "eu": "1",
              "timezone": "Europe/Berlin",
              "country_name": "Germany"
            },
            "hashmatches": []
          }
        }
      ],
      "cookies": [],
      "console": [],
      "links": [],
      "timing": {
        "beginNavigation": "2020-02-12T23:42:23.121Z",
        "frameStartedLoading": "2020-02-12T23:42:23.152Z",
        "frameNavigated": "2020-02-12T23:42:23.152Z",
        "loadEventFired": "2020-02-12T23:42:23.164Z",
        "frameStoppedLoading": "2020-02-12T23:42:23.165Z",
        "domContentEventFired": "2020-02-12T23:42:23.165Z"
      },
      "globals": [
        {
          "prop": "onformdata",
          "type": "object"
        },
        {
          "prop": "onpointerrawupdate",
          "type": "object"
        }
      ]
    },
    "stats": {
      "resourceStats": [
        {
          "count": 1,
          "size": 87613,
          "encodedSize": 9939,
          "latency": 0,
          "countries": [
            "DE"
          ],
          "ips": [
            "[2a00:1450:4001:81f::200e]"
          ],
          "type": "Document",
          "compression": "8.8",
          "percentage": null
        }
      ],
      "protocolStats": [
        {
          "count": 1,
          "size": 87613,
          "encodedSize": 9939,
          "ips": [
            "[2a00:1450:4001:81f::200e]"
          ],
          "countries": [
            "DE"
          ],
          "securityState": {},
          "protocol": "h2"
        }
      ],
      "tlsStats": [
        {
          "count": 1,
          "size": 87613,
          "encodedSize": 9939,
          "ips": [
            "[2a00:1450:4001:81f::200e]"
          ],
          "countries": [
            "DE"
          ],
          "protocols": {
            "TLS 1.3 /  / AES_128_GCM": 1
          },
          "securityState": "secure"
        }
      ],
      "serverStats": [
        {
          "count": 1,
          "size": 87613,
          "encodedSize": 9939,
          "ips": [
            "[2a00:1450:4001:81f::200e]"
          ],
          "countries": [
            "DE"
          ],
          "server": "sffe"
        }
      ],
      "domainStats": [
        {
          "count": 1,
          "ips": [
            "2a00:1450:4001:81f::200e",
            "[2a00:1450:4001:81f::200e]"
          ],
          "domain": "docs.google.com",
          "size": 87613,
          "encodedSize": 9939,
          "countries": [
            "DE"
          ],
          "index": 0,
          "initiators": [],
          "redirects": 0
        }
      ],
      "regDomainStats": [
        {
          "count": 1,
          "ips": [
            "2a00:1450:4001:81f::200e",
            "[2a00:1450:4001:81f::200e]"
          ],
          "regDomain": "google.com",
          "size": 87613,
          "encodedSize": 9939,
          "countries": [],
          "index": 0,
          "subDomains": [],
          "redirects": 0
        }
      ],
      "secureRequests": 1,
      "securePercentage": 100,
      "IPv6Percentage": 100,
      "uniqCountries": 1,
      "totalLinks": 0,
      "malicious": 0,
      "adBlocked": 0,
      "ipStats": [
        {
          "requests": 1,
          "domains": [
            "docs.google.com"
          ],
          "ip": "2a00:1450:4001:81f::200e",
          "asn": {
            "ip": "2a00:1450:4001:81f::200e",
            "asn": "15169",
            "country": "US",
            "registrar": "arin",
            "date": "2000-03-30",
            "description": "GOOGLE, US",
            "route": "2a00:1450:4001::/48",
            "name": "GOOGLE"
          },
          "dns": {},
          "geoip": {
            "range": "",
            "country": "DE",
            "region": "HE",
            "city": "Frankfurt am Main",
            "ll": [
              50.1188,
              8.6843
            ],
            "metro": 0,
            "area": 100,
            "eu": "1",
            "timezone": "Europe/Berlin",
            "country_name": "Germany"
          },
          "size": 87613,
          "encodedSize": 9939,
          "countries": [
            "DE"
          ],
          "index": 0,
          "ipv6": true,
          "redirects": 0,
          "count": null
        }
      ]
    },
    "meta": {
      "processors": {
        "download": {
          "state": "done",
          "data": []
        },
        "geoip": {
          "state": "done",
          "data": [
            {
              "ip": "2a00:1450:4001:81f::200e",
              "geoip": {
                "range": "",
                "country": "DE",
                "region": "HE",
                "city": "Frankfurt am Main",
                "ll": [
                  50.1188,
                  8.6843
                ],
                "metro": 0,
                "area": 100,
                "eu": "1",
                "timezone": "Europe/Berlin",
                "country_name": "Germany"
              }
            }
          ]
        },
        "wappa": {
          "state": "done",
          "data": []
        },
        "rdns": {
          "state": "done",
          "data": []
        },
        "asn": {
          "state": "done",
          "data": [
            {
              "ip": "2a00:1450:4001:81f::200e",
              "asn": "15169",
              "country": "US",
              "registrar": "arin",
              "date": "2000-03-30",
              "description": "GOOGLE, US",
              "route": "2a00:1450:4001::/48",
              "name": "GOOGLE"
            }
          ]
        },
        "done": {
          "state": "done",
          "data": {
            "state": "done"
          }
        }
      }
    },
    "task": {
      "uuid": "5bdbd899-8644-45a7-b55e-94c3d10954e9",
      "time": "2020-02-12T23:42:23.007Z",
      "url": "https://docs.google.com/static/document/client/css/926478166-homescreen_quantum_css_webkit_ltr.css",
      "visibility": "public",
      "options": {
        "useragent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"
      },
      "method": "api",
      "source": "d0ae021b",
      "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36",
      "reportURL": "https://urlscan.io/result/5bdbd899-8644-45a7-b55e-94c3d10954e9/",
      "screenshotURL": "https://urlscan.io/screenshots/5bdbd899-8644-45a7-b55e-94c3d10954e9.png",
      "domURL": "https://urlscan.io/dom/5bdbd899-8644-45a7-b55e-94c3d10954e9/"
    },
    "page": {
      "url": "https://docs.google.com/static/document/client/css/926478166-homescreen_quantum_css_webkit_ltr.css",
      "domain": "docs.google.com",
      "country": "DE",
      "city": "Frankfurt am Main",
      "server": "sffe",
      "ip": "2a00:1450:4001:81f::200e",
      "ptr": "",
      "asn": "AS15169",
      "asnname": "GOOGLE, US"
    },
    "lists": {
      "ips": [
        "2a00:1450:4001:81f::200e"
      ],
      "countries": [
        "DE"
      ],
      "asns": [
        "15169"
      ],
      "domains": [
        "docs.google.com"
      ],
      "servers": [
        "sffe"
      ],
      "urls": [
        "https://docs.google.com/static/document/client/css/926478166-homescreen_quantum_css_webkit_ltr.css"
      ],
      "linkDomains": [],
      "certificates": [
        {
          "subjectName": "*.google.com",
          "issuer": "GTS CA 1O1",
          "validFrom": 1579594566,
          "validTo": 1586852166
        }
      ],
      "hashes": [
        "2132a3ab5c2b1b064edbb0bd0dbf0528e1981086f2d8cf046530cc036a2724fd"
      ]
    },
    "verdicts": {
      "overall": {
        "score": 0,
        "categories": [],
        "brands": [],
        "tags": [],
        "malicious": false,
        "hasVerdicts": 0
      },
      "urlscan": {
        "score": 0,
        "categories": [],
        "brands": [],
        "tags": [],
        "detectionDetails": [],
        "malicious": false
      },
      "engines": {
        "score": 0,
        "malicious": [],
        "benign": [],
        "maliciousTotal": 0,
        "benignTotal": 0,
        "verdicts": [],
        "enginesTotal": 0
      },
      "community": {
        "score": 0,
        "votes": [],
        "votesTotal": 0,
        "votesMalicious": 0,
        "votesBenign": 0,
        "tags": [],
        "categories": []
      }
    }
  }
}

Error

The italicized, maroon text is inserted into the message payload upon a failed request.

"payload": {
    "urlscan": {
        "error": {
            "error_code": 5,
            "error_message": {
                "statusCode": [HTTP Status Code],
                "message": "[error message]",
                "total": 0,
                "input": [Domain|Hash|IP|URL]
            }
        }
    }
}

Nevelex Labs, Main Office

Metro Office Park
2950 Metro Drive, Suite 104
Bloomington, MN 55425
Phone: +1 952-500-8921

Contact Us

EULA