Crowdsourcing detections for the suspicious and impossible

There’s a wealth of knowledge and intelligence available from an application’s point of use. Crowdsourcing detections can be as simple as asking the user to confirm whether they did actually just log in and from where. Here, time can be your ally rather than your enemy. By using a sliding window of time to verify identity and intent, we can garner a lot of useful information about our current security posture including how to optimize our responses or target our resources. We will use Tines and Datadog to crowdsource detections in a few simple steps.

Crowdsourcing can be used effectively for low-fidelity but high frequency common alerts. A few examples include validating and verifying:

• Logins from suspicious IPs (and related locations)

• Impossible travel / time travel alerts

• First time logins to new services or from new devices

• When suspicious commands are run (e.g. Powershell encoded command or ‘sudo’)

• Users who received a suspicious email e.g. BEC(Business Email Compromise) fraud

• Installation of new or unusual tools
  

Let’s focus on logins from new or suspicious locations by using IP address enrichment and engaging the user. IP address ranges are hierarchical and allocated from regional registries to ISPs (Internet Service Providers) and smaller national or local registries.

Globally there are five regional registries for IPv4, IPv6 and ASN numbering assets, ARIN, RIPE NCC, LACNIC, AFRINIC and APNIC. Allocations go to a range of entities including private organisations. Although there are some PI(Provider Independent) blocks, all allocations are tied to an entity and almost all to a national or local geographic level. The boundaries of these IP NETBLOCKs are extremely useful to us, especially at the country level.

If your users frequently log in to services from the same geographic location then it is only when their location (and thus their IP address or NETBLOCK) changes that you should potentially heighten your vigilance and investigate further. The concept of logging in from disparate locations, ones that the user could not have physically been traveled to within a real-world time window, is playfully called ‘Time Travel’ or sometimes “Impossible Travel”. It can however be a serious signal that something is not right and warrants more attention. Identifying successful but illegitimate logins can help stop attackers dead in their tracks before they penetrate deeper or get a foothold in an environment. Attackers often use the path of least resistance, so they use compromised credentials as an easy way in (one we’d prefer to block earlier rather than later in the ‘kill-chain’). So, something as simple as a “successful” login, but from an unusual or new location, can be seen as a potential IoA(Indicator of Attack) or IoC(Indicator of Compromise). This is exactly the same concept used for credit card fraud detection when transactions appear from unusual locations. Luckily the process of verifying an authentic user’s intent and actual location can be simply automated, including the act of asking them.

The following example can also be used for a wide range of authentication or authorization events that generate logs (including that of web apps).

We will enable basic suspicious login detection for SSH credentials that are observed by Datadog. Best practice involves locking down SSH endpoint access to predetermined trusted and managed IP prefix ranges (or using bastion hosts) however this is not always actioned, convenient, or possible. A user or operator’s credentials should not be being used from multiple widespread geographic locations unless they are intentionally doing so, in which case we can validate this with our automated checks.

Datadog at the Edge

‍We begin with Datadog sending a webhook to Tines whenever there’s a successful login for a specific service. We then request the associated logs from Datadog before going on to check with the user if it actually was them or not. To parse these authentication logs with Datadog we create a quick log pipeline that extracts useful information from patterns in our `auth.log` message body.

We extract as much information as we can such as the port, service version, date/timestamp, service, process_id, user, and client IP address (shown below).

We also set up a simple monitor that notifies Tines via a webhook on a successful login. This is just one example of matching on an interactive login (you could match on any successful login).

We are using Datadog as the initiator due to its agent footprint and privileges on systems. The bulk of the subsequent logic and workflow are left for Tines to orchestrate, as Tines allows you to easily build logic and flexibly enrich data and integrate multiple other sources.

Once we receive the webhook we perform a subsequent log search from Tines using the Datadog logs API. This will also surface our custom pipeline attributes. By using webhooks it means we are not searching and generating API calls unless there are both active and successful logins happening. We could also expand the search if so desired to look for this login being used elsewhere but for now we leave that for the automated Incident Response playbook if necessary.

A Deeper Look‍

If we take a look at the response to our “Search Logs in Datadog” Tines API call we can see, we explode any logs returned and then deduplicate them based upon the combined attributes of “crowdsource_ip” and “crowdsource_user”. We then use AbuseIPDB to enrich our understanding of the IP address abuse rating, the registered IP owner, and the geo-location of the IP.

Prompt Confirmation

The ability to integrate human real time responses into automations is a powerful feature. Human gate-kept automation allows for a response to alter the path of an automated workflow and dictate how it proceeds. Here we make use of the Tines prompt widget to create a decision point. The automation generates a customized Slack message with two prompts and sends it directly to the user. It has simple buttons for the user to quickly signal whether it was them or not.

We use two prompts, one for the user to click if it was them logging in and another for them to click if it was not them. We don’t expect a user to recognize their current IP address so we also mention the general location, date/time, service, and the ISP or IP NETBLOCK owner. 

From an automation perspective the prompts are simple one liners in a Slack message post. Tines already has all the required integration templates for Slack messages so you just need to compose your message and drop in the prompts. The prompt widget expands to create a custom link each time the automation runs, so in this case, the button’s “url” value becomes a unique URL that, when clicked, returns your chosen string to the automation.

{
  "type": "actions",
    "elements": [
    {
      "type": "button",
      "text": {
        "type": "plain_text",
        "emoji": true,
        "text": "Yes, that was me"
      },
      "style": "primary",
      "url": "{% prompt true %}"
    },
    {
      "type": "button",
      "text": {
        "type": "plain_text",
        "emoji": true,
        "text": "No, that was not me"
      },
      "style": "danger",
      "url": "{% prompt false %}"
    }
  ]
}

This click causes the Tines Slack message Action to emit a new event containing the user’s selection. This allows for branching logic and the ability to respond to a user’s specific choice or needs. The choices are vast and the level of complexity is entirely up to you with this highly flexible and interactive feature.

When the user clicks to say it was not them we initiate another sub-Story (our Incident Response playbook). We can now take automated defensive measures such as locking the account, adding the observed IP (or whole NETBLOCK) to an infrastructure ACL(Access List) or application block list, perform triage, and/or investigate more deeply what was accessed and how. You could equally act on a positive response where, if the user clicks to say it was them, you temporarily add their new IP address as a trusted IP across other systems and services to give a better, more secure, and unified user experience. 

By keeping a deduplicated cache of the last X login IP addresses for authenticated users, you gain the upper hand with identifying malicious or spurious logins. If you simply ask the user via an alternate channel such as email or Slack, was it actually them, you can close the faster. You already have a distributed and intelligent sensor network, your people. Let Tines automate the appropriate prompts to bolster your defenses. Increase your situational awareness and improve your organization's immune system by automating what's next.

You can download the above Story/workflow and try it out in your own tenancy here.

‍

*Please note we recently updated our terminology. Our "agents" are now known as "Actions," but some visuals might not reflect this.*