Automating URL analysis with Tines and URLScan

Last updated on

Eoin HinchyCo-founder & CEO, Tines
Aaron Sandow Sales Engineer, Tines

We are delighted to announce that Tines is sponsoring one of our favorite tools, urlscan.io. Welcome URLScan users to the Tines website! In this blog, we explore URLScan automation, including how you can automate your URL analysis processes; search for IOCs within URLScan; search for leaked credentials; and share threat intelligence with the security community.

For those unfamiliar with URLScan, you’re one of today’s lucky 10,000! URLScan.io is a website scanner built by Johannes Gilger, which scans and classifies almost 100,000 URLs every day. This includes submissions from thousands of public and enterprise users and security researchers and all URLs in openphish, phishtank, certstream, urlhaus and more. URLScan runs all the analysis on its own servers and records HTTP request data; all domain interactions; all links on the scanned page; the website technologies in use; a hash of every file on the page; and SSL certificate detection, as well as related scans, IP information, google safe browsing information for the domain and more.

Even better, URLScan makes all this information available, for free, via an intuitive and well-built API. This makes automating scanning, searching, and interacting with URLScan through the Tines security automation platform incredibly easy.

For those of you visiting Tines for the first time, Tines is a Security Orchestration, Automation and Response (SOAR) platform that helps security teams automate any repetitive manual task. If you are unfamiliar with Security Automation, you can check out our ‘getting started’ guide. If you are familiar with security automation, you can read about why Tines is different than all other SOAR platforms. (hint: we don’t rely on any prebuilt integrations – you can integrate easily with every tool in your technology stack!)

Why sponsor URLScan?

At Tines, we’ve long been fans of URLScan. Before setting up Tines, we worked as security engineers in eBay/PayPal and DocuSign, some of the most phished brands in the world. When we had to analyze thousands of phishing URLs we quickly realized that manually analyzing them one by one was time-consuming, error-prone, and, frankly, boring. As a result, we turned to automation, and we started using URLScan.io. When we started Tines, URLScan's incredible API made it easy to showcase how to analyze URLs and to share threat intelligence back to the community.

It’s no surprise that many enterprise security teams rely on URLScan.io to analyze suspicious URLs. It’s also no surprise that URLScan has been mentioned heavily in other blog posts by Tines! At Tines, we want to help ensure URLScan continues to be an incredible resource for the security community.

Does Tines connect with URLScan?

Yes! URLScan is a tool that exposes all its analysis information up front in a clean and simple to use API. Because of this, it’s very easy for Tines customers to search for and submit URLs to URLScan. We have several out-of-the-box stories which harness the power of URLScan. Customers can easily customize these to suit their own needs and processes.

As mentioned above, Tines does not rely on pre-built apps to integrate with external systems. Instead, the HTTP Request Action (one of the seven Actions in Tines) provides a direct connection with the target tool, in this case, URLScan. This means consistent connections with any tool, regardless of the vendor, whether it’s open or closed-source or commercial off-the-shelf or custom-built.

More about URLScan automation in Tines 

The primary purpose of URLScan is to analyze URLs. Those familiar with URLScan will know that every page that is analyzed is categorized and given a malicious score verdict:

The verdict of a URL analysis in URLScan

The most obvious process to automate, therefore, is the analysis of URLs sent to employee or customer abuse inboxes. You can read more about Tines' in-depth and out-of-the-box abuse inbox processing here. If you are spending significant time analyzing URLs, you should consider automating that process using an automation platform like Tines.

Submitting a URL to URLScan through Tines is easy:

You can also use Tines to pull suspicious URLs from other sources, which can then be analyzed in URLScan.io. Common sources of malicious or suspicious URLs include:

  • URLs blocked by your email security solution like Proofpoint, FireEye ETP, Barracuda, Mimecast, or Microsoft APT.

  • DMARC failures or rejects

  • Suspicious uncategorized or punycode URLs from your firewall logs or DNS logs

  • New SSL Certificates registered with domains similar to your brand (e.g. from crt.sh)

  • Threat Intel sources like the Phish.ai threat intel feed which generates feeds based on the brands attacked

  • Free feeds of malicious URLs like Phishtank, Openphish, phishstats.info, or URLhaus. Note, that these feeds are often high-reputation so don’t necessarily need to be further analyzed.

Using Tines’ Phishing Story it’s easy to collect suspicious URLs from dozens of different sources automatically. Once these feeds are in Tines, it’s easy to deduplicate and classify URLs to prevent alert overload and generate more accurate metrics.

Does URLScan detect if a site is malicious? 

URLScan results in Tines

Using the information returned via the URLScan API, we can build a trigger Action to flag URLs classified as malicious. We can then take additional Actions, including blocking that URL; scanning for traffic to the domain in our environment; sending takedown notices for malicious content; creating tickets for analysts, etc.) This verdict is also returned in API calls, so we can use this information to automate the URL analysis process. This verdict considers the domain and IP classification in other security tools like GSB, OpenPhish, PhishTank, URLhaus, etc.

What else can I automate with URLScan.io? 

Automate IOC extraction 

Urlscan.io records the hash of every file it downloads as an indicator of compromise or ‘IOC’. Using the URLScan API, you can search for other pages with this same IOC. This means if you’re a highly phished brand, for example, or a researcher tracking a phishing campaign, you can search for pages with similar IOCs that may be using the same phishing kit. E.g., searching for this md5 (a PayPal logo) will return several thousand other pages impersonating PayPal.

You can automate this search using Tines to extract URLs with matching IOC every hour and issue takedown notices for pages abusing your brand, for example.

Search for leaked credentials or access tokens 

Unfortunately, users and analysts occasionally mistake legitimate emails as suspicious and use URLScan to analyze legitimate web pages. Occasionally sensitive information like document access links or password reset tokens is exposed. For many services, the URL itself can be enough to give an attacker access to an account or too sensitive information.

With a trivial amount of effort searching URLScan (no, we won’t give you the searches here!), you can find several password reset tokens for high-profile enterprise services and access links to enterprise file-sharing services like Dropbox, OneDrive, etc.

For more information on how to see if your company’s accounts or website might be affected and how you can automate the detection of these for your enterprise, you can read this Tines blog on the topic.

Share threat intelligence with the community 

Urlscan is a valuable threat intelligence tool for researchers and security professionals. However, it’s only as good as the data the community submits and shares. If you have a feed of malicious URLs you have detected privately; you can give back to the security community by sharing this information automatically to URLScan using Tines. Sharing threat intelligence with URLScan means researchers and other security teams can keep their customers, companies, and the wider internet community safer.

For more information, you can read this Tines blog on how to share threat intelligence information using URLScan.

Conclusion 

In conclusion, integrating with URLScan.io is easy with Tines. If you find yourself using URLScan frequently to analyze URLs you should consider looking at a SOAR platform to help with URLScan automation and let your team focus on more impactful risk reduction efforts.

Built by you, powered by Tines

Talk to one of our experts to learn the unique ways your business can leverage Tines.