We are delighted to announce that Tines is sponsoring one of our favorite tools, urlscan.io. Welcome urlscan users to the Tines website! In this blog you’ll learn more about urlscan automation including how you can automate your URL analysis processes; search for IOCs within urlscan; search for leaked credentials; and share threat intelligence with the security community.
For those Tines readers unfamiliar with urlscan, you’re one of today’s lucky 10,000! urlscan.io is a website scanner built by Johannes Gilger, which scans and classifies almost 100,000 urls every day. This includes submissions from thousands of public and enterprise users and security researchers and all urls in openphish, phishtank, certstream, urlhaus and more. urlscan runs all the analysis on its own servers and records http request data; all domain interactions; all links on the scanned page; the website technologies in use; a hash of every file on the page; and ssl certificate detection, as well as related scans, IP information, google safe browsing information for the domain and more.
Even better, urlscan makes all this information available, for free, via an intuitive and well built API. This makes automating scanning, searching, and interacting with urlscan through the Tines security automation platform incredibly easy.
For those of you visiting Tines for the first time, Tines is a Security Orchestration, Automation and Response (SOAR) platform that helps security teams automate any repetitive manual task. If you are unfamiliar with Security Automation, you can check out our ‘getting started’ guide. If you are familiar with security automation, you can read about why Tines is different than all other SOAR platforms. (hint: we don’t rely on any prebuilt integrations – you can integrate easily with every tool in your technology stack!)
Why sponsor urlscan?
At Tines we’ve long been fans of urlscan. Before setting up Tines we worked as security engineers in eBay/PayPal and DocuSign, some of the most phished brands in the world. When we had to analyze thousands of phishing urls we quickly realized that manually analyzing them one-by-one was time consuming, error prone and, frankly, boring. As a result, we turned to automation and we started using urlscan.io. When we started Tines, urlscan’s incredible API made it easy to showcase how to analyze urls, and to share threat intelligence back to the community.
It’s no surprise that many enterprise security teams rely on urlscan.io to analyze suspicious URLs. It’s also no surprise that urlscan has been mentioned heavily in other blog posts by Tines! At Tines, we want to help ensure urlscan continues to be an incredible resource for the security community.
Does Tines integrate with urlscan?
Yes! urlscan is a tool which exposes all its analysis information up front in a clean and simple to use API. Because of this, it’s very easy for Tines customers to search for and submit urls to urlscan. We have several out of the box stories which harness the power of urlscan. Customers can easily customize these to suit their own needs and processes.
As mentioned above, Tines does not rely on pre-built apps to integrate with external systems. Instead, the HTTP Request Action (one of the seven Actions available in Tines) provides direct integration with the target tool, in this case urlscan. This means consistent integration with any tool, regardless of the vendor, regardless of whether it’s open or closed-source, and regardless of whether it’s commercial off the shelf or custom built.
Tell me about urlscan automation in Tines!
The primary purpose of urlscan is to analyze urls. Those familiar with urlscan will know that every page that is analyzed is categorized and given a malicious score verdict:
The most obvious process to automate, therefore, is the analysis of urls sent to employee or customer abuse inboxes. You can read more about Tines in depth and out-of-the-box abuse inbox processing here. If you are spending significant time analyzing urls you should consider automating that process using an automation platform like Tines.
Submitting a url to urlscan through Tines is easy:
You can also use Tines to pull suspicious URLs from other sources which can then be analyzed in urlscan.io. Common sources of malicious or suspicious urls include:
URLs blocked by your email security solution like Proofpoint, FireEye ETP, Barracuda, Mimecast or Microsoft APT.
DMARC failures or rejects
Suspicious uncategorized or punycode URLs from your firewall logs or DNS logs
New SSL Certificates registered with domains similar to your brand (e.g. from crt.sh)
Threat Intel sources like the Phish.ai threat intel feed which generates feeds based on the brands attacked
Free feeds of malicious urls like Phishtank, Openphish, phishstats.info or Urlhaus. Note, these feeds are often are high-reputation so don’t necessarily need to be further analyzed.
Using Tines’ Phishing Story it’s easy to collect suspicious urls from dozens of different sources automatically. Once these feeds are in Tines it’s easy to deduplicate and classify urls to prevent alert overload and to generate more accurate metrics.
Does urlscan detect if a site is malicious?
Using the information returned via the urlscan API we can build a trigger Action to flag urls classified as malicious. We can then take additional Actions including blocking that URL; scanning for traffic to the domain in our environment; sending takedown notices for malicious content; creating tickets for analysts etc.)in the future note, this urlscan api feature is in beta mode so may change. This verdict is also returned in API calls, so we can use this information to automate the url analysis process. (Yes! The above screenshot shows that a verdict, or overall malicious score, is returned in the urlscan UI based on an analysis of the content on the page. This verdict takes into account the classification of the domain and IP in other security tools like GSB, openphish, phishtank, urlhaus etc.
What else can I automate with urlscan.io?
Automate IOC extraction
Urlscan.io records the hash of every file it downloads as an indicator of compromise or ‘ioc’. Using the urlscan API you can search for other pages with this same IOC. This means if you’re a highly phished brand, for example, or a researcher tracking a phishing campaign, you can search for pages with similar IOCs which may be using the same phishing kit. E.g. searching for this md5 (a PayPal logo) will return several thousand other pages impersonating PayPal.
You can automate this search using Tines to extract urls with matching IOC every hour and issue takedown notices for pages abusing your brand, for example.
Search for leaked credentials or access tokens
Unfortunately, users and analysts occasionally mistake legitimate emails as suspicious and use urlscan to analyze legitimate web pages. Occasionally sensitive information like document access links or password reset tokens are exposed. For many services the url itself can be enough to give an attacker access to an account or to sensitive information.
With a trivial amount of effort searching urlscan (no, we won’t give you the searches here!) you can find several password reset tokens for high profile enterprise services as well as access links to enterprise file sharing services like Dropbox, OneDrive etc.
For more information on how to see if your company’s accounts or website might be affected, and how you can automate the detection of these for your enterprise, you can read this Tines blog on the topic.
Share threat intelligence with the community
Urlscan is valuable threat intelligence tool for researchers and security professional, however it’s only as good as the data that the community submits and shares. If you have a feed of malicious urls you have detected privately, you can give back to the security community by sharing this information automatically to urlscan using Tines. Sharing threat intelligence with urlscan means researchers and other security teams can keep their customers, companies and the wider internet community safer.
For more information you can read this Tines blog on how to share threat intelligence information using urlscan.
In conclusion, integrating with urlscan.io is easy with Tines. If you find yourself using urlscan frequently to analyze urls you should consider looking at an SOAR platform to help with urlscan automation and let your team focus on more impactful risk reduction efforts.
*Please note we recently updated our terminology. Our "agents" are now known as "Actions," but some visuals might not reflect this.*