Login
By shifting from point-solutions to a cybersecurity platform approach, IT and security teams significantly improve their efficiency and security outcomes. Security Service Edge (SSE) projects are… Read more on Cisco Blogs
In this blog about the design, deployment and automation of the Black Hat network, we have the following sections:
Cisco is honored to be a Premium Partner of the Black Hat NOC, and is the Official Network Platform, Mobile Device Management, Malware Analysis and DNS (Domain Name Service) Provider of Black Hat.
2022 was Cisco’s sixth year as a NOC partner for Black Hat Europe. However, it was our first time building the network for Black Hat Europe. We used experiences of Black Hat Asia 2022 and Black Hat USA 2022 to refine the planning for network topology design and equipment. Below are our fellow NOC partners providing hardware, to build and secure the network, for our joint customer: Black Hat.
We are grateful to share that Black Hat Europe 2022 was the smoothest experience we’ve had in the years at Black Hat. This is thanks to the 15 Cisco Meraki and Cisco Secure engineers on site (plus virtually supporting engineers) to build, operate and secure the network; and great NOC leadership and collaborative partners.
To plan, configure, deploy (in two days), maintain resilience, and recover (in four hours) an enterprise class network, took a lot of coordination. We appreciate the Black Hat NOC leadership, Informa and the NOC partners; meeting each week to discuss the best design, staffing, gear selection and deployment, to meet the unique needs of the conference. Check out the “Meraki Unboxed” podcast – Episode 94: Learnings from the Black Hat Europe 2022 Cybersecurity Event
We must allow real malware on the Black Hat network: for training, demonstrations, and briefing sessions; while protecting the attendees from attack within the network from their fellow attendees, and prevent bad actors from using the network to attack the Internet. It is a critical balance to ensure everyone has a safe experience, while still being able to learn from real world malware, vulnerabilities, and malicious websites.
In addition to the weekly meetings with Black Hat and the other partners, the Cisco Meraki engineering team of Sandro Fasser, Rossi Rosario Burgos, Otis Ioannou, Asmae Boutkhil, Jeffry Handal and I met every Friday for two months. We also discussed the challenges in a Webex space with other engineers who worked on past Black Hat events.
The mission:
Division of labor is essential to reduce mistakes and stay laser focused on security scope. Otis took the lead working on network topology design with Partners. Asmae handled the port assignments for the switches. Rossi ensured every AP and Switch was tracked, and the MAC addresses were provided to Palo Alto Networks for DCHP assignments. Otis and Rossi spent two days in the server room with the NOC partners, ensuring every switch was operating and configured correctly. Rossi also deployed and configured a remote Registration switch for Black Hat.
In the weeks before deployment, our virtual Meraki team member, Aleksandar Dimitrov Vladimirov, and I focused on planning and creating a virtual Wi-Fi site survey. Multiple requirements and restrictions had to be taken into consideration. The report was based on the ExCel centre floor plans, the space allocation requirements from Black Hat and the number of APs we had available to us. Although challenging to create, with some uncertainties and often changing requirements due to the number of stakeholders involved, the surveys AP placement for best coverage ended up being pivotal at the event.
Below is the Signal Strength plan for the Expo Hall Floor on the 5 GHz band. The original plan to go with a dual-Band deployment was adjusted onsite and the 2.4 GHz band was disabled to enhance performance and throughput. This was a decision made during the network setup, in coordination with the NOC Leadership and based on experience from past conferences.
Upon arrival at the ExCel Centre, we conducted a walkthrough of the space that most of us had only seen as a floor plan and on some photos. Thanks to good planning, we could start deploying the 100+ APs immediately, with only a small number of changes to optimize the deployment on-site. As the APs had been pre-staged and added to the Meraki dashboard, including their location on the floor maps, the main work was placing and cabling them physically. During operation, the floor plans in the Meraki Dashboard were a visual help to easily spot a problem and navigate the team on the ground to the right spot, if something had to be adjusted.
As the sponsors and attendees filled each space, in the Meraki dashboard, we were able to see in real-time the number of clients connected to each AP, currently and over the time of the conference. This enabled quick reaction if challenges were identified, or APs could be redeployed to other zones. Below is the ExCel Centre Capital Hall and London Suites, Level 0. We could switch between the four levels with a single click on the Floor Plans, and drill into any AP, as needed.
The Location heatmaps also provided essential visibility into conference traffic, both on the network and footfalls of attendees. Physical security is also an important aspect of cybersecurity; we need to know how devices move in space, know where valuable assets are located and monitor their safety.
Below is the Business Hall at lunchtime, on the opening day of the conference. You can see no live APs in the bottom right corner of the Location heatmap. This is an example of adapting the plan to reality onsite. In past Black Hat Europe conferences, the Lobby in that area was the main entrance. Construction in 2022 closed this entrance. So, those APs were reallocated to the Level 1 Lobby, where attendees would naturally flow from Registration.
The floor plans and heatmaps also helped with the Training, Briefings and Keynote network resilience. Capacity was easy to add temporarily, and we were able to remove it and relocate it after a space emptied.
During our time in the NOC, we had the chance to work with other vendor engineers and some use cases that came up led to interesting collaborations. One specific use case was that we wanted to block wireless clients, that show some malicious or bad behavior, automatically after they have been identified by one of the SOC analysts on the different security platforms, in addition we wanted to show them a friendly warning page that guides them to the SOC for a friendly conversation.
The solution was a script that can be triggered thru the interfaces of the other security products and attaches a group policy thru the Meraki Dashboard, including a quarantine VLAN and a splash page, via the Meraki APIs. This integration was just one of the many collaboration bits that we worked on.
During the first day of training, in the Meraki dashboard Air Marshal, I observed packet flood attacks, against we were able to adapt and remain resilient.
I also observed an AP spoofing and broadcast de-authentication attack. I was able to quickly identify the location of the attack, which was at the Lobby outside the Business Hall. Should the attacks continue, physical security had the information to intervene. We also had the ability to track the MAC address throughout the venue, as discussed in Christian Clasen’s section in part two.
From our experiences at Black Hat USA 2022, we had encrypted frames enabled, blunting the attack.
The Meraki dashboards made it very easy to monitor the health of the network APs and Switches, with the ability to aggregate data, and quickly pivot into any switch, AP or clients.
Through the phases of the conference, from two days of pre-conference setup, to focused and intense training the first two days, and transition to the briefings and Business Hall, we were able to visualize the network traffic.
In addition, we could see the number of attendees who passed through the covered area of the conference, with or without connecting to the network. Christian Clasen takes this available data to a new level in Part 2 of the blog.
As the person with core responsibilities for the switch configuration and uptime, the Meraki dashboard made it very simple to quickly change the network topology, according to the needs of the Black Hat customer.
If you refer back to Black Hat USA 2022, you’d have seen that we had over 1,000 iOS devices to deploy, with which we had several difficulties. For context, the company that leases the devices to Black Hat doesn’t use a Mobile Device Management (MDM) platform for any of their other shows…Black Hat is the only one that does. So, instead of using a mass deployment technology, like Apple’s Automated Device Enrollment, the iOS devices are “prepared” using Apple Configurator. This includes uploading a Wi-Fi profile to the devices as part of that process. In Las Vegas, this Wi-Fi profile wasn’t set to auto join the Wi-Fi, resulting in the need to manually change this on 1,000 devices. Furthermore, 200 devices weren’t reset or prepared, so we had those to reimage as well.
Black Hat Europe 2022 was different. We took the lessons from US and coordinated with the contractor to prepare the devices. Now, if you’ve ever used Apple Configurator, there’s several steps needed to prepare a device. However, all of these can be actions can be combined into a Blueprint:
Instead of there being several steps to prepare a device, there is now just one! Applying the Blueprint!
For Black Hat Europe, this included:
There’s lots of other things that can be achieved as well, but this results in the time taken to enroll and set up a device to around 30 seconds. Since devices can be set up in parallel (you’re only limited by the number of USB cables / ports you have), this really streamlines the enrollment and set up process.
Now, for the future, whilst you can’t Export these blueprints, they are transportable. If you open Terminal on a Mac and type:
cd /Users/<YOUR USER NAME>/Library/Group Containers/K36BKF7T3D.group.com.apple.configurator/Library/Application Support/com.apple.configurator/Blueprints
You’ll see a file / package called something.blueprint This can be zipped up and emailed to some else so, they can then use the exact same Blueprint! You may need to reboot your computer for the Blueprint to appear in Apple Configurator.
As mentioned, the registration / lead capture / session scanning devices are provided by the contractor. Obviously, these are all catalogued and have a unique device code / QR code on the back of them. However, during setup, any device name provisioned on the device gets lost.
So, there’s three things we do to know, without having to resort to using the unwieldy serial number, what devices is what.
In the footnote, you’ll see Device Name and Device Serial in blue. This denotes that the values are actually dynamic and change per device. They include:
On the Lock Screen, it’s now possible to see the device’s name and serial number, without having to flip the device over (A problem for the registration devices which are locked in a secure case) or open systems preferences.
We also had integration with SecureX device insights, to see the security status of each iOS device.
With the ability to quickly check on device health from the SecureX dashboard.
This goes without saying, but the iOS devices (Registration, Lead Capture and Session Scanning) do have access to personal information. To ensure the security of the data, devices are wiped at the end of the conference. This is incredibly satisfying, hitting the Erase Devices button in Meraki Systems Manager, and watching the 100+ devices reset!
Deploying a network like Black Hat takes a lot of work, and repetitive configuration. Much of this has been covered in previous blogs. However, to make things easier for this event, instead of the 60 training SSIDs we had in Black Hat US 2022, the Meraki team discussed the benefits of moving to iPSKs with Black Hat NOC Leadership, which accepted the plan.
For context, instead of having a single pre shared key for an SSID, iPSK functionality allows you to have 1000+. Each of these iPSKs can be assigned its own group policy / VLAN. So, we created a script:
This only involves five API calls:
The bulk of the script is error handling (The SSID or network doesn’t exist, for example) and logic!
The result was one SSID for all of training: BHTraining, and each classroom had their own password. This reduced the training SSIDs from over a dozen and helped clear the airwaves.
Thank you to the Cisco NOC team:
Also, to our NOC partners NetWitness (especially David Glover, Iain Davidson, Alessandro Contini and Alessandro Zatti), Palo Alto Networks (especially James Holland, Matt Ford, Matt Smith and Mathew Chase), Gigamon, IronNet, and the entire Black Hat / Informa Tech staff (especially Grifter ‘Neil Wyler’, Bart Stump, Steve Fink, James Pope, Jess Stafford and Steve Oldenbourg).
For 25 years, Black Hat has provided attendees with the very latest in information security research, development, and trends. These high-profile global events and trainings are driven by the needs of the security community, striving to bring together the best minds in the industry. Black Hat inspires professionals at all career levels, encouraging growth and collaboration among academia, world-class researchers, and leaders in the public and private sectors. Black Hat Briefings and Trainings are held annually in the United States, Europe and USA. More information is available at: blackhat.com. Black Hat is brought to you by Informa Tech.
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
Our core mission in the NOC is network resilience. We also provide integrated security, visibility and automation, a SOC inside the NOC.
In part one, we covered:
In part two, we are going deep with security:
As the needs of Black Hat evolved, so did the Cisco Secure Technologies in the NOC:
The SecureX dashboard made it easy to see the status of each of the connected Cisco Secure technologies.
Since joining the Black Hat NOC in 2016, my goal remains integration and automation. As a NOC team comprised of many technologies and companies, we are pleased that this Black Hat NOC was the most integrated to date, to provide an overall SOC cybersecurity architecture solution.
We have ideas for even more integrations for Black Hat Asia and Black Hat USA 2023. Thank you, Piotr Jarzynka, for designing the integration diagram.
Below are the SecureX threat response integrations for Black Hat Europe, empowering analysts to investigate Indicators of Compromise very quickly, with one search.
The original Black Hat NOC integration for Cisco was NetWitness sending suspicious files to Threat Grid (know Secure Malware Analytics). We expanded that in 2022 with Palo Alto Networks Cortex XSOAR and used it in London, for investigation of malicious payload attack.
NetWitness observed a targeted attack against the Black Hat network. The attack was intended to compromise the network.
NetWitness extracted the payload and sent it to Secure Malware Analytics for detonation.
Reviewing the analysis report, we were able to quickly determine it was the MyDoom worm, which would have been very damaging.
The attack was blocked at the perimeter and the analysts were able to track and enrich the incident in XSOAR.
My first time at Black Hat turned out to be an incredible journey!
Thanks to the cybersecurity partnership between Paris 2024 and Cisco, I was able to integrate into the Cisco Crew, to operate the NOC/SOC as a Threat Hunter on the most dangerous network in the world for this European Edition of Black Hat.
My first day, I helped with deploying the network by installing the wireless Meraki APs on the venue, understanding how they were configured and how they could help analysts to identify and locate any client connected to the network that could have a bad behavior during the event, the idea being to protect the attendees if an attack was to spray on the network.
Following this “physical” deployment, I’ve been able to access the whole Cisco Secure environment including Meraki, Secure Malware Analytics, Umbrella, SecureX and the other Black Hat NOC partners software tools.
SecureX was definitely the product on which I wanted to step up. By having so fantastic professionals around me, we were able to dig in the product, identifying potential use cases to deploy in the orchestration module and expected integrations for Paris 2024.
Time was flying and so were the attendees to the conference, a network without user is fun but can be quite boring as nothing happens, having so many cybersecurity professional at the same place testing different security malwares, attacks and so on led us to very interesting investigations. A paradox at the Black Hat, we do not want to block malicious content as it could be part of exercises or training classes, quite a different mindset as what we, security defenders, are used to! Using the different components, we were able to find some observables/IOCs that we investigate through SecureX, SecureX being connected to all the other components helped us to enrich the observables (IPs, urls, domains…), understanding the criticality of what we identified (such as malware payloads) and even led us to poke the folks in the training classes to let them know that something really wrong was happening on their devices.
Being part of the Black Hat NOC was an incredible experience, I was able to meet fantastic professionals, fully committed on making the event a success for all attendees and exhibitors. It also helped me to better understand how products, that we use or will use within Paris 2024, could be leveraged to our needs and which indicators could be added to our various Dashboards, helping us to identify, instantaneously, that something is happening.
During the last day of Black Hat Europe, our NOC partner, NetWitness saw some files being downloaded on the network. The integration again automatically carved out the file and submitted the Cisco Secure Malware Analytics (SMA) platform. One of those files came back as a trojan, after SMA detonated the file in a sandbox environment. The specific hash is the below SHA-256:
938635a0ceed453dc8ff60eab20c5d168a882bdd41792e5c5056cc960ebef575
The screenshot below shows some of the behaviors that influenced the decision:
The result of seeing these behaviors caused SMA to give it the highest judgement score available to a detonated file:
After this judgement was made, we connected with the Palo Alto Networks team, and they found the IP address associated with the file download.
Once we had this information, we went to the Meraki dashboard and did a search for the IP address. The search returned only one client that has been associated with the address for the entire Black Hat conference.
Knowing that there has only been one client associated with the address made finding the attendee easier. We then needed to know where they were and Meraki had this figured out. After opening the client’s profile, we saw what SSID and access point (AP) they were connected to using the Meraki location map.
We then found the attendee and let them know to have their IT inspect their laptop to make sure it is clean.
Apart from the technical challenges of running a temporary network for N thousand people, the Black Hat event reminded us that success doesn’t happen without teamwork; that leadership isn’t just about keeping the project on track. It is also about looking after the team and that small details in planning, build up and tear down can be just as important, as having all the right tools and fantastically skilled Individuals using them during the event itself.
In the Cisco Secure technology stack, within the Black Hat NOC, we use SecureX Single Sign-on. This reduces the confusion of managing multiple accounts and passwords. It also streamlines the integrations between the Cisco products and our fellow NOC partners. We have an open ecosystem approach to integrations and access in the NOC, so we will provision Cisco Secure accounts for any staff member of the NOC. Logging into each individual console and creating an account is time consuming and can often lead to confusion on which tools to provision and which permission levels are needed.
To automate this process, I developed two workflows: one to create non-admin users for NOC partners and one to create administrator accounts in all the tools for Cisco staff. The workflows create accounts in SecureX, Secure Malware Analytics (Threat Grid), Umbrella DNS and Meraki dashboard, all using SecureX Single Sign-On.
Here is what the workflow looks like for creating non-admin users.
The workflow requires three inputs: first name, last name, and email. Click Run.
The sequence of API calls is as follows:
Once the workflow has completed successfully, the user will receive four emails to create a SecureX Sign-On account and accept the invitations to the various products. These workflows really improved our responsiveness to account provisioning requests and makes it much easier to collaborate with other NOC partners.
Over the previous Black Hat events, we have been utilizing Meraki scanning data to get location data for individual clients, as they roamed conference. In the initial blog post (Black Hat Asia 2022), we created a Docker container to accept the data from the Meraki Scanning API and save it for future analysis. At Black Hat USA 2022, we wrote about how to use Python Folium to use the flat text files to generate chronological heatmaps that illustrated the density of clients throughout the conference.
This time around, we’ve stepped it up again by integrating Umbrella DNS Security events and adding the ability to track clients across the heatmap using their local IP address.
To improve the portability of our data and the efficiency of our code, we began by moving from flat JSON files to a proper database. We chose SQLite this time around, though going forward we will likely use Mongo.
Both can be queried directly into Python Pandas dataframes which is what will give us the optimal performance we are looking for. We have a dedicated Docker container (Meraki-Receiver) that will validate the incoming data stream from the Meraki dashboard and insert the values into the database.
The database is stored on a Docker volume that can be mounted by our second container, the Meraki-Mapper. Though this container’s primary purpose is building the heatmaps, it also performs the task of retrieving and correlating Umbrella DNS security events. That is, any DNS query from the Black Hat network that matches one of several predefined security categories. Umbrella’s APIs were recently improved to add OAuth and simplify the URI scheme for each endpoint. After retrieving a token, we can get all security events in the time frame of the current heatmap with one call.
What we want to do with these events is to create Folium Markers. These are static “pins” that will sit on the map to indicate where the DNS query originated from. Clicking on a marker will popup more information about the query and the client who sent it.
Thanks to the Umbrella Virtual Appliances in the Black Hat network, we have the internal IP address of the client who sent the DNS query. We also have the internal IP address in the Meraki scanning data, along with the latitude and longitude. After converting the database query into a Pandas dataframe, our logic takes the IP address from the DNS query and finds all instances in the database of location data for that IP within a 5-minute window (the resolution of our heatmap).
What we end up with is a list of dictionaries representing the markers we want to add to the map. Using Bootstrap, we can format the popup for each event to make it look a bit more polished. Folium’s Popup plugin allows for an iFrame for each marker popup.
The result is a moving heatmap covering an entire day on a given conference floor, complete with markers indicating security events (the red pushpin icon).
Clicking on the pushpin shows the details of the query, allowing us in the NOC to see the exact location of the client when they sent it.
To further improve this service during the next conference, we plan to implement a web page where NOC staff can submit an IP address and immediately get map tracking that client through the conference floor. This should give us an even more efficient way to find and notify folks who are either behaving maliciously or appear to be infected.
For years we have been tracking the DNS stats at the Blackhat conferences. The post-pandemic 2022 numbers look like we never skipped a beat after the dip in DNS queries from 2021, seen in the bar graph below. This year’s attendance saw well over 11 million total DNS queries.
The Activity volume view from Umbrella gives a top-level level glance of activity by category, which we can drill into for deeper threat hunting. On trend with the previous Black Hat Europe events, the top Security categories were Dynamic DNS and Newly Seen Domains. However, it’s worth noting a proportionally larger increase in the cryptomining and phishing categories from 9 to 17 and 28 to 73, respectively, compared to last year.
These years, Black Hat saw over 4,100 apps connect to the network, which is nearly double of what was seen last year. However, still not topping over 6,100 apps seen at Black Hat USA early this year.
Should the need arise, we can block any application, such as Mail.ru above.
Black Hat Europe 2022 was the best planned and executed NOC in my experience, with the most integrations and visibility. This allowed us the time to deal with problems, which will always arise.
We are very proud of the collaboration of the team and the NOC partners.
Black Hat Asia will be in May 2023, at the Marina Bay Sands, Singapore…hope to see you there!
Thank you to the Cisco NOC team:
Also, to our NOC partners NetWitness (especially David Glover, Iain Davidson, Alessandro Contini and Alessandro Zatti), Palo Alto Networks (especially James Holland, Matt Ford, Matt Smith and Mathew Chase), Gigamon, IronNet, and the entire Black Hat / Informa Tech staff (especially Grifter ‘Neil Wyler’, Bart Stump, Steve Fink, James Pope, Jess Stafford and Steve Oldenbourg).
For 25 years, Black Hat has provided attendees with the very latest in information security research, development, and trends. These high-profile global events and trainings are driven by the needs of the security community, striving to bring together the best minds in the industry. Black Hat inspires professionals at all career levels, encouraging growth and collaboration among academia, world-class researchers, and leaders in the public and private sectors. Black Hat Briefings and Trainings are held annually in the United States, Europe and USA. More information is available at: blackhat.com. Black Hat is brought to you by Informa Tech.
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
Watch ThreatWise TV: Explorations in the spam folder
The spam folder: that dark and disregarded corner of every email account, full of too-good-to-be-true offers, unexpected shipments, and supposedly free giveaways.
You’re right to ignore this folder; few good things come from exploring it. But every once in a while one of these misleading, and sometimes malicious, emails manages to evade the filters that normally siphon them off, landing them in your inbox instead.
Fortunately, it’s easy enough to spot these emails if you know what to look for. We’ve investigated this folder once before, showcasing a variety of scams. With the holiday season in full swing, we thought this would be a good time to revisit how scammers are trying to trick unsuspecting users.
The holiday season is traditionally a time when this type of activity increases, and this year is no different. According to research published by credit reporting agency TransUnion, the average daily number of suspected digital fraud attempts was up 82 percent globally between Thanksgiving and Cyber Monday (Nov 24–Nov 28) compared to the rest of the year (Jan 1–Nov 23) and 127 percent higher for transactions originating in the US.
This level of activity makes it all the more important to be aware of these scams. With that in mind, let’s dive into the spam folder to get a picture of the types of campaigns currently circulating.
While much of the spam circulating is innocuous, many emails are phishing attempts, and some are indeed malicious. To explore these scams, we used a dedicated computer, segmented from the rest of the network, and leveraged Cisco Secure Malware Analytics to safely open the emails before clicking on links or opening attachments. The point being, we do not recommend doing this at home.
By far, the largest category of spam we saw were surveys scams. According to these emails, if you fill out a simple survey you’ll receive “exclusive offers” such as gift cards, smartphones, smart watches, power drills, or even pots and pans.
There are even some campaigns that specifically target the holiday shopping season.
Clicking the links in these emails takes the recipient to sites where they are asked to fill out a survey.
These pages often include fake testimonials that say how easy the survey is and what they did with their free gift.
The surveys are straightforward, comprising 10-20 simple questions that cover demographic information and shopping habits.
After the survey is completed, these sites offer the choice of a handful of rewards. All the recipient must do is pay for shipping. They are then brought to a page where they can fill out shipping and payment information, and the reward is supposedly shipped.
However, the attempts to make payment often appear to fail, or the recipient is informed that the prize is no longer available.
An unsuspecting user may simply give up at this point, disappointed that they won’t be getting their free gift. What they may not be aware of, is that they have just given their credit card details away in a phishing scam.
In their 2021 Internet Crime Report, the Internet Crime Complaint Center (IC3) said that Non-Payment / Non-Delivery scams such as these led to more than $337 million in losses, up from $265 million in 2020. Credit card fraud amounted to $172 million in 2021 and has been climbing continuously at a conservative rate of 15-20 percent since 2019.
According to Cisco Umbrella, many of the sites asking for credit card details are known phishing sites, or worse, host malware.
Another topic that we covered the last time we explored these types of scams was package delivery spam. These continue to circulate today. There are a variety of shipping companies impersonated in these campaigns, and some generic ones as well.
Many of these campaigns claim that a package could not be delivered. If the recipient clicks on a link in an email, they’re brought to a web page that explains that there are outstanding delivery fees that need to be paid.
The recipient is further enticed by suggestions that the package contains a big-ticket item, such as an iPhone or iPad Pro. All the recipient is required to do is enter their credit card details to cover the shipping.
While no outright malicious activity was detected while examining these emails in Secure Malware Analytics, several suspicious behaviors were flagged. Chances are the bad actors behind these campaigns are phishing for credit card details.
Sometimes the simplest approaches can work just as well as the flashiest. This certainly holds true with spam campaigns, given the prominence of plain-text messages.
The topics covered in such emails run the gamut, including medical cures, 419 scams, romance and dating, pharmaceuticals, weight loss, and many of the scam types we’ve already covered. Many of these link to phishing sites, though some attempt to establish a dialog with the recipient, tricking them into sending the scammers money.
The IC3 report says that victims of confidence fraud and romance scams lost $956 million collectively, which is up from $600 million in 2020. Healthcare fraud, such as the miracle pills and prescriptions scams, resulted in $7 million in losses in 2021, but nearly $30 million in 2020. While these types of scams seem generic and easily spotted, they still work, and so it’s important to be aware and avoid them.
Many emails hitting the spam box attempt to trick users of various services into believing that there is a problem with their account. The problems cover all sorts of services, including streaming platforms, email providers, antivirus subscriptions, and even public records.
If the links are clicked, the recipient is presented with landing pages that mimic the respective services. Any details that are entered will likely be phished, leading to account takeover and/or access to personal records. However, some domains encountered in these cases may do more than just steal information, they could deliver malware too.
Another frequently encountered scam surrounds billing. Many of these appear to be unexpected bills for services the recipient never purchased.
These emails include attachments that are designed to look like official invoices. Interestingly, most of the attachments that we looked at this time were harmless. The goal is to get the recipient to call what appears to be a toll-free number.
While we haven’t called any of these numbers, the experience usually unfolds like a standard customer service call. In the end the “agents” simply claim the charges—which never existed in the first place—have been removed. Meanwhile the scammers steal any personal or financial information provided during the call.
While most billing scams we encountered played out as described above, a few did indeed contain malware.
In this example, the email appears to come from an internet service provider, informing us that our monthly bill is ready.
An invoice appears to be attached, stored within a .zip file. If the recipient opens it and double clicks the file within, a command prompt appears.
This may seem unusual to the recipient, especially since no invoice appears, but by this point it’s too late. The file contains a script that launches PowerShell and attempts to download a remote file.
While the remote file was no longer available at the time of analysis, there is a high likelihood it was malicious. But even though we were unable to determine its contents, Secure Malware Analytics flagged the script execution as malicious.
Knowing about prevalent scams, especially during the holiday season, is a first step in guarding against them. Granted the bad actors who distribute these spam campaigns do everything they can to make their scams look legitimate.
Fortunately, there are several things that you can do to identify scams and defend against them:
But even the best of us can be fooled, and when overseeing a large operation it’s more a matter of when, rather than if, someone clicks on the wrong link. There are elements of the Cisco Secure portfolio that can help for when the inevitable happens.
Cisco Secure Malware Analytics is the malware analysis and malware threat intelligence engine behind all products across the Cisco Security Architecture. The system delivers enhanced, in-depth, advanced malware analysis and context-rich intelligence to help better understand and fight malware within your environments. Secure Malware Analytics is available as a standalone solution, as a component in other Cisco Security solutions, and through software-as-a-service (SaaS) in the cloud, on-premises, and hybrid delivery models.
Cisco Secure Email protects against fraudulent senders, malware, phishing links, and spam. Its advanced threat detection capabilities can uncover known, emerging, and targeted threats. In addition, it defends against phishing by using advance machine learning techniques, real time behavior analytics, relationship modeling, and telemetry that protects against identity deception–based threats.
Cisco Umbrella unifies multiple security functions in a single cloud service to secure internet access. By enforcing security at the DNS layer, Umbrella blocks requests to malware before a connection is even established—before they reach your network or endpoints. In addition, the secure web gateway logs and inspects all web traffic for greater transparency, control, and protection, while the cloud-delivered firewall helps to block unwanted traffic.
Cisco Secure Endpoint is a single-agent solution that provides comprehensive protection, detection, response, and user access coverage to defend against threats to your endpoints. The SecureX platform is built into Secure Endpoint, as are Extended Detection and Response (XDR) capabilities. With the introduction of Cisco Secure MDR for Endpoint, we have combined Secure Endpoint’s superior capabilities with security operations to create a comprehensive endpoint security solution that dramatically decreases the mean time to detect and respond to threats while offering the highest level of always-on endpoint protection.
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
Ampol has been Australia’s leading transport fuel company since 1900. What began over 125 years ago is now an organization that powers a country, operating 1,500 retail stores and stations across ANZ, plus 89 depots for refining and importing fuels and lubricants, and 8,200 employees throughout Australia, New Zealand, the United States, and Singapore. And while Ampol’s history goes back a century, they are a modern organization, using internet of things (IoT) technology across operational and retail locations, with sensors on everything from electric vehicle charging units to fuel tank gauges to transportation trucks to refrigeration units inside retail stores.
As a critical energy provider to a country of over 25 million people, Ampol’s security needed to match its evolving infrastructure. As Satish Chowdhary, Network Enterprise Architect, said, “At Ampol, we have implemented sensor technology across our network: from gauges in the fuel tanks to monitor fuel quality and quantity to sensors that monitor the temperature in various refrigerators across our retail sites to ensure goods stay chilled. It’s critical to manage these devices effectively and securely, and that’s where Cisco comes in…With IoT, a major security risk is posed by dodgy legacy devices left unpatched and vulnerable within your network. Cisco’s TrustSec and VLAN segregation automatically isolate vulnerable devices, not exposing the rest of the network to risks from untrusted devices.”
In addition to securing the IoT that let’s Ampol monitor and manage its critical operations, Cisco was able to create a comprehensive security environment that solved for their three strategic goals.
“Three key components of our cyber-resilient strategy were isolation, orchestration, and rapid recovery. Cisco SecureX nailed all three providing us a single interface to see all security events, and malicious files, thus expediting how fast we can isolate events and recover,” Chowdhary explained. “Before using Cisco Secure, security was a hindrance, not an enabler for our IT team, employees, and even customers,” he added.
In fact, Cisco Secure helped Ampol improve their security posture so much that they were able to quickly pivot during the early days of the pandemic.
“When Covid triggered supply challenges during lockdowns, people not being able to access groceries turned to their local service station convenience stores to get what they needed. For Ampol, maintaining that supply continuity was critical, not just for our business, but for the customers who were relying on us to get their supplies. And all of this was done when many employees were now having to work remotely… This was possible only because we could maintain our revamped locations, staff, clients, and business partners safe on our network – while still maintaining speed and efficiency. Cisco Secure was the ticket to Ampol’s resilience in the face of major change,” Chowdhary said.
In addition to enabling flexibility against supply chain fluctuations, Ampol is readily protected against threats, cyberattacks, and other vulnerabilities. Their Cisco security solution included:
“The major force for our Cisco Secure investment was simplification by integrating the entire Security portfolio…If we ever happen to have a cyber-attack, we can quickly find it and contain it,” Chowdhary said, adding, “The greatest outcome of using Cisco Secure is simplicity at its core. We achieved great efficiency integration, better visibility, and context that’s not hidden across five, ten, or fifteen consoles, and ultimately, greater security outcomes.”
To find out how else Cisco Secure is helping protect Ampol against sophisticated threats and other challenges, read the full Ampol case study.
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
In part one of our Black Hat USA 2022 NOC blog, we discussed building the network with Meraki:
In this part two, we will discuss:
Cisco is a Premium Partner of the Black Hat NOC, and is the Official Wired & Wireless Network Equipment, Mobile Device Management, DNS (Domain Name Service) and Malware Analysis Provider of Black Hat.
Black Hat USA is my favorite part of my professional life each year. We had an incredible staff of 20 Cisco engineers to build and secure the network. Also, for the first time, we had two Talos Threat Hunters from the Talos Incident Response (TIR) team, providing unique perspectives and skills to the attacks on the network. I really appreciated the close collaboration with the Palo Alto Networks and NetWitness team members. We created new integrations and the NOC continued to serve as an incubator for innovation.
We must allow real malware on the network for training, demonstrations, and briefing sessions; while protecting the attendees from attack within the network from their fellow attendees and prevent bad actors using the network to attack the Internet. It is a critical balance to ensure everyone has a safe experience, while still being able to learn from real world malware, vulnerabilities, and malicious websites. So, context is what really matters when investigating a potential attack and bringing so many technologies together in SecureX really accelerated investigation and response (when needed).
All the Black Hat network traffic was supported by Meraki switches and wireless access points, using the latest Meraki gear donated by Cisco. Our Meraki team was able to block people from the Black Hat network, when an investigation showed they did something in violation of the attendee Code of Conduct, upon review and approval by the Black Hat NOC leadership.
Cisco Secure provided all the domain name service (DNS) requests on the Black Hat network through Umbrella, whenever attendees wanted to connect to a website. If there is a specific DNS attack that threatened the conference, we supported Black Hat in blocking it to protect the network. However, by default, we allow and monitor DNS requests to malware, command and control, phishing, crypto mining, and other dangerous domains, which would be blocked in a production environment. That balance of allowing cybersecurity training and demos to occur, but ready to block when needed.
In addition to the Meraki networking gear, Cisco Secure also shipped an Umbrella DNS virtual appliance to Black Hat USA, for internal network visibility with redundancy. The Intel NUC containing the virtual appliance also contained the bridge to the NetWitness on-premises SIEM, custom developed by Ian Redden.
We also deployed the following cloud-based security software:
We analyzed files that were downloaded on the network, checking them for malicious behavior. When malware is downloaded, we confirm it is for a training, briefing or demonstration, and not the start of an attack on attendees.
During an investigation, we used SecureX to visualize the threat intelligence and related artifacts, correlating data. In the example below, an attacker was attempting remote code execution on the Registration Servers, alerted by the Palo Alto team, investigated by the NOC threat hunters, and blocked by order of the NOC leadership upon the results of the investigation.
Cisco Secure Threat Intelligence (correlated through SecureX)
Donated Partner Threat Intelligence (correlated through SecureX)
Open-Source Threat Intelligence (correlated through SecureX)
Continued Integrations from past Black Hat events
New Integrations Created at Black Hat USA 2022
The NOC partners, especially NetWitness and Palo Alto Networks, were so collaborative and we left Vegas with more ideas for future integration development
Creating Custom Meraki Dashboard Tiles for SecureX, by Matt Vander Horst
One of the biggest benefits of Cisco SecureX is its open architecture. Anyone can build integrations for SecureX if they can develop an API with the right endpoints that speak the right language. In the case of SecureX, the language is the Cisco Threat Intelligence Model (CTIM). As mentioned above, Cisco Meraki powered Black Hat USA 2022 by providing wired and wireless networking for the entire conference. This meant a lot of equipment and users to keep track of. To avoid having to switch between two different dashboards in the NOC, we decided to build a SecureX integration that would provide Meraki dashboard tiles directly into our single pane of glass: SecureX.
Building an integration for SecureX is simple: decide what functionality you want your integration to offer, build an internet-accessible API that offers those functions, and then add the integration to SecureX. At Black Hat, our Meraki integration supported two capabilities: health and dashboard. Here’s a summary of those capabilities and the API endpoints they expect:
| Capability | Description | API Endpoints |
| Health | Enables SecureX to make sure the module is reachable and working properly. | /health |
| Dashboard | Provides a list of available dashboard tiles and, after a tile is added to a dashboard, the tile data itself. | /tiles
/tile-data |
With our capabilities decided, we moved on to building the API for SecureX to talk to. SecureX doesn’t care how you build this API if it has the expected endpoints and speaks the right language. You could build a SecureX-compatible API directly into your product, as a serverless Amazon Web Services (AWS) Lambda, as a Python script with Django, and so on. To enable rapid development at Black Hat, we chose to build our integration API on an existing Ubuntu server in AWS running Apache and PHP.
After building the API framework on our AWS server, we had to decide which dashboard tiles to offer. Here’s what we ended up supporting:
| Tile Name | Description |
| Top Applications | Shows the top 10 applications by flow count |
| Client Statistics | Shows a summary of clients |
| Top SSIDs by Usage in GB | Shows the top 10 SSIDs by data usage in GB |
| Access Point Status | Shows a summary of access points |
Finally, once the API was up and running, we could add the integration to SecureX. To do this, you need to create a module definition and then push it to SecureX using its IROH-INT API. After the module is created, it appears in the Available Integration Modules section of SecureX and can be added. Here’s what our module looked like after being added to the Black Hat SecureX instance:
After adding our new tiles to the SecureX dashboard, SecureX would ask our API for data. The API we built would fetch the data from Meraki’s APIs, format the data from Meraki for SecureX, and then return the formatted data. Here’s the result:
These dashboard tiles gave us useful insights into what was going on in the Meraki network environment alongside our existing dashboard tiles for other products such as Cisco Secure Endpoint, Cisco Umbrella, Cisco Secure Malware Analytics, and so on.
If you want to learn more about building integrations with SecureX, check out these resources:
Talos Threat Hunting, by Jerzy ‘Yuri’ Kramarz and Michael Kelly
Black Hat USA 2022 was our first fully supported event, where we deployed an onsite threat hunting team from Talos Incident Response (TIR). Our colleagues and friends from various business units, connected by SecureX integration, granted us access to all the underlying consoles and API points to support the threat hunting efforts enhanced by Talos Intelligence.
The threat hunting team focused on answering three key hypothesis-driven questions and matched that with data modelling across all of the different technology stacks deployed in Black Hat NOC:
To answer the above hypothesis, our analysis started with understanding of how the network architecture is laid out and what kind of data access is granted to NOC. We quickly realized that our critical partners are key to extending visibility beyond Cisco deployed technologies. Great many thanks go to our friends from NetWitness and Palo Alto Networks for sharing full access to their technologies, to ensure that hunting did not stop on just Cisco kit and contextual intelligence could be gathered across different security products.
Daily threat hunt started with gathering data from Meraki API to identify IP and DNS level requests leaving the devices connected to wireless access points across entire conference. Although Meraki does not directly filter the traffic, we wanted to find signs of malicious activity such as DNS exfiltration attempts or connections to known and malicious domains which were not part of the class teaching. Given the level of access, we were then able to investigate network traffic capture associated with suspicious connections and check for suspected Command and Control (C2) points (there were a few from different threat actors!) or attempts to connect back to malicious DNS or Fast Flux domains which indicated that some of the attendee devices were indeed compromised with malware.
That said, this is to be expected given hostility of the network we were researching and the fact that classroom environments have users who can bring their own devices for hands-on labs. SecureX allowed us to quickly plot this internally to find specific hosts which were connecting and talking with malicious endpoints while also showing a number of additional datapoints which were useful for the investigation and hunting. Below is one such investigation, using SecureX threat response.
While looking at internal traffic, we have also found and plotted quite a few different port-scans running across the internal network. While not stopping these, it was interesting to see different tries and attempts by students to find ports and devices across networks. Good thing that network isolation was in place to prevent that! We blurred out the IP and MAC addresses in the image below.
Here is another example of really nice port scan clusters that were running across both internal and external networks we have found. This time it was the case of multiple hosts scanning each other and looking to discovery ports locally and across many of the Internet-based systems. All of that was part of the class but we had to verify that as it looked quite suspicious from the outset. Again, blurred picture for anonymity.
In a few instances, we also identified remarkably interesting clear-text LDAP traffic leaving the environment and giving a clear indicator of which organization the specific device belonged to simply because of the domain name which was requested in the cleartext. It was quite interesting to see that in 2022, we still have a lot of devices talking clear text protocols such as POP3, LDAP, HTTP or FTP, which are easy to subvert via Man-In-The-Middle type of attacks and can easily disclose the content of important messages such as email or server credentials. Below is an example of the plain text email attachments, visible in NetWitness and Cisco Secure Malware Analytics.
In terms of the external attacks, Log4J exploitation attempts were pretty much a daily occurrence on the infrastructure and applications used for attendee registration along with other typical web-based attacks such as SQL injections or path traversals. Overall, we saw a good number of port scans, floods, probes and all kind of web application exploitation attempts showing up daily, at various peak hours. Fortunately, all of them were successfully identified for context (is this part of a training class or demonstration) and contained (if appropriate) before causing any harm to external systems. Given the fact that we could intercept boundary traffic and investigate specific PCAP dumps, we used all these attacks to identify various command-and-control servers for which we also hunted internally to ensure that no internal system is compromised.
The final piece of the puzzle we looked to address, while threat hunting during Black Hat 2022, was automation to discover interesting investigation avenues. Both of us investigated a possibility of threat hunting using Jupyter playbooks to find outliers that warrant a closer look. We have created and developed a set of scripts which would gather the data from API endpoints and create a data frames which could be modeled for further analysis. This allowed us to quickly gather and filter out systems and connections which were not that interesting. Then, focus on specific hosts we should be checking across different technology stacks such as NetWitness and Palo Alto.
Unmistaken Identity, by Ben Greenbaum
An unusual aspect of the Black Hat NOC and associated security operations activities is that this is an intentionally hostile network. People come to learn new tricks and to conduct what would in any other circumstance be viewed rightfully as malicious, unwanted behavior. So, determining whether this is “acceptable” or “unacceptable” malicious behavior is an added step. Additionally, this is a heavily BYOD environment and while we do not want attendees attacking each other, or our infrastructure, there is a certain amount of suspicious or indicative behavior we may need to overlook to focus on higher priority alerts.
In short, there are broadly speaking 3 levels of security event at Black Hat:
When Umbrella alerted us (via a SecureX orchestration Webex workflow) of DNS requests for a domain involved in “Illegal Activity” it was reminiscent of an event at a previous conference where an attendee was caught using the conference network to download forged vaccination documents.
Using the Cisco Secure Malware Analytics platform’s phishing investigation tools, I loaded and explored the subject domain and found it to be a tool that generates and provides pseudo-randomized fake identities, customizable in various ways to match on demographics. Certainly, something that could be used for nefarious purposes, but is not illegal in and of itself. Physical security and access control is, however, also important at Black Hat, and if this activity was part of an effort to undermine that, then this was still a concern.
This is, however, also the kind of thing that gets taught at Black Hat…
Using the reported internal host IP from Umbrella, Meraki’s connection records, and the Meraki access point map, we were able to narrow the activity down to a specific classroom. Looking up what was being taught in that room, we were able to confirm that the activity was related to the course’s subject matter
Network owners and administrators, especially businesses, typically don’t want their network to be used for crimes. However, here at Black Hat what some would consider “crimes” is just “the curriculum”. This adds a layer of complexity to securing and protecting not just Black Hat, but also Black Hat attendees. In security operations, not every investigation leads to a smoking gun. At Black Hat, even when it does, you may find that the smoking gun was fired in a safe manner at an approved target range. Having the right tools on hand can help you make these determinations quickly and free you up to investigate the next potential threat.
25 Years of Black Hat – Musings from the show (and some DNS stats), by Alejo Calaoagan
Back in Singapore, I wrote about cloud app usage and the potential threat landscape surrounding them. My original plan at Black Hat USA was to dig deeper into this vector to see what interesting tidbits I could find on our attendee network. However, given that this was the 25th anniversary of Black Hat (and my 14th in total between Vegas, Singapore, and London), I’ve decided to pivot to talk about the show itself.
I think it’s safe to say, after two difficult pandemic years, Black Hat is back. Maybe it’s the fact that almost everyone has caught COVID by now (or that a lot of people just stopped caring). I caught it myself at RSA this year back in June, the first of consecutive summer super spread events (Cisco Live Vegas was the following week). Both of those shows were in the 15-18k attendee range, well below their pre-pandemic numbers. Black Hat USA 2022 was estimated at 27,000 attendees.
If I remember correctly, 2019 was in the 25-30K range. Last year in Vegas, there were ~3,000 people at the event, tops. 2021 in London, was even lower…it felt like there were less than 1,000 attendees. Things certainly picked up in Singapore (2-3k attendees), though that event doesn’t typically see attendee numbers as high as the other locations. All in all, while the pandemic certainly isn’t over, Las Vegas gave glimpses of what things were like before the “Rona” took over our lives.
The show floor was certainly back to the norm, with swag flying off the countertops and lines for Nike sneaker and Lego giveaways wrapping around different booths. The smiles on people’s faces as they pitched, sold, hustled, and educated the masses reminded me how much I missed this level of engagement. RSA gave me this feeling as well, before COVID sidelined me midway through the show anyway.
Not everything was quite the same. The Black Hat party scene certainly is not what it used to be. There was no Rapid 7 rager this year or last, or a happy hour event thrown by a security company you’ve never heard of at every bar you walk by on the strip. There were still some good networking events here and there, and there were some awesomely random Vanilla Ice, Sugar Ray, and Smashmouth shows. For those of you familiar with Jeremiah Grossman’s annual Black Hat BJJ throwdown, that’s still, thankfully, a thing. Hopefully, in the coming years, some of that old awesomeness returns….
Enough reminiscing, here are our DNS numbers from the show:
From a sheer traffic perspective, this was the busiest Black Hat ever, with over 50 million DNS requests made…
Digging into these numbers, Umbrella observed over 1.3 million security events, including various types of malware across the attendee network. Our threat hunting team was busy all week!
We’ve also seen an increase in app usage at Black Hat:
In a real-world production environment, Umbrella can block unapproved or high-risk apps via DNS.
The increases in DNS traffic volume and Cloud App usage obviously mirrors Black Hat’s return to the center stage of security conferences, following two years of pandemic uncertainty. I’m hopeful that things will continue to trend in a positive direction leading up to London and, hopefully, we’ll see you all there.
——
Hats off to the entire NOC team. Check out Black Hat Europe in London, 5-8 December 2022!
Acknowledgements: Special thanks to the Cisco Meraki and Cisco Secure Black Hat NOC team.
SecureX threat response, orchestration, device insights, custom integrations and Malware Analytics: Ian Redden, Aditya Sankar, Ben Greenbaum, Matt Vander Horst and Robert Taylor
Umbrella DNS: Christian Clasen and Alejo Calaoagan
Talos Incident Response Threat Hunters: Jerzy ‘Yuri’ Kramarz and Michael Kelley
Meraki Systems Manager: Paul Fidler (team leader), Paul Hasstedt and Kevin Carter
Meraki Network Engineering: Evan Basta (team leader), Gregory Michel, Richard Fung and CJ Ramsey
Network Design and Wireless Site Survey: Jeffry Handal, Humphrey Cheung, JW McIntire and Romulo Ferreira
Network Build/Tear Down: Dinkar Sharma, Ryan Maclennan, Ron Taylor and Leo Cruz
Critical support in sourcing and delivering the Meraki APs and switches: Lauren Frederick, Eric Goodwin, Isaac Flemate, Scott Pope and Morgan Mann
Also, to our NOC partners NetWitness (especially David Glover), Palo Alto Networks (especially Jason Reverri), Lumen, Gigamon, IronNet, and the entire Black Hat / Informa Tech staff (especially Grifter ‘Neil Wyler’, Bart Stump, Steve Fink, James Pope, Jess Stafford and Steve Oldenbourg).
Read Part 1:
Black Hat USA 2022: Creating Hacker Summer Camp
About Black Hat
For 25 years, Black Hat has provided attendees with the very latest in information security research, development, and trends. These high-profile global events and trainings are driven by the needs of the security community, striving to bring together the best minds in the industry. Black Hat inspires professionals at all career levels, encouraging growth and collaboration among academia, world-class researchers, and leaders in the public and private sectors. Black Hat Briefings and Trainings are held annually in the United States, Europe and USA. More information is available at: blackhat.com. Black Hat is brought to you by Informa Tech.
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
FreshRSS 