Blog

Cisco recently announced a slew of vunerabilities, including three 10 out of 10 vulnerabilities in their popular small business line of VPN routers. Some of the devices vulnerable are out of support, no updates will be available. 10 out of 10 means it is simple to exploit, without privilege, remotely. And, this is not a good thing for the device that protects your network from the world, the single bastion.

In 2010 Forrester Research published an article called ‘No More Chewy Centers: Introducing The Zero Trust Model Of Information Security’. The thesis was that network design was currently based on the concept of an M&M: a hard outer shell, a soft interior. And, that this was no longer sufficient since threats could get in, could walk sideways once in.

Like many research articles it spawned debate, discussion, but, immediately, not a lot of action. People continued to slap a single security device at the perimiter, use it to VPN to the interior. They would then ‘harden’ the access, forcing things like multi-factor for remote access, assuming that all threats came in from the outside. The walled garden surrounded by the desolate desert.

Recently Cisco has announced a list of vulnerabilities affecting common business routers. Notable in this list is 3 vulnerabilities rated 10 out of 10 (see the CVSS calculator to understand what this means). And, this got me to thinking. What are the vulnerabilities? How would they be exploited in a way that demonstrates the fallacy of the castle-and-moat security?

Well first, one of the vulnerabilities is in the SSL layer. To exploit it you do not need to have credentials, merely layer 3 access. Since these Cisco devices are typically the access router + VPN, they will indeed be network accessible. Let’s do a quick Shodan.io search. OK, confirmed. These devices inhabit the Internet. This means that your VPN is indeed now the magic carpet into that ‘Chewy Centre’, no questions asked, courtesy of the Remote Code Execution flaw.

Second, one of the vulnerabilities is in the management web UI. Let’s assume we have one of these devices solely inside our network. No inbound access from the Internet. This must be safe, right? Well, let’s hypothesise for a second. Do you micro-segment your internal network? If you were a valid user sitting at a desk would you have access to this router? The answer is nearly certainly yes, why not? Now, I wonder if I could entice one of your users to go to a web site I controlled? Maybe via spear-phishing, a doppelganger domain, some social engineering attack, an advertisement, whatever? Now, on that site I can place some JavaScript. And it can guess the name of your router. Is it ‘rtr’? ‘router’? etc. And, that JavaScript runs as you, in your browser, inside your network. Inside the Chewy bit. It has direct access sideways to this ‘protected’ router.

That’s right, you became the ‘bounce’ point, the pawn, in this game of lateral traversal. So, the inbound firewall did nothing to protect the ‘damaged’ router. Now the attacker has run some code on it, has a vantage point in your network to move sideways.

Now, you might be thinking, my network team will have patched this already, right? But, some of these devices do not have (and will not have) patches available. This is a general industry problem: embedded devices tend to run for longer than their support. And, you have a lot of embedded devices around (thermostats, smart TV, PLC’s and SCADA in manufacturing, etc.) Any one of these could have one or more vulnerabilities this critical right now. Worse, there is not a systematic publishing of vulnerabilities for the embedded space. We are talking about the Cisco ones since the info is public, what about the one we don’t know about? Someone does.

OK, now that I’ve explained a couple of the risks of the fallacy of the ‘hard outside, soft inside’ perimiter approach, what would I suggest? Well, a generic strategy of Defense In Depth. Assume you have no single strong defense, but a collection of defenses. Instead of the Maginot Line, you have a set of fallback and delay approachs. Think about limiting the ‘blast radius’. Think in terms of “when the bad thing occurs, what is my next defense” rather than “i must prevent the bad thing at all costs.”.

Here are ten suggestions to chew on:

• Multi-Factor Authentication should be for all users (not just admins)
• Multi-Factor Authentication should be for all devices (not just personal)
• Multi-Factor Authentication should be for all networks (not just external or remote)
• Use privileged access management (no layer 3 access should be possible for unauthenticated users to your infrastructure)
• Use zero-trust concepts like identity<->resource pairwise authorisation
• Have sensors, logs, tripwires so you can answer the “where did it go next?”
• Move devices and longer lived things to their own networks
• Use Private VLAN / Client Isolation to prevent devices talking to each other
• Use a Web Application Firewall as a terminating SSL gateway to prevent SSL overruns
• Call me and discuss

Yet another internet-breaking zero-day vulnerability, Log4Shell, has hit the wire and likely sent your IT teams into a patching frenzy, or at least it should have. Log4Shell is creating attack vectors just about everywhere and likely triggered a patch-a-thon for your technology teams as they secured important business systems. But what about operational technology like the HVAC or smart thermostat? 

Ignoring systems that may be deemed ‘unimportant’ in comparison to your revenue-generating technology stack will leave your organization open to compromise from the Log4Shell vulnerability. There are a number of operational devices embedded into an organization that can be compromised through Log4Shell and could become the source of lateral network traversal by a malicious actor. 

What is Log4Shell? 

Log4j is a very widely used logging ‘library’ for Java-based software. Through this library, an attacker can send crafted log messages that are able to execute any arbitrary code (spyware, malware, ransomware, etc.). This zero-day vulnerability has been named Log4Shell, identified as CVE-2021-44228.

The Log4j library is currently under the purview of the Apache Foundation. The library can be added to any Java-based software for the purpose of logging application and user inputs. The library is found in many different types of applications, servers, and services, most specifically enterprise software, IoT devices, and other external/internet-facing applications. That means everything from financial companies, to hospitality, or manufacturing and industrial operations could be vulnerable or even compromised already.

While Apache has released a patch, the vulnerability is still affecting very commonly used server software such as Apache Solr, Apache Druid, Elasticsearch, Apache Dubbo, and VMware vCenter. Nearly the entire big-data stack of self-hosted and managed operators runs this software (Spark, Druid, Hadoop, Elasticsearch), which means that ingested data could also be a risk, bypassing the firewall since it’s inserted by staff.

How it Works

What makes the Log4Shell vulnerability so concerning is that wherever an expression is invoked, a lookup mechanism can find and replace the value contained, which can then be used to run any executable file. Unfortunately, there are a variety of lookups supported by Log4j (JNDI, SYS, ENV, JAVA, lower, and upper). 

For example, an attacker can use the JNDI expressions in the logs to execute an attack where they make an HTTP request to a malicious web server. The request to the malicious web server can then download and execute a malicious payload. This example is one of the most common ways this vulnerability has so far been exploited. 

Essentially, Log4Shell can be used to execute some arbitrary code without checking who wrote it, where it came from, and what it is going to do, leaving servers, networks, and devices open to compromise. 

The popularity of Log4j means malicious actors have a huge attack surface including:

• Network headers which get logged
• URLs, filename, paths which get logged
• Instant message routing, in-game purchases, user-names, comments in blogs, and more

Nearly any field which can be set by any user can be exploitable.

Why You Should Be Concerned

There are very few companies around the world that aren’t affected by the Log4Shell vulnerability. The prevalence of the Log4j library is contributing to the panic we are seeing, especially as organizations like Amazon Web Services, Broadcom, Cisco, ConnectWise, Fortinet, IBM, Okta, VMware have been impacted.

As these major organizations ensure the patches are in place and your IT team has told you “don’t worry”, or that you “are good”, ask yourself, “did anybody check the HVAC?” What about the IoT thermostat? How about the smart projector, printer, or router?

Protecting against the Log4Shell vulnerability goes beyond updating HTTP server headers. Technology teams need to take a holistic approach to review every application or device in the organization that could be using the Log4j Java library and what it is connected to.

There is a huge universe of products, devices, and services we use daily that are also exposed to the Log4Shell vulnerability. Aside from cranking the thermostat up or down, the risk to an organization comes in when these devices are connected to or able to connect to an internal corporate network. Oftentimes these devices are a cybersecurity afterthought and can leave an organization vulnerable to lateral network traversal attacks. 

In the case of the Log4Shell vulnerability, an unpatched afterthought like a printer or other operational device can be used to mount a man-in-the-middle attack and gain network access. After an attacker gains network access they will be able to traverse your network, snoop, steal, damage, and execute malicious software like ransomware. The net effect is that your network is crippled, your resources are compromised, and the business might find itself facing down a ransomware threat.

Another example of where Log4Shell can be used to gain network access is through any desktop software within your firewall or on your VPN is exploited. If a user can be influenced to open a URL or path with this magic string jndi://ldap, you may suddenly find yourself under attack from the inside.

So when you hear, “we’re good”, it might be appropriate to think twice about those cybersecurity afterthoughts and ask your team, “Has anyone checked the Smart Thermostat?”

What You Should Do

Evaluating your organization’s tech stack beyond the technology powering business activities, and keeping other devices in mind is a good step. The technology powering your facilities could very well be the doorway into your corporate network and if it is vulnerable to Log4Shell, malicious code could be easily executed on your network.

What should you do? Patching this zero-day vulnerability is of critical importance. If it can’t be patched, any exposed devices should be isolated from your network. You might also think twice about stopping at simply implementing a patch and instead start thinking about how you can harden your cyber posture with tools like Web Application Firewalls, or fine-grained Access Controls.

Another step in improving your resilience to these types of cyber risk events is ensuring your IT organization is using modern security frameworks like resource segmentation, Zero-Trust, Multi-Factor Authentication, containerized servers, and egress firewalls. Finally, having a great audit system could help you identify whether you’ve had an issue already and take the necessary steps to mitigate the event.

The depth of risk that Log4shell creates is showing the world that modern security problems require modern cybersecurity solutions. Advanced cybersecurity services can help you mitigate the risk of these types of zero-day attacks, which are only increasing in frequency with new ones discovered from one week to the next.

Agilicus helps organizations of all sizes implement advanced cybersecurity measures that are cost-accessible, easy to deploy, and introduce identity-based access control without the need for a VPN.

Two hikers see a bear. One bends over to tie shoes. Other says, you can’t out-run a bear. First says, just need to out-run you. Pause laughter.

OK, this is the subject of a long running gag, but the lesson is sound. You don’t need to be perfect to be successful. Perfect is the enemy of good enough.

To bring this back to Cyber Security. Let’s talk about consequences. Ransomware. Its very successful, it extracts $10’s of thousands of dollars from a lot of companies. And, this amount is too small to be worth it for law enforcement to deal with the hassle of international communication and warrants etc. So, there are no (not much) consequences for the bad actors.

Now here’s a little anecdote for you. A large percentage of the worlds malware will not activate if the computer has the Russian language installed. Why? presumably since the miscreants are based there and don’t want to face the wrath of local authorities.

So, if we wanted to protect our country (in my case Canada), we can go down the path of hardening everything, training, etc. And its a good path. But what about another complementary idea? What if we made it a zero-tolerance zone for cyber crime? Even if the cost of policing was very high? We would make the country as a whole less appetising. If the RCMP would investigate and prosecute every cyber crime, from $1 upwards, no quarter given. It would take a lot of funding, but it would mean there was systematically consequences in Canada for cyber crime, regardless of jurisdiction, regardless of how long it took for the long arm of the law to arrive.

This approach would tilt the playing field. Canada would become the slightly faster runner. Sure the bear would still feast on the slower runners, sure the bear could still theoretically catch us, but, well, we would be quite a bit less appetising.

I’ll leave you with this link.

Multi-Factor-Authentication. 2 or more distinct of I AM. I HAVE. I KNOW. Much strength. Little cost. You would think *money* would be the first thing that had multi-factor, no?

A few years ago I wrote of the trials and tribulations of trying to get a Canadian Bank to give me multi-factor. TL;DR: for some transactions you can get a SecureID fob, but not for all, not for login. And, you can recover the password of a file w/ the last balance, so the login is enough to totally destroy all defenses.

So it was with some excitement that i saw that the Royal Bank of Canada had implemented a push-based multi-factor using their mobile App. Except… it neither works, nor is secure. It is what I would refer to as “Security Theatre”.

First, the doesn’t work. It will not send me a notification. No chirps, not a beep. Android 12, Pixel 4. So, when you try and login to the web site using the desktop you get this first message:

Seems promsing. But, I did not get the notification. Am i blocked? Wait, no? What are my options then? Surely it cannot be to just flat out bypass the multi-factor authentication, can it?

Of course it can. Any attacker that can get the first factor can simply bypass the second factor by saying “no thanks”. They then get this option (below).

OK, um, resend. I guess that is ok. As long as there is some limit to avoid e.g. a DoS attack.

Send a text message? I am not a fan of SMS, its not secure, but i guess its better than nothing, it would prove *someone* owned a sim card or knew how to hack SMS/SS7. So a bit better than nothing, a last resort.

But what’s this? A drivers license or passport? My passport was scanned in quite a lot of world hotels and airports. Its not “Something I Have” and its not “Something I AM”. Its “Something I and thousands of other people know”. Remember a few years ago when Starwood hotels was breached? And Marriott? Yeah, well, that data is permanently out there.

To add insult to injury, the “Personal Verification Question”. This is not a second factor. I’ve written about this before. You honestly think the name of my first high school is an inherently unknowable property of the universe?

So what is the intent of this “fake” multi-factor authentication system? It seems its to make things seem more secure, without actually doing anything. An item in a brochure saying “RBC has multi-factor”. But in practice, its just an irritant to the real users, and nothing much to the attackers.

Why is it in 2021 my Github, my Gmail, is more secure than my Bank? Google plans to roll this out by default to billions of people, is the average ability of those people less than the average bank customer?

Fix this. You collect $B in fees. Your profit keeps going up. When the breach occurs, you will blame me somehow, with your army of lawyers and agreements. But it was you that put this house of cards in place.

You are a small service company. You do work for your customers, you bill them for it. Suddenly, some miscreant sneaks in through your lightly secured infrastructure, sends them a change in billing information. Who is at fault? You for not securing? The customer for not checking? Well clearly the criminal owns some of the blame 🙂 These are the questions a judge in Mahoning County is being asked to solve in this lawsuit.

In BOARDMAN MOLDED PRODUCTS INC v INVOLTA LLC, the case is based on Boardman’s claim that Involta (their MSP) insufficiently secured its own (Involta’s) network, allowing a bad actor access to customer lists, and to send email with invalid invoices.

This plastic’s fabrication company then paid the invoices, totalling ~ $1.8M USD.

Involta was notified through a trouble ticket (and closed it as Fake News!).

So, where does the blame lie? Is Boardman responsible for overseeing all of their risk, including suppliers? Is Involta responsible for overseeing their own risk to their customers? Both?

Boardman hired Involta to handle complex technical matters, running a network, patching, etc, and thus relied on the advice and best practices of their supplier.

The details suggest there was no multi-factor authentication used, which is indeed a table-stakes best practice.

Interestingly, Boardman is sueing for the costs of dealing with the attack, but not the costs they paid out, so $25K rather than $1.8M.

The CompTIA 2021 National Survey of Local Government Cybersecurity and Cloud Initiatives is out. Its a (US) national survey of local government cyber-security programs, and gives you an idea of the strengths, weakness, strategies, and areas of increased spend ahead.

A few things stood out to me. The first was around Cyber Awareness Training.

Ninety-two percent of respondents state their jurisdiction provides employees with cyber awareness training – what to do and what not to do when it comes to Cybersecurity. Fiftynine percent state that training is provided on an on-going basis throughout the year; 34% state that training is provided once a year.

When asked if elected officials, their staff and senior leadership are exempted from awareness training, 24% responded yes.

CompTIA 2021 Survey

92% is a good number to be providing the training. But, 24% allow the most senior people, and their elected officials, to skip. This is super worrying, they have the easiest to guess email addresses, the most external mail, and, often the greatest internal ‘power’ or ‘security’. This means they are the most interesting target. Why would we exempt anyone from taking (and passing, and, re-taking if not passing)? It’s probably hard to tell the Mayor to take a course, but its certainly less embarrassing than explaining the bad outcome later.

Another thing that stood out to me was Cloud Computing adoption.

I’m a huge proponent. Properly done, your security goes up, your operational cost goes down, and you get a better experience for all. The cloud environments offer better security tools overall, from audit to DDoS to backup and disaster recovery.

But, that is a big chunk who have no plans, And a big chunk who are just starting.

Another thing stood out to me: lack of awareness of what to do following a breach. The vast majority are unsure. What is their plan, look it up on the locked down machines that the ransomware gang is busily ex filtrating your data from?

So, the good news, increasing progress, hardening, awareness in the public sector. There was no question around the use of multi-factor authentication, which is one I would love to know. What organisations require this for all users (and do not exempt anyone). What organisations require this on all networks (not just external)? And all applications (not just “important“)?

There exists a concept called Minimum Viable Product’. The intent is to find the least amount you can do that will be sell-able, and get that done as quickly as you can. The idea is to do what is needed, but not more, so that you can start gaining traction before others do. You later go back and fill in the areas that are not part of that absolute minimum. But what of security? Is there a Minimum Viable Secure Product?

Many would argue that constraints like ‘security’ get cut or ignored here (since the MVP tends to be feature-focused, value-focused). So, a few folks got together and defined what the “Minimum Viable Secure Product” would be.

They define a few categories: Business Controls, Application Design Controls, Application Implementation Controls, Operational Controls. I would spoil the content (check it out here), but a few specific areas warmed my heart:

• Content Security Policy: Set a minimally permissive Content Security Policy
• Single Sign On: Implement single sign-on using modern and industry standard protocols

One thing they missed… They suggest single-sign-on (and you should read this as OpenID Connect), but they allow for password authentication. And, here they miss multi-factor authentication as a requirement. So i’m conflicted. I think the Minimum Viable Secure Product should *not* allow local authentication at all (neither a first or a second factor), relying solely on a shared identity provider. But, if they do allow it, it for sure should require a second factor. Hmm.

Want to know more about Content Security Policy? See my video.

In the early innocent days of the Internet a few protocols reigned supreme. Telnet. SMTP. Both were architected with negligible security, it was a simpler time, people were more worried about the impending death of disco and where to buy leg warmers.

Fast forward 40 years. SMTP has been hardened via a variety of bolt-on solutions. SPF, DMARC, DKIM, TLS, etc. It now suffices. Telnet? Not so much. Open port 23 and start typing. So it should come as no surprise that I (and everyone else) suggests: get rid of it. You cannot secure it no matter how much fancy firewall and multi-factor authentication you put in front. And, this is not new advice, you’ve had years to delete this protocol from your fleet.

So it surprised me substantially today when, as I was explaining what Telnet was to someone who was not yet around in when Tron premiered and Telnet launched, I opened Shodan to show how dead it was… and… Sacre Bleu! There it is. Starting me in the face. Not only is Telnet alive, but, Canada is half of the worlds Telnet.

What can possibly be the reason the Great White North is a safe-haven for the devil’s port? TekSavvy and Primus are the majority. Perhaps they don’t filter port 23 and all the other worlds’ ISPs do? Perhaps Tim Horton’s uses it in Point-Of-Sale? What can be the reason?

When we look at risky protocols (like Telnet), we often think about ‘mitigation’. E.g. smart firewalls, forced multi-factor, nonce-based-one-time-passwords.

But when the underlying transport is insecure, the option negotiation is fraught with peril, the password is in the clear, and, well, SSH exists, there is just no reason.

Friends don’t let friends run Telnet. If you know why its still alive here, drop me a line.

Multi-factor authentication is your first line of defense in protecting your applications, websites, and data by requiring a second or more ‘factors’ to prove your identity, in order to grant access. Many use SMS as a second factor by sending a text with a login verification code. Understanding multi-factor authentication and how to best implement it is critical. Not all ‘factors’ are created equal.

A recent hack suggests that the method of presenting a second factor for authentication can be nearly as important as requiring multi-factor in the first place. Syniverse, a company you most likely never heard of, but likely unknowingly used, was compromised. Syniverse, quietly mentioned the breach in a couple of paragraphs of their 333-page SEC filing. Here is an excerpt from the filing:

“May 2021, Syniverse became aware of unauthorized access to its operational and information technology systems by an unknown individual or organization (the “May 2021 Incident”)…the unauthorized access began in May 2016…individual or organization gained unauthorized access to databases within its network on several occasions, and that login information allowing access to or from its Electronic Data Transfer (“EDT”) environment was compromised for approximately 235 of its customers.”

In short, Syniverse was compromised for five years and affected 235 customers. And according to the company, the compromises had no impact on day-to-day operations. No harm, no foul. Again, who cares? 235 customers is not many; however, it includes all major US carriers, servicing more than 340 million Americans. Syniverse’s global customers are the largest mobile carriers (think AT&T, China Mobile, Vodafone) and through Syniverse’s inter-connection services, they reach seven billion mobile devices and process four billion transactions daily. SMS, MMS and information like call records (‘to’, ‘from’, location) and billing particulars may have been exposed.

If you use SMS for multi-factor authentication, it may have been compromised through this 3rd party. So sure, Syniverse’s day-to-day operations were not impacted and their customers were notified, but what about their customers’ customers?

The Syniverse May 2021 Incident is troubling: a third party may have exposed your private information and they only felt the need to mention it because of a corporate transaction. This example highlights the vulnerability of text messaging as a factor of authentication.

SMS is a popular choice as a second factor; it’s accessible by everyone, but consequently it is the least secure choice. There are many reasons why you should not choose SMS as a second factor. For starters, it is not encrypted. But it is better than nothing at all. We may never know the extent and nature of the Syniverse breach but it’s likely not as benign as implied in the filing. However, turning on multi-factor authentication is a must.

Learn how to add multi-factor authentication to older legacy applications.

Multi-factor authentication is combining 2 or more of something you know (password, pin), something you have (usb key, phone, …) and something you are (fingerprint, etc.). It dramatically lowers your risk if you require at least 2 different types from this list. But why?

Let’s first talk about the risks for losing control of one of these factors. And, demonstrate how they are uncorrelated, independent risks.

First, the thing you are. Your fingerprint, your face. What are the risks you lost control of that feature? If you do, you normally have higher risks. Some of these can be mitigated (e.g. retina scanners that ensure blood is flowing, temperature sensors for fingerprints). But normally we consider these outside the scope.

Second, the thing you have. Your phone, a USB key. How do you lose this? Someone has to be physically near you. They break in to your house, etc. This pool of people is relatively small.

Third, the thing you know. If you use a different password on every site, then it comes down to keyloggers or single-site-breaches. If you share the password across sites, the risk goes up. The risk here is global, anyone, anywhere.

Now, lets examine the risks across these. The criminal gang somewhere that has purchased a dump of passwords from some old forum you used, they have no overlap with the burglar that stole your phone. And this is the key. Its not that each factor is e.g. 50% likely to be lost, so 2 factor is 1/4 likely to be lost. Its exponential, they are not correlated.

Multi-factor, as long as you use 2 of the above, is very strong, and very simple.

What to watch out for? Don’t get tricked into using 2 factors from the same category. A common error is using SMS, thinking its something you have (phone). But, actually, that SIM card in the phone is something you know… and, via sim-jacking, your phone number can be hijacked. Systems that use a password + pin… that is still just 2 things you know (same goes for personal verification questions).

What else to watch out for? lateral traversal. Multi-factor authentication is for all users, all environments. Its no something you use “outside the VPN”, or for “managers” or for “financial applications”. Don’t leave a door open for that criminal to wiggle in and walk sideways. Starve them.

I AM. I HAVE. I KNOW. The trifecta of simple and secure. Low cost, high return, all users, all applications.