Blog

  • Securely Updating Software: The Update Framework

    Securely Updating Software: The Update Framework

    Software. You know it will need updating. Every consumer product you have ever bought the first thing it did was to install the “real” version. How do we securely update software? Let’s talk about The Update Framework.

    Freshness is one of the battles in the software operations world. It seems like every piece of software requires loving hand feeding and care for hourly updates. Wouldn’t it be great if software could just care for itself and stay up to date? What’s that you say? You are concerned about auto-updating software? How will that be secure? Well, pull up a chair and let me tell you the tale of The Update Framework, achieving the goal of auto-update and secure together at last.

    Nearly all of our software is run in our cloud, by us. We use Dependabot and our CI and CD to keep it fresh. But one of our Features is a connector that runs on your site. And, we want that to be zero cost to you to keep up to date, safely, securely.

    The Update Framework is a CNCF project. It was originally envisaged for deeply embedded systems such as Automobiles (you can see Uptane). In a nutshell you create a set of signing keys:

    • root key: used to sign and rotate the other keys
    • target key: used to sign a new binary (target)
    • snapshot key: used to sign the set of binaries, e.g. “this is version 3, use these 9 files together”.
    • refresh key: used to show the repo itself is being updated

    In your software you ship to your customer site you put a library (there are implementations for several languages of The Update Framework). This periodically polls your online repo. If it finds there are updates, it pulls and runs them.

    For security the signing keys make sure that the clients at the customer site will ignore any tampering. But, to make this secure, you need to segregate and control those signing keys above. And this is where it gets complex.

    First, the root key, you want this offline only. You want to only access this once a year or so, when you rotate the other keys. It should be nearly impossible for a bad-actor to get it. I choose to cipher it with a multi-word passphrase. The passphrase I used said “Approximate Crack Time: 1,482,947,715,526,880,400 centuries”, so, well, that probably won’t be the method they get me. The key itself is strong (perhaps not quantum proof, but certainly strong enough). So the private key, ciphered w/ that passphrase, now we need to protect it. For this I have chosen not to use a Hardware Security Module (HSM). Instead, I’ve placed it on a pair of USB keys which I will place in secure locations.

    Now, use the root key to create the target/snapshot/refresh key. These are placed into Google Secret Manager, and then we use IAM to indicate who can access these, and when. This provides my audit trail.

    Our CI is taught to run a daily refresh job, it will handle the refresh job, we’ll deal with the target updates manually, when we decide what to publish.

    All in all, not too complex. Using the update framework we have obtained high integrity update chain, fully automatic, low effort for customer, low effort for us, low risk of tampering.

  • Embrace Failures: find the start times of Kubernetes pods

    Embrace Failures: find the start times of Kubernetes pods

    Cloud Native means embracing failures. At Agilicus, our strategy for security is Defense in Depth. In a nutshell, assume bad things will happen and have a fallback position, rather than dying on the hill of the first line. Similarly for reliability we assume Strength in Numbers. Rather than spending large time and money on a single infinitely reliable thing, we assume each component will fail, and have a strategy to make that invisible.

    A big part of our strategy is the Pre-emptible node. This means that the underlying machines our software runs on can be powered off without notice. This might sound like a bad thing, but, consider, kernels panic, hardware fails, its going to happen anyway. Would you rather make it infrequent enough you don’t know what to do when it happens? Or would you rather make it part of a normal workday and have a solution? Embrace Failure.

    The Agilicus strategy to embracing the failure involves Kuberentes, Istio, retries, etc. Many pieces. And sometimes one of those needs improving or investigating. And, the first question is always: “what restarted when”? For this, you would think, “kubectl -n namespace get pods“, right? Well, turns out that kubectl is a liar. It misses events like a node panic, showing an age which doesn’t correlate to when it started.

    But, all is not lost, kubectl is capable of telling you the truth, its just not the default. So, this small script below was invented. It uses the jsonpath output and a wee bit of formatting to make it beautiful readable.

    $ cat ~/bin/p-times 
    #!/bin/bash
    ns=$1
    
    (
    echo -e "Pod\tnodeName\tstartTime\tstartedAt"
    kubectl -n "$1" get pods -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.nodeName}{"\t"}{.status.startTime}{"\t"}{.status.containerStatuses[0].state.running.startedAt}{"\n"}{end}'
    ) | column -t
    

    Running it gives this output. Adding the nodeName makes it quite wide, feel free to remove that. We use it to correlate to node-logs in Elasticsearch if needed.

    $ p-times istio-system
    Pod                                      nodeName                                startTime             startedAt
    istio-citadel-849cc4bfb6-ksklm           gke-noc-noc-preempt-pool-356e72e3-heye  2021-01-15T15:13:55Z  2021-01-15T15:15:13Z
    istio-citadel-849cc4bfb6-l5cqs           gke-noc-noc-preempt-pool-bc2cfcf6-gmcm  2021-01-14T05:59:18Z  2021-01-14T21:59:04Z
    istio-galley-6476d5f4df-qx5q5            gke-noc-noc-preempt-pool-bc2cfcf6-gmcm  2021-01-14T05:59:15Z  2021-01-14T21:56:13Z
    istio-galley-6476d5f4df-x5vp5            gke-noc-noc-preempt-pool-356e72e3-heye  2021-01-15T15:13:54Z  2021-01-15T15:14:35Z
    istio-ingressgateway-6d7d9cc94-2cr62     gke-noc-noc-preempt-pool-356e72e3-heye  2021-01-15T15:13:54Z  2021-01-15T15:14:47Z
    istio-ingressgateway-6d7d9cc94-dc2g6     gke-noc-noc-preempt-pool-bc2cfcf6-lw40  2021-01-14T05:59:18Z  2021-01-15T01:37:03Z
    istio-pilot-89497d7c9-kndft              gke-noc-noc-preempt-pool-bc2cfcf6-lw40  2021-01-14T05:59:18Z  2021-01-15T01:37:45Z
    istio-pilot-89497d7c9-pjgjp              gke-noc-noc-preempt-pool-356e72e3-heye  2021-01-15T15:13:58Z  2021-01-15T15:15:28Z
    istio-policy-7589588588-b8dhk            gke-noc-noc-preempt-pool-bc2cfcf6-lw40  2021-01-14T05:59:19Z  2021-01-15T01:37:58Z
    istio-policy-7589588588-vkc4s            gke-noc-noc-preempt-pool-356e72e3-heye  2021-01-15T15:13:53Z  2021-01-15T15:14:21Z
    istio-sidecar-injector-67d9c85d6f-l52hr  gke-noc-noc-preempt-pool-bc2cfcf6-gmcm  2021-01-14T05:59:15Z  2021-01-14T22:00:42Z
    istio-sidecar-injector-67d9c85d6f-pcll7  gke-noc-noc-preempt-pool-356e72e3-heye  2021-01-15T15:13:58Z  2021-01-15T15:15:33Z
    istio-telemetry-6f6fb677bd-jj7h8         gke-noc-noc-preempt-pool-356e72e3-heye  2021-01-15T15:13:54Z  2021-01-15T15:14:50Z
    istio-telemetry-6f6fb677bd-jt4dg         gke-noc-noc-preempt-pool-bc2cfcf6-gmcm  2021-01-14T05:59:15Z  2021-01-14T22:01:00Z
    
  • Internet Redirect & Alias: The CNAME

    Internet Redirect & Alias: The CNAME

    You have probably used a product which used, or seen talk of, a CNAME. What is this? Why would you want it? How would you configure it? What does it do? When would you use it?

    A CNAME is a “Canonical Name Record“. It is an entry in an Internet Name Server that maps one name onto another. In a nutshell, an alias. They were originally created to allow running multiple services on the same host (e.g. ‘www’ and ‘ftp’ and ‘smtp’). CNAME’s were standardised in RFC 1034 in 1987. Perhaps before you were born!

    As an adminstrator, the most common use the CNAME has today is for services provided by a 3rd party that use your domain name. An example would be a Web Application Firewall. You might purchase such as service from a 3rd party, and, want them to handle the traffic to www.example.com. So, you would create a CNAME for www.example.com pointing to fw.waf.com, if they were called ‘waf.com’. This means that when one of your users opens their browser and types in ‘www.example.com’, they get back a record saying “if you really want to know where that is, we suggest looking up fw.waf.com”.

    Why would you use a CNAME for this, rather than an A record (address, e.g. an IP address)? Well, ‘waf.com’ might have big plans. Someday they will move out of their parent’s basement and into a real cloud provider. They will have hundreds of servers in tens of countries. They don’t want to have to have all customers update DNS records for this growth. So, by giving you a pointer to something they control, waf.com can make their updates independent of you, the administrator of example.com.

    How do you configure a CNAME? Well, it will depend on your DNS provider. It is very likely they have a web-based interface, and it might vary. Below is how I would configure a CNAME for ‘*.cloud.example.com’ pointing to ‘ca-1.agilicus.ca’. Where did ‘example.com’ go in this screenshot you ask? Well, it is implicit, I am configuring the DNS for example.com, so the records omit that.

    What does ‘*.cloud’ mean? Well, it means that foo.cloud.example.com, bar.cloud.example.com will both (and any name) resolve to ca-1.agilicus.ca.

    Google DNS CNAME

    Now, how would you confirm you have created your CNAME properly? My favourite method is the command-line and a tool called ‘dig’.

    $ dig -t cname foo.cloud.zero-trust.ca
     ; <<>> DiG 9.16.6-Ubuntu <<>> -t cname foo.cloud.zero-trust.ca
    ...
     ;; QUESTION SECTION:
     ;foo.cloud.zero-trust.ca.    IN  CNAME
     ;; ANSWER SECTION:
     foo.cloud.zero-trust.ca. 3600    IN  CNAME   ca-1.agilicus.ca.

    But, you can also use online web sites that will help you, e.g. https://dnslookup.online. You can see the result below. DNS takes time to propogate, so give it a few minutes. Also, the TTL (time to live) controls how long others will cache this. If in doubt, make this 5 minutes until you have your CNAME setup properly, and then use a longer time like 1 hour.

    c2d8e494 image
  • Let’s Encrypt X3 to R3 and Certificate Transparency Logs

    Let’s Encrypt X3 to R3 and Certificate Transparency Logs

    Recently Let’s Encrypt switched how they issue their intermediate certificates. You can read all about it. The main goals are reducing certificate size (moving from RSA to ECDA), speed, reliance.

    As of September 2020, the Let’s Encrypt hierarchy looks like:

    Recently Agilicus had a problem with one of our customers. Our IPSEC VPN dropped after rotating the Let’s Encrypt certificate from the X3 to the R3. And, we wanted to see when that happened, what was issued, etc.

    Enter the Certificate Transparency Log. You can try this at https://crt.sh. You go to that site, enter a domain name, and it shows all the certificates issued (by any provider supporting transparency). This would be a key activity you would do:

    1. When obtaining a new domain name. Check if there are any outstanding certificates someone else is in possession of
    2. If you are worried about spearphishing of your domain via typosquatting, you could check for certificates being issued to doppleganger domains
    3. When wanting to understand when it was rotated. This is the case we used.

    By looking at the report for our domain, we see that it started using the ‘R3’ issuer today. It would appear that the other end (Palo Alto) is having some trouble with it. I’m not clear why, since the change is to add a 2nd cross-signed root (X2), not removing X1.

  • Security.txt: how to document receiving security vulnerabilities

    Security.txt: how to document receiving security vulnerabilities

    The ostrich has a novel approach to threats: pretend they don’t exist. Commonly known as the head-in-sand approach. Do you want to be an ostrich when it comes to vulnerabilities in your software, your web site? Or do you want to be a bit more open-minded?

    Security researchers look for vulnerabilities. You may not even know of the existence of the security researcher nor the vulnerability they are researching. But, you may be susceptible to the attack they research. Getting that info into your hands will make it more likely you are on of the first to resolve the issue, the last to be attacked. And this can save you money.

    Some organisations even offer “bug bounties”, recognising that paying a small amount may obviate the need to pay a large amount later.

    Now, the challenge. How do you document in a simple way for all to understand your policies, procedures, etc? Who should the security researcher contact? How should they authenticate themself? How will you authenticate yourself to them? Etc. Well the good folks at the IETF have a standard in progress on the topic called “A File Format to Aid in Security Vulnerability Disclosure“. The TL;DR is “add a file called /.well-known/security.txt to your web site”. A slightly more involved “overview’ can be found at “securitytxt.org“.

    Will this standard change the world? No. Is it easy to implement? Yes. Does it have any downside? None that I can see.

  • OAuth 2.0 Protected Resource Threats

    OAuth 2.0 Protected Resource Threats

    This is part of a larger series on OAuth 2.0 Threats. The other parts are linked below.

    1. OAuth 2.0 Threat Model and Security Considerations
    2. OAuth 2.0 Client Threats
    3. OAuth 2.0 Authorisation Endpoint Threats
    4. OAuth 2.0 Token Endpoint Threats
    5. OAuth 2.0 Refresh Token Threats
    6. OAuth 2.0 Protected Resource Threats

    Finally we get to the protected resource itself. The OAuth 2.0 client and server have done their work, an access token has been generated. If the Access Token is a JWT, ensure that the resource always validates it against the signing authority, otherwise it is useless.

    Consider using a single-point enforcement (an Identity Aware Web Application Firewall, an API Gateway) so that you can have a single location of audit, of logging.

    If using a JWT-based access token, you can encode the scopes inside it, and, the protected resource can be involved in ensuring those scopes and the request are correct.

    Consider using simple schemes like Cross-Origin-Request-Sharing to ensure the clients are using the access token appropriately.

    Also consider, can the resource itself be spoofed? E.g. could the user be encouraged to send a connection to a fake resource, and thus expose their access token? TLS, Certificates, well written clients are your defense here.

    Also, from an east-west perspective, the client might use the same access token for more than one resource. If one of the resources becomes compromised, will it use the access token on behalf of the user to access another resource? Protect against this with resource-specific access tokens and scopes.

    Your security infrastructure might itself be used against you. Have you put in a place a TLS-inspecting firewall? Does it log lots of useful information for later forensics? Does this include the Authorization Header? This means that the log infrastucture now must be secured at a higher level than everything else combined.

  • OAuth 2.0 Refresh Token Threats

    OAuth 2.0 Refresh Token Threats

    This is part of a larger series on OAuth 2.0 Threats. The other parts are:

    1. OAuth 2.0 Threat Model and Security Considerations
    2. OAuth 2.0 Client Threats
    3. OAuth 2.0 Authorisation Endpoint Threats
    4. OAuth 2.0 Token Endpoint Threats
    5. OAuth 2.0 Refresh Token Threats
    6. OAuth 2.0 Protected Resource Threats

    The OAuth 2.0 refresh token was a bit controversial. It allows long-lived impersonation and recreation of access tokens (with the same or reduced scope). An attacker gaining access to a refresh token has more time, more lattitude than simply obtaining an access token.

    The defense for refresh tokens is similar to the other endpoint threats: always use TLS, use a public Certificate Authority, never allow insecure mode.

    Like other tokens it is common practice in an OAuth 2.0 server to store them in a persistent database. This might be a Key-Value store like Redis, it might be a RDMS like Postgresql. If these systems are breached (e.g. a backup is lost, etc), you have inadvertently given access to your entire universe.

    Refresh tokens can take many implementation-specific forms. The most common form may be a JWT, but, other handle-based designs could be used which could be susceptible to guessing.

    As with the other client threats, watch the storage.

  • OAuth 2.0 Token Endpoint Threats

    OAuth 2.0 Token Endpoint Threats

    This is part of a larger series on OAuth 2.0 Threats. The other parts are linked below.

    1. OAuth 2.0 Threat Model and Security Considerations
    2. OAuth 2.0 Client Threats
    3. OAuth 2.0 Authorisation Endpoint Threats
    4. OAuth 2.0 Token Endpoint Threats
    5. OAuth 2.0 Refresh Token Threats
    6. OAuth 2.0 Protected Resource Threats

    The Token Endpoint creates and transmits an access token to a client. Any eavesdropping here will give (limited by your scope) access to the bad actor.

    As a mitigation, always use TLS. Use a public Certificate Authority. Never allow insecure mode or self-signed certificates. Avoid using wildcard certificates. Keep the encryption all the way to the token endpoint. Use Defense In Depth.

    Many OAuth 2.0 implementations maintain state. This might be in an in-memory Key-Value database (e.g. Redis), or, could be in a relational database (e.g. Postgresql). This state sture must be even more secure than the transport: any loss of this information gives *all* the access tokens.

    In addition, the Token Endpoint could be a convenient way for an attacker to do password stuffing: guessing client-id/password pairs. A Fail-2-ban approach can be a low risk, low-complexity, high-effective risk reduction to this approach.

  • Your password policy is wrong: NIST SP 800-63B

    Your password policy is wrong: NIST SP 800-63B

    Password policies. Every site has a different one. How often they should be changed. Special characters. Do they display as you enter, etc. By 2021 we have all come to the realisation that the password is imperfect. But, its all we know. Does that mean we cannot improve?

    The US National Institute of Standards has a special publication, NIST 800-63B that talks about Identity guidelines. I cover it in more detail here. It is a set of best practices for password policies, codified. And, I will wager, your system is not compliant. Let’s look at a couple of examples:

    Password Complexity (Composition Rules)

    6e2c8613 image

    Verifiers SHOULD NOT impose other composition rules (e.g., requiring mixtures of different character types or prohibiting consecutively repeated characters) for memorized secrets.

    https://pages.nist.gov/800-63-3/sp800-63b.html, 5.1.1.2 Memorized Secret Verifiers

    Huh. Is that what your password policy says? Or does it say “at least one special character, at least one upper case, one lower case and 2 numbers”? Now, I’m not suggesting you should use “12345“, but, forcing too much complexity is not increasing the odds people will remember a more complex passphrase.

    Password Change Frequency

    Verifiers SHOULD NOT require memorized secrets to be changed arbitrarily (e.g., periodically). However, verifiers SHALL force a change if there is evidence of compromise of the authenticator.

    https://pages.nist.gov/800-63-3/sp800-63b.html, 5.1.1.2 Memorized Secret Verifiers

    Huh, is that what your password policy says? Or does it require you to change it every 19 days +- 3?

    Show The Password

    In order to assist the claimant in successfully entering a memorized secret, the verifier SHOULD offer an option to display the secret — rather than a series of dots or asterisks — until it is entered. This allows the claimant to verify their entry if they are in a location where their screen is unlikely to be observed. The verifier MAY also permit the user’s device to display individual entered characters for a short time after each character is typed to verify correct entry. This is particularly applicable on mobile devices.

    https://pages.nist.gov/800-63-3/sp800-63b.html, 5.1.1.2 Memorized Secret Verifiers

    As someone who has poor vision, this is a password policy that bothers me. I know the password I’m creating. I’m not sure I’m typing it properly. Let me see please.

    It also goes on to talk about salt+hashed passwords, something that is inexcusable if sites don’t do it. I’ll let you in on a secret: if your system can retrieve your password, it is not salt+hashed.

  • OAuth 2.0 Authorisation Endpoint Threats

    OAuth 2.0 Authorisation Endpoint Threats

    This is part of a larger series on OAuth 2.0 Threats. The other parts are linked below.

    1. OAuth 2.0 Threat Model and Security Considerations
    2. OAuth 2.0 Client Threats
    3. OAuth 2.0 Authorisation Endpoint Threats
    4. OAuth 2.0 Token Endpoint Threats
    5. OAuth 2.0 Refresh Token Threats
    6. OAuth 2.0 Protected Resource Threats

    The front-door of your OAuth 2.0 authorisation system is the Authorisation Endpoint. This should be well-known and documented (ideally with an OpenID Discovery Document), so don’t even think about security by obscurity.

    Your first line of defense is TLS and Certificates. Use a public-signed certificate for your OAuth 2.0 Endpoints. Never use self-signed certificates. This is your primary defense against DNS poisoning, ARP spoofing, doppleganger domains, etc. If a bad actor can encourage your users to hit a counterfit authorisation server, they could use this to steal password information. In your client, always verify your TLS certificates, never allow bad or self-signed.

    An Authorisation Server may ‘cache’ information on previously authorised clients. This can be used to reduce the computational cost, to reduce the user complexity. But, it leaves a replay attack open. You can protect against this by limiting the number of access tokens which may be (re) issued, by forcing pre-registered public redirect URI.

    As with all services, use a set of best practices. Never use wildcard TLS certificates. Use read-only fileystems. Protect against the east-west threat from other services in the same domain or scope or cluster.

  • OAuth 2.0 Client Threats

    OAuth 2.0 Client Threats

    This is part of a larger series on OAuth 2.0 Threats. The other parts are linked below.

    1. OAuth 2.0 Threat Model and Security Considerations
    2. OAuth 2.0 Client Threats
    3. OAuth 2.0 Authorisation Endpoint Threats
    4. OAuth 2.0 Token Endpoint Threats
    5. OAuth 2.0 Refresh Token Threats
    6. OAuth 2.0 Protected Resource Threats

    The Client. In OAUTH2 terminology this is the part that is asking for Authorisation. It might be a web client directly. It might be a web client via a backend server. It might be a server on its own. When OAuth 2.0 was envisaged the web clients were more ‘smart forms’ engines, e.g. the backend had all the rendering smarts in something like PHP and there was no Angular Typescript Single Page Applications.

    As a consequence, the initial architecture envisaged a secret that could be protected, something running on that web server which did the authorisation on behalf of a user. Some implementations started to experiment with a web client directly talking to the OAuth 2.0 server, and the Implicit flow was born. The Implicit flow in turn ended up fairly weak and the PKCE flow was born. If you are writing a new Web Application today you should use the PKCE flow. If you have an existing application using the Implicit flow, make sure you understand your Content-Security-Policy and your upstream code base is well trusted, else you should update.

    NEVER EVER ISSUE A CLIENT SECRET TO A WEB CLIENT. There is no way to do this safely.

    If you have an implementation where the backend server is resonsible for Authorising the user, make sure it has a secure means of storing the Client Secret, and, consider rotating that secret (with revocation).

    A threat that is often overlooked on the Client is the refresh token. The Refresh Token can be used to obtain new Access Tokens, so losing it can be problematic. To reduce this risk, make sure your clients use specific scopes, and, the minimum set they can. Verify the client id on the refresh. And consider refresh token rotation.

    An Access Token (typically a JWT) is “you-in-a-string”. So clearly you want to ensure the client does not leak this to unauthorised parties. Clients have a bad history of storing these in shared persistent storage. Mitigation includes limiting the duration, auditing applications to ensure they use similar protection as for refresh tokens.

    A common client model which emerged was a mobile application which used an embedded browser for the OAuth 2.0 phase. This has been deprecated and many providers no longer support it. The risk is a malicious application phishing a user, owning the embedded browser, getting a cleartext copy of the users authentication information.

    As always, employ Defense In Depth, employ Zero Trust. The client may be part of your security solution, but it can never be considered secure. Secure it using best practices, and then move on with the assumption it is compromised.

  • Fail to ban. Simple. Strong. Defense in Depth

    Fail to ban. Simple. Strong. Defense in Depth

    For many years I have run a simple defense on my Linux systems: Fail to ban. Its very simple: source IP that attempt to login, after a fixed number of failures, are blocked for some time duration. The simplicity is high, but the effectiveness is off the chart.

    Attackers have a (high but) limited number of IP to come from. If you have a user with a weak password, lets say it will only take 100K guesses, well at 1 guess per second they would have it in a day. But, if you block guesses for 15 minutes after 3 bad guesses in a row, now it will take them 1041 days, or about 3 years.

    Is it a single perfect form of security? No. Its a valid part of your Defense In Depth strategy. It helps switch the cost from defending (you) to attacking (them).

  • OAuth 2.0 Threat Model and Security Considerations

    OAuth 2.0 Threat Model and Security Considerations

    OAuth 2.0 has simplified authentication and authorisation for many applications, shifting from custom code to simple library import. However, as more applications come to rely on it, this makes its weaknesses more interesting. An attacker can gain access to a broader set of data via a smaller set of tactics and techniques. First lets understand the threat areas, and then, the best current practices for addressing them.

    This is a multi-part blog series. We will cover individual threat areas in 6-parts:

    1. OAuth 2.0 Threat Model and Security Considerations
    2. OAuth 2.0 Client Threats
    3. OAuth 2.0 Authorisation Endpoint Threats
    4. OAuth 2.0 Token Endpoint Threats
    5. OAuth 2.0 Refresh Token Threats
    6. OAuth 2.0 Protected Resource Threats

    But first, we will talk about the assumptions and security features of OAuth 2.0. One of the main controversial areas is the reliance on a Bearer Token. In “OAuth 2.0 Proof of Possession” I talked about some of the controversy around this decision. If you leak your Bearer Token, whoever has possession can use it as you without control (until it is revoked or expires). You might lose your Bearer Token through inappropriate logging, through browser cache/cookie/storage manipulation, etc. Its a permanent blank cheque, unprotected from replay. I discussed a new draft standard “OAuth 2.0 Access Token Demonstrate Proof Of Possession” which seeks to reduce this risk by moving to replay-protected model with a new set of cryptographic headers inside the Token.

    Much of the protection of OAuth 2.0 comes from the Public Key Infrastrcture which drives TLS, guaranteeing you are talking to who you think, that there is no interception or modification. As such, you should never use OAuth 2.0 without TLS, and, you must never disable certificate checking. In “Let’s Encrypt Staging Safely” I show how to safely use the Let’s Encrypt staging certificates, preventing you from a desire to “turn it all off because It’s just dev mode”. Its all too common to find code which disables CA Certificate checking: don’t let this happen to you.

    OAuth 2.0 relies on Client ID and Client Secrets for pairwise authentication to each client. However, for many clients (particularly web-based or mobile-app-based) there is no secure method to store a Client Secret. For these applications you should always use the PKCE Code Flow (not the implicit flow, and not the basic-auth via Client Secret).

    The TL;DR of this section, use TLS properly, don’t use Client Secrets is your main take-away.

    If we look at the Authorisation Server, we find that it has a host of issues to concern ourself with. Unlike the client this is in a more protected domain of control. Still, this system has usernames, possibly passwords (surely stored encrypted, salted, hashed). It may have plaintext copies of the client secrets. It might even have a plaintext copy of the client refresh token or access token. It also is the server-endpoint of our TLS, so it has a TLS private key. In addition, it is the state transition engine for the OAuth 2.0 dance, meaning if its nonces or states get compromised or are guessable, an attacker can interfect.

    Now that we’ve got a background of the basic security underpinnings, let’s dig into the indvidual threats in the next 6 parts.

    For more information, the best starting source of information is the IETF RFC 6819 OAuth 2.0 Threat Model and Security Considerations.

  • Merger, Acquisition: Federated Identity And Zero Trust

    Merger, Acquisition: Federated Identity And Zero Trust

    A Merger or Acqusition is an exciting, disruptive event. Formerly competitive companies might be merged, causing a long battle of who has the best system. Formerly orthogonal companies might suddenly be called to cooperate. The quicker we can merge, the quicker we can start aligning interests, the quicker we can be successful. Amongst the barriers is our IT systems. Identity, Access. What is the best approach to quickly align these teams? Mergers and Acquisitions: Zero Trust.

    20 years ago now my company was acquired by Cisco Systems. Cisco parachuted in a large team who executed a well thought out plan. Work stopped on friday as company A, and work started monday as Company B. To make that happen, a lot of people re-imaged a lot of machines, and a lot of changes happened. At the team we were acquired we were hundreds of staff. If we had been thousands, I’m not sure this would have succeeded as a strategy. Instead, I think a much messier mesh of VPN’s and partially cross-linked identity systems would have been lingering for days, weeks, months, maybe even years.

    Fast forward to 2020. We have new technology and practices. We can achieve our merger, acquisition with Zero Trust.

    First, we need a Federated Identity system. Don’t create new accounts and logins. Instead, create a new Federated Identity system which allows for a user to be working for one of the companies, or the other of the companies. Make no differentiation in ease of use nor security nor access.

    Second, we need a means of authorising access to the resource. Zero Trust Network Architecture allows us to treat resources in isolation. Make no differentiation in ease of use nor security nor access for any resource, regardless of which of the 3 entities “owns” it. A Merger or Acquisition with Zero Trust allows authorisation regardless resource or user location or ownership.

    Don’t hire a firewall or a VPN to do the job, they are not fit for purpose. They have a time and a place. But not here.

    Once achieved, we can a la carte build the team and tools that will make the merged entity successful. And, in the unlikely event the merger fails and needs to be divested, its trivial to disconnect. Fast to start and try, fast to fail if needed. This is what continuous is all about. M&A, Zero Trust.

  • Joint Venture: The Case For Federated Identity And Zero Trust

    Joint Venture: The Case For Federated Identity And Zero Trust

    A Joint Venture allows 2 unrelated companies to share resources, ownership in a new special purpose entity. For cost reasons you might want the new JV to have access to partial resources of one or more of the parent entities. Users might have permissions in both the JV and the parent organisation. How does one achieve this? A mesh of VPN’s? A new Identity system? These seem likely to lead to heart ache and security woes. Business needs to be fast and efficient. Starting a new JV we should not have to worry about who can login to what. What’s the best approach? A Joint Venture with Zero Trust.

    First, we need a Federated Identity system. Don’t create new accounts and logins. Instead, create a new Federated Identity system which allows for a user to be working for one of the parents, or the other of the parents, or the JV directly. Make no differentiation in ease of use nor security nor access.

    Second, we need a means of authorising access to the resource. Zero Trust Network Architecture allows us to treat resources in isolation. Make no differentiation in ease of use nor security nor access for any resource, regardless of which of the 3 entities “owns” it. A Joint Venture with Zero Trust allows authorisation regardless resource or user location or ownership.

    Don’t hire a firewall or a VPN to do the job, they are not fit for purpose. They have a time and a place. But not here.

    Once achieved, we can a la carte build the team and tools that will make the Joint Venture successful. And, in the unlikely event the Joint Venture fails and needs to be wound up, its trivial to disconnect. Fast to start and try, fast to fail if needed. This is what continuous is all about. Joint Venture, Zero Trust.

  • OAuth 2.0 Proof of Possession

    OAuth 2.0 Proof of Possession

    OAuth 2.0 simplified its problem space by making Proof Of Possession tokens optional, replacing them with Bearer tokens as the standard choice. This was controversial, (partially) causing one of the authors to quit. But, good news, Proof Of Possession tokens are back for OAuth 2.0.

    Now, back to the author who quit. He gave an entertaining talk on how bad OAuth 2 is. Yes, OAuth 2.0 was designed prior to modern web applications (it viewed the world through a PHP lens), but, OpenID Connect was built on top of it and we’ve moved on.

    OK, now that you’ve watched that and are somewhat concerned about your own use of OAuth 2.0, you are wondering if you should switch from Bearer to PoP. There is also the in-progress draft for Demonstration of Proof of Concept, confusing you more.

    Well, lets start with the basics. Content Security Policy. Cross-Site-Scripting. Once these are in place, evaluate, how can my Bearer (Access) token be stolen? If you can come up with a reasonable way, fix it. While you are doing that, evaluate these two drafts on Proof of Possession. As they progress, if they become standard, adopt.

    Is Proof of Possession better? Yes. Is OAuth 2.0 better than “proprietary auth scheme written by unknown vendor”? Yes. So you are better using a spec which is being improved, than something hidden which is unknown.

  • OAuth 2.0 Access Token Demonstrate Proof of Possession

    OAuth 2.0 Access Token Demonstrate Proof of Possession

    An Access Token is a cryptographicly-signed key, allowing a system to identify and authorise itself. The most common format is called JSON Web Token (JWT). In the most common means of use, you open a browser, you login as yourself. An Access Token is generated representing you, this application, this session. That Access Token is then stored in your browser. Each web page you open uses it to generate some API calls after the fact. That is the normal case.

    The abnormal case, however, is not as good. If that web page stores the token as a cookie, it may be intercepted. If it stores it in browser local storage, it could be scraped from the disk. Malicous JavaScript hosted on the same page might use cross-site-scripting request forgery to get access to it. How do you protect yourself?

    Well, first start by using a set of best practices. Use a strong Content-Security-Policy. Secure your cookies. Use the XSS features. Use the PKCE Code Flow. Achieving this in your web application will make it much stronger, reducing the risk.

    What if that is not enough? What if your data is more valuable, your app more risky than the average? Well, a new IETF draft (OAuth 2.0 Demonstrating Proof-of-Possession at the Application Layer (DPoP)) seeks to enhance, to lower the risk. The specification defines a mechanism for sender-constraining OAuth 2.0 tokens via a proof-of-possession mechanism on the application level. This mechanism allows for the detection of replay attacks with access and refresh tokens.

    Now, in a full Zero-Trust model, we would use client-certificates (mutual TLS) to identify the sender. However, in a typical web application this is not possible. So, they replace it with a private-key/public-key pair. The applicaton which generated the Access Token in the first place also has its own private key. Each subsequent use it signs in such a way the server can see who signed it, but that replace is not possible.

    Now, this does not solve all attacks. In particular, a browser does not have a lot of options for secure storage of local secrets. So this private key could be stolen in the same way the access token is. But, it does protect against access tokens stolen by network sniffing, from caches, logs, etc.

  • Zero-Trust Reduces Ransomware Risk

    Zero-Trust Reduces Ransomware Risk

    Ransomware is one of the most successful cyber crimes of recent years. A criminal gains access to your network, encrypts your data, asks for money to return it. If you don’t pay, they threaten to release it publicly. Most organisations pay. The payments are large enough, and frequent enough, to be a motivator for the criiminals, and small enough to not be a motivator for the international police co-operation needed. Let’s fix this with Zero-Trust.

    The most successful ransomware follows a simple pattern. A distribution campaign tricks one user into running something (through spear phishing, social engineering, etc.). This malicious code fetches the ransomware, which then scans the local machine, and all connected machines. It encrypts everything it finds and then generates a ransom note.

    The most common means of sharing files in todays corporate world is the Microsoft SMB (CIFS) protocol. You see this as a ‘share’. And these shares are a fertile breeding ground for ransomware. In a typical small corporate environment all your desktops directly access a small number of servers. And, importantly, the servers can all access each other. This system is only as strong as its weakest link. And here, the weakest link, is you. And the route from you to your corporate world is the SMB protocol. Click a spearphishing link? It takes out your machine, and everything your machine can access, and everything those machines can access. You are 2 or 3 hops away from your entire company.

    What can we do about this. Well, educate users, install endpoint security, that sort of thing. Nothing wrong there, certainly best practices. But it hasn’t worked. Bad ransomware happens to good people. So ransomware happens, people pay, the cycle continues. What can we do better?

    First, we adopt a principle of Defense In Depth. Instead of a single (presumed) infinitely strong security point (the Firewall, the VPN), we assume each layer of our defense will be breached. We switch to a set of fallback positions, delay the attacker, shift the cost from us to them. Think trench warfare rather than a single combat line.

    Zero-Trust is the principal of switching from a perimeter-based trust model (you -> VPN -> building) to a user -> resource model (you to application-1). In doing so, you end up with a set of point-to-point links rather than a point to “whatever” link. In adopting Zero Trust, we are adopting Defense in Depth. We assume one user might become compromised. From there, rather than them having access to everything directly, they only have access to what they need. And, those systems in turn only have access to what they need. We have limited the blast radius.

    But we can go one step further. We can stop using the SMB protocol to share in a hub-and-spoke model everything to everything. We can instead use WebDav. Same affect for the user: a remote directory is shared locally. But, more importantly, we can apply fine grained role-based access control. Since WebDav is HTTP, it doesn’t suffer from the same complex-protocol weakness that has enabled SMB to become such a transmission route. Users are now more loosely coupled, they don’t need to have a direct link to the Domain Controller, to the File Servers.

    Coming back to what the criminals do here. They encrypt your data and ask for money. But, did they merely encrypt it? Or did they also exfiltrate it? When you refuse to pay you find they suggest they’ll leak your private data to the web. When ransomware happens the first thing you will ask is: what data, where did it go, what accessed it? In a traditional ‘walled-garden’ approach, we cannot ascertain this. Once inside the firewall & VPN you have unfettered access to everything, without access control, without audit. Zero Trust fixes this two ways. First, removing the walled garden means we are authenticating to each resource, so we can see who accessed. But, the Zero Trust agent can also product audit logs of what was accessed.

    Zero-Trust implemented means a dramatic reduction in the risk of ransomware. Both the risk of getting it, and, the risk of it spreading. It allows you to ascertain who got what, when while simultaneously reducing the amount of who, the amount of what.

    Simple for the users, Secure for the company. Defense in Depth via Zero-Trust is better.

  • Button down your 2-factor-auth and dnssec lest ye be hijacked

    Button down your 2-factor-auth and dnssec lest ye be hijacked

    Got a DNS name? (if not, why not?). Make sure the ‘console’ you use to access/control/edit it is locked down with 2-factor authentication. Ideally its e.g. OAUTH2 off your Google account which uses a Yubikey or at least your Android phone for push. Not sure I’d even both with SMS, but I suppose if its that or nothing. I’ll wait, this is important. Its not hard to do, just enable it (and maybe do your Amazon, Github, Gmail while you are at it).

    OK, done? Good. You just neutralised half of the horror you are about to read. Now, before I give you the details, while you are in your DNS console, enable DNSSEC, and then check it here. May as well double check your CAA and SPF while you are in there.

    OK, back, done? Good. You’ve neutralised most of the rest. So lets read what the good folks at Fireeye have said. Its a method of broadly harvesting all kinds of information by spoofing a machine, without much chance of the victim detecting.

    Now, the next step, and my ask of you. For a couple of sites you care about, do a quick check. Do they have DNSSEC enabled? If not, contact them, open a support ticket, give them a tsk tsk sound as you pass them in the hall, whatever you gotta do.

    Here's an example, my bank. No DNSSEC. Why not? Lets check the CAA:
    $ dig -t caa royalbank.com
    ... NO RECORDS...

    This is no good. A CAA record helps prevent you from being spoofed. The above attack, if it can get a hold of your DNS, can sign a certificate w/ Let’s Encrypt. They might not notice the CAA if you have it there.

    So, anyone involved with the Royal Bank of Canada want to comment on their DNSSEC practices, why its disabled? Or anyone else want to check your bank and comment here as to whether it has DNSSEC (and CAA)? Takes 1 second to test here.

    Its got another bad practice:

    http://royalbank.ca redirects to http://www.rbcroyalbank.com. This in turn does not redirect you to the SSL site but serves content, fetches from remote CDNs. You don’t want that. TLS or bust.

    $ curl -q www.rbcroyalbank.com |grep https

    Is DNSSEC perfect? No. Its part of your Defense in Depth.

  • Your VPN Hates Your Video Conferencing. Here’s Why

    Your VPN Hates Your Video Conferencing. Here’s Why

    Move the company network out to the users via VPN, break their video conferencing. Didn’t know what was a thing? Assumed that all the problems with latency, lag, quality, etc where due to poor home networks, older devices? It might surprise you that its could be how your route your traffic. The corporate VPN.

    In the video below I go through the dreaded hairpin, that line in red in the below animation.

  • Blurring the web and the app: web push

    Blurring the web and the app: web push

    If you do nothing else with this article, please click the Bell in the lower right. Thanks!

    The web is the great leveller. The great deliverer of information. But, sometime between the early days of a read-only medium, and today, it became the primary application delivery vehicle. And the browser became the operating system.

    At first we had simple web forms. You were presented some information, you supplied some information, you hit “Submit”. It worked for signing up to newsletters, but beyond that it was not that great.

    Then we had a false start. Java applets, Flash came on the scene, seeking to make the web a delivery for executables. Billions were spent, security was compromised. RIP you evolutionary dead-end.

    Now we have a technology called Progressive Web Applications. These behave as a local rich application, but, run in the web. Google Docs is a great example. You use these every day and don’t know. If you have a web-app hybrid you like, there’s a good chance its a PWA. You may hear words like Reactive, Angular, AJAX, etc.

    However, one of the challenges was… what if the browser were not open to the page. How would you receive a push notification from something like Email? This is where the Push API came in. It allows a server to efficiently wake up and deliver some data to a sleeping web application. Efficient in battery usage, in network usage.

    At Agilicus we use the Web Push for multiple uses. We use it as part of the multi-factor authentication feature in our product (locking a user to a browser, waking it up with a “Was this you logging in”?).

    But, we also use it on this very web page to help you know when a new blog post is available. And, you see, because we care, it doesn’t auto-suggest you turn it on. You see that bell in the lower right? Click it. It will ask if you want to Subscribe. Web Push is anonymous, we don’t know who you are. You are in control, you can unsubscribe any time. This is much better than subscribing via email which is neither anonymous nor easy to guarantee unsubscribe.

    And yet, like much technology, people don’t understand, and don’t trust, and thus use something worse for them. More people subscribe to this site via email than via webpush.

    So, I am hoping you will do two things:

    1. Subscribe to this site using that bell
    2. Use web push subscriptions in the future on sites you trust, and you want to receive updates from

    So, give it a try. It won’t bite. I promise.

  • Zero Trust Audit Logging: Reliable Meets Simple

    Zero Trust Audit Logging: Reliable Meets Simple

    For years we have used syslog and netflow style logging, convinced that we could easily correlate events, find security issues, forensically see what happened. Then reality sank in. Multiple devices on the same IP due to NAT. Proxy servers. Spoofed IP. Big investments in SIEM and correlation became big work to get the most out of them. There has to be a better way!

    Well, there is. We swap that perimeter-security and port-forward-DMZ-firewall world for Zero Trust Network Architecture. We place a cryptographically-secure header (JWT) on each transaction. We audit based on the contents of the JWT.

    Now it doesn’t matter about IP address, 5-tuple-lookup, who had that DHCP address, spoofing. Each transaction has a GUID, called a SUB (Subject, e.g. the user). Correlation becomes exact match, no complexity, no confusion, no time horizons, no inside the NAT vs outside. Simple. Secure. Learn more in the below video!

  • Reject The Status Quo: Zero Trust Status Future

    Reject The Status Quo: Zero Trust Status Future

    The myth of the VPN, the Firewall as the only and best method of remote access has lived for 20 years. Let’s retire it together. I discuss the myth, and, an outbound-only, no firewall reconfiguration method, no client method of achieving your goals of happier productive users accessing their data and applications.

    In this short video I discuss Zero Trust Networking as a means of, no-client, no-vpn, no-inbound-through-firewall means of achieving higher security, better performance, and simpler end user experience.

    The VPN is breaking your video conferencing. its costing you money. Its frustrating your users. Let’s break down the myth that its the best or only solution to accessing data and applications. Its not.

  • Access an Internal Resource with no Inbound Connections

    Access an Internal Resource with no Inbound Connections

    Zero Trust Network Access. The principle of managing security between a user and a resource, rather than a user and a perimeter. By Identifying the user, providing Authorisation in the network, and then remote access to individual resources, we increase security and simplify at the same time. But, did you know you can do this with 0 inbound connections open? Nothing. No ports open to see on a pentest, on nmap, on Shodan. How? In this video I explain. In a nutshell, the resource makes a permanent outbound connection to the cloud. The user makes an outbound connection to the same cloud. Identity is confirmed, Authorisation is confirmed. The connections are stitched together. Simple and secure, no reconfiguration of firewalls. No client VPN software.

  • The Agilicus Philosophy: Continuous Learn. Do. Teach.

    The Agilicus Philosophy: Continuous Learn. Do. Teach.

    The unnoficial slogan of Agilicus is Learn. Do. Teach. We favour continuous learning, small batches, agile. Its in the name.

    Over the course of this year our trajectory has been nudged around by many forces, both internal and external. The basic principles of Zero Trust Network Architecture evolve. Our customer space evolves. The technology we operate on evolves. And, the technology we create evolves.

    Small batches. Continuous integration. Small merge requests. Continuous communication. I discuss the philosophy in the video below. Enjoy!

  • My team works outside, why can’t their Kronos Timesheet?

    My team works outside, why can’t their Kronos Timesheet?

    The application with the highest adoption in many companies is timesheet management. And, Kronos is one of the best ways of managing labour tracking, time, attendance. But what if your workforce is outside? Do you make them update their timesheet on return to the service centre at the end of shift? For efficiency would it not be better to enable people to update their timesheet where they are, with the device they have?

    I know, you are saying, the VPN. Its my panacea. People can just VPN in. But, is that the best solution? Its certainly not the easiest solution, managing the client on various devices. Its certainly not the most popular solution. But at least its the most secure, right? Wrong. Your VPN is doing little to secure, and much to annoy, your team and your tools. When you connect via VPN your device suddenly behaves as if it were on the internal network. With access to more than it needs.

    Zero Trust Networking is the principle of moving the protection from perimeter (moat and castle) to user + resource. It allows you to take that Kronos application, installed years ago, and make it behave as if it were a standalone Internet destination. With the same login. With higher security. With perfect audit. And, most importantly, with higher efficiency for that workforce you are trying to optimise.

    Without re-architecting your network, without upgrading your Kronos, you could be having your team members access it directly. From any device. From any network. With less training, less complaining.

    Want to learn more? Or change my mind? Email me, info at agilicus.com

  • Stupid Simple Security Trick: Add a DNS CAA Record

    Stupid Simple Security Trick: Add a DNS CAA Record

    RFC 8659 gave us the DNS CAA record. In a nutshell, you add a record to your DNS, to prevent mis-issuance of TLS certificates for your domain.

    Worried that someone on your team will create a wildcard for your domain without understanding the risk?
    Worried that a CA will make a mistake an issue a certificate for you?
    Got DNS?
    You can solve this for free right now! And while you are at it, join the preload list.

    Watch, learn.

  • You want me to sign in with what now?

    You want me to sign in with what now?

    Have you seen or used a web application that suggests “Sign in with Google” or “Sign in with Apple”? Did your eyes skip over that and head to the familiar comfort of “create new account” while you tried to remember a unique password based probably on your cat and your birthday? After all, why would you sign into “random-app” with “gmail”? Well, let me explain. The tldr is “if you do this, you are more secure, and its simpler for you, you are giving nothing up”. The longer is, well,. check the short video.

  • The Firewall Emperor Has No Clothes

    The Firewall Emperor Has No Clothes

    Many people buy a network firewall and hoist a mission accomplished flag over security. The magic box prevents the bad of the Internet from infecting the good of the interior. But did you know the firewall itself can be taken over, or other internal resources can be traversed, just by your own browser?

    As a user, whatever your browser can get to, JavaScript running in it can get to it. Your firewall allows all outbound. Thus if I can influence you to a site with some bad JavaScript, I can connect to something inside. That might be the administrative interface of your firewall. It could be your payroll system. O firewall where art thou in my time of need?

    The solution is Defense in Depth. Stop thinking about security as a perimeter, start thinking about it as user+resource pairs. Look at a Zero Trust Architecture, its simpler than the other micro-segmentation approaches out there.

  • The JWT Bearer Token: You On The Wire

    The JWT Bearer Token: You On The Wire

    The Internet and its tools started out fully trusting. Protocols like SMTP (email) have no authentication built in. HTTP later added basic-authentication, which i seffectively a username+password on each transaction. The Secure Sockets Layer (SSL, later standardised as Transport Layer Security, TLS) came along and gave us privacy, but still not authentication.

    Initially we viewed the problem as a User logging in, as session setup: a one-time activity. But later, API’s came along, progressive web applications blurred the lines between a web page and an always-on application. We needed a safe means of acting as a user, without the user being involved in each transaction.

    Enter the humble JSON Web Token (JWT). The JWT allows an application to embed assertions (claims) and identity into a cryptographically secure, transportable object. An application can present the JWT to act on behalf of the user.

    Today you will find JWT present in nearly everything you do. Your banking, your email, your mobile applications. You can see one by looking in the web console of your browser (usually by hitting ctrl-shift-I) and looking for a cookie or local storage item with the word “token” in the name. It will be a long gibberish-looking string. You can paste it into https://jwt.io/ to see what it has inside. Now, is this safe? Should you paste your security token into a random web page on the Internet? (hint: no, not really). If you need to do this frequently, you might try:

    npm install -g jwt-cli

    And now you have a command called jwt you can use to see the same info, locally, without sharing the security credentials.

    Inside your JWT you will see a set of fields like iat, nbf, etc. These are all defined in RFC 7519 JSON Web Token (JWT). (iat means issued-at, nbf means not-before, two of the time-fields).

    If you are building a new application today, whether web-based or not, or building API today, you must look at JWT. It is a far better standard and method than propogating username+password around, and far better than arbitrary secrets (called API keys). Its the core of Zero Trust, and can dramatically simplify your firewall configuration and security.