18 December 2019

runp: run shell commands in parallel


I'm using shell (bash specifically) on daily basis. From time to time a need arises to run multiple commands in parallel. For example my .bashrc runs commands like these to download or clone vim plugins I use:

curl -L -o $HOME/.git-completion.bash https://raw.githubusercontent.com/git/git/master/contrib/completion/git-completion.bash
rm -rf $HOME/.vim/pack/plugins/start/grep.vim && git clone https://github.com/yegappan/grep.git $HOME/.vim/pack/plugins/start/grep.vim
# https://tpaschalis.github.io/vim-go-setup/
rm -rf $HOME/.vim/pack/plugins/start/vim-go && git clone https://github.com/fatih/vim-go.git $HOME/.vim/pack/plugins/start/vim-go

The problem is that these commands run sequentially and it takes a while until they are done. I was thinking of a way how to speed them up. So to scratch my itch I came up with runp.

Why and how to use it

Now I can run those commands (I stored them in install-my-stuff.txt) in parallel:

Let's see how much time did I save:
$ time bash install-my-stuff.txt
<...>
real    0m15.690s
user    0m3.440s
sys    0m0.902s

$ time runp install-my-stuff.txt
<...>
real    0m3.678s
user    0m3.904s
sys    0m0.880s
Hmm, around 12 seconds. Not bad I think :-).
If you don't want runp to add any output of its own (that is sent to stderr by the way) use the -q flag to quiet it:
$ runp -q install-my-stuff.txt 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 70071  100 70071    0     0   145k      0 --:--:-- --:--:-- --:--:--  145k
Cloning into '/home/reisinge/.vim/pack/plugins/start/vim-go'...
Cloning into '/home/reisinge/.vim/pack/plugins/start/grep.vim'...

How to install it

runp is easy to install. It's a single binary that you download and make it executable:
# choose your system and architecture
export SYS=linux  # or darwin
export ARCH=amd64 # or arm

# download it an make it executable
curl -L https://github.com/jreisinger/runp/releases/latest/download/runp-$SYS-$ARCH -o ~/bin/runp
chmod u+x ~/bin/runp

More examples

The commands to execute can be supplied also via stdin. It means that runp can be used within pipelines like this one:
$ for dir in $HOME /etc /tmp; do echo sudo "du -sh $dir"; done | runp -q | sort -h
13M    /tmp
17M    /etc
370G    /home/reisinge
Here we generate the commands to run in a bash for loop. Then we pipe the commands into runp. Finally the runp's output (stdout) is sorted.
We can simplify by using the -p flag which adds a prefix string to the final command that will be run:
$ echo -e "$HOME\n/etc\n/tmp" | runp -q -p 'sudo du -sh' | sort -h
13M    /tmp
17M    /etc
370G    /home/reisinge
The final example shows how to find open ports from a list of hosts and ports:
# file with host-port pairs
$ cat host-port.txt
localhost 80
localhost 81
127.0.0.1 443
127.0.0.1 444
localhost 22

# find out which ports are listening
$ cat host-port.txt | runp -p 'netcat -v -w2 -z' -q 2>&1 | egrep 'open$'
localhost [127.0.0.1] 443 (https) open
localhost [127.0.0.1] 80 (http) open
localhost [127.0.0.1] 22 (ssh) open

28 February 2018

Quick Docker

(Up-to-date source of this post.)

Docker is a container technology. It's a well timed fusion of

  • kernel features
  • filesystem tricks
  • networking hacks

Think of a container not as a virtual machine but a very lighweight wrapper around a single Unix process.

Docker revision controls

  1. filesystem layers
  2. image tags

Terminology

Docker server - the docker command run in daemon mode on a Linux host:

$ sudo docker -d -H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375

Docker image - one or more filesystem layers and metadata that represent all the files required to run a Dockerized application

Docker container - a Linux container that has been instantiated from a Docker image

Working with Docker images

To launch a container

  • download a public image
  • create your own

To create a custom image you need a Dockerfile - each line in a Dockerfile creates a new image layer that is stored by Docker

Build an image:

git clone https://github.com/spkane/docker-node-hello.git
cd docker-node-hello
docker build -t example/docker-node-hello:latest .

Run an image (or a container?):

docker run -d -p 80:8080 example/docker-node-hello:latest
  • -p 80:8080 tells Docker to proxy the container's port 80 on the host's port 8080 (port binding)
  • example/docker-node-hello:latest is a tag

Remove an image:

docker images
docker rmi <image_id>

Remove all images on your Docker host:

docker rmi $(docker images -q -)

Working with Docker containers

A container is a self-contained execution environment that shares the kernel of the host system and which is (optionally) isolated from other containers in the system.

Containers are a Linux only technology.

Create a container (see also "Run an image" above):

docker run --rm -ti ubuntu /bin/bash 
  • run - create + start
  • --rm - delete the container when it exits
  • -t - allocate a pseudo-TTY
  • -i - interactive session, e.i. keep STDIN open
  • /bin/bash - executable to run within the container

Get into a running container:

docker ps
docker exec -it <container_id> /bin/bash

Stop a container:

docker stop <container_id>

Remove a container:

docker ps -a
docker rm <container_id>

Remove all containers on your Docker host:

docker rm  $(docker ps -a -q)

Sources

  • Docker: Up & Running (2015)
  • Unix and Linux System Administration Handbook, 5th ed. (2017)

18 July 2017

DevOps

(Up-to-date source of this post.)

The opposite of DevOps is despair. -- Gene Kim

If HW and SW are sufficiently fault tolerant, the remaining problems are human. -- TPoCSA


DevOps is a set of techniques to solve the chronic conflict between Development an IT Operations. This conflict causes:

  • increased time to market
  • problematic code deployments
  • increased number of Sev 1 outages
  • unplanned work
  • technical debt

The techniques are:

  • architecture and processes
  • technical practices
  • cultural norms

DevOps is an emerging field in operations. The practice of DevOps typically appears in web application and cloud environments, but its influence is spreading to all parts of all industries.

The principles behind DevOps work patterns are the same principles that transformed manufacturing.

DevOps has been formed by:

  • Lean principles
  • Agile Community
  • infrastructure as code
  • CI, CD
  • Innovation Culture
  • Toyota Kata

"The Three ways" are the underpinning principles that all the DevOps patterns can be derived from.

The First Way - workflow (process)

What

  • left-to-right flow of work from Dev to Ops to the customer
  • each step is done in a repeatable way
  • small batch sizes and intervals of works
  • never passing defects to downstream work centers
  • local optimization must not degrade global performance (optimmize for global goals)

Practices

  • continuous build, CI, CD
  • environments on demand
  • limiting WIP
  • building safe (to change) systems

The Second Way - feedback (communication)

What

  • constant flow of fast feedback from right-to-left at all stages of the values stream

Practices

  • stopping the production line when builds/tests go wrong
  • constantly elevating the improvements of daily work over daily work
  • fast automated test suites
  • shared goals and shared pain between Devs and Ops
  • pervasive telemetry

The Third Way - experimentation (trying new things)

Culture that fosters:

  • continual experimentation (risk taking and learning from success and failure)
  • understanding that repetition and practice is the prerequisite to mastery
  • faults are introduced into the system to increase resilience (fire drills)

DevOps without Devs

It's possible to apply these DevOps principles to any complex process:

1) Turn chaotic processes into repeatable, measurable ones.

  • document process until consistent => automate => self-service
  • don't do IT tasks but maintain the system that does the IT tasks

2) Talk about problems and issues within and between teams.

  • don't obscure or suffer through problems

3) Try new things, measure the results, keep the successes, and learn from the failures.

  • create a culture of learning and experimentation

4) Do work in small batches so we can learn and pivot along the way.

  • it's better to deliver some results each day than to hold back and deliver the entire result at the end
  • you'll get feedback and fix problems sooner and avoid too much waisted effort

5) Drive configuration and infrastructure from machine readable sources kept under a source control system.

  • IaC is flexible, testable and can benefit from software engineering techniques

6) Always be improving.

  • always be identifying the next big bottleneck and experiment to fix it
  • don't wait for a major disaster

7) Pull, don't push.

  • determine the desired weekly output, allocate resources, and pull the work through the system to completion

8) Build community.

  • you, your team, your company and world's IT community are interdependent

Injecting modern concepts into legacy processes

1) Identify the team’s value streams — processes done for the business, or requested by the business.

2) Each step in the process needs to be clearly defined so that it can be done in a repeatable fashion. That is, a reasonably well-trained person should be able to do the step and the result will be the same as the result from another reasonably trained person.

3) Once the process is defined, amplify the feedback loops. That is, make sure each step has a way to raise the visibility of problems so that they are worked on, not ignored. Collect measurements on the length, frequency, and failure rate of the steps. Make this data available to all involved.

4) Based on the feedback find pain points (steps that are the most error prone, unreliable, or slow) and build business case

  • What problem really needs to be solved?
  • Gather metrics showing how things can improve.
  • Make business case showing why we need change and what we will get.

5) Find allies, share and collaborate

  • Ex. find people outside your team who can benefit from the change.

6) Start small and build from the bottom up

  • Start with something easy to build confidence while working out issues.
  • Ex.: document => share documentation => automate (script) => automate more (Jenkins, RunDeck)

Source

17 October 2016

Creating a Module::Build Distribution

(Up-to-date source of this post.)

Creating a Module::Build Distribution

We show here how to create a Perl distribution using Module::Build build system with Module::Starter. The other Perl build system (we don't show here) is ExtUtils::MakeMaker. For sophisticated distribution creation see Dist::Zilla.

Create config file ~/.module-starter/config:

author: Foo Bar
email: foo@bar.org
builder: Module::Build
verbose: 1
# Allow adding new modules to existing distro.
plugins: Module::Starter::AddModule

... or use module-starter (see below) with command line arguments like:

--author="Foo Bar" \
--email=foo@bar.org \
--mb \
--verbose \
--plugin=Module::Starter::AddModule

Run basic commands

  • install needed modules: cpanm Module::Build Module::Starter Module::Starter::AddModule
  • create (a working skeleton of) module distribution: module-starter --module=Animal
  • change to the created distro directory: cd Animal
  • create the Build script: perl Build.PL
  • build the distro (modules from lib copied to blib staging area and embedded documenation translated into Unix manpage in blib/libdoc): ./Build
  • make sure the tests pass: ./Build test (or run individual tests - see below)
  • test the distro: ./Build disttest
  • create the distro: ./Build dist

Add modules

  • add new modules: module-starter --module=Sheep,Cow,Horse --dist=Animal
  • add new modules (we are inside our distribution directory): module-starter --module=Sheep,Cow,Horse --dist=.

Run individual tests

  • rebuild distro and run test including modules from blib/lib: ./Build && perl -Iblib/lib -T t/Cow.t
  • rebuild distro and run test including modules from blib/lib: ./Build && perl -Mblib -T t/Cow.t

Measure out test coverage

  • run testcover target: ./Build testcover
  • turn the collected statistics into human-readable reports: cover

The following is a typical release cycle for github users:

  1. Work on changes until all tests pass
  2. Make sure the Changes file documents all major changes
  3. Make your changes atomic, all changes related to a particular fix or feature should go in a single commit, including the Changes entry.
  4. Bump the version
  5. Upload to PAUSE
  6. Tag with the version. By convention for version 1.01 the tag would be 'v1.01'
  7. Push to github

For more see:

12 September 2016

Finding a Good CPAN Module

(Up-to-date source of this post.)

Anyone can contribute to CPAN. It is also pretty huge. This means you might want to separate the wheat from the chaff. To do that these hints might help:

  1. What's the code activity? The more the better.

  2. How many issues do we have? The open bugs show community involvement, the closed ones are about the maintainer's diligence.

  3. What aboout the quality and quantity of tests and their results.

  4. Are there any reverse dependencies (they use the same basic principle as Google PageRank)? The more the better.

  5. Follow the advice of experts in the Perl field, ex. Task::Kensho.

NOTE: don't check any of the hints above in isolation but all/more of them together.

Resources:

  • https://www.usenix.org/system/files/login/articles/logindec1410_blank-edelman.pdf

17 May 2016

tcpdump - standard Unix tool for analyzing network packets

(Up-to-date source of this post.)

  • despite its name it can do much more than capturing TCP headers
  • can sniff traffic on many network types (including 802.1Q VLAN)
  • de facto standard for command line packet analysis in Unix environment

Useful options:

-D -- list available interfaces

-i INTERFACE -- listen on INTERFACE (default: lowest numbered interface)

-w FILE -- write raw packets to FILE

-r FILE -- read packets from FILE

-nn -- turn off host and protocol name resolution (to avoid generating DNS packets)

-s0 -- set snaplength to 0, i.e. read the whole packet not just first 68 bytes (default if version >= 4.0)

-t -- turn off timestamp entries

-c COUNT -- capture COUNT packets and stop

Examples:

tcpdump -nni eth1 -w packets.pcap
tcpdump -nnr packets.pcap

Output format will vary based upon what protocols are in use:

  • TCP

    timestamp L3_protocol sIP.sPort > dIP.dPort: TCP_flags,
    TCP_sequence_number, TCP_acknowledgement_number, TCP_windows_size,
    data_length
    
  • UDP

    timestamp L3_protocol sIP.sPort > dIP.dPort: L4_protocol, data_length
    
  • use up to -vvv to provide more information on headers

  • use -x to get entire packets (including data not just headers) in hex format
  • use -A to get entire packets in hex and ASCII format
  • use -X to get entire packets in hex and ASCII format

Packet Filtering

  • utilizes the Berkeley Packet Filter (BPF) format
  • added to the end of the command (recommended to use single quotes)

    tcpdump -nnr packets.pcap 'tcp dst port 8080' -w packets_tcp8080.pcap
    tcpdump -nnr packets.pcap -F known_good_hosts.bpf
    

BPF

           operator
 primitive   |      primitive
     |       |         |
+---------+  | +----------------+
|         |  | |                |
udp port 53 && dst host 192.0.2.2
 |        |
 |        value
qualifier

Qualifiers

  • host
  • net - network in CIDR notation
  • port
  • src - communication source
  • dst - communication destination
  • ip - IP protocol
  • tcp - TCP protocol
  • upd - UPP protocol

Logical operators

  • && - true when both conditions are true
  • || - true when either condition is true
  • ! - true when a condition is NOT met

Examples

  • host 192.0.2.100 - match traffic to/from 192.0.2.100
  • dst host 2001:db8:85a3::8a2e:370:7334 - match traffic to the IPv6 address
  • ether host 00:50:56:98:60:92 - match traffic to the specified MAC address
  • !port 22 - match any traffic not to/from port 22
  • icmp - match all ICMP traffic
  • !ip6 - match everything that is not IPv6

Resources

  • Applied Network Security Monitoring

13 May 2016

Common Vagrant Tasks

(Up-to-date source of this post.)

Search and add a box (virtual machine image):

vagrant box add https://atlas.hashicorp.com/ubuntu/boxes/trusty64
  • added box is global to the vagrant install
  • this is the base box (used to start the VM from the clean state)
  • base boxes are stored in ~/.vagrant.d/boxes

Initialize vagrant environment:

mkdir ubuntu-trusty64
cd ubuntu-trusty64
vagrant init ubuntu/trusty64
  • Vagrantfile is created

Start vagrant environment:

vagrant up
  • vagrant "imports" (copies) the base box to provider specific location (ex. ~/.VirtualBox)

Check box(es) status:

vagrant status

Check box(es) SSH configuration:

vagrant ssh-config

Ssh to a box:

vagrant ssh

Clean up:

# save VM's state; fastest to start again; eats most diskspace (hard disk + saved state of RAM)
vagrant suspend

# graceful shutdown; slower to start again, still eats disk space (hard disk)
vagrant halt

# power down and remove all of the guest hard disks; even slower to
# start again (reimport of the base box and reprovisioning)
vagrant destroy

Show status of all vagrant environments on the host (independent of the directory you're in):

vagrant global-status [--prune]

To share a folder from the host on the guest, add following to Vagrantfile:

config.vm.synced_folder "../../eset-repos", "/shared/eset-repos",
  owner: "jreisinger", group: "jreisinger"

Resources

  • https://docs.vagrantup.com
  • https://sysadmincasts.com/episodes/4-vagrant
  • http://docs-v1.vagrantup.com/v1/docs/