2018-09-26

Perl one-liners

I think Perl one-liners are still super useful. They are small Perl programs that are run directly from command line. Like this one from the Kubernetes job documentation:

perl -Mbignum=bpi -wle 'print bpi(2000)' # calculate PI to 2000 digits

perl is the Perl language interpreter. -M and -wle are command line switches (or flags or options) that modify the perl’s behaviour. See below for explanation of what they mean. The string within the quotes is the Perl code that gets executed. In this case it uses the bpi subroutine from the bignum module to calculate the PI with accuracy of 2000 digits. The command will take a while to finish.

Switches

These are some of the most used command line switches:

  • -e '<code>'execute <code>
  • -E '<code>'Execute <code> enabling new features for your version of Perl
  • -w – enable warnings (generally advisable)
  • -p – loop through lines, reading and printing them (in-script equivalent: while (<>) { [<code>] print })
  • -n – loop through lines, reading but not printing them
  • -l – print a newline ($/ actually) after each line of output and chomp input newlines if used with -n or -p
  • -i[<.ext>] – edit files in-place, optionally create backups with <.ext> extension
  • -aautosplit the $_ default variable into @F array (space is the default separator, change it with -F, ex. -F:)
  • -M<module>[=<subroutine>,...] – load subroutine(s) from a Module

See perlrun for more.

Cut

Cut out 2nd and 1st space separated field (column):

$ cat birthdays.txt
2890-09-22 Bilbo Baggins
2968-09-22 Frodo Baggins
$ perl -wlane 'print join " ", @F[1,0]' birthdays.txt
Bilbo 2890-09-22
Frodo 2968-09-22

The field numbering starts at 0. We use join to put a space between the cut out fields.

Find lines in logs that contain error or warning:

perl -wne '/error|warning/i && print' /var/log/*.log

The thing between slashes is a regular expression. It means match string error or string warning anywhere in the log line. i says to Perl to ignore the case. So it will match ERROR, error, Warning etc. If the regex finds a match (i.e. evaluates to true) the && logical operator runs the print statement that will print the line containing the match.

Get IP addresses from logs:

journalctl --since "00:00" | perl -wlne '/((?:\d{1,3}\.){3}\d{1,3})/ && print $1' | \
sort | uniq > /tmp/ips.txt

The IP address regex explained:

(               # capturing parenthesis to be retrieved via $1
    (?:         # non capturing parenthesis, only for grouping
        \d{1,3} # one to three decimal numbers
        \.      # literal dot
    ){3}        # three times all within innermost parenthesis
        \d{1,3} # one to three decimal numbers
)               

For a more serious program where you want to cover possible edge cases you should use a well tested module Regexp::Common as suggested by PerlMonks.

Replace

Replace /bin/sh with /bin/bash and emit the transformed passwd file to STDOUT:

perl -wpe 's#/bin/sh$#/bin/bash#' /etc/passwd

We used # instead of / as delimeter for better readibility since the strings themselves contain slashes. $ means end of the string.

Replace colour with color in all text files. The original files will be kept with .bak suffix:

perl -i.bak -wpe 's/colour/color/g' *.txt

g (global) means replace all occurences in a string not just the first one.

Convert between DOS and Unix newline:

perl -i -wpe 's/\r//'  <file1> <file2> ... # dos-to-unix
perl -i -wpe 's/$/\r/' <file1> <file2> ... # unix-to-dos

Calculate

Calculate the total size of log files older than 30 days:

find /opt/splunk/syslog/ -iname "*log*" -type f -mtime +30 | \
perl -wlne '$sum += (stat)[7]}{print $sum'

The stat function returns a 13-element list of status info about a file. We take the 8th element (with index 7) which is the size of a file. We loop over the found files and add the size of each into the $sum variable. The handy Eskimo Greeting Operator is for printing the $sum when the loop is over (suggested by PerlMonks).

More

perl1line.txt

For a deeper dive see Famous Perl One-Liners Explained. If you want a book have a look at Minimal Perl for UNIX and Linux People.

2018-02-28

Quick Docker

(Up-to-date source of this post.)

Docker is a container technology. It's a well timed fusion of

  • kernel features
  • filesystem tricks
  • networking hacks

Think of a container not as a virtual machine but a very lighweight wrapper around a single Unix process.

Docker revision controls

  1. filesystem layers
  2. image tags

Terminology

Docker server - the docker command run in daemon mode on a Linux host:

$ sudo docker -d -H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375

Docker image - one or more filesystem layers and metadata that represent all the files required to run a Dockerized application

Docker container - a Linux container that has been instantiated from a Docker image

Working with Docker images

To launch a container

  • download a public image
  • create your own

To create a custom image you need a Dockerfile - each line in a Dockerfile creates a new image layer that is stored by Docker

Build an image:

git clone https://github.com/spkane/docker-node-hello.git
cd docker-node-hello
docker build -t example/docker-node-hello:latest .

Run an image (or a container?):

docker run -d -p 80:8080 example/docker-node-hello:latest
  • -p 80:8080 tells Docker to proxy the container's port 80 on the host's port 8080 (port binding)
  • example/docker-node-hello:latest is a tag

Remove an image:

docker images
docker rmi <image_id>

Remove all images on your Docker host:

docker rmi $(docker images -q -)

Working with Docker containers

A container is a self-contained execution environment that shares the kernel of the host system and which is (optionally) isolated from other containers in the system.

Containers are a Linux only technology.

Create a container (see also "Run an image" above):

docker run --rm -ti ubuntu /bin/bash 
  • run - create + start
  • --rm - delete the container when it exits
  • -t - allocate a pseudo-TTY
  • -i - interactive session, e.i. keep STDIN open
  • /bin/bash - executable to run within the container

Get into a running container:

docker ps
docker exec -it <container_id> /bin/bash

Stop a container:

docker stop <container_id>

Remove a container:

docker ps -a
docker rm <container_id>

Remove all containers on your Docker host:

docker rm  $(docker ps -a -q)

Sources

  • Docker: Up & Running (2015)
  • Unix and Linux System Administration Handbook, 5th ed. (2017)