owned this note
# GöHPCoffee - Bash ## Brace expansion Easy way to rename files or populate loops. Examples: ```bash [nimboden@glogin3 ~]$ echo {a,b} a b [nimboden@glogin3 ~]$ echo /home/marcus/{file2,file1} /home/marcus/file2 /home/marcus/file1 [nimboden@glogin3 ~]$ echo /home/marcus/{file2,file1}.cpp /home/marcus/file2.cpp /home/marcus/file1.cpp [nimboden@glogin3 ~]$ touch file1 [nimboden@glogin3 ~]$ cp file1{,.bak} [nimboden@glogin3 ~]$ ls file1* file1 file1.bak [nimboden@glogin3 ~]$ echo /home/marcus/{file2,file1{a,b,c}} /home/marcus/file2 /home/marcus/file1a /home/marcus/file1b /home/marcus/file1c [nimboden@glogin3 ~]$ echo {a,b,c}{1,2,3} a1 a2 a3 b1 b2 b3 c1 c2 c3 [nimboden@glogin3 ~]$ echo {a..c} a b c [nimboden@glogin3 ~]$ echo {c..a} c b a [nimboden@glogin3 ~]$ echo {1..5}{a..z} 1a 1b 1c 1d 1e 1f 1g 1h 1i 1j 1k 1l 1m 1n 1o 1p 1q 1r 1s 1t 1u 1v 1w 1x 1y 1z 2a 2b 2c 2d 2e 2f 2g 2h 2i 2j 2k 2l 2m 2n 2o 2p 2q 2r 2s 2t 2u 2v 2w 2x 2y 2z 3a 3b 3c 3d 3e 3f 3g 3h 3i 3j 3k 3l 3m 3n 3o 3p 3q 3r 3s 3t 3u 3v 3w 3x 3y 3z 4a 4b 4c 4d 4e 4f 4g 4h 4i 4j 4k 4l 4m 4n 4o 4p 4q 4r 4s 4t 4u 4v 4w 4x 4y 4z 5a 5b 5c 5d 5e 5f 5g 5h 5i 5j 5k 5l 5m 5n 5o 5p 5q 5r 5s 5t 5u 5v 5w 5x 5y 5z ``` [Explanation](https://www.howtogeek.com/725657/how-to-use-brace-expansion-in-linuxs-bash-shell/) ## Command and Process Substitution Command Substitution lets you use the output of a command in another command or variable assignment. You can either save the output in a variable or directly use it in the next command. ```bash [nimboden@glogin3 ~]$ ps PID TTY TIME CMD 26866 pts/12 00:00:00 bash 28727 pts/12 00:00:00 ps [nimboden@glogin3 ~]$ var1=$(ps) [nimboden@glogin3 ~]$ echo "$var1" PID TTY TIME CMD 26866 pts/12 00:00:00 bash 28749 pts/12 00:00:00 ps [nimboden@glogin3 ~]$ sbatch --dependency=afterany:$(sbatch --parsable testjob.sh) dep.sh Submitted batch job 3645412 ``` [Command Substitution](https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_03) Process Substitution creates temporary files from the output of a process/command to use in other commands. This is useful if a cli program expects a file, not something via arguments or stdin: ```bash #Two unsorted files, list2 has an extra letter in it [nimboden@glogin3 coffee]$ cat list1 d b e a [nimboden@glogin3 coffee]$ cat list2 b d c e a [nimboden@glogin3 coffee]$ diff <(sort list1) <(sort list2) 2a3 > c [nimboden@glogin3 coffee]$ diff -y <(sort list1) <(sort list2) a a b b > c d d e e ``` [Process Substitution](https://mywiki.wooledge.org/ProcessSubstitution) ## loops ### worth-while examples ```bash [nimboden@glogin3 coffee]$ while read l; do echo prefix: $l;done < list2 prefix: b prefix: d prefix: c prefix: e prefix: a #change file a bit: [nimboden@glogin3 coffee]$ cat list2 b 1 d 2 c 1 e 3 a 8 [nimboden@glogin3 coffee]$ while read l; do echo prefix: $l;done < list2 prefix: b 1 prefix: d 2 prefix: c 1 prefix: e 3 prefix: a 8 #Put each element into its own different var and reverse the order (number first, then letter) [nimboden@glogin3 coffee]$ while read a b ; do echo prefix: $b $a; done < list2 prefix: 1 b prefix: 2 d prefix: 1 c prefix: 3 e prefix: 8 a #Change delimiter with IFS [nimboden@glogin3 coffee]$ sacct -pX JobID|JobName|Partition|Account|AllocCPUS|State|ExitCode| 3645386|testjob.sh|large96|nimboden|192|COMPLETED|0:0| 3645408|testjob.sh|large96|nimboden|192|COMPLETED|0:0| [nimboden@glogin3 coffee]$ while IFS="|" read id name part acc cpus state exc; do echo $acc,$id,$part;done< <( sacct -Xp) Account,JobID,part nimboden,3645386,large96 nimboden,3645408,large96 ``` ### foraging with for-loops Consider this dir: ```bash [nimboden@glogin3 coffee]$ ls -l total 0 -rw-r----- 1 nimboden nimboden 8 May 25 08:19 list1 -rw-r----- 1 nimboden nimboden 20 May 25 15:11 list2 -rw-r----- 1 nimboden nimboden 83 May 25 11:41 script.sh drwxr-x--- 2 nimboden nimboden 4096 May 25 11:17 tmp -rw-r----- 1 nimboden nimboden 179 May 25 11:52 trap.sh ``` If you want to go through files in a directory, **DO NOT** do this ```bash [nimboden@glogin3 coffee]$ for f in $(ls *);do echo $f;done list1 list2 script.sh trap.sh tmp: ``` Reason: File names can be anything! More on this [here](https://mywiki.wooledge.org/ParsingLs) Better: ```bash [nimboden@glogin3 coffee]$ for f in * ;do echo $f;done list1 list2 script.sh tmp trap.sh [nimboden@glogin3 coffee]$ for f in ./* ;do echo $f;done ./list1 ./list2 ./script.sh ./tmp ./trap.sh ``` But still not ideal. What happens if dir is empty? ```bash [nimboden@glogin3 coffee]$ cd tmp [nimboden@glogin3 tmp]$ ls -l total 0 [nimboden@glogin3 tmp]$ for f in * ;do echo $f;done * ``` Asterisk is not expanded and stays the `*`. This can cause problems: ```bash [nimboden@glogin3 tmp]$ for f in *;do cp $f ../;done cp: cannot stat ‘*’: No such file or directory ``` If you want to iterate over files in a dir, set the `nullglob` shell option. This makes sure that globs (such as `*`) evaluate to an empty string, if there are no files matching it: ```bash [nimboden@glogin3 tmp]$ shopt -s nullglob [nimboden@glogin3 tmp]$ for f in *;do cp $f ../;done [nimboden@glogin3 tmp]$ ``` ## Variable expansion and substrings This topic is... extensive to say the least. And I personally can never remember the syntax options. But it is good to know what's possible so you know what to google. ```bash #empty var [nimboden@glogin3 tmp]$ echo $TEST_PATH #Give default value ("moep" in this case) if empty [nimboden@glogin3 tmp]$ echo ${TEST_PATH:-moep} moep [nimboden@glogin3 tmp]$ echo $TEST_PATH #Still empty. Next: Set var to default value if empty [nimboden@glogin3 tmp]$ echo ${TEST_PATH:=moep} moep [nimboden@glogin3 tmp]$ echo $TEST_PATH moep ``` Getting the basepath and the filename (in pure bash without `dirname` and `basename`): ```bash [nimboden@glogin3 coffee]$ VAR=$(realpath trap.sh) [nimboden@glogin3 coffee]$ echo $VAR /home/nimboden/coffee/trap.sh [nimboden@glogin3 coffee]$ echo ${VAR##*/} trap.sh [nimboden@glogin3 coffee]$ echo ${VAR#*/} home/nimboden/coffee/trap.sh [nimboden@glogin3 coffee]$ echo ${VAR%/*} /home/nimboden/coffee ``` This is a small subset of possibilities. Read more [here](https://wiki.bash-hackers.org/syntax/pe) and [here](https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html). You may wonder: "Why should I do this in bash? There are cli tools that do that for me." First of all: Because you can! And secondly: Calling a new program takes time. Barely noticeable, when working on the cli, but if you have large loops, it will become more significant. ## Some safety features (`set -euxo pipefail`) I find these options generally helpful when writing scripts. There are [some caveats](https://mywiki.wooledge.org/BashPitfalls#set_-euo_pipefail), but for me they do more good then bad. I will give a brief introduction here. Please have a look at [this blog](https://vaneyckt.io/posts/safer_bash_scripts_with_set_euxo_pipefail/) for more examples ### set -e This causes a bash script to exit when there is an error. In many cases the rest of the script won't work anymore anyway. ### set -u Exit on undefined variables. Bash usually doesn't complain, if you use a variable that hasn't been filled before. It's just an empty string then. This options make bash terminate on a unset variable. Why? Consider the following: `for f in $TMP_PATH/*;do rm -rf $f; done` On first glance, that will remove all files in `$TMP_PATH`. But if `$TMP_PATH` is not set, it will be empty and the loop will essentiall be: `for f in /*;do rm -rf $f; done` This removes your whole root directory without a warning (if you're root). Yay. ### set -o pipefail If using `set -e`, piping commands into each other can have an unexpected effect: If the first command of the pipe fails, the script will **not** stop. So the `-e` options essentially only counts towards the last command of a pipe. This changes the behaviour, so that if any command fails, the script will stop. ### set -x Print the lines to the console before running them. Brilliant for debugging. To also get filename and line number in the output, run `export PS4='+$0 l $LINENO: '`. ## Simple math You can do simple math directly in the shell: ```bash [nimboden@glogin3 coffee]$ echo $((4 + 5 )) 9 [nimboden@glogin3 coffee]$ a=4 [nimboden@glogin3 coffee]$ echo $((a + 5 )) 9 ``` ## Shell script or function arguments The arguments of a shell script or function are handed over in a special variable and can be accessed like this: ```bash #!/bin/bash echo Scriptname: $0 echo Number of args $# echo First arguments $1 echo Second arguments $2 echo loop throgh arguments: for arg in "$@"; do echo $arg done ``` Example: ```bash [nimboden@glogin3 coffee]$ bash script.sh test test2 blah Scriptname: script.sh Number of args 3 First arguments test Second arguments test2 loop throgh arguments: test test2 blah ``` ## Traps [It's A Trap!](https://www.youtube.com/watch?v=4F4qzPbcFiA) You can send signals to your bash script. These are usually used to tell the script to stop (SIGTERM, SIGINT) or forcefully terminate (SIGKILL). `kill $pid` sends the SIGTERM signal and `kill -9 $pid` sends the SIGKILL signal to the programm with the process ID $pid. There are more signals (`kill -l` lists all) and you can use some of them (e.g. SIGUSR1, SIGUSR2) to communicate with your script. Bash has the `trap` function to define what to do when a signal arrives: ```bash #!/bin/bash trap "echo Booh!" SIGINT SIGTERM echo "pid is $$" sleep 60 & wait ``` This scripts will sleep for 60 seconds. If you try to kill it with `CTRL-C` or `kill $pid`, it will echo "Booh!": ```bash [nimboden@glogin3 coffee]$ bash trap.sh pid is 155040 ^CBooh! [nimboden@glogin3 coffee]$ ``` The `&` after sleep runs the command in the background and the wait command wait for all background processes to finish. This is necessary, as the sleep process would otherwise block execution of trap and it would only be executed after sleep is done. You can use traps in Slurm. `sbatch` lets you define a [`--signal`](https://slurm.schedmd.com/sbatch.html#OPT_signal) option to send user defined signals before a job runs into the timelimit. You can catch the signal with a trap and, for example, copy temporary checkpoint files back to your home directory: ``` can #!/bin/bash #SBATCH --signal=B12@600 trap "cp -r $TMP_LOCAL $HOME/backup" 12 echo "pid is $$" while :; do sleep 60 & # This script is not really doing anything. wait done ``` You can find more information on traps (and my shamelessly copied script) [here](https://tldp.org/LDP/Bash-Beginners-Guide/html/sect_12_02.html). # Link collection My go to resource for bash scripting: https://devhints.io/bash Interesting snippets: https://github.com/dylanaraps/pure-bash-bible Lots of tips, tricks and ideas about shell scripting: https://mywiki.wooledge.org/ Especially the [pitfalls](https://mywiki.wooledge.org/BashPitfalls).