--- title: Awk date: 2020-12-31 15:18:34 background: bg-slate-600 tags: - bash - text - script categories: - Linux Command intro: | This is a one page quick reference cheat sheet to the [GNU awk](https://www.gnu.org/software/gawk/manual/gawk.html), which covers commonly used awk expressions and commands. plugins: - copyCode --- Getting Started --------------- ### Have a try ```shell script {.wrap} $ awk -F: '{print $1, $NF}' /etc/passwd ``` ---- | - | - | - | |---|---------------|---------------------------| | | `-F:` | Colon as a separator | | | `{...}` | Awk program | | | `print` | Prints the current record | | | `$1` | First field | | | `$NF` | Last field | | | `/etc/passwd` | Input data file | {.left-text} ### Awk program ``` BEGIN {} {} {} ... END {< final actions >} ``` #### Example ``` awk ' BEGIN { print "\n>>>Start" } !/(login|shutdown)/ { print NR, $0 } END { print "<<30 {print $1}' /etc/passwd ``` See: [Conditions](#awk-conditions) ### Generate 1000 spaces ``` awk 'BEGIN{ while (a++ < 1000) s=s " "; print s }' ``` See: [Loops](#awk-loops) ### Arrays ``` awk 'BEGIN { fruits["mango"] = "yellow"; fruits["orange"] = "orange" for(fruit in fruits) { print "The color of " fruit " is " fruits[fruit] } }' ``` See: [Arrays](#awk-arrays) ### Functions ``` # => 5 awk 'BEGIN{print length("hello")}' # => HELLO awk 'BEGIN{print toupper("hello")}' # => hel awk 'BEGIN{print substr("hello", 1, 3)}' ``` See: [Functions](#awk-functions) Awk Variables --------- ### Build-in variables | - | - | |----------------|-----------------------------------------------------| | `$0` | Whole line | | `$1, $2...$NF` | First, second… last field | | `NR` | `N`umber of `R`ecords | | `NF` | `N`umber of `F`ields | | `OFS` | `O`utput `F`ield `S`eparator
_(default " ")_ | | `FS` | input `F`ield `S`eparator
_(default " ")_ | | `ORS` | `O`utput `R`ecord `S`eparator
_(default "\n")_ | | `RS` | input `R`ecord `S`eparator
_(default "\n")_ | | `FILENAME` | Name of the file | ### Expressions | - | - | |---------------------|------------------------------------| | `$1 == "root"` | First field equals root | | `{print $(NF-1)}` | Second last field | | `NR!=1{print $0}` | From 2nd record | | `NR > 3` | From 4th record | | `NR == 1` | First record | | `END{print NR}` | Total records | | `BEGIN{print OFMT}` | Output format | | `{print NR, $0}` | Line number | | `{print NR " " $0}` | Line number (tab) | | `{$1 = NR; print}` | Replace 1st field with line number | | `$NF > 4` | Last field > 4 | | `NR % 2 == 0` | Even records | | `NR==10, NR==20` | Records 10 to 20 | | `BEGIN{print ARGC}` | Total arguments | | `ORS=NR%5?",":"\n"` | Concatenate records | ### Examples Print sum and average ``` awk -F: '{sum += $3} END { print sum, sum/NR } ' /etc/passwd ``` Printing parameters ``` awk 'BEGIN { for (i = 1; i < ARGC; i++) print ARGV[i] }' a b c ``` Output field separator as a comma ``` awk 'BEGIN { FS=":";OFS=","} {print $1,$2,$3,$4}' /etc/passwd ``` Position of match ``` awk 'BEGIN { if (match("One Two Three", "Tw")) print RSTART }' ``` Length of match ``` awk 'BEGIN { if (match("One Two Three", "re")) print RLENGTH }' ``` ### Environment Variables | - | - | |-----------|-----------------------------------------------------------| | `ARGC` | Number or arguments | | `ARGV` | Array of arguments | | `FNR` | `F`ile `N`umber of `R`ecords | | `OFMT` | Format for numbers
_(default "%.6g")_ | | `RSTART` | Location in the string | | `RLENGTH` | Length of match | | `SUBSEP` | Multi-dimensional array separator
_(default "\034")_ | | `ARGIND` | Argument Index | ### GNU awk only | - | - | |---------------|-----------------------| | `ENVIRON` | Environment variables | | `IGNORECASE` | Ignore case | | `CONVFMT` | Conversion format | | `ERRNO` | System errors | | `FIELDWIDTHS` | Fixed width fields | ### Defining variable ``` awk -v var1="Hello" -v var2="Wold" ' END {print var1, var2} ' ` - `<=` - `>=` {.cols-3 .marker-none} ### Examples ``` awk 'BEGIN { if ("foo" ~ "^fo+$") print "Fooey!"; }' ``` #### Not match ``` awk 'BEGIN { if ("boo" !~ "^fo+$") print "Boo!"; }' ``` #### if in array ``` awk 'BEGIN { assoc["foo"] = "bar"; assoc["bar"] = "baz"; if ("foo" in assoc) print "Fooey!"; }' ``` Awk Functions ---------- ### Common functions {.col-span-2} | Function | Description | |-----------------------|---------------------------------------------------------------------------------| | `index(s,t)` | Position in string s where string t occurs, 0 if not found | | `length(s)` | Length of string s (or $0 if no arg) | | `rand` | Random number between 0 and 1 | | `substr(s,index,len)` | Return len-char substring of s that begins at index (counted from 1) | | `srand` | Set seed for rand and return previous seed | | `int(x)` | Truncate x to integer value | | `split(s,a,fs)` | Split string s into array a split by fs, returning length of a | | `match(s,r)` | Position in string s where regex r occurs, or 0 if not found | | `sub(r,t,s)` | Substitute t for first occurrence of regex r in string s (or $0 if s not given) | | `gsub(r,t,s)` | Substitute t for all occurrences of regex r in string s | | `system(cmd)` | Execute cmd and return exit status | | `tolower(s)` | String s to lowercase | | `toupper(s)` | String s to uppercase | | `getline` | Set $0 to next input record from current input file. | ### User defined function ``` awk ' # Returns minimum number function find_min(num1, num2){ if (num1 < num2) return num1 return num2 } # Returns maximum number function find_max(num1, num2){ if (num1 > num2) return num1 return num2 } # Main function function main(num1, num2){ result = find_min(num1, num2) print "Minimum =", result result = find_max(num1, num2) print "Maximum =", result } # Script execution starts here BEGIN { main(10, 60) } ' ``` Awk Arrays --------- ### Array with index ``` awk 'BEGIN { arr[0] = "foo"; arr[1] = "bar"; print(arr[0]); # => foo delete arr[0]; print(arr[0]); # => "" }' ``` ### Array with key ``` awk 'BEGIN { assoc["foo"] = "bar"; assoc["bar"] = "baz"; print("baz" in assoc); # => 0 print("foo" in assoc); # => 1 }' ``` ### Array with split ``` awk 'BEGIN { split("foo:bar:baz", arr, ":"); for (key in arr) print arr[key]; }' ``` ### Array with asort ``` awk 'BEGIN { arr[0] = 3 arr[1] = 2 arr[2] = 4 n = asort(arr) for (i = 1; i <= n ; i++) print(arr[i]) }' ``` ### Multi-dimensional ``` awk 'BEGIN { multidim[0,0] = "foo"; multidim[0,1] = "bar"; multidim[1,0] = "baz"; multidim[1,1] = "boo"; }' ``` ### Multi-dimensional iteration ``` awk 'BEGIN { array[1,2]=3; array[2,3]=5; for (comb in array) { split(comb,sep,SUBSEP); print sep[1], sep[2], array[sep[1],sep[2]] } }' ``` Awk Conditions ---------- ### if-else statement ``` awk -v count=2 'BEGIN { if (count == 1) print "Yes"; else print "Huh?"; }' ``` #### Ternary operator ``` awk -v count=2 'BEGIN { print (count==1) ? "Yes" : "Huh?"; }' ``` ### Exists ``` awk 'BEGIN { assoc["foo"] = "bar"; assoc["bar"] = "baz"; if ("foo" in assoc) print "Fooey!"; }' ``` #### Not exists ``` awk 'BEGIN { assoc["foo"] = "bar"; assoc["bar"] = "baz"; if ("Huh" in assoc == 0 ) print "Huh!"; }' ``` ### switch ``` awk -F: '{ switch (NR * 2 + 1) { case 3: case "11": print NR - 1 break case /2[[:digit:]]+/: print NR default: print NR + 1 case -1: print NR * -1 } }' /etc/passwd ``` Awk Loops ---------- ### for...i ``` awk 'BEGIN { for (i = 0; i < 10; i++) print "i=" i; }' ``` #### Powers of two between 1 and 100 ``` awk 'BEGIN { for (i = 1; i <= 100; i *= 2) print i }' ``` ### for...in ``` awk 'BEGIN { assoc["key1"] = "val1" assoc["key2"] = "val2" for (key in assoc) print assoc[key]; }' ``` #### Arguments ``` awk 'BEGIN { for (argnum in ARGV) print ARGV[argnum]; }' a b c ``` ### Examples {.row-span-3} #### Reverse records ``` awk -F: '{ x[NR] = $0 } END { for (i = NR; i > 0; i--) print x[i] } ' /etc/passwd ``` #### Reverse fields ``` awk -F: '{ for (i = NF; i > 0; i--) printf("%s ",$i); print "" }' /etc/passwd ``` #### Sum by record ``` awk -F: '{ s=0; for (i = 1; i <= NF; i++) s += $i; print s }' /etc/passwd ``` #### Sum whole file ``` awk -F: ' {for (i = 1; i <= NF; i++) s += $i; }; END{print s} ' /etc/passwd ``` ### while {.row-span-2} ``` awk 'BEGIN { while (a < 10) { print "- " " concatenation: " a a++; } }' ``` #### do...while ``` awk '{ i = 1 do { print $0 i++ } while (i <= 5) }' /etc/passwd ``` ### Break ``` awk 'BEGIN { break_num = 5 for (i = 0; i < 10; i++) { print i if (i == break_num) break } }' ``` ### Continue ``` awk 'BEGIN { for (x = 0; x <= 10; x++) { if (x == 5 || x == 6) continue printf "%d ", x } print "" }' ``` Awk Formatted Printing --------- ### Usage #### Right align ``` awk 'BEGIN{printf "|%10s|\n", "hello"}' | hello| ``` #### Left align ``` awk 'BEGIN{printf "|%-10s|\n", "hello"}' |hello | ``` ### Common specifiers | Character | Description | |---------------|-----------------------| | `c` | ASCII character | | `d` | Decimal integer | | `e`, `E`, `f` | Floating-point format | | `o` | Unsigned octal value | | `s` | String | | `%` | Literal % | ### Space ``` awk -F: '{ printf "%-10s %s\n", $1, $(NF-1) }' /etc/passwd | head -n 3 ``` Outputs ```shell script root /root bin /bin daemon /sbin ``` ### Header ``` awk -F: 'BEGIN { printf "%-10s %s\n", "User", "Home" printf "%-10s %s\n", "----","----"} { printf "%-10s %s\n", $1, $(NF-1) } ' /etc/passwd | head -n 5 ``` Outputs ``` User Home ---- ---- root /root bin /bin daemon /sbin ``` Miscellaneous ------------- ### Regex Metacharacters - `\` - `^` - `$` - `.` - `[` - `]` - `|` - `(` - `)` - `*` - `+` - `?` {.cols-3 .marker-none} ### Escape Sequences | - | - | |------|---------------------| | `\b` | Backspace | | `\f` | Form feed | | `\n` | Newline (line feed) | | `\r` | Carriage return | | `\t` | Horizontal tab | | `\v` | Vertical tab | ### Run script ```shell script $ cat demo.awk #!/usr/bin/awk -f BEGIN { x = 23 } { x += 2 } END { print x } $ awk -f demo.awk /etc/passwd 69 ``` Also see -------- - [The GNU Awk User's Guide](https://www-zeuthen.desy.de/dv/documentation/unixguide/infohtml/gawk/gawk.html) _(www-zeuthen.desy.de)_ - [AWK cheatsheet](https://gist.github.com/Rafe/3102414) _(gist.github.com)_