Files
reference/source/_posts/awk.md
T
2023-06-15 10:44:51 +08:00

779 lines
16 KiB
Markdown

---
title: Awk
date: 2020-12-31 15:18:34
background: bg-slate-600
tags:
- bash
- text
- script
categories:
- Linux Command
intro: |
This is a one page quick reference cheat sheet to the [GNU awk](https://www.gnu.org/software/gawk/manual/gawk.html), which covers commonly used awk expressions and commands.
plugins:
- copyCode
---
Getting Started
---------------
### Have a try
```shell script {.wrap}
$ awk -F: '{print $1, $NF}' /etc/passwd
```
----
| - | - | - |
|---|---------------|---------------------------|
| | `-F:` | Colon as a separator |
| | `{...}` | Awk program |
| | `print` | Prints the current record |
| | `$1` | First field |
| | `$NF` | Last field |
| | `/etc/passwd` | Input data file |
{.left-text}
### Awk program
```
BEGIN {<initializations>}
<pattern 1> {<program actions>}
<pattern 2> {<program actions>}
...
END {< final actions >}
```
#### Example
```
awk '
BEGIN { print "\n>>>Start" }
!/(login|shutdown)/ { print NR, $0 }
END { print "<<<END\n" }
' /etc/passwd
```
### Variables {.row-span-2}
```bash
$1 $2/$(NF-1) $3/$NF
▼ ▼ ▼
┌──────┬──────────────────┬───────┐
$0/NR ▶ │ ID │ WEBSITE │ URI │
├──────┼──────────────────┼───────┤
$0/NR ▶ │ 1 │ cheatsheets.zip │ awk │
├──────┼──────────────────┼───────┤
$0/NR ▶ │ 2 │ google.com │ 25 │
└──────┴──────────────────┴───────┘
```
---
```
# First and last field
awk -F: '{print $1,$NF}' /etc/passwd
# With line number
awk -F: '{print NR, $0}' /etc/passwd
# Second last field
awk -F: '{print $(NF-1)}' /etc/passwd
# Custom string
awk -F: '{print $1 "=" $6}' /etc/passwd
```
See: [Variables](#awk-variables)
### Awk program examples {.col-span-2 .row-span-2}
```
awk 'BEGIN {print "hello world"}' # Prints "hello world"
awk -F: '{print $1}' /etc/passwd # -F: Specify field separator
# /pattern/ Execute actions only for matched pattern
awk -F: '/root/ {print $1}' /etc/passwd
# BEGIN block is executed once at the start
awk -F: 'BEGIN { print "uid"} { print $1 }' /etc/passwd
# END block is executed once at the end
awk -F: '{print $1} END { print "-done-"}' /etc/passwd
```
### Conditions
```
awk -F: '$3>30 {print $1}' /etc/passwd
```
See: [Conditions](#awk-conditions)
### Generate 1000 spaces
```
awk 'BEGIN{
while (a++ < 1000)
s=s " ";
print s
}'
```
See: [Loops](#awk-loops)
### Arrays
```
awk 'BEGIN {
fruits["mango"] = "yellow";
fruits["orange"] = "orange"
for(fruit in fruits) {
print "The color of " fruit " is " fruits[fruit]
}
}'
```
See: [Arrays](#awk-arrays)
### Functions
```
# => 5
awk 'BEGIN{print length("hello")}'
# => HELLO
awk 'BEGIN{print toupper("hello")}'
# => hel
awk 'BEGIN{print substr("hello", 1, 3)}'
```
See: [Functions](#awk-functions)
Awk Variables
---------
### Build-in variables
| - | - |
|----------------|-----------------------------------------------------|
| `$0` | Whole line |
| `$1, $2...$NF` | First, second… last field |
| `NR` | `N`umber of `R`ecords |
| `NF` | `N`umber of `F`ields |
| `OFS` | `O`utput `F`ield `S`eparator <br> _(default " ")_ |
| `FS` | input `F`ield `S`eparator <br> _(default " ")_ |
| `ORS` | `O`utput `R`ecord `S`eparator <br> _(default "\n")_ |
| `RS` | input `R`ecord `S`eparator <br> _(default "\n")_ |
| `FILENAME` | Name of the file |
### Expressions
| - | - |
|---------------------|------------------------------------|
| `$1 == "root"` | First field equals root |
| `{print $(NF-1)}` | Second last field |
| `NR!=1{print $0}` | From 2nd record |
| `NR > 3` | From 4th record |
| `NR == 1` | First record |
| `END{print NR}` | Total records |
| `BEGIN{print OFMT}` | Output format |
| `{print NR, $0}` | Line number |
| `{print NR " " $0}` | Line number (tab) |
| `{$1 = NR; print}` | Replace 1st field with line number |
| `$NF > 4` | Last field > 4 |
| `NR % 2 == 0` | Even records |
| `NR==10, NR==20` | Records 10 to 20 |
| `BEGIN{print ARGC}` | Total arguments |
| `ORS=NR%5?",":"\n"` | Concatenate records |
### Examples
Print sum and average
```
awk -F: '{sum += $3}
END { print sum, sum/NR }
' /etc/passwd
```
Printing parameters
```
awk 'BEGIN {
for (i = 1; i < ARGC; i++)
print ARGV[i] }' a b c
```
Output field separator as a comma
```
awk 'BEGIN { FS=":";OFS=","}
{print $1,$2,$3,$4}' /etc/passwd
```
Position of match
```
awk 'BEGIN {
if (match("One Two Three", "Tw"))
print RSTART }'
```
Length of match
```
awk 'BEGIN {
if (match("One Two Three", "re"))
print RLENGTH }'
```
### Environment Variables
| - | - |
|-----------|-----------------------------------------------------------|
| `ARGC` | Number or arguments |
| `ARGV` | Array of arguments |
| `FNR` | `F`ile `N`umber of `R`ecords |
| `OFMT` | Format for numbers <br> _(default "%.6g")_ |
| `RSTART` | Location in the string |
| `RLENGTH` | Length of match |
| `SUBSEP` | Multi-dimensional array separator <br> _(default "\034")_ |
| `ARGIND` | Argument Index |
### GNU awk only
| - | - |
|---------------|-----------------------|
| `ENVIRON` | Environment variables |
| `IGNORECASE` | Ignore case |
| `CONVFMT` | Conversion format |
| `ERRNO` | System errors |
| `FIELDWIDTHS` | Fixed width fields |
### Defining variable
```
awk -v var1="Hello" -v var2="Wold" '
END {print var1, var2}
' </dev/null
```
#### Use shell variables
```
awk -v varName="$PWD" '
END {print varName}' </dev/null
```
Awk Operators
---------
### Operators
| - | - |
|------------------|-------------|
| `{print $1}` | First field |
| `$2 == "foo"` | Equals |
| `$2 != "foo"` | Not equals |
| `"foo" in array` | In array |
#### Regular expression
| - | - |
|-----------------|-------------------|
| `/regex/` | Line matches |
| `!/regex/` | Line not matches |
| `$1 ~ /regex/` | Field matches |
| `$1 !~ /regex/` | Field not matches |
#### More conditions
| - | - |
|------------------------|-----|
| `($2 <= 4 || $3 < 20)` | Or |
| `($1 == 4 && $3 < 20)` | And |
### Operations
#### Arithmetic operations
- `+`
- `-`
- `*`
- `/`
- `%`
- `++`
- `--`
{.cols-3 .marker-none}
#### Shorthand assignments
- `+=`
- `-=`
- `*=`
- `/=`
- `%=`
{.cols-3 .marker-none}
#### Comparison operators
- `==`
- `!=`
- `<`
- `>`
- `<=`
- `>=`
{.cols-3 .marker-none}
### Examples
```
awk 'BEGIN {
if ("foo" ~ "^fo+$")
print "Fooey!";
}'
```
#### Not match
```
awk 'BEGIN {
if ("boo" !~ "^fo+$")
print "Boo!";
}'
```
#### if in array
```
awk 'BEGIN {
assoc["foo"] = "bar";
assoc["bar"] = "baz";
if ("foo" in assoc)
print "Fooey!";
}'
```
Awk Functions
----------
### Common functions {.col-span-2}
| Function | Description |
|-----------------------|---------------------------------------------------------------------------------|
| `index(s,t)` | Position in string s where string t occurs, 0 if not found |
| `length(s)` | Length of string s (or $0 if no arg) |
| `rand` | Random number between 0 and 1 |
| `substr(s,index,len)` | Return len-char substring of s that begins at index (counted from 1) |
| `srand` | Set seed for rand and return previous seed |
| `int(x)` | Truncate x to integer value |
| `split(s,a,fs)` | Split string s into array a split by fs, returning length of a |
| `match(s,r)` | Position in string s where regex r occurs, or 0 if not found |
| `sub(r,t,s)` | Substitute t for first occurrence of regex r in string s (or $0 if s not given) |
| `gsub(r,t,s)` | Substitute t for all occurrences of regex r in string s |
| `system(cmd)` | Execute cmd and return exit status |
| `tolower(s)` | String s to lowercase |
| `toupper(s)` | String s to uppercase |
| `getline` | Set $0 to next input record from current input file. |
### User defined function
```
awk '
# Returns minimum number
function find_min(num1, num2){
if (num1 < num2)
return num1
return num2
}
# Returns maximum number
function find_max(num1, num2){
if (num1 > num2)
return num1
return num2
}
# Main function
function main(num1, num2){
result = find_min(num1, num2)
print "Minimum =", result
result = find_max(num1, num2)
print "Maximum =", result
}
# Script execution starts here
BEGIN {
main(10, 60)
}
'
```
Awk Arrays
---------
### Array with index
```
awk 'BEGIN {
arr[0] = "foo";
arr[1] = "bar";
print(arr[0]); # => foo
delete arr[0];
print(arr[0]); # => ""
}'
```
### Array with key
```
awk 'BEGIN {
assoc["foo"] = "bar";
assoc["bar"] = "baz";
print("baz" in assoc); # => 0
print("foo" in assoc); # => 1
}'
```
### Array with split
```
awk 'BEGIN {
split("foo:bar:baz", arr, ":");
for (key in arr)
print arr[key];
}'
```
### Array with asort
```
awk 'BEGIN {
arr[0] = 3
arr[1] = 2
arr[2] = 4
n = asort(arr)
for (i = 1; i <= n ; i++)
print(arr[i])
}'
```
### Multi-dimensional
```
awk 'BEGIN {
multidim[0,0] = "foo";
multidim[0,1] = "bar";
multidim[1,0] = "baz";
multidim[1,1] = "boo";
}'
```
### Multi-dimensional iteration
```
awk 'BEGIN {
array[1,2]=3;
array[2,3]=5;
for (comb in array) {
split(comb,sep,SUBSEP);
print sep[1], sep[2],
array[sep[1],sep[2]]
}
}'
```
Awk Conditions
----------
### if-else statement
```
awk -v count=2 'BEGIN {
if (count == 1)
print "Yes";
else
print "Huh?";
}'
```
#### Ternary operator
```
awk -v count=2 'BEGIN {
print (count==1) ? "Yes" : "Huh?";
}'
```
### Exists
```
awk 'BEGIN {
assoc["foo"] = "bar";
assoc["bar"] = "baz";
if ("foo" in assoc)
print "Fooey!";
}'
```
#### Not exists
```
awk 'BEGIN {
assoc["foo"] = "bar";
assoc["bar"] = "baz";
if ("Huh" in assoc == 0 )
print "Huh!";
}'
```
### switch
```
awk -F: '{
switch (NR * 2 + 1) {
case 3:
case "11":
print NR - 1
break
case /2[[:digit:]]+/:
print NR
default:
print NR + 1
case -1:
print NR * -1
}
}' /etc/passwd
```
Awk Loops
----------
### for...i
```
awk 'BEGIN {
for (i = 0; i < 10; i++)
print "i=" i;
}'
```
#### Powers of two between 1 and 100
```
awk 'BEGIN {
for (i = 1; i <= 100; i *= 2)
print i
}'
```
### for...in
```
awk 'BEGIN {
assoc["key1"] = "val1"
assoc["key2"] = "val2"
for (key in assoc)
print assoc[key];
}'
```
#### Arguments
```
awk 'BEGIN {
for (argnum in ARGV)
print ARGV[argnum];
}' a b c
```
### Examples {.row-span-3}
#### Reverse records
```
awk -F: '{ x[NR] = $0 }
END {
for (i = NR; i > 0; i--)
print x[i]
}
' /etc/passwd
```
#### Reverse fields
```
awk -F: '{
for (i = NF; i > 0; i--)
printf("%s ",$i);
print ""
}' /etc/passwd
```
#### Sum by record
```
awk -F: '{
s=0;
for (i = 1; i <= NF; i++)
s += $i;
print s
}' /etc/passwd
```
#### Sum whole file
```
awk -F: '
{for (i = 1; i <= NF; i++)
s += $i;
};
END{print s}
' /etc/passwd
```
### while {.row-span-2}
```
awk 'BEGIN {
while (a < 10) {
print "- " " concatenation: " a
a++;
}
}'
```
#### do...while
```
awk '{
i = 1
do {
print $0
i++
} while (i <= 5)
}' /etc/passwd
```
### Break
```
awk 'BEGIN {
break_num = 5
for (i = 0; i < 10; i++) {
print i
if (i == break_num)
break
}
}'
```
### Continue
```
awk 'BEGIN {
for (x = 0; x <= 10; x++) {
if (x == 5 || x == 6)
continue
printf "%d ", x
}
print ""
}'
```
Awk Formatted Printing
---------
### Usage
#### Right align
```
awk 'BEGIN{printf "|%10s|\n", "hello"}'
| hello|
```
#### Left align
```
awk 'BEGIN{printf "|%-10s|\n", "hello"}'
|hello |
```
### Common specifiers
| Character | Description |
|---------------|-----------------------|
| `c` | ASCII character |
| `d` | Decimal integer |
| `e`, `E`, `f` | Floating-point format |
| `o` | Unsigned octal value |
| `s` | String |
| `%` | Literal % |
### Space
```
awk -F: '{
printf "%-10s %s\n", $1, $(NF-1)
}' /etc/passwd | head -n 3
```
Outputs
```shell script
root /root
bin /bin
daemon /sbin
```
### Header
```
awk -F: 'BEGIN {
printf "%-10s %s\n", "User", "Home"
printf "%-10s %s\n", "----","----"}
{ printf "%-10s %s\n", $1, $(NF-1) }
' /etc/passwd | head -n 5
```
Outputs
```
User Home
---- ----
root /root
bin /bin
daemon /sbin
```
Miscellaneous
-------------
### Regex Metacharacters
- `\`
- `^`
- `$`
- `.`
- `[`
- `]`
- `|`
- `(`
- `)`
- `*`
- `+`
- `?`
{.cols-3 .marker-none}
### Escape Sequences
| - | - |
|------|---------------------|
| `\b` | Backspace |
| `\f` | Form feed |
| `\n` | Newline (line feed) |
| `\r` | Carriage return |
| `\t` | Horizontal tab |
| `\v` | Vertical tab |
### Run script
```shell script
$ cat demo.awk
#!/usr/bin/awk -f
BEGIN { x = 23 }
{ x += 2 }
END { print x }
$ awk -f demo.awk /etc/passwd
69
```
Also see
--------
- [The GNU Awk User's Guide](https://www-zeuthen.desy.de/dv/documentation/unixguide/infohtml/gawk/gawk.html) _(www-zeuthen.desy.de)_
- [AWK cheatsheet](https://gist.github.com/Rafe/3102414) _(gist.github.com)_