UNIX SHell

practice / master these

bulk file renaming

for f in ./*; do mv "$f" "${f/Silicon_Valley/S01E01}"; done
# unix 'sr' tool(symbolic link rename) works well.
sr app State::ProductItems::ProductItemMapping State::BaseUnitMapping

# linux: echo to another session (if you want to open multiple terminal windows via a webpage and share between these windows)
echo 'ROFLCOPTER' > /dev/pts/3

Hex dump

# hex dump
xxd file1.bk | head
# 00000000: 7265 7175 6972 6520 2762 756e 646c 6572  require 'bundler
# 00000010: 2f67 656d 5f74 6173 6b73 270a 7265 7175  /gem_tasks'.requ
# 00000020: 6972 6520 276a 656b 796c 6c27 0a72 6571  ire 'jekyll'.req

Run script multiple times

for i in {1..10}; do you_script; done

# linux: echo to another session (if you want to open multiple terminal windows via a webpage and share between these windows)
echo 'ROFLCOPTER' > /dev/pts/3

Hex dump

# hex dump
xxd file1.bk | head
# 00000000: 7265 7175 6972 6520 2762 756e 646c 6572  require 'bundler
# 00000010: 2f67 656d 5f74 6173 6b73 270a 7265 7175  /gem_tasks'.requ
# 00000020: 6972 6520 276a 656b 796c 6c27 0a72 6571  ire 'jekyll'.req

Run script multiple times

for i in {1..10}; do you_script; done

basic rename of strings in files

sr () { ag -0 -l $2 $1 | xargs -0 sed -i '' -e "s/$2/$3/g" }

Every Bash script

a must do for every bash script. (However consider Ruby for anything that is harder than elementary) (massive credit to Mr. SE)

#!/usr/bin/env bash
set -euo pipefail

The -e option stands for “exit immediately” The -u option stands for “unbound (or unset) variables”. This causes the script to exit with an error if it attempts to use a variable that has not been set. By default, bash would use an empty string as the value for such variables, which can lead to subtle bugs. The -o is required pipefail

also can add ‘debugger’ to show what scripts are running

set -x

Always use [[ ]] instead of [ ]

Always quote your variables

Always use latest bash (as of 2024-jan I had 3.2.57(1) Mac M2, brew’ed it to 5.2.21)

`yes`

auto-respond

# will type `y`
yes | some_command
# or provide an ARG, which will type 'no' in this case
yes no | some_command 

name of current directory

basename $(pwd)

`PIPESTATUS` (Bash and ZSH differ)

is an array variable in Bash that holds the exit status of the last foreground pipeline (a sequence of one or more commands separated by the pipe | operator). Each element of the array corresponds to the exit status of a command in the pipeline.

echo 'lala' | grep 'da' |  curl non_existent.com
# curl: (6) Could not resolve host: non_existent.com

# BASH
echo ${PIPESTATUS[@]}
# 0 1 6

# ZSH
echo $pipestatus[@]
# 0 1 6

`jq`

cat db/tickets.json | jq '.[].submitter_id' | wc -l

`sed`

echo howtogonk | sed 's/gonk/geek/'
# howtogeek
echo 's/hello/world/' > myscript.sed
sed -f myscript.sed input.txt > output.txt

# Print lines
sed -n '1,6p' README.md
sed -n -e '1,6p' -e '10,16p' README.md

# delete lines and create a backup
sed -i'.bak' '/AWS/d' README.md

take the first line

# take the first line returned
some_command | sed -n '1p'

# or awk 
some_command | awk 'NR==1'

# or head
some_command | head -n 1

diff in USB devices

ls /dev > temp.patch
# plug or unplug device
diff <(ls /dev ) <(cat temp.patch ) | grep -E "^[<>]" | sed 's/[<>] //'

diff in USB devices

ls /dev > temp.patch
# plug or unplug device
diff <(ls /dev ) <(cat temp.patch ) | grep -E "^[<>]" | sed 's/[<>] //'

`pbcopy` Apple and `xclip` Linux

cat file.txt | pbcopy

combo with jq / head / copy

bat db/tickets.json | jq '.[]._id' -r | head -n 1 | xclip -selection clipboard

xargs + find

find . -maxdepth 1 -name "*.md" | xargs tail
find . -name "*.txt" | xargs rm
ls | xargs rm
echo "dir1 dir2 dir3" | xargs mkdir
# Parallelizing commands:
find . -name "*.txt" | xargs -P 4 gzip

Parallel

echo "file1 file2 file3" | parallel command-to-run {}
parallel echo "Hello, {}" ::: Alice Bob Charlie
ls *.txt | parallel wc -l
find *.md | xargs tail | parallel echo '------------------' | less

Background jobs / Vi session

you can send current session into the background by ctrl +z (at least in VI) use fg to list all jobs in the background, or switch into a job(tab after fg)

Piping grep + awk + xargs

your_script | grep keyword_in_line | awk '{print $1}' | command_to_use_return_value_from_the_previous_step

## grab args
```sh
jobs -p | ag lala | awk '{print $3}' | xargs kill -9
## or
ps aux | grep ruby | awk '{ print $2 }'
##  with ruby
ps aux | ruby -ne 'puts $_.split("/").last'

echo foo$(you_script)bar

Data Streams `stdin` `stdout` `stderr`

Great Tutorial

## stdin 0<
## stdout 1> # OR JUST stdout >
## stderr 2>

find /qwerty -type f 1> ./lala 2> ./err
## OR to APPEND instead of overwriting
find /qwerty -type f 1>> ./lala 2>> ./err

Permissions

chmod 666

In Unix-based systems, file permissions are represented by a series of three numbers, each representing a different group of users: the owner, the group, and everyone else.

The “chmod” command stands for “change mode,” and “666” is a code that represents the permissions being granted.

Remember only, 1-execute, 2-write, 4-read. The sum of these will result in appropriate permission level

No permissions
Execute permission
Write permission
Write and execute permissions
Read permission
Read and execute permissions
Read and write permissions
Read, write, and execute permissions

Error debugging tools

lsof: “lsof” stands for “list open files.” It displays information about files and network connections that processes have open on the system. It can help identify which processes are using specific files, sockets, or network connections.
htop: “htop” is an interactive process viewer that provides a real-time overview of system resource usage. It displays CPU and memory utilization, running processes, and allows for easy process management, such as killing processes or changing their priorities.
ps: “ps” is a command-line tool used to view information about running processes. It provides a snapshot of active processes, their resource usage, and process attributes like process IDs (PIDs), parent PIDs, CPU and memory usage, and more.
strace: “strace” is a powerful diagnostic tool that traces and records system calls made by a process and the signals received by it. It helps in analyzing the behavior of a program or troubleshooting issues related to system calls.
tcpdump: “tcpdump” is a network packet analyzer that captures and displays network traffic` in real-time. It can help debug network-related issues by capturing packets and allowing you to analyze their contents and flow.
netstat: “netstat” is a command-line tool used to display active network connections, routing tables, and various network interface statistics. It can provide information about open ports, established connections, and network-related statistics.
dmesg: “dmesg” displays the kernel ring buffer, which contains messages related to the system’s hardware and software. It can be useful for troubleshooting hardware-related issues, device initialization problems, or kernel-level error messages.
gdb: “gdb” is a powerful debugger that allows you to analyze and debug programs. It enables you to inspect variables, set breakpoints, step through code, and track down the root cause of program crashes or abnormal behavior.

lsof -i -P | grep -i "listen"
# or
ps aux | grep postgres

brew services issues might be down and need / (re)starting

brew services list

brew services start postgres@12

sometimes temp file record pid needs to be cleared manually

cat /opt/homebrew/var/postgres@12/postmaster.pid
rm /opt/homebrew/var/postgres@12/postmaster.pid
brew info postgresql@12

Cron

crontab -l # list

crontab -e 
crontab -u root -e # for specific user, you can also set cron jobs for other users that you can't log into

@hourly echo 'hourly cron'
@reboot echo 'cron message triggered by reboot'

* * * * * echo "hello world"
* * 1 * * echo "happy new month!"
0 9 * * 1 echo "happy Monday"
0 4 * * * root /usr/sbin/reboot


0 9 * * 1 /usr/local/bin/script.sh # state full path of a command for other users
# save and exit! tail and see your echos

tail -f /var/log/syslog | grep echo

Cheatsheet / generator https://crontab.guru

Signals

kill -l # list signals

kill(3123, 15) # this is for sending signal 15 to a process 3123, but the syntax to shell is different in an opposite order
raise(signum) # to send a signal yourself

## Find a process that uses port and kill it

```sh
lsof -i :9292
kill -9 <PID>
# or
lsof -i :9292 | grep IP | awk '{print $2}' | xargs kill -9

handy quirks

cat

Append to file using cat and HEREDOC

cat <<EOF >> filename.txt
This is the first line to append.
This is the second line to append.
EOF

rename

mv db/{oldName,newName}_structure.sql

send session into the background / jobs

In VIM ctrl + Z Backgrounds the job jobs To list all jobs fg Foreground last job (fg 1 or 2 to select specific)

Calendar

cal 2018
cal -3

`auto start on boot` DIY service

Create new service


vi /etc/systemd/system/my_services.service

Often it is important to include user, I had python scripts not working since by default it was executed by system and not my User

[Unit]
Description=MyServices

[Service]
Type=simple
ExecStart=/usr/bin/python3 /full/path/app.py
User=anton
Restart=always
WorkingDirectory=/full/path/dir/

[Install]
WantedBy=multi-user.target

available options for the Restart:

no: Do not automatically restart the service.
on-success: Restart the service only if it exits with a clean (zero) exit status.
on-failure: Restart the service only if it exits with a non-zero exit status.
on-abnormal: Restart the service if it exits with a non-zero exit status, or if it terminates unexpectedly (e.g., due to a signal).
on-abort: Restart the service only if it was terminated due to a signal.
always: Always restart the service regardless of the exit status or how it was terminated.

daemon reload

sudo systemctl daemon-reload

enable service

sudo systemctl enable my_services.service

start service

sudo systemctl start my_services.service

check status

sudo systemctl status my_services.service

soft reboot

sudo shutdown -r now

Symlinking, iNode

ls -li

# ubuntu 
sudo debugfs /dev/mapper/ubuntu--vg-ubuntu--lv
stat file.name

mkdir and go into it

mkdir new_dir && cd $_

#fullpath
mkdir -p ./not_yet_created_dir/new_dir && cd $_

# multiple dirs
mkdir -p ./not_yet_created_dir/{a,b}/{x,y,z} && cd $_

Anton, Friendlyantz

UNIX SHell

practice / master these

bulk file renaming

Run script multiple times

Run script multiple times

basic rename of strings in files

Every Bash script

`yes`

name of current directory

`PIPESTATUS` (Bash and ZSH differ)

`jq`

`sed`

take the first line

diff in USB devices

diff in USB devices

`pbcopy` Apple and `xclip` Linux

combo with jq / head / copy

xargs + find

Parallel

Background jobs / Vi session

Piping grep + awk + xargs

Data Streams `stdin` `stdout` `stderr`

Permissions

Error debugging tools

Cron

Signals

handy quirks

cat

rename

send session into the background / jobs

Calendar

`auto start on boot` DIY service

soft reboot

Symlinking, iNode

mkdir and go into it

Leave a comment

Anton, Friendlyantz

practice / master these

bulk file renaming

Run script multiple times

Run script multiple times

basic rename of strings in files

Every Bash script

yes

name of current directory

PIPESTATUS (Bash and ZSH differ)

jq

sed

take the first line

diff in USB devices

diff in USB devices

pbcopy Apple and xclip Linux

combo with jq / head / copy

xargs + find

Parallel

Background jobs / Vi session

Piping grep + awk + xargs

Data Streams stdin stdout stderr

Permissions

Error debugging tools

Cron

Signals

handy quirks

cat

rename

send session into the background / jobs

Calendar

auto start on boot DIY service

soft reboot

Symlinking, iNode

mkdir and go into it

Leave a comment

`yes`

`PIPESTATUS` (Bash and ZSH differ)

`jq`

`sed`

`pbcopy` Apple and `xclip` Linux

Data Streams `stdin` `stdout` `stderr`

`auto start on boot` DIY service