Work notes/tips

 

A somewhat-random collection of hints, tips, notes, syntax examples, and other data that have allowed me to filter data, troubleshoot issues, automate processes, & get work done.  Subject to editing at any time.  Mostly for my own reference – but if someone else finds something useful here, that’s great too.

 

Assorted bash notes:

Take a directory with file names like ‘ASI-soapui-project-ASI-P2-CI.xml’ & remove the first three name segments & the file extension:

   for i in $( ls -1 ./*.xml ); do basename $i .xml; done | awk -F “-” ‘{$1=$2=$3=””; print $0}’ – | awk ‘{$1=$1};1’ | tr [:blank:] ‘-‘ > ~/svclist.txt

 

Readable Tomcat/ps output (1225 is the Tomcat pid):
   ps aux | egrep [1]225 | tr “[:space:] ” “\n” | egrep -v “^$”

This example inserts commas into a file full of columns of long numbers (rewrites
the file on disk):
    sed -i ‘:a;s/\B[0-9]\{3\}\>/,&/;ta’ numbers.txt

 

Double-space a file which already has blank lines in it. Output file
should contain no more than one blank line between lines of text:
   sed ‘/^$/d;G’

 

List the File Descriptors that a process has open:
    sudo ls -Al /proc/<PID#>/fd

To assign the output of a command to a variable & also print it out:

    echo “$(ps aux | egrep [p]rocname | egrep [j]ava | awk ‘{print $2}’)” | tee /var/run/procname.pid
OR
    echo “$!” | tee /var/run/procname.pid

 

To find/search for a file named exactly NAME (not *NAME*), use
    locate -b ’\NAME’
(Because \ is a globbing character, this disables the implicit replacement
of NAME by *NAME*.)

 

Get cumulative size of all of the files in the “subcommand”:
    du -ch $( subcommand ) | tail -1 | cut -f 1

 

Useful for killing processes that spawn more quickly than you can
manually ‘kill’ them:
    for i in `ps aux | egrep [p]rocname | tr -s ‘ ‘ | cut -d ‘ ‘ -f 2`; do kill $i; done

 

To filter processes by a specific column (in this case, process state):
    ps aux | awk ‘$8 ~ “Z”‘

 

To see processes associated with a specific terminal:
    ps lt <pts/1>

 

Editing iptables filtering rules:
(If passing this output to another command, remember that the IPs are
displayed with dashes between the octets, not dots!!)
  iptables -L –line-numbers     (to show rules)
  iptables -D <Chain> <Line #>  (to delete a rule)

Convert Linux file permissions from ‘rwx’ format to numeric:
  stat –format %a <filename>

Show both ctime & atime for a file:
    stat -c ‘%x %z’ filename.txt

Remove empty lines & lines that have ‘#’ as the 1st non-whitespace
character:
    egrep -v “(^\s*#|^$)”

Display three lines of context before & after the line containing the
target term:
    egrep -n -A3 -B3 <pattern> <filename>

Show only the lines that differ between two files:
    diff -ys –suppress-common-lines <file1> <file2>

To see all processes on the system, ordered by PID number:
    ps -j ax

Sort by directory names:
   ls -AlFhrt /etc/ | egrep “^dr” | sort -k9

CRON reference
 — Minute (0 – 59)
 — Hour  (0 – 23)
 — DoM   (1 – 31)
 — Month (1 – 12)
 — DoW   (0 – 6)

Run a cron job every five minutes:
    */5 * * * *  /path/to/executable

List most-recent backup files:
 for i in $(sudo cat /home/adminuser/tmpstuff/backups.txt | egrep -v nexus | egrep -v lost );
do echo “$i”;ls -lt /mnt/backups/$i/filesystem/ | head -3 | egrep -v total; done
 for i in $(sudo cat /home/adminuser/tmpstuff/backups.txt | egrep -v nexus | egrep -v lost ); do echo “$i”;ls -lt /mnt/backups/$i/mysql/ | head -3 | egrep -v total; done

Use ‘top’ to monitor a specific process:
  top -c -p $(pgrep -d’,’ -f <>)
 (OR ‘pidstat 1‘)

 

---

vim – Visual Block Mode (big thanks to Nick for sharing this tip!)

First, move the cursor to the first char of the first line in the block of
code you want to comment, then type:

  CTRL + V

vim will go into VISUAL BLOCK mode.

Move the cursor down until you reach the last line of your code block.
Then type:

  Shift + I

now vim goes to INSERT mode and the cursor is at the first character of
the first line.

Finally, type # then ESC and the code block is now commented.

To uncomment, do the same things but instead of typing Shift + I,
you just type x to remove all # after highlighting them in VISUAL BLOCK
mode.

—————————————————————————————————————
Use vi editor identify multi lines to comment and comment them

  : set number
  :10,20s/^/#/

—————————————————————————————————————

vi editor – :set ignorecase

—————————————————————————————————————

Basic use of the Windows version of netcat (documentation seems to be rather scarce for this):
    ncat.exe -vuz <IP address> <dest port>

tshark -n -r /root/Capture1.pcap -z dests,tree -q

/usr/sbin/tshark -i eth1 -O udp -f "dst port 25826" -w ./Test1.pcap

The stdout macro, which is specified by POSIX and used by many
environments as the underlying I/O mechanism, behaves differently when
connected to a terminal and to a pipe: When connected to a terminal, the
output is “line buffered”, i.e. the buffer is flushed when a newline is
printed. When connected to a pipe, the stream is “fully buffered”, i.e.
output is only actually written if a fixed-size buffer is full. This
means that the output can actually appear immediately when testing the
code in the command line, but still suffer from the problem when
actually running with <program>.

 

---

 

Ansible (ad-hoc commands):
(On this system, I aliased ‘an’ to the ‘ansible’ command, and aliased ‘ap’ to the ‘ansible-playbook’ command – saved quite a few keystrokes.)

See a list of all of the ansible variables on a system (warning – verbose!):
    ansible -m setup hostname

Run an ansible ad-hoc task as the systemadmin user:
    an all -u systemadmin -m shell -a “cd /usr/local/corporate/scripts;git status”

View root’s crontab on every server managed by Ansible:
    an all -b -m shell -a “crontab -lu root”

Exclude the corps and localhost systems from running the ‘sample’
playbook:
    ap –limit ‘all:!corps:!localhost’ /home/systemadmin/playbooks/sample.yml

“…you can do something like `ansible somehosts -m shell -a “egrep ‘Invalid message’ /path/to/directory/* || exit 0”` to suppress the error.”

 

 

---

 

SSL:
   openssl x509 -in certificate.crt -text -noout

To simultaneously check the certificates of multiple domains at the command-line on your local server (requires creation of ‘servers.txt’):
    for i in $(cat servers.txt); do echo $i && curl –insecure -v $i 2>&1 | awk ‘BEGIN { cert=0 } /^\* SSL connection/ { cert=1 } /^\*/ { if (cert) print }’; done

 

---

 

GIT:

Show everything that’s happened in each of the commits between “1st” and “2nd”:
    git log -p <1st commit> <2nd commit>

Set git file permissions to executable:
    git update-index –chmod=+x buildwar.sh deploy.sh  (performs change locally, then you push to origin)

 

 

---

 

MongoDB:

“When performing a mongodump, all collections within the designated
databases will be dumped as BSON output. If no database is specified,
MongoDB will dump all databases except for the admin, test and local
databases as they are reserved for internal use.”

Get the list of Mongo DBs:
    mongo localhost:27017/localdb –eval “db.adminCommand(‘listDatabases’)” |
egrep name | sed ‘s/^[ \t]*//;s/[ \t]*$//’ | sed ‘s/”name” : //’ | sed ‘s/,//’ | tr -d ‘”‘  (replace w/ some variant of ‘awk $NF’??)

/usr/bin/mongorestore -d test2 /mnt/nfs/backups/hostname/mongo/temp/dump/localdb/mongo testrestore –eval “db.dropDatabase()”

> db[“MongoBackup_06-18-2018-03:07”].drop()
true

> db.stats()
{
        “db” : “localdb“,
        “collections” : 2,
        “views” : 0,
        “objects” : 378,
        “avgObjSize” :
3270.6931216931216,
        “dataSize” : 1236322,
        “storageSize” : 892928,
        “numExtents” : 0,
        “indexes” : 2,
        “indexSize” : 65536,
        “fsUsedSize” : 13432729600,
        “fsTotalSize” : 42147577856,
        “ok” : 1
}

 

 

---

 

 

Miscellaneous uncategorized:

for i in $( find . -type f -name *.xml | tr [:blank:] ‘-‘ | egrep -v “web\.xml” | egrep -v log4j ); do /bin/basename $i .xml; done | egrep -v “\([0-9]\)” > ~/svclist.txt

netstat -ln | egrep “*\.sock” | awk ‘{ print $9 }’

rsync `find . -name “*.pdf” -mtime -60` user@anothermachine:/tmp/

echo “$( date ‘+%F-%R’ )”

touch “$( date ‘+%c’ )”

netstat -ln | egrep “*\.sock” | awk ‘{ print $9 }’

—————————————————————————————————————
[adminuser@SYS1 ~]$
/usr/sbin/showmount -a
All mount points on SYS1:
{IP ADDRESS}:/mnt/volume-1
{IP ADDRESS}:/mnt/backups
{IP ADDRESS}:/mnt/backups
{IP ADDRESS}:/mnt/backups
{IP ADDRESS}:/mnt/backups
{IP ADDRESS}:/mnt/backups
{IP ADDRESS}:/mnt/volume-1
{IP ADDRESS}:/mnt/backups
{IP ADDRESS}:/mnt/backups
{IP ADDRESS}:/mnt/backups
{IP ADDRESS}:/mnt/volume-1
{IP ADDRESS}:/mnt/backups
{IP ADDRESS}:/mnt/volume-1
{IP ADDRESS}:/mnt/volume-1
{IP ADDRESS}:/mnt/backups

[adminuser@SYS1 ~]$
/usr/sbin/showmount -e

Export list for SYS1:
/mnt/monitor        {IP SUBNET}/16
/mnt/backups        {IP SUBNET}/16
/mnt/volume-1 {IP SUBNET}/16
/opt/data          
{IP ADDRESS},{IP ADDRESS}

To show all RPC services registered
with version 2 of the rpcbind protocol:
    rpcinfo -p <remote IP/hostname>

———————————————————————————————

xargs  –show-limits
Your environment variables take up 1869 bytes
POSIX upper limit on argument length (this system): 2617523
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 2615654
Size of command buffer we are actually using: 131072

Execution of xargs will continue now, and it will try to read its input
and run commands; if this is not what you wanted to happen, please type
the end-of-file keystroke.

—————————————————————————————————————