UNIX Commands with usage and examples

UNIX :

lsof

·         Find the process running on a specific port :  losf –i TCP:<port>

·         Kill all activity of Particular user : kill -9 ‘lsof  -t –u  tecmint’

Q. Write a command to list all the links from a directory?        

           ls –ltr | grep “^|”

Q. Create a readonly file in your home directory?    

           Chmod 400 <filename>

Q. Command to find which operating system we are using?

           Uname –a

Q. How do you know which process is taking how mch cpu and memory?

           top

Q. How to check how much space is used on the system?

           df -h

Tip - Zombie : The process is dead but have not been removed from the process table.

Q. How to find how many times a particular word is repeated in a file? 

          grep -c "word" filename

Q. How do you set the environment variable?  

          export

Q. How to replaces a word in a files?

         sed s/<odlword>/<newword>/g filename

Q. How to display particular coloum in a file?

         cut -f<column numebr> filename

Q. How to display nth line of a file?

         head -n filename | tail -1

Q. How to remove the header from a file?

         sed -i '1 d' filename

Q. How to remove the footer from a file?

         sed -i '$ d' filename

Q. How to find the length of a line in a file?

        set -n '<n> p' filename.

SCP Command Examples:

Let see the examples of scp command in unix or linux system.

1. Copying with in the same system

You can use the scp command just like the cp command to copy files from one directory to another directory. 

scp Unix-storage.dat /var/tmp/

This command copies the file unix-storage.dat from current directory to the /var/tmp directory.

2. Copy file from local host to remote server

This is most frequently used operation to transfer files in unix system. 

scp filename user@remotehost:/remote/directory/

This command connects to the remote host and copies the specified file to the /remote/directory/. 

3. Copy files from remote host to local server.

This operation is used when taking backup of the files in remote server.

scp user@remotehost:/usr/backup/oracle_backup.dat .

This command copies the oracle backup file in the remote host to the current directory.

4. Copying files between two remote servers

The scp command can also be used to copy files between two remote hosts.

scp source_user@source_remote_host:/usr/bin/mysql_backup.sh target_user@target_remote_host:/var/tmp/

The above command copies the mysql bakup shell script from the source remote host the /var/tmp directory of target remote host.

5. Copying a directory.

To copy all the files in a directory, use the -r option with the scp command. This makes the scp command to copy the directory recursively.

scp -r directory user@remotehost:/var/tmp/

The above command copies the directory from local server to the remote host.

6. Improving performance of scp command

By default the scp command uses the Triple-DES cipher/AES-128 to encrypt the data. Using the blowfish or arcfour encryption will improve the performance of the scp command.

scp -c blowfish filename  user@remoteserver:/var/
scp -c arcfour  localfile user@remoteserver:/var/

7. Limit bandwidth

You can limit the bandwidth used by the scp command using the -l option.

scp -l bandwidth_limit filename user@hostname:/usr/backup/
Here bandwidth_limit is numeric to be specified in kilobits per second.

8. Specifying the port number

We can make the scp command to copy the files over a specified port number using the -P option.


scp -P 6001 storage_backup.bat username@hostname:/tmp/

FIND COMMAND IN UNIX AND LINUX EXAMPLES

Find is one of the powerful utility of Unix (or Linux) used for searching the files in a directory hierarchy. The syntax of find command is

find [pathnames] [conditions]

Let see some practical exercises on using find command.

1. How to run the last executed find command?

!find

This will execute the last find command. It also displays the last find command executed along with the result on the terminal.

2. How to find for a file using name?

find -name "sum.java"
./bkp/sum.java
./sum.java

This will find all the files with name "sum.java" in the current directory and sub-directories.

3. How to find for files using name and ignoring case?

find -iname "sum.java"
./SUM.java
./bkp/sum.java
./sum.java

This will find all the files with name "sum.java" while ignoring the case in the current directory and sub-directories.

4. How to find for a file in the current directory only?

find -maxdepth 1 -name "sum.java"
./sum.java

This will find for the file "sum.java" in the current directory only

5. How to find for files containing a specific word in its name?

find -name "*java*"
./SUM.java
./bkp/sum.java
./sum.java
./multiply.java

It displayed all the files which have the word "java" in the filename

6. How to find for files in a specific directory?

find /etc -name "*java*"

This will look for the files in the /etc directory with "java" in the filename

7. How to find the files whose name are not "sum.java"?

find -not -name "sum.java"
.
./SUM.java
./bkp
./multiply.java

This is like inverting the match. It prints all the files except the given file "sum.java".

8. How to limit the file searches to specific directories?

find -name "sum.java"
./tmp/sum.java
./bkp/var/tmp/files/sum.java
./bkp/var/tmp/sum.java
./bkp/var/sum.java
./bkp/sum.java
./sum.java

You can see here the find command displayed all the files with name "sum.java" in the current directory and sub-directories.

a. How to print the files in the current directory and one level down to the current directory?

find -maxdepth 2 -name "sum.java"
./tmp/sum.java
./bkp/sum.java
./sum.java

b. How to print the files in the current directory and two levels down to the current directory?

find -maxdepth 3 -name "sum.java"
./tmp/sum.java
./bkp/var/sum.java
./bkp/sum.java
./sum.java

c. How to print the files in the subdirectories between level 1 and 4?

find -mindepth 2 -maxdepth 5 -name "sum.java"
./tmp/sum.java
./bkp/var/tmp/files/sum.java
./bkp/var/tmp/sum.java
./bkp/var/sum.java
./bkp/sum.java

9. How to find the empty files in a directory?

find . -maxdepth 1 -empty
./empty_file

10. How to find the largest file in the current directory and sub directories

find . -type f -exec ls -s {} \; | sort -n -r | head -1

The find command "find . -type f -exec ls -s {} \;" will list all the files along with the size of the file. Then the sort command will sort the files based on the size. The head command will pick only the first line from the output of sort.

11. How to find the smallest file in the current directory and sub directories

find . -type f -exec ls -s {} \; | sort -n -r | tail -1

Another method using find is

find . -type f -exec ls -s {} \; | sort -n  | head -1

12. How to find files based on the file type?

a. Finding socket files

find . -type s

b. Finding directories

find . -type d

c. Finding hidden directories

find -type d -name ".*"

d. Finding regular files

find . -type f

e. Finding hidden files

find . -type f -name ".*"

13. How to find files based on the size?

a. Finding files whose size is exactly 10M

find . -size 10M

b. Finding files larger than 10M size

find . -size +10M

c. Finding files smaller than 10M size

find . -size -10M

14. How to find the files which are modified after the modification of a give file.

find -newer "sum.java"

This will display all the files which are modified after the file "sum.java"

15. Display the files which are accessed after the modification of a give file.

find -anewer "sum.java"

16. Display the files which are changed after the modification of a give file.

find -cnewer "sum.java"

17. How to find the files based on the file permissions?

find . -perm 777

This will display the files which have read, write, and execute permissions. To know the permissions of files and directories use the command "ls -l".

18. Find the files which are modified within 30 minutes.

find . -mmin -30

19. Find the files which are modified within 1 day.

find . -mtime -1

20. How to find the files which are modified 30 minutes back

find . -not -mmin -30

21. How to find the files which are modified 1 day back.

find . -not -mtime -1

22. Print the files which are accessed within 1 hour.

find . -amin -60

23. Print the files which are accessed within 1 day.

find . -atime -1

24. Display the files which are changed within 2 hours.

find . -cmin -120

25. Display the files which are changed within 2 days.

find . -ctime -2

26. How to find the files which are created between two files.

find . -cnewer f1 -and ! -cnewer f2

So far we have just find the files and displayed on the terminal. Now we will see how to perform some operations on the files.

1. How to find the permissions of the files which contain the name "java"?

find -name "*java*"|xargs ls -l

Alternate method is

find -name "*java*" -exec ls -l {} \;

2. Find the files which have the name "java" in it and then display only the files which have "class" word in them?

find -name "*java*" -exec grep -H class {} \;

3. How to remove files which contain the name "java".

find -name "*java*" -exec rm -r {} \;

This will delete all the files which have the word “java" in the file name in the current directory and sub-directories.

Similarly you can apply other Unix commands on the files found using the find command. I will add more examples as and when i found.

EXECUTE A COMMAND AT A GIVEN TIME IN UNIX

Q) How to execute a command in unix or linux at a specific point of time.?

Recently I got a requirement from one of our client to execute a command at midnight of the day. Initially, I thought of using the crontab command to schedule the script at midnight. However the cron executes the job every day at midnight. My client wants to run that command only one time and after that the command should not run. Luckily unix has a built-in command, at, which allows us to run a script at specified time only once.

AT COMMAND EXAMPLES

The following at command runs the command "ls -lrt" at midnight of the day only once. 

> echo "ls -lrt" | at midnight

You can also execute a script at a specified time using the at command. An example is shown below: 

at -m 01:35 < scriptFile

The m option is used to send an email to the user after the job completes.

To list all the scheduled jobs, use the -l option 

> at -l
119     2014-07-01 00:00 a username

Here 119 is the jobname.

To remove a scheduled job, use the -r option. 

> at -r 119

CONNECT TO POSTGRES DATABASE IN UNIX SHELL SCRIPT

Q) How to connect to postgres database from unix shell or bash script and run sql queries?

The postgres command line utility, psql, can be used to connect to the postgres database and run some queries. The syntax of psql command is 

psql [option] [dbname[username]]

The psql command has so many options which we are not going to discuss here. In this article, we will see how to connect to postgres database from shell script.

CONNECTING TO POSGRES DATABASE

The following shell script connects to the postgres database, runs the select query and prints the result. 

#!/bin/sh
DATABASE=pg_database_name
USERNAME=pg_username
HOsTNAME=pg_hostname
export PGPASSWORD=pg_db_password

psql -h $HOSTNAME -U $USERNAME $DATABaSE << EOF
select * from table_name
EOF

In the above script, the password is exported to PGAPASSWORD variable in the shell script. If you dont want your password (hide password) to be in the script file, then put that export statment in the .bash_profile file. An other way is to use the .pgpass file where you can save your postgres connection details.

PGPASS FILE TO STORE CONNECTION DETAILS

The connection details in the file should contain lines in the following format. 

hostname:port:database:username:password

Place the pgpass file in the users home directory. If you want to place the pgpass file in any other directory, then reference the path of the file by PGPASSFILE variable.

In the shell script, you can skip all the connection details and simply use the psql command as shown in the following script: 

#!/bin/sh
psql << EOF
select * from table_name
EOF

That is it on how to connect to postgres database from unix shell script. If you have any questions, you can comment here.

CHECK IF A DIRECTORY EXISTS IN SHELL SCRIPT - UNIX

How to check whether a directory exists or not in unix or linux shell script?

I want to check for the directory existence in the shell script and if the directory exists, then I want to remove that directory. How can I do this in a shell script?

SHELL SCRIPT FOR DIRECTORY EXISTENCE CHECK

The following shell script checks for the given directory and then deletes that directory if it exists.

Shell Script: 

> cat directory_remove.sh

#!/usr/bin/bash
DIR=$1

if [[ -d $DIR ]]
then
  echo "deleting $DIR directory"
  rmdir $DIR
else
 echo "$DIR directory does not exists"
fi

Execute the above shell script as shown below: 

> bash directory_remove.sh directory-name

The -d option in the if clause is used to test for the directory existence in the shell script.

Note: The rmdir command deletes the directory only if it is empty. To remove the non-empty directories use the rm command with -r option as shown below: 

rm -r $DIR

SYMBOLIC LINK TO DIRECTORY

If you have created a symbolic link to a directory and you are making the directory check on the symbolic link in the if clause, then the if condition will satisfy. However the rmdir command will throw an error as it is a symbolic link and not a directory. This is shown in the following example: 

> mkdir directory-name
> ln -s directory-name symbolic-link

Now run the above bash script by passing the symbolic-link

> bash directory_remove.sh symbolic-link
deleting symbolic-link directory
rmdir: symbolic-link: Not a directory

After checking for the directory existence, then you have to check whether it is symbolic or not. The updated shell script is shown below: 

#!/usr/bin/bash
DIR=$1

if [[ -d $DIR ]]
then
  if [[ -L $DIR ]]
  then
    echo "$DIR is a symbolic link"
    rm $DIR
  else
    echo "$DIR is a directory"
    rmdir $DIR
  fi
else
 echo "$DIR directory does not exists"
fi

SED COMMAND TO DELETE LINES - BASED ON POSITION IN FILE

In the following examples, the sed command removes the lines in file that are in a particular position in a file. 

1. Delete first line or header line

The d option in sed command is used to delete a line. The syntax for deleting a line is: 

> sed 'Nd' file

Here N indicates Nth line in a file. In the following example, the sed command removes the first line in a file. 

> sed '1d' file
unix
fedora
debian
ubuntu

2. Delete last line or footer line or trailer line

The following sed command is used to remove the footer line in a file. The $ indicates the last line of a file. 

> sed '$d' file
linux
unix
fedora
debian

3. Delete particular line

This is similar to the first example. The below sed command removes the second line in a file. 

> sed '2d' file
linux
fedora
debian
ubuntu

4. Delete range of lines

The sed command can be used to delete a range of lines. The syntax is shown below: 

> sed 'm,nd' file

Here m and n are min and max line numbers. The sed command removes the lines from m to n in the file. The following sed command deletes the lines ranging from 2 to 4: 

> sed '2,4d' file
linux
ubuntu

5. Delete lines other than the first line or header line

Use the negation (!) operator with d option in sed command. The following sed command removes all the lines except the header line. 

> sed '1!d' file
linux

6. Delete lines other than last line or footer line

> sed '$!d' file
ubuntu

7. Delete lines other than the specified range

> sed '2,4!d' file
unix
fedora
debian

Here the sed command removes lines other than 2nd, 3rd and 4th. 

8. Delete first and last line

You can specify the list of lines you want to remove in sed command with semicolon as a delimiter. 

> sed '1d;$d' file
unix
fedora
debian

9. Delete empty lines or blank lines

> sed '/^$/d' file

The ^$ indicates sed command to delete empty lines. However, this sed do not remove the lines that contain spaces. 

SED COMMAND TO DELETE LINES - BASED ON PATTERN MATCH

In the following examples, the sed command deletes the lines in file which match the given pattern. 

10. Delete lines that begin with specified character

> sed '/^u/d' file
linux
fedora
debian

^ is to specify the starting of the line. Above sed command removes all the lines that start with character 'u'. 

11. Delete lines that end with specified character

> sed '/x$/d' file
fedora
debian
ubuntu

$ is to indicate the end of the line. The above command deletes all the lines that end with character 'x'. 

12. Delete lines which are in upper case or capital letters

> sed '/^[A-Z]*$/d' file

13. Delete lines that contain a pattern

> sed '/debian/d' file
linux
unix
fedora
ubuntu

14. Delete lines starting from a pattern till the last line

> sed '/fedora/,$d' file
linux
unix

Here the sed command removes the line that matches the pattern fedora and also deletes all the lines to the end of the file which appear next to this matching line. 

15. Delete last line only if it contains the pattern

> sed '${/ubuntu/d;}' file
linux
unix
fedora
debian

Here $ indicates the last line. If you want to delete Nth line only if it contains a pattern, then in place of $ place the line number. 

Note: In all the above examples, the sed command prints the contents of the file on the unix or linux terminal by removing the lines. However the sed command does not remove the lines from the source file. To Remove the lines from the source file itself, use the -i option with sed command. 

> sed -i '1d' file

If you dont wish to delete the lines from the original source file you can redirect the output of the sed command to another file. 

sed '1d' file > newfile

REPLACE STRING WITH AWK/SED COMMAND IN UNIX

Replace String With Awk/Sed Command In Unix:

You might have used the Sed Command often to replace the text in file. Awk can also be used to replace the strings in a file.

Here i will show you how to replace the string with awk command. To learn about replacing text with sed command go though the link, Replace String with Sed Command

Replace text with Awk command

1. First we will see a simple example of replacing the text. The source file contians the below data

>cat file.txt
Learn unix
Learn linux

We want to replace the word "unix" with "fedora". Here the word "unix" is in the second field. So, we need to check for the word "unix" in the second field and replace it with workd "fedora" by assigning the new value to the second field. The awk command to replace the text is

awk '{if($2=="unix") {$2="fedora"} print $0}' file.txt
Learn fedora
Learn linux

2. Now we will see a bit complex example.Consider the text file with the below data

>cat file.txt

left
(
In left
)
right
(
In top
)
top
(
In top
)
bottom
(
In bottom
)

Now replace the string, "top" in right section with the string "right". The output should look as

left
(
In left
)
right
(
In right
)
top
(
In top
)
bottom
(
In bottom
)

Here the delimiter in the text file is brace. We have to specify the delimiters in awk command with the record separators. The below awk command can be used to replace the string in a file

awk -vRS=")" '/right/{ gsub(/top/,"right"); }1' ORS=")" file.txt

Here RS is the input record separator and ORS is the output record separator.

SED COMMAND IN UNIX AND LINUX EXAMPLES

Sed is a Stream Editor used for modifying the files in unix (or linux). Whenever you want to make changes to the file automatically, sed comes in handy to do this. Most people never learn its power; they just simply use sed to replace text. You can do many things apart from replacing text with sed. Here I will describe the features of sed with examples.

Consider the below text file as an input.

>cat file.txt
unix is great os. unix is opensource. unix is free os.
learn operating system.
unixlinux which one you choose.

SED COMMAND EXAMPLES

1. Replacing or substituting string

Sed command is mostly used to replace the text in a file. The below simple sed command replaces the word "unix" with "linux" in the file.

>sed 's/unix/linux/' file.txt
linux is great os. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you choose.

Here the "s" specifies the substitution operation. The "/" are delimiters. The "unix" is the search pattern and the "linux" is the replacement string.

By default, the sed command replaces the first occurrence of the pattern in each line and it won't replace the second, third...occurrence in the line.

2. Replacing the nth occurrence of a pattern in a line.

Use the /1, /2 etc flags to replace the first, second occurrence of a pattern in a line. The below command replaces the second occurrence of the word "unix" with "linux" in a line.

>sed 's/unix/linux/2' file.txt
unix is great os. linux is opensource. unix is free os.
learn operating system.
unixlinux which one you choose.

3. Replacing all the occurrence of the pattern in a line.

The substitute flag /g (global replacement) specifies the sed command to replace all the occurrences of the string in the line.

>sed 's/unix/linux/g' file.txt
linux is great os. linux is opensource. linux is free os.
learn operating system.
linuxlinux which one you choose.

4. Replacing from nth occurrence to all occurrences in a line.

Use the combination of /1, /2 etc and /g to replace all the patterns from the nth occurrence of a pattern in a line. The following sed command replaces the third, fourth, fifth... "unix" word with "linux" word in a line.

>sed 's/unix/linux/3g' file.txt
unix is great os. unix is opensource. linux is free os.
learn operating system.
unixlinux which one you choose.

5. Changing the slash (/) delimiter

You can use any delimiter other than the slash. As an example if you want to change the web url to another url as

>sed 's/http:\/\//www/' file.txt

In this case the url consists the delimiter character which we used. In that case you have to escape the slash with backslash character, otherwise the substitution won't work.

Using too many backslashes makes the sed command look awkward. In this case we can change the delimiter to another character as shown in the below example.

>sed 's_http://_www_' file.txt
>sed 's|http://|www|' file.txt

6. Using & as the matched string

There might be cases where you want to search for the pattern and replace that pattern by adding some extra characters to it. In such cases & comes in handy. The & represents the matched string.

>sed 's/unix/{&}/' file.txt
{unix} is great os. unix is opensource. unix is free os.
learn operating system.
{unix}linux which one you choose.

>sed 's/unix/{&&}/' file.txt
{unixunix} is great os. unix is opensource. unix is free os.
learn operating system.
{unixunix}linux which one you choose.

7. Using \1,\2 and so on to \9

The first pair of parenthesis specified in the pattern represents the \1, the second represents the \2 and so on. The \1,\2 can be used in the replacement string to make changes to the source string. As an example, if you want to replace the word "unix" in a line with twice as the word like "unixunix" use the sed command as below.

>sed 's/\(unix\)/\1\1/' file.txt
unixunix is great os. unix is opensource. unix is free os.
learn operating system.
unixunixlinux which one you choose.

The parenthesis needs to be escaped with the backslash character. Another example is if you want to switch the words "unixlinux" as "linuxunix", the sed command is

>sed 's/\(unix\)\(linux\)/\2\1/' file.txt
unix is great os. unix is opensource. unix is free os.
learn operating system.
linuxunix which one you choose.

Another example is switching the first three characters in a line

>sed 's/^\(.\)\(.\)\(.\)/\3\2\1/' file.txt
inux is great os. unix is opensource. unix is free os.
aelrn operating system.
inuxlinux which one you choose.

8. Duplicating the replaced line with /p flag

The /p print flag prints the replaced line twice on the terminal. If a line does not have the search pattern and is not replaced, then the /p prints that line only once.

>sed 's/unix/linux/p' file.txt
linux is great os. unix is opensource. unix is free os.
linux is great os. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you choose.
linuxlinux which one you choose.

9. Printing only the replaced lines

Use the -n option along with the /p print flag to display only the replaced lines. Here the -n option suppresses the duplicate rows generated by the /p flag and prints the replaced lines only one time.

>sed -n 's/unix/linux/p' file.txt
linux is great os. unix is opensource. unix is free os.
linuxlinux which one you choose.

If you use -n alone without /p, then the sed does not print anything.

10. Running multiple sed commands.

You can run multiple sed commands by piping the output of one sed command as input to another sed command.

>sed 's/unix/linux/' file.txt| sed 's/os/system/'
linux is great system. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you chosysteme.

Sed provides -e option to run multiple sed commands in a single sed command. The above output can be achieved in a single sed command as shown below.

>sed -e 's/unix/linux/' -e 's/os/system/' file.txt
linux is great system. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you chosysteme.

11. Replacing string on a specific line number.

You can restrict the sed command to replace the string on a specific line number. An example is

>sed '3 s/unix/linux/' file.txt
unix is great os. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you choose.

The above sed command replaces the string only on the third line.

12. Replacing string on a range of lines.

You can specify a range of line numbers to the sed command for replacing a string.

>sed '1,3 s/unix/linux/' file.txt
linux is great os. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you choose.

Here the sed command replaces the lines with range from 1 to 3. Another example is

>sed '2,$ s/unix/linux/' file.txt
linux is great os. unix is opensource. unix is free os.
learn operating system.
linuxlinux which one you choose.

Here $ indicates the last line in the file. So the sed command replaces the text from second line to last line in the file.

13. Replace on a lines which matches a pattern.

You can specify a pattern to the sed command to match in a line. If the pattern match occurs, then only the sed command looks for the string to be replaced and if it finds, then the sed command replaces the string.

>sed '/linux/ s/unix/centos/' file.txt
unix is great os. unix is opensource. unix is free os.
learn operating system.
centoslinux which one you choose.

Here the sed command first looks for the lines which has the pattern "linux" and then replaces the word "unix" with "centos".

14. Deleting lines.

You can delete the lines a file by specifying the line number or a range or numbers.

>sed '2 d' file.txt
>sed '5,$ d' file.txt

15. Duplicating lines

You can make the sed command to print each line of a file two times.

>sed 'p' file.txt

16. Sed as grep command

You can make sed command to work as similar to grep command.

>grep 'unix' file.txt
>sed -n '/unix/ p' file.txt

Here the sed command looks for the pattern "unix" in each line of a file and prints those lines that has the pattern.

You can also make the sed command to work as grep -v, just by using the reversing the sed with NOT (!).

>grep -v 'unix' file.txt
>sed -n '/unix/ !p' file.txt

The ! here inverts the pattern match.

17. Add a line after a match.

The sed command can add a new line after a pattern match is found. The "a" command to sed tells it to add a new line after a match is found.

>sed '/unix/ a "Add a new line"' file.txt
unix is great os. unix is opensource. unix is free os.
"Add a new line"
learn operating system.
unixlinux which one you choose.
"Add a new line"

18. Add a line before a match

The sed command can add a new line before a pattern match is found. The "i" command to sed tells it to add a new line before a match is found.

>sed '/unix/ i "Add a new line"' file.txt
"Add a new line"
unix is great os. unix is opensource. unix is free os.
learn operating system.
"Add a new line"
unixlinux which one you choose.

19. Change a line

The sed command can be used to replace an entire line with a new line. The "c" command to sed tells it to change the line.

>sed '/unix/ c "Change line"' file.txt
"Change line"
learn operating system.
"Change line"

20. Transform like tr command

The sed command can be used to convert the lower case letters to upper case letters by using the transform "y" option.

>sed 'y/ul/UL/' file.txt
Unix is great os. Unix is opensoUrce. Unix is free os.
Learn operating system.
UnixLinUx which one yoU choose.

Here the sed command transforms the alphabets "ul" into their uppercase format "UL"

SED COMMAND TO PRINT MATCHING LINES - UNIX / LINUX

We use sed command mostly to do operations like search and replace. We can also use the sed command just like a grep to print only the matching lines.

The syntax of sed command to print matching lines is shown below: 

sed -n '/pattern/p' filename

The syntax of sed command to print lines using line addresses is: 

sed -n 'Np' filename

Examples:

Create the following file in your unix or linux machine: 

unix virtual server
linux virtual hosting
fedora os

1. Display lines matching a pattern

The following examples prints the lines that contain the pattern virtual using sed and grep commands. 

> grep 'virtual' filename
> sed -n '/virtual/p' filename

2. Print lines with specific line addresses.

The below sed command displays the line with number 2: 

sed -n '2p' filename

GREP COMMAND IN UNIX AND LINUX EXAMPLES

Grep is the frequently used command in Unix (or Linux). Most of us use grep just for finding the words in a file. The power of grep comes with using its options and regular expressions. You can analyze large sets of log files with the help of grep command.

Grep stands for Global search for Regular Expressions and Print.

The basic syntax of grep command is

grep [options] pattern [list of files]

Let see some practical examples on grep command.

1. Running the last executed grep command

This saves a lot of time if you are executing the same command again and again.

!grep

This displays the last executed grep command and also prints the result set of the command on the terminal.

2. Search for a string in a file

This is the basic usage of grep command. It searches for the given string in the specified file.

grep "Error" logfile.txt

This searches for the string "Error" in the log file and prints all the lines that has the word "Error".

3. Searching for a string in multiple files.

grep "string" file1 file2
grep "string" file_pattern

This is also the basic usage of the grep command. You can manually specify the list of files you want to search or you can specify a file pattern (use regular expressions) to search for.

4. Case insensitive search

The -i option enables to search for a string case insensitively in the give file. It matches the words like "UNIX", "Unix", "unix".

grep -i "UNix" file.txt

5. Specifying the search string as a regular expression pattern.

grep "^[0-9].*" file.txt

This will search for the lines which starts with a number. Regular expressions is huge topic and I am not covering it here. This example is just for providing the usage of regular expressions.

6. Checking for the whole words in a file.

By default, grep matches the given string/pattern even if it found as a substring in a file. The -w option to grep makes it match only the whole words.

grep -w "world" file.txt

7. Displaying the lines before the match.

Some times, if you are searching for an error in a log file; it is always good to know the lines around the error lines to know the cause of the error.

grep -B 2 "Error" file.txt

This will prints the matched lines along with the two lines before the matched lines.

8. Displaying the lines after the match.

grep -A 3 "Error" file.txt

This will display the matched lines along with the three lines after the matched lines.

9. Displaying the lines around the match

grep -C 5 "Error" file.txt

This will display the matched lines and also five lines before and after the matched lines.

10. Searching for a sting in all files recursively

You can search for a string in all the files under the current directory and sub-directories with the help -r option.

grep -r "string" *

11. Inverting the pattern match

You can display the lines that are not matched with the specified search sting pattern using the -v option.

grep -v "string" file.txt

12. Displaying the non-empty lines

You can remove the blank lines using the grep command.

grep -v "^$" file.txt

13. Displaying the count of number of matches.

We can find the number of lines that matches the given string/pattern

grep -c "sting" file.txt

14. Display the file names that matches the pattern.

We can just display the files that contains the given string/pattern.

grep -l "string" *

15. Display the file names that do not contain the pattern.

We can display the files which do not contain the matched string/pattern.

grep -L "string" *

16. Displaying only the matched pattern.

By default, grep displays the entire line which has the matched string. We can make the grep to display only the matched string by using the -o option.

grep -o "string" file.txt

17. Displaying the line numbers.

We can make the grep command to display the position of the line which contains the matched string in a file using the -n option

grep -n "string" file.txt

18. Displaying the position of the matched string in the line

The -b option allows the grep command to display the character position of the matched string in a file.

grep -o -b "string" file.txt

19. Matching the lines that start with a string

The ^ regular expression pattern specifies the start of a line. This can be used in grep to match the lines which start with the given string or pattern.

grep "^start" file.txt

20. Matching the lines that end with a string

The $ regular expression pattern specifies the end of a line. This can be used in grep to match the lines which end with the given string or pattern.

grep "end$" file.txt

TOP EXAMPLES OF AWK COMMAND IN UNIX

Awk is one of the most powerful tools in Unix used for processing the rows and columns in a file. Awk has built in string functions and associative arrays. Awk supports most of the operators, conditional blocks, and loops available in C language.

One of the good things is that you can convert Awk scripts into Perl scripts using a2p utility.

The basic syntax of AWK:

awk 'BEGIN {start_action} {action} END {stop_action}' filename

Here the actions in the begin block are performed before processing the file and the actions in the end block are performed after processing the file. The rest of the actions are performed while processing the file.

Examples:

Create a file input_file with the following data. This file can be easily created using the output of ls -l.

-rw-r--r-- 1 center center  0 Dec  8 21:39 p1
-rw-r--r-- 1 center center 17 Dec  8 21:15 t1
-rw-r--r-- 1 center center 26 Dec  8 21:38 t2
-rw-r--r-- 1 center center 25 Dec  8 21:38 t3
-rw-r--r-- 1 center center 43 Dec  8 21:39 t4
-rw-r--r-- 1 center center 48 Dec  8 21:39 t5

From the data, you can observe that this file has rows and columns. The rows are separated by a new line character and the columns are separated by a space characters. We will use this file as the input for the examples discussed here.

1. awk '{print $1}' input_file

Here $1 has a meaning. $1, $2, $3... represents the first, second, third columns... in a row respectively. This awk command will print the first column in each row as shown below.

-rw-r--r--
-rw-r--r--
-rw-r--r--
-rw-r--r--
-rw-r--r--
-rw-r--r--

To print the 4th and 6th columns in a file use awk '{print $4,$5}' input_file

Here the Begin and End blocks are not used in awk. So, the print command will be executed for each row it reads from the file. In the next example we will see how to use the Begin and End blocks.

2. awk 'BEGIN {sum=0} {sum=sum+$5} END {print sum}' input_file

This will prints the sum of the value in the 5th column. In the Begin block the variable sum is assigned with value 0. In the next block the value of 5th column is added to the sum variable. This addition of the 5th column to the sum variable repeats for every row it processed. When all the rows are processed the sum variable will hold the sum of the values in the 5th column. This value is printed in the End block.

3. In this example we will see how to execute the awk script written in a file. Create a file sum_column and paste the below script in that file

#!/usr/bin/awk -f
BEGIN {sum=0} 
{sum=sum+$5} 
END {print sum}

Now execute the the script using awk command as

awk -f sum_column input_file.

This will run the script in sum_column file and displays the sum of the 5th column in the input_file.

4. awk '{ if($9 == "t4") print $0;}' input_file

This awk command checks for the string "t4" in the 9th column and if it finds a match then it will print the entire line. The output of this awk command is

-rw-r--r-- 1 pcenter pcenter 43 Dec  8 21:39 t4

5. awk 'BEGIN { for(i=1;i<=5;i++) print "square of", i, "is",i*i; }'

This will print the squares of first numbers from 1 to 5. The output of the command is

square of 1 is 1
square of 2 is 4
square of 3 is 9
square of 4 is 16
square of 5 is 25

Notice that the syntax of “if” and “for” are similar to the C language.

Awk Built in Variables:

You have already seen $0, $1, $2... which prints the entire line, first column, second column... respectively. Now we will see other built in variables with examples.

FS - Input field separator variable:

So far, we have seen the fields separted by a space character. By default Awk assumes that fields in a file are separted by space characters. If the fields in the file are separted by any other character, we can use the FS variable to tell about the delimiter.

6. awk 'BEGIN {FS=":"} {print $2}' input_file
OR
awk -F: '{print $2} input_file

This will print the result as 

39 p1
15 t1
38 t2
38 t3
39 t4
39 t5

OFS - Output field separator variable:

By default whenever we printed the fields using the print statement the fields are displayed with space character as delimiter. For example

7. awk '{print $4,$5}' input_file

The output of this command will be

center 0
center 17
center 26
center 25
center 43
center 48

We can change this default behavior using the OFS variable as

awk 'BEGIN {OFS=":"} {print $4,$5}' input_file

center:0
center:17
center:26
center:25
center:43
center:48

Note: print $4,$5 and print $4$5 will not work the same way. The first one displays the output with space as delimiter. The second one displays the output without any delimiter.

NF - Number of fileds variable:

The NF can be used to know the number of fields in line

8. awk '{print NF}' input_file
This will display the number of columns in each row.

NR - number of records variable:
The NR can be used to know the line number or count of lines in a file.

9. awk '{print NR}' input_file
This will display the line numbers from 1.

10. awk 'END {print NR}' input_file
This will display the total number of lines in the file.

String functions in Awk:
Some of the string functions in awk are:

index(string,search)
length(string)
split(string,array,separator)
substr(string,position)
substr(string,position,max)
tolower(string)
toupper(string)

Advanced Examples:

1. Filtering lines using Awk split function

The awk split function splits a string into an array using the delimiter.

The syntax of split function is
split(string, array, delimiter)

Now we will see how to filter the lines using the split function with an example.

The input "file.txt" contains the data in the following format

1 U,N,UNIX,000
2 N,P,SHELL,111
3 I,M,UNIX,222
4 X,Y,BASH,333
5 P,R,SCRIPT,444

Required output: Now we have to print only the lines in which whose 2nd field has the string "UNIX" as the 3rd field( The 2nd filed in the line is separated by comma delimiter ).
The ouptut is:

1 U,N,UNIX,000
3 I,M,UNIX,222

The awk command for getting the output is:

awk '{ 
        split($2,arr,","); 
        if(arr[3] == "UNIX") 
        print $0 
} ' file.txt

EXAMPLES OF AWK COMMAND IN UNIX - PART 2

1. Inserting a new line after every 2 lines

We will see how to implement this using the awk command with an example.
The input "file.txt" contains the below data:

1 A
2 B
3 C
4 D
5 E
6 F

Let say, we want to insert the new line "9 Z" after every two lines in the input file. The required output data after inserting a new line looks as

1 A
2 B
9 Z
3 C
4 D
9 Z
5 E
6 F
9 Z

The awk command for getting this output is

awk '{ 
 if(NR%2 == 0) 
 {
  print $0"\n9 Z";
 } 
 else 
 {
  print $0
 }
     }' file.txt

2. Replace the Nth occurrence of a pattern

The input file contains the data.

AAA 1
BBB 2
CCC 3
AAA 4
AAA 5
BBB 6
CCC 7
AAA 8
BBB 9
AAA 0

Now we want to replace the fourth occurrence of the first filed "AAA" with "ZZZ" in the file.
The required output is:

AAA 1
BBB 2
CCC 3
AAA 4
AAA 5
BBB 6
CCC 7
ZZZ 8
BBB 9
AAA 0

The awk command for getting this output is

awk 'BEGIN {count=0} 
 { 
  if($1 == "AAA") 
  { 
   count++
  } 
  if(count == 4) 
  { 
   sub("AAA","ZZZ",$1) 
  } 
 } 
 {
  print $0
 }' file.txt

3. Find the sum of even and odd lines separately
The input file data:

A 10
B 39
C 22
D 44
E 75
F 89
G 67

You have to get the second field and then find the sum the even and odd lines.
The required output is

174, 172

The awk command for producing this output is

awk '{ 
 if(NR%2 == 1) 
 {
  sum_e = sum_e + $2
 } 
 else 
 {
  sum_o = sum_o + $2
 }
      } 
      END { print sum_e,sum_o }' file.txt

4. Fibonacci series using awk command

Now we will produce the Fibonacci series using the awk command.

awk ' BEGIN{ 
 for(i=0;i<=10;i++) 
 { 
  if (i <=1 )
  {
   x=0;
   y=1;
   print i;
  }
  else
  {
   z=x+y;
   print z; 
   x=y;
   y=z;
  }
 } 
    }'

The output is

0
1
1
2
3
5
8
13
21
34
55

5. Remove leading zeros from a file using the awk command. The input file contains the below data.

0012345
05678
01010
00001

After removing the leading zeros, the output should contain the below data.

12345
5678
1010
1

The awk command for this is.

awk '{print $1 + 0}' file.txt
awk '{printf "%d\n",$0}' file.txt

READ PASSWORD WITHOUT SHOWING / DISPLAYING ON THE SCREEN - UNIX BASH SCRIPT

Q) My requirement is to write a bash script which prompts for the username, Password and then do some operations. When the password is entered on the unix terminal / screen, the password should not be displayed on the terminal. How to do this?

Solution:

The following unix or linux shell script reads the username, password from the terminal and then prints the values on the output screen: 

> cat pass_script.sh

#!/bin/bash
echo -n "Enter username: "
read username
echo -n "Enter password: "
read -s passwd
echo
echo "$username, the password entered is $passwd"
#Statements to connect to remote box
#using the entered username and password.

The -s option to the read command makes the value to be not printed on the screen. 

> bash pass_script.sh
Enter username: root
Enter password: 
root, the password entered is hiddenpasswd

FILE TEST OPERATORS / OPERATIONS EXAMPLES IN UNIX / LINUX SHELL SCRIPT

 In linux and unix operating systems every thing is a file. When you are using files in your shell or bash script, it is a good idea to do some tests on the file before using it.

The file tests include:

• Checking for existence of the file.
• File is readable, writeable or executable.
• Type of the file and so on.

The file test operators are mostly used in the if clause of the bash script. The syntax is shown below:

if [ -option filename ]
then
  do something
else
  do something
fi

The different file test operators are listed below:

• a : True if the file exists.
• b : True if the file exists and is a block special file.
• c : True if the file exists and is a character special file.
• d : True if the file exists and is a directory.
• e : True if the file exists.
• f : True if the file exists and is a regular file.
• g : True if the file exists and its SGID bit is set.
• h : True if the file exists and is a symbolic link.
• k : True if the file exists and its sticky bit is set.
• p : True if the file exists and is a named pipe (FIFO).
• r : True if the file exists and is readable.
• s : True if the file exists and has a size greater than zero.
• t : True if file descriptor is open and refers to a terminal.
• u : True if the file exists and its SUID (set user ID) bit is set.
• w : True if the file exists and is writable.
• x : True if the file exists and is executable.
• O : True if the file exists and is owned by the effective user ID.
• G : True if the file exists and is owned by the effective group ID.
• L : True if the file exists and is a symbolic link.
• N : True if the file exists and has been modified since it was last read.
• S : True if the file exists and is a socket.

File Test Operator Example:

The following shell script checks for the existence of a regular file:

#!/bin/bash
#assign file name to the variable
FILE="linux-server.dat"

if [ -f $FILE ]
then
  echo "$FILE exists and is a regular file"
else
  echo "Either $FILE does not exist or is not a regular file"
fi 

READ COMMAND LINE ARGUMENTS - UNIX / LINUX BASH SCRIPT

Q) How to read the arguments or parameters passed to a shell script from the command line?

In general the command line arguments are passed to the bash or shell script to change the behavior of the script. In this article, I am going to provide a simple shell script to read and print the command line parameters.

Take a look at the following unix shell script:

> cat OS_Print.sh
#!/bin/bash
echo "Script execution starts"
echo "$@"
echo "$0"
echo "$1"
echo "$2"
echo "$#"
echo "Script execution ends"

The basic functionality of the above script is to print the values stored in the $ variables. Now we will run the above script by passing some arguments.

> OS_Print.sh unix linux
Script execution starts
unix linux
OS_Print.sh
unix
linux
2
Script execution ends

You can see, the command line arguments passed here are unix and linux. Command line arguments are a list of parameters separated by space delimiters passed to the shell script.

Explanation of $ variables:

• $@ : contains all the arguments
• $0 : contains script name
• $1 : First argument
• $2 : Second argument
• $n : Nth argument
• $# : Count of arguments passed.

Examples:

1. Script to iterate through arguments.

The following script prints the parameters using for loop.

#!/bin/bash

for value in $@
do
  echo $value
done

2. Print only the last argument.

There are many ways to display the last argument. The following script shows the different ways of printing the last argument.

#!/bin/bash

echo "${@: -1}"
echo "${BASH_ARGV[0]}"
echo "${@: $#}"
echo "${!#}"

for value in $@; do :; done
echo $value

GREP COMMAND IN UNIX SHELL SCRIPT

Q) How to use the grep command in unix or linux bash scripts to search for a pattern match?

You might have used the grep command to search for a string in a file on the unix command line. Here we will see how to use the grep command in a bash script.

Consider the below file with data

> cat sample_file.txt
linux storage
unix distributed system
linux file server
debian server
fedora backup server

Let see the bash script that prints the lines which contain the word server in it. The bash script code is shown below:

>vi grep_script.sh

#!/bin/bash

WORD=server
INPUT_FILE=sample_file.txt
grep "$WORD" $INPUT_FILE

Now we will execute this script and see what the output is

> bash grep_script.sh

linux file server
debian server
fedora backup server

Here in the script we have hardcoded the filename and the search pattern. We will see how to pass these as arguments from the command line to the script. The bash script is:

>vi grep_script_command_line.sh

#!/bin/bash

INPUT_FILE=$1
WORD=$2
grep "$WORD" $INPUT_FILE

Now run the script as shown below:

> bash grep_script_command_line.sh sample_file.txt server

BASIC UNIX AND LINUX COMMANDS WITH EXAMPLES

Learning unix operating system is very easy. It is just that you need to understand the unix server concepts and familiar with the unix commands. Here I am providing some important unix commands which will be used in daily work.

Unix Commands With Examples:

1. Listing files

The first thing after logging into the unix system, everyone does is listing the files in a directory. The ls command is used to list the files in a directory.

>ls

add.sh
logfile.txt
prime.pl

If you simply execute ls on the command prompt, then it will display the files and directories in the current directory.

>ls /usr/local/bin

You can pass a directory as an argument to ls command. In this case, the ls command prints all the files and directories in the specific directory you have passed.

2. Displaying the contents of a file.

The next thing is to display the contents of a file. The cat command is used to display the contents in a file.

>cat file.txt
This is a sample unix file
Learning about unix server is awesome

3. Displaying first few lines from a file.

The head command can be used to print the specified number of lines from the starting of a file. The below head command displays the first five lines of file.

>head -5 logfile.dat

4. Displaying last few lines from a file.

The tail command can be used to print the specified number of lines from the ending of a file. The below tail command displays the last three lines of file.

>tail -3 logfile.dat

5. Changing the directories

The cd command can be used to change from one directory to another directory. You need to specify the target directory where you want to go.

>cd /var/tmp

After typing this cd command you will be in /var/tmp directory.

6. Creating a file.

The touch command simply creates an empty file. The below touch command creates a new file in the current directory.

touch new_file.txt

7. copying the contents of one file into another.

The cp command is used to copy the content of source file into the target file. If the target file already have data, then it will be overwritten.

>cp source_file target_file

8. Creating a directory.

Directories are a way of organizing your files. The mkdir command is used to create the specified directory.

>mkdir backup

This will create the backup directory in the current directory.

9. Renaming and moving the files.

The mv command is used to rename the files and it also used for moving the files from one directory into another directory.

Renaming the file.

>mv file.txt new_file.txt

Moving the file to another directory.

>mv new_file.txt tmp/

10. Finding the number of lines in a file

The wc command can be used to find the number of line, words and characters in a file.

>wc logfile.txt
21  26 198 logfile.txt

To know about the unix command, it is always good to see the man pages. To see the man pages simply pass the command as an argument to the man.

DELETE EMPTY LINES USING SED / GREP COMMAND IN UNIX (OR LINUX)

In Unix / Linux you can use the Sed / Grep command to remove empty lines from a file. For example, Consider the below text file as input

> cat file.txt
Remove line using unix grep command

Delete lines using unix sed command

How it works

Now we will see how to remove the lines from the above file in unix / linux

1. Remove lines using unix sed command

The d command in sed can be used to delete the empty lines in a file.

sed '/^$/d' file.txt

Here the ^ specifies the start of the line and $ specifies the end of the line. You can redirect the output of above command and write it into a new file.

sed '/^$/d' file.txt > no_empty_lines.txt

2. Delete lines using unix grep command

First we will see how to search for empty lines using grep command.

grep '^$' file.txt

Now we will use the -v option to the grep command to reverse the pattern matching

grep -v '^$' file.txt

The output of both sed and grep commands after deleting the empty lines from the file is

Remove line using unix grep command
Delete lines using unix sed command
How it works