Shell Programming Blog Shell Programming Blog

Wednesday, 31 October 2012

Linux Bash Scripting: How to combine two lines by removing newline character using sed

abu sittik | 08:57 | | Be the first to comment!

Bash Linux Unix shell scripting

In this post we will see how to remove the newline character and combine two lines in a text file using "sed".

In sed, we have single line version and multi-line version of few commands. One such command is the next command. Click here to see an example for single-line version of next command in sed.

You might want to look at :

The processing space of the sed editor is known as the pattern space. The multi-line next command (N) adds the next line to the existing pattern space which already contains the previous line. So, the sed editor can treat two lines as a single line. Below is an example for the N command.


The above sed editor script searches for the line of text that contains the word “first” in it. When it finds the line, it uses the N command to combine the next line with that line. It then uses the substitution command (s) to replace the newline character with a space. The result is that the two lines in the text file appear as one line in the sed editor output.

Tuesday, 30 October 2012

Linux Bash Scripting: How to create directories for each day of the current month

abu sittik | 09:59 | | 2 Comments so far

Bash Linux Unix shell scripting

 In this post I have written a script that creates an empty directory for each day of the current month. I used cal command to get the date and month instead of using date command. The reason is that the date command does not work in Solaris. To make it portable, I have used cal command.

The below script does the following
  • It takes the total number of days of the current month 
  • It gets the current month name from the cal command
  • It gets the current year from the cal command
  • It creates an empty directory for each day of the current month
  • Before creating, this checks whether the directory already exists, if yes, the script exits by displaying a message.

You can change the "dir" to the directory in which you want the folders to be created.

Sunday, 28 October 2012

Linux Bash Scripting: How to remove a blank line after a particular line using sed

abu sittik | 07:35 | | Be the first to comment!

Bash Linux Unix shell scripting
Removing a blank line after a particular line can be achieved using the single-line next command (n) of the sed editor.

The lowercase n command tells the sed editor to move to the next line of text in the data stream, without going back to the beginning of the sed commands. Normally the sed editor processes all of the defined commands on a line before moving to the next line of text in the data stream. The single‐line next command alters this flow.

In the below example, the data file that contains five lines, two of them empty. The goal is to remove the blank line after the header line but leave the blank line before the last line intact. If you write a sed script to just remove blank lines, you will remove both blank lines.


This happened because there was no specific way to find the unique blank line that you wanted to delete. The same can be achieved using the n command as shown in the below script.

The above script searches for the keyword "header" and then moves to the next line when it executes the n command and then applies the d command to delete the blank line below it.

Tuesday, 23 October 2012

Linux Bash Scripting: How to modify the ls command output using pr command

abu sittik | 11:29 | | Be the first to comment!

I have received the below question in our "Ask a question" section of this blog.
Q: Using set -A write a script to print the output of the ls command in 5 columns with two spaces between each column. Pretend that ls does not have multi column output.

Normally we will get a multi column output if we use the command ls -A as shown below.

To print the output of the ls command in five columns with two spaces between each column. You can use the following command.

The pr utility is a printing and pagination filter for text files.

From the man page,

-COLUMN, --columns=COLUMN
output COLUMN columns and print columns down, unless -a is used. Balance number of lines in the columns on each page.

-a, --across
print columns across rather than down, used together with -COLUMN 

-SSTRING, --sep-string[=STRING]
separate columns by STRING, without -S: Default separator <TAB> with -J and <space> otherwise (same as -S" "), no effect on column options

-T, --omit-pagination
omit page headers and trailers, eliminate any pagination by form feeds set in input files

The output of the command would be,

suppose if you want to have delimiter as :::: instead of spaces, then you can change the delimiter in the following command.

you can see the difference in the order in which the files are displayed from the above two commands. That is because I purposely did not use -a option to show the difference.

To pretend that ls does not have a multi column output, you need to add the above command as the alias to the ls command as below.

After adding alias, if you use the ls command, you will get the desired output.

Sunday, 21 October 2012

Linux Bash Scripting: Standard file descriptors STDIN, STDOUT and STDERR and input/ouput redirection

abu sittik | 08:52 | | Be the first to comment!

 Bash Linux Unix shell scripting

Linux system handles every object as a file. This includes input and output processes also. The linux identifies each file using a file descriptor.

What is a file descriptor?
A file descriptor is a non negative integer which uniquely identifies the open files. Each process can have upto 9 open files at a time. The bash shell reserved three file descriptors(0, 1 and 2) for special purposes.

Linux Standard File Descriptors
File descriptor
Standard Input
Standard output
Standard Error

The above three file descriptor controls the input and output of your scripts. The shell uses these descriptors to assign the input and output to appropriate locations (default is the monitor or the terminal).  Now let us see each one in detail.


The STDIN file descriptor refers the keyboard in the terminal environment. So, the Linux will read the input from keyboard always unless it is changed.


The above “cat” command accepts input from the keyboard.  If you want to force the “cat” command to accept the input other than STDIN, you can use STDIN redirect symbol (<).


The STDOUT descriptor refers to the standard output of the shell (i.e. the monitor). So, the Linux directs all the output to the monitor unless it is changed.

You can also append the data to a file by using the >> symbol.

But whenever the command produces an error, it will be displayed in the monitor even if the output is redirected to a file. That is because the error details have a separate file descriptor (STDERR).


By default the STDERR descriptor refers to monitor.

The above example shows that the error message is displayed even after the output is redirected to the file called test. If you want to redirect the error message to some other file, you can do it as below.

If you want to redirect the output and the error message to different files, you can do as below.

If you want to redirect both the error message and the output to the same file, you can use the & operator as shown below.

Wednesday, 17 October 2012

Linux Bash Scripting: how to rename or delete a file which has special characters in its filename

abu sittik | 19:46 | | Be the first to comment!

Bash Linux Unix shell scripting

Suppose if you have created a file with a special character in its name because of a typo, you may not be able to access it. You may not even be able to delete the file. For example, see the below files.

If you see the output of ls command, you will not be able to identify the space at the end of the file. Hence you may not be able to access the file unless you specify a space at the end of the file using quotes.

The ? is a wild character for which any character can be substituted. Hence you may end up deleting or reading some other file.

To handle this situation, we can make use the inode of the file. Every file has a unique inode number with which we can identify the file. 

To rename the file:

You can specify the inode number in the find command as below to rename the files. The below command renames the file using  the inode number.

To delete the file:

You can delete the file using the inode number of the file with the help of "find" command as below.


Monday, 15 October 2012

Linux Bash Scripting: Perform floating point calculation/arithmetic/operation using BC

abu sittik | 03:07 | | Be the first to comment!

 Bash Linux Unix shell scripting

As we have discussed on performing Arithmetic operations on our previous post, now we will look at the floating point operations in the shell script in this post.

You might want to look at : 
Perform Arithmetic operations or mathematical calculations in Shell scripting - expr command

A huge limitation in the Bash shell is that it supports only integer arithmetic. There are several solutions to overcome this limitation. The famous solution is Bash Calculator.

Basics of BC(Bash Calculator):

The bash calculator is actually a programming language that allows you to enter floating point expressions at the command line and then interprets the expressions, calculates and then returns the result. The Bash calculator recognizes the following:
  • Numbers (both integer and floating point)
  • Variables (both simple variables and arrays)
  • Comments (lines starting with a pound sign or the C language /* */ pair
  • Expressions
  • Programming statements (such as ifthen statements)
  • Functions
 The Bash calculator can be invoked from the command line using "bc" command.

As you can see from the above example, each time you enter the expression, you get the result immediately. To exit the BC you need to type quit.

Floating point decimal is controlled by a variable called "scale". Using this variable we can specify no. of decimal places that we would like to see in our result.

The default value of the scale is zero. As you can see from the above example, the Bash calculator returns the answer with zero decimal places. After setting the scale to 4, it displays the result with 4 decimal places.

The -q parameter suppresses the lengthy welcome banner of the Bash Calculator. In addition to the normal numbers, the bash calculator also understands the variables.

Once a variable is defined you can use the variable throughout the Bash calculator session. The "print" statement is used to print the variables and numbers.

Using BC in the scripts:

We can use the BC in our script in two ways.

1) Using the back tick character to run the "bc" command and assign the result to the variable as shown below.

        variable=‘echo “options; expression” | bc’

Below is a sample script which illustrates the above usage.

This method will be useful for small calculations. For the calculations which involves more complex expressions, the next method will be useful.

2) Using the inline input redirection, you can perform calculations which involves more numbers. The syntax of this method would be,

        variable=‘bc << EOF

The EOF text string indicates the beginning and end of the inline redirection data. Remember that the backtick characters are still needed to assign the output of the bc command to the variable.

Below is an example script to show the usage of input redirection in BC.

Note: The variables declared inside the Bash calculator is valid only inside the BC and can't be used in the shell script.

Sunday, 14 October 2012

Linux Bash Scripting: How to ignore specific commands from being stored in the history in Linux

abu sittik | 19:39 | | 4 Comments so far

Bash Linux Unix shell scripting

If you see the history of your commands using the "history" command, you can see all the commands that you executed in the command line. If you don't want certain basic commands to be stored in your history, you can ignore them using HISTIGNORE. You can specify all the commands that you want to ignore in the HISTIGNORE. 

In the below code, I have ignored the commands pwd, ls, ls -ltr and l using HISTIGNORE. Then I am executing some commands in the command line. Finally I'm checking the output of the history command.

[Note that history did not record ls, l, ls -ltr and pwd]

Please note that adding ls does not exclude the command ls -al from being included in the history. So, you need to specify the complete command to make it as an exception.


Shell Programming Copyright © 2012 Shell Programming theme is Designed by Abusittik, Shell Programming