Shell Scripting

Oh, a final word of warning. If you haven't noticed already, Unix and Linux people have a pretty wacky sense of humor. I'm no exception. If you find some of my jokes and quips in this course offensive, you're definitely taking this whole Linux thing WAY TOO SERIOUSLY. Take a chill pill and re-read it and relax!

The diagram below illustrates the structure of the info pages. Refer to it while reading this section

Figure 1.1. The structure of the info pages

Info pages are like man pages in many ways, except they provide a lot more information than man pages (well mostly anyway). Info pages are available on all the Linux distributions currently available, and they are similar to the man pages, although in some instances give you more information.

If we look at a typical way of invoking info, it would be the word info, followed by a space, followed by the name of the command that you want to invoke info on. For example, for the command ls:

info ls
---^-
                

	Note
	Type the commands just as you see them here. I have placed minus signs under the command and it's arguments, and a caret (^) under the space. This is to illustrate that the command should be typed EXACTLY as is.

This should give us an information page on the ls command. We could invoke info on a whole range of utilities by typing:

info coreutils 
---^------
                

where coreutils is just another group of info pages. Coreutils is a grouping of info pages, containing commands such as ls, cd, cp, mv or list directory contents (ls), change directory (cd), copy files and directories (cp) and move or rename files (mv). One could, for instance, type:

info mv
                

The way info is structured, is that when you first start it, it will display the node at which you are beginning your search.

In our coreutils example, on the top of the page (right at the top) the first line looks as follows:

File: coreutils.info,  Node: Top,  Next: Introduction,  Up: (dir)
                

Starting on the left-hand side, we see the file that we're "info'ing" the coreutils.info file.

The filename that contains information about the ls, cp, mv and cd commands amongst others is coreutils.info.

The current Node is Top, which indicates that we're at the top of the coreutils info page and cannot go higher within this group of pages.

From the top level of the info page on coreutils we can now do a couple of things:

We can go to the next node (by typing 'n'), which is the next topic, called Introduction. You will notice that there is no link to a Previous topic, since we are at the top of this group of pages.

We could scroll down to the menu and select a particular topic (node) we want displayed.

File: info.info,  Node: Top,  Next: Getting Started,  Up: (dir)

Info: An Introduction
*********************

   The GNU Project distributes most of its on-line manuals in the "Info
format", which you read using an "Info reader".  You are probably using
an Info reader to read this now.

   If you are new to the Info reader and want to learn how to use it,
type the command `h' now.  It brings you to a programmed instruction
sequence.

   To read about expert-level Info commands, type `n' twice.  This
brings you to `Info for Experts', skipping over the `Getting Started'
chapter.

* Menu:

* Getting Started::             Getting started using an Info reader.
* Expert Info::                 Info commands for experts.
* Creating an Info File::       How to make your own Info file.
* Index::                       An index of topics, commands, and variables.















--zz-Info: (info.info.gz)Top, 24 lines --All-------------------------------------------------------------------------


                

If you were to scroll down to the Directory listing line, you'll see that on the left-hand side there's an asterisk, followed by the topic, followed by a double colon and what is inside the info group:

* Directory listing:: ls dir vdir d v dircolors
                

These are the topics covered in this particular node.

If you hit enter at this stage. You should see that the node has changed. The top line of your page will look as follows:

           
File: coreutils.info, Node: Directory listing, Next: Basic operations,Prev: Operating on characters, Up: Top
                

This is similar to the top of the coreutils.info page as described above, but this example includes a previous node, which is "Operating on characters", with the next node being "Basic operations".

Once I've scrolled down (using my arrow keys) to * Directory listing, I may want to go and look at information about the ls command to see what I can do with ls. Again you use the up or down arrow key and scroll to "ls invocation" and hit Enter

Once there you can read the ls info page to see what it tells you about the ls command.

How do you go back to the Directory listing info page? Type u for UP, which should take you to the previous listing.

How do you go from "Directory listing" to "Basic operations", when you're currently at the "Directory listing" node? n will take you to the NEXT node (taking you from the "Directory listing" to "Basic operations").

If you want to go right to the top, in other words, back to the coreutils group, press t for TOP.

You can do searches within info by using the forward slash (/) and then include a pattern, which might be something like

           
/Directory
                

This tells info that you want to look for the pattern Directory. Bingo! We find Directory listing, as it is the first entry that matches the pattern. If you want to use the same pattern to search again, press forward slash followed by enter:

/<ENTER>
                

This will allow you to search for the pattern "Directory" once again. How do you quit info? q will quit info.

If you want to go one page up at a time, then your backspace key will take you one page up at a time.

Finally, to obtain help within info, type '?'. This will get you into the help page for info. To leave help, press CTRL-x-0.

That is essentially how info works. Part of the reason for moving to info rather than man pages is to put everything into texinfo format rather than gz-man format. In the future, much more software will include manual pages in texinfo format, so it's a good idea to learn how the info pages work.

Having covered the info pages, we need to look at man pages since man is the standard on most UNIX and Linux systems. 'man' is short for manual. This is not a sexist operating system. There are no woman pages but we can find out how to make some a little later (to keep man company).} Manual pages are available on every operating system. (If your system administrator hasn't installed them, ask him politely to do so, as no Linux system should be running without man pages.).

The man command is actually run through a program called less, which is like more except it offers more than the more command does.

	Note
	Mark Nudelman, the developer of less, couldn't call it more, since there was already a more command, so it became less. Linux people do have a sense of humor.

To invoke man pages type:

	
man <command>
                

For example, the ls command that we info'ed above,

$ man ls | less
                

Looking at our example above, the manual page on the ls command is run through the less command.

What can you do from within man?

Well, pretty much the things you can do with info. Instead of a menu system, and nodes, we're looking at a single page detailing all the options and switches of the ls command.

If we want to search for a particular pattern we would use forward slash (/) just like we did in info.

For example, we could type

/SEE ALSO
                

This pattern search would take us to the end of the man page to look at the SEE ALSO section.

We could type question mark with a pattern, which does a reverse search for the specified pattern. Remember forward slash does a forward search and question mark does a reverse search.

?NAME
                

This pattern search will reverse search up the man page to look for the pattern NAME.

	Note
	You will notice that I'm not saying look for the string NAME, rather I'm saying look for the pattern NAME. This is because pattern matching is a critically important part of UNIX and Linux and shell scripting. We'll learn more about patterns as we go through the course.

If we want to scroll down one page at a time within the man page (i.e. we've looked at page 1 and we've read and understood it, and we want to go to page 2), then the space bar takes us forward by a page.

Similarly if we want to reverse up the man page, we press b for back, which will scroll backwards through the man page.

How do we get back to our prompt? The 'q' key comes in handy again. 'q' for quit.

man pages are generally broken down into a host of different sections. There's a SYNOPSIS section, a DESCRIPTION section, and a SEE ALSO section. Read through the man page and you will understand the different sections.

If you need help on moving around through the man page, type 'h' for help, which will give you a listing of all the help commands. You will see that it has displayed the help commands NOT for man but for less. Why? Because the pager for man, (pager, the tool that gives you one page at a time instead of just scrolling the man page past you too fast to read), is the less command

We will cover the less command a bit later but you can look it up with the info pages as follows:

info less
                

So 'h' within the man page will show you help on the 'less' command at the same time as displaying the requested manual page.

Sometimes you need to read a man page three or four times before you completely understand it, and of course sometimes you may never understand it! Don't be deterred. That's what separates the kanga's from the roo's.

The whatis database is usually rebuilt on a Linux system at night. The job of the whatis database is to search through every manual page on your system looking for the NAME section within these man pages, classifying them and placing them into a database, to facilitate searching in the future.

The whatis database is useful in that it gives us the ability to quickly look up what a particular command does. So if I asked you to tell me what the nl command does, you could look it up in the man pages or you could look it up in the whatis database.

man nl
                

or

whatis nl
                

The latter method should return with the NAME section from the man page, showing you what the commands job is on the system. It should tell you that nl numbers the lines. Similarly wc counts words, lines and characters for you.

The whatis database is very useful because it allows you to quickly find out, what a particular command on the system does.

If the whatis database is not up-to-date, it is quite simple to update it. Generally though, the updating of the whatis database is a simple automated process. Once a night, the operating system should go about updating the whatis database. Even if the system administrator has installed a whole host of new software on the system, by virtue of the fact that the man pages for that software would be installed at the same time as your new application is installed, your whatis database should pick up those pages and add them to its database each night.

As a side note, updating the whatis database manually is simply a matter of

$ makewhatis -u -w
                

and the whatis database will be updated.

The who command is designed to tell you who's logged on to the system.

If we run the who command without any switches, the left hand column shows the user id. This the user currently logged on to the system. In your case, you might be logged on as root, or perhaps as your user. The second column indicates where you are logged in.

riaan@debian:~> who
riaan    :0           Apr 30 11:13 (console)
riaan    pts/0        Apr 30 11:13
riaan    pts/3        Apr 30 11:14
riaan    pts/4        Apr 30 11:30
riaan    pts/5        Apr 30 13:19
riaan    pts/6        Apr 30 12:07
riaan    pts/7        Apr 30 12:09
riaan@debian:~>
                

So if you look at the who command output, my user riaan is logged in from :0 which is the X console. He's also logged on to

pts/0 and 
pts/1
                

These are pseudo terminals, indicating he's also logged into two pseudo terminals.

The final, third column indicates what time the user logged on.

The who command tells us about users using our system. That's great!

What are the other switches that we can use with who.

who --help 
                

This will show you the various switches that we can use with the who; command. So if we use a:

who -H
                

it prints a heading line for us. The output should look as follows:

$ who -H
NAME             LINE     TIME         FROM
heidi            ttyp1    Nov 27 17:29 (168.210.56.177:S)
mwest            ttyp2    Nov 10 15:04 (apotheosis)
heidi            ttyp4    Nov 11 13:18 (168.210.56.177:S)
                

To view a short listing which is the default listing for the who command:

who -s 
                

Using the -u switch:

who -u
                

will show the users and their process id''s.

In scripts, one can use the same commands as on the command line, including all the switches those commands use. One can run any command and produce standard text output, which one can use. We'll talk about how you can use the output later.

Run the command

$ who -u
root     tty2         Aug  4 10:41   .          2839
riaan    :0           Aug  4 07:53  old         2836 (console)
                

to identify which users are logged into your system and from which processes they are logged on.

This will show how long a terminal has been idle. It will show not only which users are logged on and what process ids they are but also how long that user has been idle. Idle users might have gone out for lunch or they might have left for the day. In default mode, most of these systems don't log you out if you're idle for longer than 10 or 15 minutes. In the old days, most systems were configured to automatically log you out after 15 minutes.

	Note
	On Debian, the -i switch does not add any extra output, it simply prints a message suggesting that you not use -i as it will be removed in future releases. Use -u as above. However the -i switch may work with other brands of Linux.

Okay, so that's the who command. We're going to use these commands later to build a system to monitor our system automatically, because we want to be spending our time doing things we enjoy.

What does the w command do? You could run:

riaan@debian:~> whatis w
w (1)                - Show who is logged on and what they are doing.
riaan@debian:~>
                

riaan@linux:~> w
 21:40:17 up 11:03,  6 users,  load average: 0.30, 0.34, 0.30
USER     TTY        LOGIN@   IDLE   JCPU   PCPU WHAT
root     tty2    	21:40    8.00s    0.06s  0.06s -bash
riaan    :0        	10:38    ?xdm?  58:31   0.04s -:0
riaan    pts/0     10:38   11:01m  0.00s  2.08s kdeinit: kwrited
riaan    pts/3     11:18   10:22m 14:37   2.63s /usr/lib/java/bin/java -mx32m -jar /home/riaan/jedit/4.2pre9/jedit.j
riaan    pts/4     11:28     1:07m  0.21s   0.21s /bin/bash
riaan    pts/5     11:28     0.00s   0.17s   0.03s w
                

which should print some information about the w command.

The w command not only tells us who are logged in, but also what they're doing. Are these users running applications? What actual processes are they running at this time? Perhaps someone's running an application like OpenOffice. w will tell us this.

If you look at the output of this command, it's got a list of headings that are fairly similar to the format of the who command.

Later we'll have a look at modifying the report columns, to get the output into a different format that may be more useful.

One of the reasons for taking you through these commands is that we're going to start writing our first shell scripts using these commands, so it is as well that we understand them now.

The date command is a useful command that can do all sorts of nifty things for us (apart from printing out the date).

It can convert between Unix time, (which is the number of seconds since 1/1/1970 - commonly known as the epoch) and a human readable (normal) date and time.

Conversely, it can also convert back from date time today to the number of seconds that have elapsed since the 1/1/1970. It can format output in a whole variety of ways. Let's look at some examples of the date command.

For that I'm going to do:

info date
                

If you scroll down, you will see a section with examples. Looking at the example:

date +"     "
                

We may now include a string describing the format inside these quotation marks.

In the shell there's a big distinction between double quotes, single quotes (which is another lesson altogether, see Chapter 5), and back quotes - let's not get them confused for now.

Within this double quoted string we can include any number of arguments. What arguments can we include? Each argument starts with a percentage sign.

To display the time, we could use:

%H - -will give us the hours in 24 hour format (0-23).

%M - -will give us the minutes (0-59) of the day
                

If we had the following string:

date +"%H:%M"
                

we will end up with hours and minutes of the day on our system. The result of the above command should be similar to:

15:04
                

But let's say that we want the hours in 12-hour format rather than 24-hour format. We could then replace the %H with a %l. The result would then be:

3:04
                

There's a host of other things that we could do. For example if we are in 12-hour format, 3:04 doesn't indicate whether it is morning or afternoon. Thus we could include %p:

date +"%l:%M %p"
                

This would show us that the time is actually:

3:04 PM
                

rather than 3:04 AM.

That's for time, but what about for the date? What happens if we want to show the date, which is:

24-Nov-2003
                

then, we should in theory be able to create a date string to reflect this format.

A way we can do is this is using the following:

date +"%d-%b-%Y"
                

where %b is a short format for month to produce Nov instead of November.

If we want to combine the date and time:

date +"%d-%b-%Y %l:%M %p"
                

This would give us the full date and time report:

24-Nov-2003  3:04 PM
                

There are a lot of other parameters that you can use within the date command. You can view these by looking at the relevant info page with :

info date
                

We're going to use this command in our script, because in almost every script that you will write you are going to want to know what time the script started, what time the script ended, when it did a particular job within the script, etc.

The final command I want to describe is a command used to send output to the screen: echo.

We've seen so far that we were able to run commands but, as yet, we don't know how to simply output some text to the screen. We may want to print a string to the screen, prior to printing the date.

Something such as:

Today's date is:
24-Nov-2003  3:04 PM
                

We would need some way of echoing that to the screen, wouldn't we?

In order to do this, there is the echo command. echo can be a bit of a nasty gremlin because there are at least two echo commands on your system. One is a shell built-in, and one is an external command and therefore it can be a bit tricky.

We're going to start off with a vanilla case. Later on we will see how we can choose which echo command we want to use.

So by way of an example, we'll use it to format the date command.

echo "Today's date is: "
date +"%d-%b-%Y %l:%M %p"
                

This would be a good time to show you how to create your first shell script. We're going to edit a file and for this you can use any editor^[1]

Open your editor (whatever editor you prefer) and put the following commands in the first lines of the file:

echo "Today's date is: "
date +"%d-%b-%Y %l:%M %p"
                

Save that file as myfirstscript and exit your editor.

You've just created your first shell script. Great! How easy was that? How do you run it to make it actually do its job?

chmod +x myfirstscript
                    

./myfirstscript
                    

Today's date is: 
24-Nov-2003  3:04 PM
                    

This section is going to cover file commands. File commands are commands such as ls (list).

Notice again, how the laziness of the Unix people comes to the fore. They could have had a command called list, but that would have required two more characters (and two more carpals - fingers!) and clearly that was a lot more effort, so we just have the ls command. The ls command shows us a listing of files in a particular directory.

This is an appropriate place to take a detour on our tour de scripting and have a look at file matching and wildcard matching. It may be something that you're familiar with, but let's have a look at it and come back to ls in a moment.

ls *
                        

ls -la ab?a
                        

abba
ab1a
ab_a
ab9a
abca
...	
                        

[abc] 	
                        

[ab]cde	
                        

ham01 	
                        

ham10
                        

ham[0-9][0-9]
                        

ls [abc123] 
                        

a
b
c
1
2
3
                        

[a-z] 
                        

ls [!a-c]*
                        

	
touch {planes,trains,boats,bikes}_{10,1,101,1001}.{bak,bat,zip,tar}
                        

ls p* 
                        

ls b* 
                        

ls bik*
                        

ls b*_??.* 
                        

boats_10.bak
boats_10.bat
boats_10.zip
boats_10.tar
bikes_10.bak
bikes_10.bat
bikes_10.zip
bikes_10.tar
                        

touch {fred,mary,joe,frank,penny}_{williams,wells,ferreira,gammon}.{1,2,3,4,5}
                        

touch {fred,mary,joe,frank,penny}_{williams,wells,ferreira,gammon}.{1,2,3,4,5}
  ------Name----------    ----------Surname-------    -Category-
                        

. 	this file is our current directory
.. 	this file is our previous or parent directory. 
                    

There are many system commands that we can use. We're going to start using these in our shell scripts.

Remember, a shell script is nothing more than a group of Linux commands working together to produce a new command.

In order to build our system that is going to manage our Linux machine, we're going to need to know a little bit about system commands. System commands such as:

df shows the disk free space

du shows the disk usage

fdisk shows the partitioning on our disk

iostat shows the input output status

vmstat shows the virtual memory status

free shows the amount of free memory

We will use these commands, but they are a subset of the commands available to us for monitoring our system.

df -h
                    

du - s 
                    

df - hT
                    

BlackBeauty:/install # fdisk -l

Disk /dev/hda: 10.0 GB, 10056130560 bytes
240 heads, 63 sectors/track, 1299 cylinders
Units = cylinders of 15120 * 512 = 7741440 bytes

   Device 	Boot      Start         End      Blocks   		Id  System
/dev/hda1   *           1          760     	5745568+ 	83 	Linux
/dev/hda2             761        1299     4074840    	5  	Extended
/dev/hda5             761         827      506488+  	82  	Linux swap
/dev/hda6             828        1299     3568288+  	83  	Linux

                    

baloo:/home/riaan# free
	     total       used       free     shared    buffers     cached
Mem:        498572     493308       5264          0      48700     230492
-/+ buffers/cache:     214116     284456
Swap:       706852       8196     698656
                    

baloo:/home/riaan# vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0   3792   3508  14208  59348    0    0    32    72 1065   818 11  4 84  2

                    

the iostat command

Finally, the iostat command.

iostat Linux 2.6.4-52-default (debian) 09/02/04 avg-cpu: %user %nice %sys %iowait %idle 2.51 0.03 1.99 0.82 94.64 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn fd0 0.00 0.00 0.00 4 0 hda 3.93 304.75 23.83 3868650 302512 hdd 0.01 0.59 0.00 7524 0

This command gives you information about input and output on your system, and how well (or otherwise) it is performing.

We'll take a closer look at the system performance commands in more detail later.

	Note
	In order to provide you with further information on the performance of your system, you should install the sysstat package. rpm -ivh sysstat.x.y.z-r1.rpm (RedHat system) (see the appendix for other distributions) The iostat command is part of the sysstat package, so if you don't install sysstat, then skip the iostat stuff)

Generally standard input, referred to as "stdin", comes from the keyboard.

When you type stuff, you're typing it on stdin (a standard input terminal). A standard input device, which is usually the keyboard, but Linux also allows you take standard input from a file.

For example:

cat < myfirstscript 
                

This would tell cat to take input from the file myfirstscript instead of from the keyboard (This is of course the same as: cat myfirstscript).

Essentially what this boils down to is that the input for this command (cat) is no longer coming from where it was expecting to come from (the keyboard), but is now coming from a file.

Linux associates the file descriptor 0 with standard input. So, we could have said:

cat 0< myfirstscript
                

which would have produced the same as the previous command.

Why could we leave out the 0?

Since, at the time of creating a process, one standard input is associated with the process.

Standard output, as created at process creating time, goes to the console, your terminal or an X terminal. Exactly where output is sent clearly depends on where the process originated.

Our console or terminal should be the device that is accepting the output. Running the command:

cat file
                

would [con]catenate the file, by default, to our standard output i.e. our console or terminal screen. (Where the process originated.)

We can change this and redirect the output to a file.

Try the following:

ls -al myfirstscript > longlisting
                

This redirects the output not to the screen, but rather to a new file 'longlisting'. The process of redirection using the '>' will create the file 'longlisting' if it was not there. Alternately, if the file 'longlisting' existed, it would remove it, (removing the contents too of course) and put a new file there with the directory listing of "myfirstscript" within it.

How would we see the contents of the file?

cat longlisting
                

This will show the size, the owner, the group and the name of the file myfirstscript inside the file 'longlisting'.

In this example, the output of the ls command has not gone to the standard output (the screen by default), but rather into a file called 'longlisting'.

Linux has got a file descriptor for standard output, which is 1 (similar to the 0 for standard input file descriptor).

The above ls -la example can be rewritten as:

ls -al myfirstscript 1> longlisting
                

which, would do the same thing as leaving out the file descriptor identifier and just using the greater than sign.

In the same way we use our standard input as < (or a 0<), we use a > (or a 1> ) to mean standard output.

Returning to the cat example above, we could type data on the command line that gets sent directly to a file. If the file is not there it will create it and will insert the content we typed on the command line, into the file. This is illustrated below:

$ cat > test
This is the first line.
This is the second line.
This is the final line. < Ctrl-d pressed here >
$ cat test
This is the first line.
This is the second line.
This is the final line.
                

Doing this does not return us to a prompt. Why? What is it waiting for?

It's waiting for us to actually enter our string into a buffer. You should start typing a sentence then another sentence, and another, etc. Each time you type a character, it's getting appended to the file 'newfile'.

When you have finished typing in what you want, press Ctrl-d. The Ctrl-d (^D) character will send an end of file signal to cat thereby returning you to your prompt.

If you list your directory contents using:

ls -al 
                

you should see the file 'newfile'. This file is the one that you've just created on the command line.

cat newfile
                

will show you the contents of 'newfile' displayed onto the standard output.

Now why did all of this work? It worked because cat was taking its input from standard input and was putting its output not to standard out as normal, but was rather redirecting output to the file 'newfile'.

On typing the command and hitting enter, you are not returned to your prompt since the console is expecting input to come from stdin; you type line after line of standard input followed by ^D. The ^D stopped the input, by sending an end-of-file character to the file, hence the file 'newfile' is created.

Question: What do you think tac > newFile will do?

If you decide you want to copy the contents of two files to another file (instead of using the cp command - there is more than one way to skin a cat in Linux) you could do the following:

cat < myfirstscript > mynewscript 
                

Incidentally, this is equivalent to

cp myfirstscript mynewscript
                

Well that's fine and dandy, but what happens if we don't want to delete our longlisting script and want to rather append it to a file that's already there.

Initially we typed:

ls -al myfirstscript > longlisting
                

If you did this a second time, it would overwrite the first file longlisting. How could you append to it? Simply adding two greater than signs, immediately following one another as in the example below, would append output to the file 'longlisting'

ls -al myfirstscript >> longlisting
                

Each time you ran this command, it would not clobber (remove the contents of) the longlisting file but would rather append to it and as a result, the file 'longlisting' would grow in size.

A final note on standard output and standard input is that redirection must be the final command that you execute on the command line. In other words, you can't do any other command after the redirection. We will talk about this in a later section on pipes.

The final component in this dialog of file descriptors is standard error.

Every command could send it's output to one of two places: a) it could be valid output or b) it could be an error message.

It does the same with the errors as it does with the standard output; it sends them directly to your terminal screen.

If you run the command (as your user):

find / -name "*" -print
                

you would find that find would find a load of things, but it would also report a lot of errors in the form of 'Permissions denied. . .'.

Perhaps we're not interested in 'Permission denied...' - we may wish to discard these messages (as root, no error messages would be returned).

If we ran the command, we could put standard error into a file (remember standard error by default is going to the console; the same place as stdout). In this case I'm going to put it into a special file on the Linux system called /dev/null.

/dev/null is similar to the "Recycle Bin" under Windows except it's a waste paper basket with a point of no return - the Linux black hole! Once information has gone into /dev/null, it's gone forever.

Initially, I'm going to put any errors that the find command generates into /dev/null, because I'm not interested in them.

We saw that standard input was file descriptor 0, the standard output was file descriptor was 1, so no prizes for guessing that standard error has file descriptor 2.

Thus, the command

find / -name "*" -print 2> /dev/null
                

discards any errors that are generated by the find command. They're not going to pollute our console with all sorts of stuff that we're not interested in.

	Note
	Notice there is no space between the 2 and the >

We could do this with any command, we could for example say:

ls -al 2> myerror
                

Which would redirect all the error messages into a file called "myerror".

To recap we can use either:

<	OR	0<	for standard input
>	OR	1>	 for standard output
but for standard error I have to use 2> 
                

It's not optional to leave off the number two (2). Leaving it off would mean that the standard output would go to "myerror", including a 2 means standard error.

In the listing case of:

ls -al  2> myerror 
                

puts any errors into a file called 'myerror'.

If we wanted to keep all those error messages instead of using a single greater than sign, we would use double greater than signs.

By using a single greater than sign, it would clobber the file 'myerror' if it exists, no different to standard output. By using a double greater than sign, it will append to the contents of the file called myerror.

ls -al 2>> myerror 
                

Thus the contents of 'myerror' would not be lost.

With our new found knowledge, let's try and do a couple of things with the find command. Using the find command, I want to completely ignore all the error messages and I want to keep any valid output in a file. This could be done with:

find / -name "*" -print 2> /dev/null > MyValidOutput
                

This discards any errors, and retains the good output in the file "MyValidOutput". The order of the redirection signs is unimportant. Irrespective of whether standard output or standard error is written first, the command will produce the correct results.

Thus, we could've rewritten that command as:

find / -name "*" -print >MyValidOutput 2>/dev/null
                

Finally I could've appended the output to existing files with:

find / -name "*" -print >> output 2>> /dev/null
                

Clearly appending to /dev/null makes no sense, but this serves to illustrate the point that output and errors can both be appended to a file.

What happens if we want to take our standard output and put it into the same file as standard error? What we can do is this:

find / -name "*" -print 2> samefile 1>&2
                

This means that standard error goes into a file called samefile and standard output goes to the same place as file descriptor 2 which is the file called samefile.

Similarly we could've combined the output by doing:

find / -name "*" -print 1> file 2>&1
                

This captures the output of both standard output and standard error into the same file.

Clearly, we could've appended to a particular file instead of overwriting it.

find / -name "*" -print 1>> file 2>>&1
                

Up to now, we've seen that we can run any command and we can redirect its output into a particular file using our redirection characters (>, <, >>, 2> or 2>>).

It would be good if we could redirect the output of one command into the input of another. Potentially we may want the output of the next command to become the input of yet another command and so forth. We could repeat this process over and over until we achieve the desired output.

In Linux, this is affected using the pipe character, (which is a vertical bar '|'). An example is:

ls -la | less 	
                

This would pass the output of the ls -al command to the input of the less command.

The effect would be to page your output one page at a time, rather than scrolling it to the standard output all in one go - too fast for you to be able to read, unless of course you are Steve Austin!.

What makes the pipe command powerful in Linux, is that you can use it over and over again.

We could type:

ls -l | grep myfirstfile | less
                

Instead of grep's standard input coming from a file (or keyboard) it now comes from the ls command. The grep command is searching for a pattern (not a string) that matches myfirstfile. The output of the grep command becomes the input of the less command. Less could take its input from a keyboard, or from a file, but in this case it's taken its input from the command grep.

How many of these pipes can we have on a single line? As many as we want! We will see this and how it's used to good effect in our shell scripts from the next chapter onwards.

If pipes were not available, we may have to archive the above command in two or more steps. However, with pipes, this task is simplified (speeded up).

If we take a step back to run the who command:

who | grep <your user name> 
                

	Note
	The < and > here don't mean redirection!

We will see only the people that are logged on as your user (hopefully that is only you!!). Perhaps you want to only see people who are logged on to pseudo terminals, then:

who | grep pts
                

which would tell us only the usernames of the people logged on to pseudo terminals.

In these examples we are using the output of the who command as the input to the grep command.

Additionally we could redirect the output of this command to a file called outfile:

who | grep pts > outfile
                

This would produce the file 'outfile' containing only those users who have logged onto pseudo terminals. That's nifty.

We will see this put to good effect as we start scripting, building very complex filters where we use the output of one command to be the input of the next, and the output of that to be the input of the next.

These exercises should assist in your understanding of the login and non-login shell initialisation files.

The shell interprets commands. A command typed in at the prompt will know how to be executed because the shell will use its PATH to search for the command.

Typing cd, the command interpreter knows this is a built-in command , and will not search for it in the PATH.

Typing pwd, it understands that it needs to display the local working directory.

Using mv, the shell must know to run an external program, as this is not a shell built-in. ^[3].

Equally the shell is responsible for parsing the command line to detect errors in syntax. For instance typing

cd.. 
                

produces an error because there is no white space between the cd and the .. (this is a problem ex-DOS people often stumble over). In this instance the command interpreter would typically give you feedback such as

cd..
bash:cd..: command not found
                

The shell allows variables to be set. Variables such as your PATH variable, or your input field separator (IFS), or setting your shell HISTORY size.

The shell is also responsible for input, output and error redirection (consult the previous chapter to refresh your memory on IO redirection).

The shell understands and allows pipes.

The final job of the shell is to allow you, a user, to customise your environment. Setting variables, changing your prompt, running scripts automatically are all things that allow the user some control over their environment.

In summary, a shell script is like a batch file in DOS. Unlike the DOS command interpreter, the shell incorporates a powerful, almost full programming environment.

For instance, it allows 'if' statements, 'while' loops, 'for' loops, 'arrays', as well as a host of other programming techniques.

Scripting will make you life easier as a system administrator.

The first of our pattern sequences: a fullstop (or a period as the Americans call it) matches any character.

We might say:

sed 's/Linu./LINUX/g' bazaar.txt
                

This sed expression means:

s (search for) / Linu.  / (replace with) LINUX / g (globally) <filename to search>
 ----------^-------^-----------------^---------
                

Looking at the command in detail: The pattern 'Linu.' says match any set of characters that begins with a uppercase 'l', followed by i, an 'n' and a 'u' followed by any other character - the fullstop matches the "any other character". In the file bazaar.txt the following strings appear, that would match this pattern:

Linus
Linux
                

The pattern we used in the sed above, will match occurrences of Linux and Linus.

Using the fullstop in place of 's' or 'x' ensures that these two strings are matched. However, the pattern will also match:

Linup
                

Why? Because it matches the pattern 'Linu' followed by any single character.

	Important
	The fullstop in regular expression terms matches any character.

Pipe the sed expression through nl, and look at line 9 ... "Linus Torvalds" has been changed to "LINUX Torvalds".

sed 's/Linu./LINUX/g' bazaar.txt | nl
                

Sed is an acronym for "Stream Editor". The "stream", in our example above, comes from the file bazaar.txt.

Besides the input stream sed must also have a command pattern-command combination.

SYNTAX:
sed [command] / pattern / [replace sequence]  / [modifier] [command]
                

In this case our command is 's' for search while the pattern we're searching for is enclosed in forward slashes (forward slashes are not strictly required, but we'll not going to complicate matters right now).After the second forward slash, we have a replace sequence.

sed will search for a pattern (Linu.) and on finding it, will replace it with the replace sequence (LINUX).

Finally we have a modifier in this case a 'g' meaning "globally" i.e. search for a pattern and replace it as many times as you find it on a line. If there were 10 instances of 'Linu<any character>' on a line, it would replace all occurrences.

Since sed is a stream editor, it considers each line in the file bazaar.txt independently (in essence, "finish processing this line, then get the next line from the input file"). The stream ends when an end-of-file character is reached. Thus the "globally" modifier only operates on the current line under consideration.

If we just wanted to replace only the second instance and not the first or the third, etc. we could replace the g with a 2. sed would then only replace the second instance of the desired pattern. As you go through this chapter, you will become friendly with sed, and work with many patterns.

To Summarise: a fullstop (.) as a regular expression matches any single character.

Square brackets mean a range of characters. If we tried [abc], ( you should remember this from earlier), it means match a single character which is either and 'a' or a 'b' or a 'c'.

A caret ( ^ ) matches a start of line and the dollar ( $ ) the end of the line.

Now I'm going to use these together to create more complex RE's. We're going to write a sed expression that's going to match lines (not search and replace as before, ) that begin with 'a', 'e' or i and print ( p ) them.

sed '/^[aeI]/p' bazaar.txt
                

You will notice that before we were doing a search-replace, this time around we're doing a pattern match, but merely printing the matched lines.

This regular expression would match lines that begin with either 'a' or 'e' or i. Now, you'll notice when we ran this command, the lines that begin with 'a', 'e' or i are printed twice while every non-matching line is printed only once. sed parsed the entire file, line by line and each time it matched a line that began with 'a', 'e' or i, the line was printed (which is why the lines were duplicated). In our example we can see that line 6 begins with an i - hence a match:

I believe that most important software....
                

Similarly, line 8 is also printed twice:

editor) needed to be built....
                

How would we match both 'e' and 'E'? Simply include 'E' in the pattern so that the RE becomes:

sed '/^[aeEI]/p' bazaar.txt
                

This time if you run it, you will notice that line 16 is also matched:

Extract taken from....
                

We've seen two things:

Now what makes this slightly more complex is that if this caret appeared inside the square brackets, it's meaning becomes altered.

Examine the following Regular Expression:

sed '/^[^aeEI]/p 
                

This causes every line that does NOT begin with 'a', 'e', 'E' or i to be printed. What's happening here? Well, the caret inside the square bracket means "do not match".

The caret outside the square bracket says match the start of the line, the caret inside the square bracket says do not match a,e,E or I. Reading this RE left to right:

"any line that starts with NOT an 'a' or an 'e' or an 'E' or and i - print it".

What happens if we replace the 'p' with a 'd'?

sed '/^[^aeEI]/d'
---------------^
                

means:

"any line that starts with NOT an 'a' or an 'e' or an 'E' or and i - delete "it".^[4]

Here are the new concepts you've learned:

Just the same way that the caret means the beginning of the line, the $ means the end of the line. An expression such as:

sed '/[aeEI]$/!d' bazaar.txt
                

means

"don't ( ! ) delete any line that ENDS in either an 'a', an 'e' an 'E' or an 'I'".
                

We've used the following expressions:

Perhaps we want to print only the lines beginning with 'a', 'e', 'E' or i.

How can sed achieve this? We could try,

"delete all lines NOT beginning with an 'a,e,E or I'"

sed '/^[^aeEI]/d' bazaar.txt
                

Bingo. However it also produced a series of blank lines. How do we remove blank lines, leaving only the lines that we are interested in? We could pipe the command into yet another sed, where we could look for blank lines. The pattern for a blank line is:

^$
                

So the following command would delete the unwanted blank lines:

sed '/^[^aeEI]/d' bazaar.txt | sed '/^$/d' 
                

Bingo (again), we end up with the lines we wanted. You might want to pipe this command through nl just to see the line numbers:

sed '/^[^aeEI]/d' bazaar.txt | sed '/^$/d' | nl
                

Notice that the first sed is acting on a file, while the second sed is acting on a stream of lines as output by the initial sed. Well we could have actually simplified this slightly, because sed can accommodate multiple command-pattern-command sequences is they are separated by a ';' Hence, a modified command:

sed '/^[^aeEI]/d;/^$/d' bazaar.txt | nl
 ----------------^----------------------
[ notice the ^ indicating the ; ]
                

These examples illustrate two concepts:

By way of re-enforcing this, run the following commands:

time sed '/^[^aeEI]/d' bazaar.txt |sed '/^$/d' |nl
time sed '/^[^aeEI]/d;/^$/d' bazaar.txt |nl

[ the 'time' command will time the commands ]
                

This will show the elapsed time in addition to a host of other information about how long this command took to run. In this case since our RE is so simple and the file we're operating on is so small, the time difference is marginal. If however this were a 100Mb file, invoking sed twice would be a significant impairment on the speed with which your script executes.

sed is a stream editor, but what's a stream? A stream is just like a river, in which information is flowing. sed is able to edit the stream as it 'flows' past. We've been invoking sed using a file as an argument, however we could alternatively have used sed as part of a pipe :

cat bazaar.txt | sed '/^[^aeEI]/d;/^$/d'
                

This would produce the same results as invoking sed earlier. sed is one of the commands that you should be comfortable using since it can be used in many and varied ways. Now, as part of the pipe, sed is searching for a pattern. On finding the pattern it's modified and sent on to stdout.

The splat (*) matches 0, one or more occurrences OF THE PREVIOUS PATTERN.

Supposing we wanted to match Line or Linux or Linus the pattern:

sed '/Lin.*/p' bazaar.txt	
                

would match lines containing these words.

The splat says "match 0, one or more of the previous pattern (which was the Full-stop, and the full-stop means one occurrence of any single character)".

Lets looks at another example:

/a*bc[e-g]*[0-9]*/
                

matches:

aaaaabcfgh19919234
bc
abcefg123456789
abc45
aabcggg87310
                

Let's looks at another example:

/.*it.$/
                

matches any number of alphanumeric characters followed by and i followed by a 't' followed by the end-of-line.

The plus operator will match the preceding pattern 1 or more times. To match the character 'a' or 'b' or 'c', one or more times, we could use:

[abc+]
                

Perhaps we want to match 19?? in the bazaar.txt file (Here we would want to find any year, 1986 or 1999 whichever you would like to find.)

19[0-9+]
                

To match the character a, one or more times, we would use

a+
                

	Note
	Note that in the previous examples, the plus character is not matched, since this ( + ) has special meaning in a RE. If we wanted to search for a plus sign (or any of the RE pattern matching tools) in a pattern, we would need to escape the plus sign.

How do we escape characters that are special in RE's? We need to escape them with a backslash ( \ ). Thus to search for the pattern a+ we would use:

a\+
                

Similarly if we wanted to match a splat ( * ), we would have to match it with:

a\*
                

So, the plus is a special character, which matches one or more of THE PREVIOUS PATTERN.

a{4}
                    

sed '/19{3}/p' bazaar.txt
                    

sed \%\<[a-z][a-z][a-z]\>%p' /usr/share/dict/words
                    

/home/hamish/some_where
                    

'/\/home\/hamish\/some_where/!d'
                    

%/home/hamish/some_where%!d
                    

\%/home/hamish/some_where%!d
                    

sed 's/.../321/g' bazaar.txt
                    

sed 's/\<...\>/321/g' bazaar.txt
                    

The above RE ( sed \%\<[a-z][a-z][a-z]\>%p' /usr/share/dict/words ) is a little long, so we could shorten it using the splat to:

sed '/\<[a-z]\{3\}\>/p' /usr/share/dict/words
                

(this may be hard to see that you are in fact getting the results you are after. You could, instead, not delete words that are 3 charaters in length by replacing the "p" with a "!d" (don't delete) in the sed expression above:

sed '/\<[a-z]\{3\}\>/!d' /usr/share/dict/words )

sed '/19\{3\}/p' bazaar.txt
                

The command now executes as expected and only one duplicate line is output from the file, that which contains the text 1999. So {n} matches exactly n occurrences of the expression.

If we wanted to match a string with a minimum of 4 a's, up to .... well infinity a's we could use the pattern:

a\{4,\} 
                

This regular expression says match no upper limit, but the string must contain at least four a's. Thus it would match four a's, forty a's or even four hundred a's following one another, but it would not match three a's.

Let's now match the letter m at least once and with no upper limit. We would do this by:

sed '/m\{1,\}/p' bazaar.txt
                

If we change the 1 to a 2, our pattern becomes:

sed '/m\{2,\}/p' bazaar.txt
                

This would match only those lines with the words: community, programming etcetera (i.e. any words containing at least two m's).

The following expression would match a minimum of four a's but a maximum of 10 a's in a particular pattern:

a\{4,10\}
                

Let's say we wanted to match any character a minimum of 3 times, but a maximum of 7 times, then we could affect a regular expression like:

.\{3,7\}
                

This allows us a huge degree of flexibility when we start combining these operators.

What does the following RE match?

^[aeEI]\{1,3\}
                

This RE means: "look for any line that starts with any of the characters a,e,E,I a minimum of one time but a maximum of 3 times. Thus it would match any of the following:

aaa
a
aE
e
E
I
                

Would it match abd or adb or azz for that matter, or only lines that start with any of the characters in the RE, followed by up to 2 other characters from the RE?

It would not match the following:

aaEI
EIea
bEaa
IIEEaae
iEE
                

(why?-- you should answer this.)

If you think this is bizarre, hang in there, it gets more bizarre. Let me finish off RE's with two concepts. The first is 'greediness'. RE's are greedy, which means that they will match as much as they possibly can.

Assuming you have an expression:

ham.*
                

This will match as much as it possibly can within that expression. So it would match

	
ham
                

but if we had:

hammock
                

it will match the entire word hammock, because it tries to grab as much as it possibly can - it's greedy. RE's are greedy and sometimes they'll be matching a lot more than you expect them to match. The closer you can get your RE to the actual thing that you're looking for, the less the greediness will affect your results. Let's look at some examples of that.

Exercises

The following exercises will show you how sed's greediness affects the output, and how to create RE's that will only give you the results you want.

I have included 3 files, emails{1,2,3}.txt in the examples directory you should have downloaded these previously.

In order to REALLY come to terms with RE's, work through these exercises using these 3 files:

1. Match the subject lines in these files. Subject lines look as follows:

   Subject:

2. List only the 'id' of each message. This can be found with the string 'id', but there is a catch!

3. What mail clients have people used?

4. Obtain a listing of all za domains, all com domains, etc in these emails.

5. Given the following RE's what would they match?

ht\{2\}p:\/\/ ht\{2\}p:\/\{2\} ht\{2\}p:\/\/w\{3\}.*za$ ht\{2\}p:\/\{2\}.*\/.\{9\}\/

	Note
	You will have noticed that in order to understand these, you have to work through them systematically, left to right, understanding each part as you go!

Placeholders are a way of keeping the pattern that you've matched.

In your example files, there's a second file called columns.txt. This file has two columns:

name		age
                

I want to swap the two columns around so that the file contains the age column on the left, and the name column on the right.

Now, if you start thinking about how to do that, it might become quite a complex thing to achieve (without using tools like awk or perl etc.).

With RE's and sed, it's very simple using placeholders. So let's first try and develop a pattern that matches name and a pattern that matches age. Notice that the two columns in the file are separated by a single space. The expression for the name column would be:

[a-z]* 
                

Assuming that no one in our file is 100 years or older we can use the following expression to match the values of the age column:

[0-9]\{1,2\}
                

That should match any age (in the file) because it means match any digit in the range 0-9 a minimum of once but a maximum of twice. So it should match a person whose age is: 1, 9 or 99.

Now the sed expression would then be:

sed '/^[a-z]* [0-9]\{1,2\}$/p'
                

This only searches for lines matching and prints them.

How do I swap the name and the age around? I'm going to enclose the name in round brackets (remember you have to escape round brackets). Similarly I'm going to enclose the age expression in round brackets.

Our sed expression now looks like:

sed 's/^\([a-z]*\) \([0-9]\{1,2\}\)$/\2,\1/' columns.txt
----^__------__^__------------__^-__-___
       1 2   3     4 5 6         7           8 9 10  11

	1 = Caret (start of line)
	2 = Start of placeholder for the name RE
	3 = Name RE
	4 = End placeholder for the name RE
	5 = Space between the name and the age in the file
	6 = Start placeholder for the age RE
	7 = The Age RE
	8 = End placeholder for the age RE
	9 = Dollar (end of line)
	10= Placeholder 2 (the age)
	11= Placeholder 1 (the name)
                

The first set of round brackets contains the 'name' RE, while the second set of round brackets enclose the 'age' RE. By encompassing them in round brackets, I've marked the expressions within placeholders. We could then use \2 to represent the 'age' placeholder, and \1 to represent the 'name' placeholder. Essentially this expression says "search for the name and age, and replace it with the age and then name". Thus we've switched the two columns.

The above final expression looks very complex but I tackled this regular expression in byte-size chunks.

I said let's write a regular expression to match the name. Now let's write a regular expression to match the age. Once I had these two individual expressions, I combined them. When I combined them into a single regular expression I then just included round brackets to create placeholders. Later in sed, we were able to use these placeholders in our search-replace expression. Now try and do that in other operating systems!

Try these:

free | sed '/^Mem/!d'
free | sed '/^Mem/!d';  '/  */,/g'
VAR=`free | sed '/^Mem/!d';  '/  */,/g'`
echo $VAR
                

A final useful trick is that of word boundaries. We've seen them a little earlier, but here is a formal explanation. Suppose we are wanting to search for all words 'the':

sed 's/the/THE/g' bazaar.txt
                

would probably be our first try. Problem is, this will also match (and change) 'there', 'them', 'then', 'therefore', etc. Problem, yes?

Solution? Well, the solution is to bound our word with word boundary markers (the official term is word anchors).

Let's rewrite our pattern with this in mind:

sed 's/\<the\>/THE/g' bazaar.txt
                

This time, we only match the whole word 'the' and not any of the others. So the word anchors will restrict the pattern to complete words and not segments of words.

1. For the next example I'd like you to make sure that you've logged on as a user (potentially root) on one of your virtual terminals.

How do you get to a virtual terminal? Ctrl-Alt plus F1 or F2 or F3 etcetera.

It should prompt you for a username and a password. Log in as root, or as yourself or as a different user and once you've logged in, switch back to your X terminal with Alt-F7. If you weren't working on X at the beginning of this session, then the Ctrl + Alt + F1 is not necessary. A simple Alt + F2 would open a new terminal, to return to the first terminal press Alt+F1.

2. Run the who command:

who 
                

This will tell us who is logged on to the system. We could also run the w command:

w
                

This will not only tell us who is logged on to our system, but what they're doing. Let's use the w command, since we want to save information about what users are doing on our system. We may also want to save information about how long they've been idle and what time they logged on.

3. Find out who is logged on to your system. Pipe the output of the w command into the input of cut. This time however we're not going to use a delimiter to delimit fields but we're going to cut on characters. We could say:

w |cut -c1-8
                

This tells the cut command the first eight characters. Doing this you will see that it cuts up until the first digit of the second. So in my case the time is now

09:57:24
                

and it cuts off to

09:57:2
                

It also cuts off the user. So if you look at this, you're left with USER and all the users currently logged onto your system. And that's cutting exactly 8 characters.

4. To cut characters 4 to 8?

w | cut -c4-8
                

This will produce slightly bizarre-looking output.

So cut cannot only cut fields, it can cut exact characters and ranges of characters. We can cut any number of characters in a line.

Often cutting characters in a line is less than optimal, since you never know how long your usernames might be. Really long usernames would be truncated which clearly would not be acceptable. Cutting on characters is rarely a long-term solution.. It may work because your name is Sam, but not if your name is Jabberwocky!

1. Let's do a final example using cut. Using our password file:

cat /etc/passwd
                

I'd like to know all usernames on the system, and what shell each is using.

The password file has 7 fields separated by a ':'. The first field is the login username, the second is the password which is an x (because it is kept in the shadow password file), the third field is the userid, the fourth is the group id, the fifth field is the comment, the sixth field is the users home directory and the seventh field 7 indicates the shell that the user is using. I'm interested in fields 1 and 7.

2. How would we extract the particular fields? Simple:^[6]

cat /etc/passwd |cut -d: -f1,7
                

cut -d  -f1,7	
or
cut -d" " -f 1,7
                

If we do this, we should end up with just the usernames and their shells. Isn't that a nifty trick?

3. Let's pipe that output to the sort command, to sort the usernames alphabetically:

cat /etc/passwd | cut -d: -f1,7 | sort
                

So this is a fairly simple way to extract information out of files. The cut command doesn't only work with files, it also works with streams. We could do a listing which that would produce a number of fields. If you recall, we used the tr command earlier to squeeze spaces.

ls -al 
                

If you look at this output, you will see lines of fields. Below is a quick summary of these fields and what they refer to.

I'm particularly interested in the size and the name of each file.

1. Let's try and use our cut command in the same way that we used it for the password file:

ls -al |cut -d' ' -f5,8
                

The output is not as expected. Because it is using a space to look for separate fields, and the output contains tabs. This presents us with a bit of a problem.

2. We could try using a \t (tab) for the delimiter instead of a space, however cut only accepts a single character (\t is two characters). An alternative way of inserting a special character like tab is to type Ctrl-v then hit the tab key.

^v + <tab>
                

That would replace the character by a tab.

ls -al | cut -d"       " -f5,8
                

That makes the delimiter a tab. But, we still don't get what we want, so let's try squeezing multiple spaces into a single space in this particular output. Thus:

ls -la |tr -s ' '|cut -d' ' -f5,8
                

3. And hopefully that should now produce the output we're after. If it produces the output we're after on your system, then we're ready for lift-off. If it doesn't, then try the command again.

Now what happens if we want to swap the name with the size? I'll leave that as an exercise for you.

There are three versions of the grep command:

If you're using egrep it's the slowest, you can test this using the following:

time egrep "[Ll]inu." bazaar.txt
time grep "[Ll]inu." bazaar.txt
time fgrep "[Ll]inu." bazaar.txt
                

The times should decrement from top to bottom. grep by default isn't very fast, so if you're trying to do the same job that was done with grep and sed, sed would be significantly faster than grep.

I'm not going to cover egrep or fgrep because they work almost identically to grep, the only difference being that you can use extended regular expressions (egrep) or no regular expressions at all (fgrep).

Use fgrep, egrep and grep where you think it appropriate to achieve the following:

From the emails*.txt files, show only the lines NOT containing linuxrus.

Using the concept of grep as a filter, explain why this would be a useful command to use on large files.

Let's understand a couple of things about Linux. Linux doesn't care about extensions, it's not interested in what the extension of this particular file is. My advice however, is for your reference (i.e. for the sake of readability), to append an .sh on the end of every shell script file. That will immediately alert you to the fact that this file is a script without having to perform any operation on the file.^[13]

Of course if you don't do that, it doesn't make any difference, it will still be a script. But my convention, (there are as many conventions as there are system administrators and Linux distributions) encourages the .sh on the end of a script file name. This tells me in no certain terms that this is meant to be a script.

At this point we should be able to run that script. So type the following on the command line:

script.sh 
                

When you do that, you will notice the following error:

script.sh: Command not found. 
                

The reason that the command is not found is that it looks in your search PATH for this "new" command.

Of course your PATH hopefully (if you're a halfway decent system administrator) doesn't have a '.' in it. In other words your PATH doesn't include your current directory.

In order to run this script you need to precede the script by the PATH to the script. Thus:

./script.sh
                

When you do this, it still won't run! Why? You haven't changed the script to be executable. The way that you do this is:

chmod +x script.sh
                

From thereon, the script is interpreted as an executable file and you can rerun that script by using the following command:

./script.sh 
                

You have to make every script executable with the chmod command. If you don't change its mode, it won't run. Every time you run it, it will show you a list of unique shells that are being used by your system.

You could give other users access to this script, or you could place this script in relevant home directories so that it could be executed.

Or you could put it into a place on the system that everybody has access to (e.g. /usr/bin).^[14]

It's important, now that you're learning to write scripts (which will ultimately take you on to writing programs and ultimately to becoming a fully-fledged open source developer), that you document your scripts well.

Since we're all such good programmers we will definitely want to do this. How? We can put comments in our scripts using a hash (#) to show that a particular line is a comment.

Edit script.sh as follows:

vi script.sh
                

Insert a hash or two at the top of this file and write a comment about what this script does, who wrote it, when you wrote it and when it was last updated.

# Student name - written February 2004.
# A script to produce the unique shells using the /etc/passwd file

cut -d: -f7 /etc/passwd |sort -u 






: w script.sh

                

This is the bare minimum comment you should make in a script. Because even if you don't maintain your scripts, there's a good chance that somebody in the future will have to; and comments go a long way to proving that you're a capable coder.

	Note
	It's a vital part of open source - to provide documentation. Comments can appear anywhere in a file, even after a command, to provide further information about what that particular command does.

Variables are a way of storing information temporarily. For example I may create a variable called NAME and I assign it a value of "Hamish":^[15]

NAME="Hamish"
                

A couple of conventions that we need to follow: variables usually appear in uppercase, for example I have assigned to a variable called 'NAME' the value 'Hamish'. My variable name is in uppercase. There is no white space between the variable name ('NAME') and the equals sign.

Similarly, without any white space enclose the value in double quotes. This process allocates space (memory) within the shell calling the reserved memory 'NAME', and allocates the value 'Hamish' to it.

How do we use variables?

In this case, we will use the echo command to print the output to the screen.

echo "Hello $NAME"
                

which would print:

Hello Hamish
                

to the screen. We could create a file with:

touch $NAME
                

This would create a file called 'Hamish', or else type:

rm $NAME
                

which would remove a file called 'Hamish'. Similarly, we could say:

vi $NAME
                

which would open the file 'Hamish' for editing. In general, we assign a variable with:

NAME=value
                

And we can use the variable in a variety of ways.

Does the variable have to be a single string? No, we could've assigned a variable with:

HELLO="Hello World"
                

Please set this variable from the command line and then test the following :

touch $HELLO
                

List your directory to see what it has produced.

Remove the file using the variable name:

rm $HELLO
                

What happens? Why?

So setting a variable is a case of assigning it using an equals sign.

Using a variable is achieved by preceding the variable name with a dollar sign.

As I indicated, the convention is to keep the variable name uppercase, however we don't necessarily need to adhere to it. My advice is to stick with the convention and keep them uppercase.

So far we've written very simple scripts. Our scripts have entailed simply an echo statement and maybe one other command. In order to achieve a higher degree of complexity, we need to tell the script what shell it's going to run under.

One might find that a little strange because we're already running a shell, so why do we need to tell the script what shell to run as? Well perhaps, even though we're running the bash as our default shell, users of this script may not be running the bash as their default shell. There are a couple of ways of forcing this script to run under the bash shell. One means of running our script using the bash may be:

sh script.sh
                

This would execute the script using the bourne shell (sh). This looks like a lot of work to repeat every time - insisting on the shell at the prompt. So instead, we use a shebang.

A shebang is really just a sequence of two characters - a hash sign followed by an exclamation mark. It looks like this:

#!
                

This is known as the shebang. Comments also start with a hash, but because this particular comment starts at the top of your script, and is followed immediately by a bang (an exclamation mark), it's called the shebang. Directly after the shebang, we tell the script what interpreter it should use.

If we had the following line at the top of our script:

#!/bin/ksh
                

This would run the contents of script.sh using the korn shell. To run the script using the bash we would have:

#!/bin/bash
                

If this was a perl program, we would start the script off with:

#!/usr/local/bin/perl
                

A sed:

#!/bin/sed
                

All subsequent commands would then be treated as if they were sed commands. Or perhaps we want to use awk:

#!/bin/awk
                

This assumes awk lives in our /bin directory. It might live in /usr/bin in which case it would be:

#!/usr/bin/awk
                

So we can include the shebang at the top of every script, to indicate to the script what interpreter this script is intended for.

While we have not included the shebang at the top of scripts written thus far, I'd encourage you to do so for the sake of portability. Meaning that the script will run correctly, wherever it is run.

We've seen a standard way of starting a script (the shebang), now I need to tell you about the standard way of ending a script.

Before we do that, we must understand what exit values are. Every program in Linux that completes successfully will almost always exit with a value of 0 - to indicate that it's completed successfully. If the program exits with anything other than 0, in other words, a number between 1 - 255, this indicates that the program has not completed successfully.

Thus, on termination of every script, we should send an exit status to indicate whether the script has completed successfully or not. Now if your script gets to the end and it does all the commands that it's supposed to do correctly, the exit status should be zero (0). If it terminated abnormally, you should send an exit status of anything but zero. I will therefore end every script with the command:

exit 0 
                

Thus, if no error is encountered before the end of the shell, the exit value will be zero.

Exit statuses also come in useful when you're using one script to call another. In order to test whether the previous script completed successfully, we could test the exit status of the script.

This is discussed in more detail later the section called “Exit status of the previous command”

There are some variables that need special attention, namely NULL and unset variables.

For example, if a variable called NAME was assigned with the following:

NAME=""
                

then the variable is set, but has a NULL value. We could have said:

NAME=
                

which too would have meant a NULL value. These are distinctly different from:

NAME=" "
                

A space between quotes is no longer a NULL value. So if you assign:

NAME="hamish"
                

this has a non-NULL value, while if you assign nothing to the NAME variable it's a NULL value. This distinction can sometimes catch you out when you're programming in the shell especially when doing comparisons between values. If the variable NAME were never set, a comparison like:

if [ $NAME = "hamish" ]; then
....
                

would return an error, as the test command requires a variable = value comparison. In the case of the NULL/unset variable it would test:

[ = "hamish" ]
                

which would be an error.

One method of handling NULL values in scripts, is to enclose the value in quotation marks, or surround them with "other characters". To display a NULL value NAME,

echo $NAME
                

would return a blank line. Compare this to:

echo :$NAME:
                

which would return

::
                

since the value is NULL. This way we can clearly see that a NULL value was returned. Another method of checking for NULL values in expressions is as follows:

if [ "${NAME}x" = "x" ]; then
.....
                

Here, if NAME is unset (or NULL), then:

"${NAME}x" would be "x"
                

and the comparison would be TRUE, while if

NAME="hamish"
                

then

"${NAME}x" would be "hamishx"
                

and thus the comparison would be FALSE.

What happens is if the value is not set at all? For example, what occurs if you unset a variable:

unset NAME
                

A similar result to the NULL variable occurs, and we can treat it in the same was as a NULL variable.

In sum then, the unset/NULL variables are very different from a variable that has an empty string as in

VAR="      "
                

Similarly, another question is: When does the shell do the interpretation of a variable?

In the statement:

echo $NAME
                

it does the $NAME variable substitution first before invoking the echo command.

What happens if we typed:

file="*"
ls $file
                

The output is equivalent to saying:

ls *
                

What happened in our example above? The variable file is being interpreted first, it then gets an asterisk (splat) which matches all files in the current directory and lists those files on the command line.

This illustrates that substitution of the variable occurs first, before any further command is executed.

What happens if I want to echo the following string?

hamishW
                

and my name variable NAME is currently set to 'hamish'? Can I do this:

echo $NAMEW
                

What's going to happen here?

The shell attempts to look for a variable NAMEW which clearly does not exist, but there is a variable NAME.

How do we make a distinction between the variable name and anything we want to follow the variable name? The easiest way to do that, is to use the curly brackets:

{}
                

Trying that again, we could write:

echo ${NAME}W
                

and the shell will now interpret the {NAME} as the shell variable and understand that 'W' is not part of the variable name.

In essence:

$NAME
                

is equivalent to

${NAME}
                

They achieve the same purpose, the only distinction between them is if one added a 'W' to the second example, it would not be considered as part of the variable name.

Since we're covering the topic of variables, now is a good time to make a distinction between environment and shell variables. Environment variables are set for every shell, and are generally set at login time. Every subsequent shell that's started from this shell, get a copy of those variables. So in order to make:

NAME="Hamish" 
                

an environmental variable, we must export the variable:

export NAME
                

By exporting the variable, it changes it from a shell variable to an environment variable.

What that implies, is that every subsequent shell (from the shell in which we exported the variable) is going to have the variable NAME with a value 'Hamish'. Every time we start a new shell, we're going to have this variable set to this value. It should go on and on like that. By exporting it, that's what we call an environment variable.

If a variable is not exported, it's called a shell variable and shell variables are generally local to the current shell that we're working in.

In other words, if we set a variable:

SURNAME="Whittal"
                

and at the prompt we now say:

bash 
                

starting a new shell, then:

echo $SURNAME
                

It will return a blank line. Why is there a blank line? Primarily because that shell variable wasn't exported from the previous shell to the new shell and is thus not an environmental variable. Shell variables are only available in the original shell where we issue the assignment of the variable.

We now have an understanding of variables, how we can set them and, in the next chapter we will look at quoting, specifically how we can run commands and assign the output of those commands to variables.

We've done basic shell scripting, but it would be nice to be able to do some basic arithmetic in the shell. While the shell is able to do basic integer arithmetic, it cannot do floating-point arithmetic. However, there are some ways of getting around this limitation. If we wanted to do floating point arithmetic we can use a utility called:

bc
                

which is a calculator.

We will have a chance to look at this later in the course. If you need to do lots of floating point arithmetic - I think you need to take a step up from this course and do a perl, Java or C course.

Let's concentrate on integer arithmetic.

There are a number of ways of doing integer arithmetic in the shell. The first is to enclose your expression in double round brackets:

$(())
                

Assuming you set a shell variable i:

I=10
                

You could then say:

$((I=I+5))
echo $I
                

It would return:

15
                

Arithmetic operators are as follows:

Read the man pages (man 1 bash or info) to find out about others. Within these $(()), you could do:

$((I=(15*I)-26))
                

By enclosing stuff inside round brackets within the arithmetic operator, you can change the precedence. The precedence within the shell is the good, old BODMAS (Brackets, Order, Division, Multiplication, Addition, Subtraction - see http://www.easymaths.com/What_on_earth_is_Bodmas.htm ).

So, the shell does honour the BODMAS rules.

Changing the order of the expression requires brackets.

$((J=I+5))
J=$((I+5))
J=$(( I + 5 ))
$(( J = I + 5 ))
                

all mean the same thing.

However, the following will produce errors due to the spaces surrounding the '=':

J = $(( I + 5 ))
                

We could, for example say:

I=$((k<1000))
                

What would happen here? This function would result in a true(0) or false(1) value for I.

If k<1000 then i=0 (true), but if k>=1000 then i=1 (false). 
                

You can do your operations like that, assuming that you have calculated the value of k before this step.

Although we currently do not have sufficient knowledge to perform loops, (we'll see later on how we use loops Chapter 8), I've included a pseudo-code loop here to illustrate how shell arithmetic can be used practically:

COUNT=0
loop until COUNT=10
    COUNT=$((COUNT+1))
done
                

COUNT, the variable, starts at 0, and increments by 1 each time round the loop. On count reaching 10, the loop exits.

Practically let's use the df command to do some examples. We're going to create a script called mydisk.sh.

At the top of your script include the shebang relevant to your shell, and at the end include your exit status.

#!/bin/sh
# This script will squeeze all spaces from the df command.
#
# First set the date and time
DATE=20031127
TIME=09:52

# Now squeeze the spaces from df
df -h|tr -s ' '

# We're done - no errors - exit 0
exit 0
                

Let's work this in sizeable bite-chunks. If you remember:

df -h|tr -s ' '
                

will pipe the diskfree output (size, percentage free and mount points) in human readable form to the translate command which will then squeeze multiple sequential spaces into a single space, giving:

/dev/hda6 3.8G 2.9G 744M 80% /
/dev/hda9 12G 10G 1.1G 90% /mnt
/dev/hda5 3.8G 2.5G 1.0G 70% /debian

df -h | tr -s ' ' | tr ' ' ',' | sed '/^\/dev/!d'; \

/dev/hda6,3.8G,2.9G,744M,80%,/
/dev/hda9,12G,10G,1.1G,90%,/mnt
/dev/hda5,3.8G,2.5G,1.0G,70%,/debian
                

From the output of the df command we are only interested in the partition that the device is on (in this example, da0s1{aeh} - nice to know RE's hey!), the size, the percentage free and the mount point in that order:

mount point,part,size,%free

df -h|tr -s ' '|tr ' ' ','|sed '/^\/dev/!d; \

s%/dev/\(hda[1-9]\+\),\([0-9]\+\.\?[0-9]\?[GMK]\?\),.*%\1;\2%g
-^_____^----^_____^--^_^-----^_^-^_^----_^_____-^-		^^^^^__-^_^
-1-----2----3----4---5-6-----7-8-9-1-----1------1-11111---1-2
-----------------------------------0-----1------2-34567---9-0

1=Start of search using the % not the / since / are in the RE
2=(hda) Start of the group (to match the hdaX on your linux machine)
3=(hda[0-9]). Match the range hda1, hda4, hda12, etc. 
- Note hda10 will not be matched here. Why not?
4=Match 0 or more of them (i.e. match hda3 or hda11)
5=Follow the hdaX by a comma
6=Start of the group 12G or 3.8G. Match a 0-9
7=Match 0-9 one or more times
8=Followed by an optional full stop (\.\?)
9=Optional full stop. See 8 above.
10=Followed immediately by an optional number (e.g. the .8 in 3.8)
11=The optional number in 10 above.
12=Followed by a G, K or M for Gigabytes, Kilobytes of Megabytes,
Optionally...
13=End of group to match the 3.8G or the 12G
14=Followed by a comma
15=Followed by anything
16=Zero or more times (for 15 above)
17=End of pattern started in 1
18=First placeholder (the hdaX)
19=Second placeholder (the 3.8G, 12G, etc.)
20=End of RE.

--This command looks like the following when it is printed on one line:--
df -h |tr -s ' '|tr ' ' ','|sed '/^\/dev/!d; s%/dev/\hda/\(hda[1-9]\+\),\([0-9]\+.\?[GMK]\?\),.*%\1;2%g'
                

Phew. I'll leave you to modify the RE to encompass all other fields we need. It's really not that difficult, just a little tricky.

As you remember we must make this script executable before we can run it, so:

chmod +x mydisk.sh
                

Now, let's run the script:

./mydisk.sh
                

Have you got the result we are after?

Of course, we could have achieved the above RE with a cut command, but there are even better ways of skinning this cat. Stay tuned.

Ensure the following for you scripts:

None of these scripts should be longer than 10 lines (at the outside)

What do the following commands do. Explain why.

What do the following commands do. Explain why.

I would personally use the second option, because it is easier to read, and not as confusing.

Using a special variable called $#.

One of the ways we can use this (we'll see when we come to decisions and if-then statements) is to check how many arguments were used to run a script. Let's imagine that we want our script eatout.sh to be executed with at least one argument.

We would include something like:

if $# < 1
echo "Usage: ...."
exit 1
                

in our script.

Why would we exit with a value of 1? Because we didn't execute the script correctly.

We can use a number of positional arguments to print out useful information about how to use our script.

We've got yet another useful construct, $*, which tells us all the positional arguments.

Let's look at an example of this:

#!/bin/bash
DATE=$(date +"%d %b %Y %H:%M")
TYPE=$1
echo "The arguments asked for: $*"
echo "Queried on $DATE"
grep $TYPE restaurants.txt |sort +3n
exit 0
                

If we then ran our script eatout.sh with:

./eatout.sh italian 5
                

We would get the following output:

riaan@debian:~> ~/ham$ ./eatout.sh italian 5
The arguments asked for: italian 5
Queried on 01 Dec 2003 21:36
italian,Bardellis,6973434,5
                

So even if we had 20 positional arguments, it would show us each one of them storing them all in $*.

I said we only have 9 positional parameters, but that's not true, you can have many more than 9. How do we get hold of them?

If we thought of the positional arguments as a list: we've got positional argument 0 which is the name of the shell script or the name of the program, but we never modify that one.

Then we have positional arguments 1 to 9 and we may have additional positional arguments.

To obtain the arguments, we can use the shift command, which will shift arguments to the left. Thus argument 2 would become 1 after a shift, 3 would become 2, 4 would become 3, etc. At the end of the list, 9 would become 8, 10 would become 9, etc.

Figure 6.1. Using Shift Command to access parameters

If we said:

shift 5
                

it would shift the first 5 positional arguments off the front, and bring 5 additional arguments from the end.

If you were crazy enough to have more than 9 positional arguments for a script, you could shift repeatedly until you get all the positional arguments. Thus you can shift positional arguments off the command line, which allows you to get hold of additional arguments above the 9 positional parameter limit.

It's a good idea to save your positional arguments so that you can use them at a later stage. You never know when you're going to actually need to use an original argument sent into the script. In the eatout.sh script, I saved the type of restaurant ($1) storing it in the variable TYPE.

A final dollar command for now, is $?. $? tells you the exit status of the previous command. So if you ran your eatout.sh script, and as soon as it's finished, you echo $? - assuming that it all ran correctly - you would expect that it would return a 0. Why is that? Because you exited your script with an exit status value of 0.

Let's assume you tried to do something in your script that doesn't work, then you could say:

if $?<>0
echo "Some previous command failed"
                

This test checks whether the previous command ran correctly and, if not (i.e. The output of the previous command was non-zero) a message to that effect is printed to the screen. So 0 is the exit status of the previous command.

If you run a command like:

ping -c1 199.199.199.1
                

Wait for it to complete. And then run:

echo $?
                

you should get a non-zero value. Why? Because the command (ping) never exited properly, and thus a non-zero value is returned. Compare that to the output of:

ping -c1 127.0.0.1
echo $?
                

which should always return a value:

0
                

Why? Because it can ping your local loop-back address. So every command exits with an exit value and you can test the exit value that it exits with, using $?.

Even if you did an:

echo "Hello"
echo $?
                

It would return a value 0, because it was able to print the string 'Hello' to the screen.

So we're going to be talking about the if-then condition. Before we do that, we need to understand the command 'test'.

info test
                    

ping -c1 199.199.199.1
                    

echo $?
                    

who; echo $?
                    

who |grep root; echo $?
                    

who |grep root; test $?
                    

The different types of tests that we can do are:

NAME="hamish"
test $NAME = hamish
echo $?
                    

test $NAME = joe
echo $?
                    

BLANKS="     "
                    

test $blanks
echo $?
                    

test "$blanks"
echo $?
                    

test '$blanks'
echo $?
                    

test "$NAME" = "hamish"
                    

test "$NAME" = hamish
                    

test "${NAME}x" = x
                    

blanks="     "
test -z "$blanks"	
echo $?
                    

test -n "$blanks"	
echo $?
                    

x=101
                    

$x < 10
                    

test "$x" -lt 10
echo $?
                    

test "$x" -lt 102
echo $?
                    

info test 
                    

test -f file
                    

test -d . 	
                    

test -d .bashrc
                    

Using the following expressions, determine whether the outcome will be TRUE (0) or FALSE (1) or unknown.

First set some variables:

MOVIE="Finding Nemo"
CHILD1="Cara"
CHILD2="Erica"
AGE_CH1=4
AGE_CH2=2
HOME=ZA
                

How do we add if statements? An if statement has the following form:

if condition
then
	do some operations
fi
                

Now, since I'm lecturing you, I might as well lecture you in good structured programming style.

When you start an 'IF' statement, put the 'THEN' statement on the next line, and make sure that you indent all the commands that you want within the 'THEN', by at least one tab or a couple of spaces. Finally, end your "IF" statement in a nice block format using the 'FI'. It's going to make maintaining your scripts much easier.

In our eatout.sh

if [ "$#" -lt 1 ]
then 
	echo "Usage: $0 <parameter>
	echo "where parameter is: italian|thai|smart|steakhouse"
	exit 1
fi
                

If we add this to the top of eatout.sh, our script will stop running if the user does not provide at least one positional parameter, or argument. Furthermore it will echo the usage command to explain how to use the script correctly and to avoid the error message.

Equally, 'IF' has an associated construct, the 'ELSE':

if condition
then
	... <condition was TRUE, do these actions> ...
else
	... <condition was FALSE, do these actions> ...
fi
                

If a user runs the eatout.sh script with a correct parameter, then you can show them your favourite eating places, and if they don't it will exit with a status of 1 as well as a usage summary.

Notice that the condition that I used is a very simple one: I'm checking whether the number of parameters is less than one.

I'll leave it as an exercise for the user to check that the parameter that the user has entered is one of the allowed words (italian/steakhouse/smart).

To give you a hint, you could use:

[ $# -lt 1 -a "$1" = 'italian' or "$1" = 'steakhouse' or ..."
                

So you're to check that the number of parameters is at least one AND the $1 is equal to one of the allowed words. Is there a better way of doing this? There sure is.

What we might want to do is, if the restaurant we choose is a steakhouse, we might want to allow the user to choose between 5 different ways of doing their steak. For that we're going to want to do more than one test:

if $1 steakhouse
then 
	... ask how they like their steak done ...
else 
	if $1 smart
	then
		...
	else
		if $1 thai
		then
			...
		fi
	fi
fi
                

Note that there have to be matching fi's for every if statement.

As you can see reading this becomes quite difficult due to all the embedded if statements. There is an alternative construct called an elif which replaces the else-if with an elif and this makes the readability easier.

Look below for the syntax:

if $1 steakhouse
then 
	...
elif $1 smart
  then 
	...
  elif $1 thai
    then 
	...
    elif $1 italian 
      then 
	   ...
      else
	..
fi
                

Note that the final else is tied to the closest if. So in our example, the else statement will only be executed if $1 is NOT an italian resturant.

Is the 'IF' statement the best way of doing things? If you're going to do else if, else if, else if, etc. - then the answer is NO! It's bad programming practice to do this else-if, else-if nonsense. So how do we do things?

Well we've got a 'CASE' statement. The structure of a 'CASE' statement is as follows:

case $1 in 
    pattern) ...
    	 ...
    	 ;;
    pattern) ...
    	 ...
    	 ;;
    *) ...
       ...
       ;;
esac
                

This means that we will match $1 to a pattern. The pattern will allow us to execute a series of statements and to finish this pattern we use a double semi-colon. We can then match the next pattern and, if it matches we do another whole series of things, ending with another double semi-colon.

If $1 matches none of the patterns, then it will be caught by the asterisk pattern since an a asterisk matches everything as we've seen in our regular expression and pattern theory.

The case statement makes your code a lot more legible, easier to maintain and allows you to match patterns.

Look at another example:

case $1 in 
	[Tt][Hh][Aa][Ii]) ...
		 ...
		 ;;
	Steakhouse) ...
		 ...
		 ;;
	*) echo "Sorry this pattern does not match any restaurant"
	   ...
	   ;;
esac
                

In this CASE statement, the first pattern matches ThAI or thAI or Thai, etc.

There's a better way of making your patterns case-insensitive. You could put the following line at the top of your script which would translate every character in your parameter $1 to uppercase:

RESTURANT_TYPE=(echo $1 |tr '[a-z]' '[A-Z]')
                

This will remove the long complicated pattern:

[Tt][Hh][Aa][Ii]) 
                

and we could instead just look for the pattern:

THAI
                

Similarly, if the user of our eatout.sh script only wants to type out part of the keyword for example, using:

./eatout.sh steak
                

instead of

./eatout.sh steakhouse
                

or

./eatout.sh meat 
                

instead of

./eatout.sh steakhouse 
                

These choices can be matched with the following pattern

steak|steakhouse|meat
                

Similarly this pattern

pasta|pizza|italian 
                

would match all of the following uses of our script:

eatout.sh pasta
                

and

eatout.sh pizza
                

and

eatout.sh italian
                

So you can match ranges of alternatives separating each with a vertical bar - the pipe character. So the case statement is most certainly the far better way to match alternatives with a script.

Using the uptime from /proc/uptime, write a script which will determine how long your Linux machine has been 'UP', printing the following output to the console accoring to the results:

0 - 1 hour	"Obviously you're new to Linux. What's \
                    all this rebooting your machine nonsense"
1 - 5 hours	"Still a novice I see, but perhaps I could be wrong"
1 - 5 days	"Mmmm. You're getting better at this Linux thing!"
                

In the script, we may want to do a null command. There is a special command:

if : 	#This condition will always be true (as : is always true)#
then
	:
else
	:
	:
fi
                

The null command is a colon. We could for example produce a never-ending while loop using nloop by saying:

while :
do		echo "hello $NAME"
done
                

"noop" is always TRUE! Earlier on we tried

`date | cut...`
                

in order to obtain 'SAST' (SA std time) as output. Doing this on the command line, it said:

SAST: command not found
                

Modifying this however to:

: `date |cut...`
                

would have not produced an error, since the colon will always produces TRUE output. The shell thus executes the no-op instead of the output of the date/cut command.

If you said:

if :
then 
	...
fi 
                

It would always do the THEN part of the IF statement.

You can achieve multiple commands on the same line by using the && and ||. How does this work? By way of example, you could say:

grep italian restaurants.txt || echo "sorry no italians here"
                

What this means is:

"if there are italians inside resturants.txt then return them
	OR [else]
return the string 'sorry, no italians here'"
                

In shell terms, it means:

if the result of the grep command is TRUE (0), 
then you  will get the lines from the file resturants.txt containing the word italian 
BUT if  there are no lines in the file containing the word italian 
(i.e.the outcome of the grep is FALSE (1)) 
	then print
 'sorry no italians here'
                

As before, this is a shortcut way of doing things. Enclosing this command in parentheses can change the order of execution as in:

\( grep italian restaurants.txt || echo "sorry no italians here" \) 
                

which could also allow:

\( cmd1 || cmd2 \) && \( cmd3  && cmd4 \)
                

Here is another example:

echo "Oh, you're looking for italian, here they are: " \
&& grep italian restaurants.txt
                

Echo will always return TRUE(0), so it would print out the echo statement and then the list of italian restaurants, if there are any. Very useful!

Using the || and && constructs, perform the following operations:

Let's start with the for loop, which has the following syntax:

for variable in list
do
	...
	...
done
                

A more specific example of this case is:

for i in 1 2 3 4
do
	echo ${i}
done
                

If you run the above, you will get four numbers printed to the output:

1
2
3
4
                

So the for loop says:

"for every element in the list (1,2,3,4 in our case) do something (echo $i in our case)"
                

In this example, we are just echoing the output. No rocket science there, but it's a good means of introducing us to for loops.

The lists could be anything, they could say:

for NAME in hamish heidi matthew riaan simone
do
	echo "people involved in this project: "
	echo $NAME
done
                

This would produce:

people involved in this project:
hamish
people involved in this project:
heidi
people involved in this project:
matthew
people involved in this project:
riaan
people involved in this project:
simone
                

You'll notice that the echo commands were printed 5 times, once for every argument in the list. This means that everything enclosed in the DO-DONE block will be executed every time that FOR loops.

Just a quick note, the file command tells us the type of a file. You could say:

file restaurants.txt
                

and hopefully it will return:

restaurants.txt: ASCII text
                

Now, we could equally use a for-loop list in another way - we could say for example:^[19]

for files in `ls -1`
do
	echo "file: `file $files`"
done
                

Remember, from earlier in the course, we saw that ls -l or $(ls -1) executes the ls command and produces some output. What this FOR loop is doing, is listing every file in our current directory with the ls -1. For each one listed, it runs the file command on the file.

The output from the above example might look something like:

bash: file: Desktop/: directory: No such file or directory
bash: file: Maildir/: directory: No such file or directory
bash: file: _viminfo: ASCII text: command not found
bash: file: blah.txt: ASCII text: command not found
bash: file: tmp/: directory: No such file or directory
bash: file: urls: ASCII English text: command not found
bash: file: windows.profile/: directory: No such file or directory
bash: file: winscp.RND: data: command not found
                

As long as you provide the "for" loop with a list, it's happy.

Another example of doing a for loop is as follows:

for count in `seq 20`
do
	echo $count
done
                

This will produce a sequence of 20 numbers from 1 through 20.

Do an info on the 'seq' command to find out what else it can do.

Okay, so provided that you supply for with a list, it can cycle through that list and do a command or sequence of commands once for every item on the list.

There's another type of "for" loop, and that's using the for without the 'in' statement.

Here, the for loop uses the arguments supplied on the command line as the list ($1, $2, $3, etc.). Using the general syntax of the "for" loop as follows:

for var 
do
	...
	...
done
                

Cycle through the arguments on the command line with the script:

#!/bin/bash
for arg
do
	echo $arg
done
exit 0
                

Make the script executable, and then run it:

chmod +x for.sh

./for.sh one two three four
                

When run, the script will cycle through the "for" loop four (4) times, echoing your parameters one by one. Let's make this for loop a bit snazzier.

We're going to set a variable count at the top of our script to the value 1, which will keep track of the number of parameters:

#!/bin/bash
count=1
for arg
do
	echo "Argument $count is $arg"
	$((count=count+1))		
done
exit 0
                

This script will not only count up the number of arguments, but will also print the value of each argument. Save the above script, make it executable and run it. If you're using a shell that does not recognise the line with $(()) in it, then you can use the line:

count=`expr $count + 1`
                

You will notice a couple of things. First-off, although it seems to be incrementing the count it also gives us some errors. Something like:

Argument 1 is one
./for.sh: line 6: 2: command not found
Argument 2 is two
./for.sh: line 6: 3: command not found
Argument 3 is three
./for.sh: line 6: 4: command not found
Argument 4 is four
./for.sh: line 6: 5: command not found
                

The errors stem from line 6, the "$((count=count+1))". This line produces a number, and the command not found is this number (i.e. The shell is looking for the command 2, or 3 or 4, etc.) So, one way of getting around this is to put a noop in front of the line:

#!/bin/bash
count=1
for arg
do
	echo "Argument $count is $arg"
	: $((count=count+1))		
done
exit 0
                

This will execute the increment of the count without giving you any sort of error messages.

Running the script will produce:

[riaan@debian] ~$ ./for.sh one two three four
Argument 1 is one
Argument 2 is two
Argument 3 is three
Argument 4 is four
                

Alternatively you could replace line 6 with:

count=$((count+1))	
                

This might be a little more intuitive anyway. Any which way you do it, you should end up with the same four lines of output.

A "for" loop without an 'in' allows you to cycle through your arguments irrespective of the number of arguments.

The final permutation of the "for" loop, although not available under all shells, is one based on the "for" loop in C.

An example may be:

for ((i=0; i<=10; i=i+1))	#can replace i=i+1 with i++
do
	echo $i
done
                

This will count from 0 to 10.

Note that the syntax is like this:

for ((start value; comparison; count increment))
                

If we wanted to count down from 10 to 0, we would do the following:

for ((i=10; i>=0; i=i-1))	#can replace i=i-1 with i--
do
	echo $i
done
                

This version of the "for" loop is useful as it allows a means of iterating a defined number of times based upon a counter rather than a list.

Clearly we could have achieve the same thing with:

for i in `seq 10`
do 
	echo $i
done
                

	Note
	the seq command would also allow you to count in reverse

In summary, there are three means of using the FOR loop:

You would generally use for loops when you know the exact number of times that you want your loop to execute. If you don't know how many times you are going to execute the loop, you should use a while or an until loop.

| 1:1   | 1:2	| 1:3 | 1:4 | 1:5 | 1:6	|
| 2:1	| 2:2	| 2:3 | 2:4 | 2:5 | 2:6	|
| 3:1	| 3:2	| 3:3 | 3:4 | 3:5 | 3:6	|
| 4:1	| 4:2	| 4:3 | 4:4 | 4:5 | 4:6	|
                    

A while loop has the following syntax:

while <condition is true>
do 
	...
	...
done
                

And the until loop has the following syntax:

until <condition is true> 
do 
	...
	...
done
                

You should notice the [subtle] difference between these two loops.

The while loop executes ONLY WHILE the condition is TRUE(0), whereas the until loop will continue to execute UNTIL the condition BECOMES TRUE(0).

In other words, the UNTIL loop continues with a FALSE (1) condition, and stops as soon as the condition becomes TRUE(0).

Prior to beginning the UNTIL loop, the condition must be FALSE(1) in order to execute the loop at least once.

Prior coming into the while loop however, the condition must be TRUE(0) in order to execute the block within the while statement at least once.

Let's have a look at some examples. Here you could say:

i=5
while test "$i" -le 10
do
	echo $i
done
                

Or we could rewrite the above example as:

i=5
while [ "$i" -le 10 ]
do
	echo $i
done
                

Since the square brackets are just a synonym for the test command.

Another example:

while [ !-d `ls` ]
do 
	echo "file" 
done
                

which says:

"while a particular file is not a directory, echo the word 'file'"

So we could do tests like that where we want to test a particular type of file, and we could do all sorts of conditions.

Remember back to the test command, we could combine the tests with (an -a for AND and -o for OR) some other test condition. So we can combine tests together as many as we want.

while [ somecondition ] -a [ anothercondition ] -o [ yetanothercondition ]
do
	something
done
                

We will also look at the while loop again when we do the read command.

The until command is similar to the while command, but remember that the test is reversed.

For example, we might want to see whether somebody is logged in to our systems. Using the who command, create a script called aretheyloggedin.sh:

user=$1
until `who | grep "$user" > /dev/null`
do 
	echo "User not logged in"

done
                

This runs the who command piping the output to grep, which searches for a particular user.

We're not interested in the output, so we redirect the output to the Linux black hole (/dev/null).

This script will spew out tonnes of lines with:

User not logged in
                

We therefore might want to include a command to sleep for a bit before doing the check or printing the message again. How do we do that?

Simply add the following line:

sleep 10
                

The script becomes:

#!/bin/bash
user=$1
until who |grep "$user"> /dev/null
do 
	echo "User not logged in"
	sleep 10
done
echo "Finally!! $user has entered the OS"
exit 0
                

Until the user logs in, the script will tell you that the user is not logged on. The minute the user logs on, the script will tell you that the user has logged on and the script will then exit.

If we did not want to print anything until the user logged on, we could use the noop in our loop as follows:

#!/bin/bash
user=$1
until who |grep "$user"> /dev/null
do 
	:
	sleep 10
done
echo "Finally, $user logged in"
exit 0
                

And so there's a script that will monitor our system regularly to find out whether a particular user has logged in. As soon as they log on, it will inform us.

When you run the script, it will merely sit there -staring blankly at you. In fact, it is performing that loop repeatedly, but there is no output.

We've looked at the three types of loops that you're going to need when programming in the shell: for loops, while loops and until loops.

These should suffice for most scripts, and unless you're writing particularly complex scripts (in which case you should be writing them in perl!) they should serve your (almost) every need.

The break and continue commands

During execution of the script, we might want to break out of the loop. This time we're going to create a script called html.sh, which is going to produce an html table.

Now an HTML table is built row by row in HTML a table can only built a row at a time. We start by telling the browser that what follows is an HTML table, and every time we start a row we have to enclose the row with a row indicator ( <TR>) and end the row with a row terminator (</TR>) tag.

Each element in the row is enclosed in a table-data tag (<TD>) and terminated in a end-table-data tag (</TD>)

A snippet of how to write a table in HTML (I've set the border of our table to 1):

<TABLE BORDER="1"> <TR><TD>element</TD></TR> <TR><TD>element</TD></TR> </TABLE>

The easiest way to generate a table of 4 rows, and 3 columns is to use a for loop since we know the exact number of times that we want to execute the loop.

Adding the following to html.sh:

#!/bin/bash echo "<TABLE BORDER='1'>" for row in `seq 4` do echo "<TR></TR"> done echo "</TABLE>" exit 0

should create a table with 4 rows, but no columns (table-data).

As usual make the script executable and run it with the following commands:

chmod +x html.sh ./html.sh > /tmp/table.html

Open your favourite browser (Mozilla, Opera, Galleon, Firebird) and point the browser at this new file by entering the URL:

file:///tmp/table.html

You should see a whole lot of nothing happening, because we haven't put any elements in our table.

Let's add some table data, as well as some extra rows.

#!/bin/bash# Start by warning the browser that a table is starting echo "<TABLE BORDER='1'>" # Start the ROWs of the table (4 rows) for row in `seq 4` do # Start the row for this iteration echo "<TR>" # Within each row, we need 3 columns (or table-data) for col in `seq 3` do #If this row 2, then break out of this inner (column) loop, returning to the next ROW above. if [ $row -eq 2 ] then break; fi # If this is NOT row 2, then put the cell in here. echo " <TD>$row,$col</TD>" done # End this ROW echo "</TR>" done#End this table. echo "</TABLE>" exit 0

	Note
	This time, inside each row, we put some data. Previously we placed no data in the rows. Also, notice that when ROW 2 is reached, we "BREAK" out of this inner loop, continuing with the outer loop (i.e. incrementing to the next ROW).

If you hold the shift key down and click the reload button of your web browser, you should see now that you have data in the table. Not really that exciting?!

Let's make this a LOT more fun, I have included the script below. Read through it, work out what it does and then saving it in a script called runfun.sh, run it using the following command line:

./runfun.sh > index.html

Again, point your browser at the resulting file (index.html) and enjoy.

	Note
	For this to work propperly you will need to make sure that the index.html file is created in the directory where you have the gif gif.tar.gz files stored.

#!/bin/bash ANIM=`ls -1 *.gif` NUM=`echo "$ANIM" | wc -l` echo "<TABLE BORDER='1' bgcolor='FFFFFF'>" for file in `seq 2` do echo "<tr>" for row in `seq 3` do file=`echo "$ANIM" | head -1` NUM=$(( NUM - 1 )) ANIM=`echo "$ANIM" | tail -$NUM` echo "<td>" # This is probably the only part you may have difficulty understanding. Here we include # an image in the cell rather than text. For this to work, you will need the couple of GIF # images packaged with this course. echo " <img src=$file alt='Image is: $file'>" echo "</td>" done echo "</tr>" done echo "</TABLE>"

This should produce a table for us with 3 rows and 3 columns.

So what happens if we wanted to skip column two, in other words, we didn't want any data in column 2? Well we could add the following if-then statement:

if [ "$col" -eq 2 ] then break fi

The break command would break out of the inner loop. So we would find that we don't have any data for column 2, but we do have data for column 1 and 3. You can add an argument to the break command such as:

break 2

which would break out of the two inner-most loops.

Thus, break is a way of immediately terminating a loop. A couple of pointers, even if you broke out of the loops, the exit status is still run. All the break statement is doing is exiting out of the inner loop and then out of the outer loop because we did a 'break 2'.

There's nothing wrong with using break as programming practice goes - it's used by C programmers all over the world.

There might also be instances where you have a loop and on a condition you want it to continue. On a condition that we may want to continue the loop without executing the commands that follow the continue statement.

For example:

loop do condition continue ... ... done

Continue tells the loop to skip any commands found on the lines following the continue beginning again at the top of the loop. This is the opposite of what the break command does, which terminates the loop.

A final word on loops. Suppose we wanted to save the output of the loop to a file, we would do this by redirecting the output to a file at the END of the loop as follows:

for ((i=0;i<10;i++)) do echo "Number is now $i" done > forloop.txt

We will see further uses of this when we come to the read command later.

Perhaps we want to take the output of this FOR loop and pipe it into the translate command. We could say:

for ((i=0;i<10;i++)) do echo "Number is now $i" done |tr '[a-z]' '[A-Z]'

We could achieve piping and redirection as per all the previous commands we have done:

for ((i=0;i<10;i++)) do echo "Number is now $i" done |tr '[a-z]' '[A-Z]' >forloop.txt

Note that the pipe or redirect must appear AFTER the 'done' and not after the 'for'.