Home CPSC 225

Saving Commands in Shell Scripts

Overview

A shell script is a set of Unix commands which are placed in a file and are executed from top to bottom, as if you typed them in manually.

Shell scripts allow us to write Unix "programs" which can automate tasks done in the shell. Thus far we have used the shell to execute single line commands but, as we will see, the shell allows for many programming constructs as well.


Creating a Shell Script

A shell script is just a text file containing commands. Sometimes scripts are given the ".sh" extension, and sometimes they are given no extension at all. I will use the .sh extension for clarity. We can create a simple script with Vim:

finlaysoni@myvm:~$ vim hello.sh 

Then type in the following:


#!/bin/bash

echo "Hello World!"

The first line is called a shebang and tells the shell which interpreter should run the script. While not strictly necessary, it is a good idea to put in place. Here we specify that the script should be executed by bash.

Note: A shebang can be used for programs as well. For instance, if you place "#!/usr/bin/python3" at the top of a Python program, then you can run it by name, without needing the python3 command.

Any line in a shell script which begins with a pound sign (including the shebang line), is not executed by the shell. This allows us to put comments in scripts.

The second line is a command to execute, which just runs the echo command to print a "Hello World!" message to he the screen.


Running a Shell Script

To run the shell script, we can either pass it as an argument to bash:

finlaysoni@myvm:~$ bash hello.sh 
Hello World!

Or, as is more common, we can execute it directly. To do this, we must first give the script executable permissions:

finlaysoni@myvm:~$ chmod u+x hello.sh
finlaysoni@myvm:~$ ls -l hello.sh    
-rwxrw-r-- 1 finlaysoni finlaysoni 22 Jul 15 10:38 hello.sh

We now can execute it directly:

finlaysoni@myvm:~$ ./hello.sh
Hello World!

The ./ portion specifies that we should run the file called hello.sh which is in the current directory. Without this, the shell would search our PATH environment variable.

If we want to be able to run the script from any working directory, we need to put it in a location which is in our PATH. I create a directory for my scripts at "~/bin" and make sure that directory is in my PATH. You can refresh your memory of the PATH environment variable on this page.

If we do this, then we can simply run our script as:

finlaysoni@myvm:~$ hello.sh
Hello World!

Variables

We have seen environment variables which are variables holding some string of text which are available to any command running in your shell. Scripts can also create their own variables for storing things.

The following example uses a variable called name:


#!/bin/bash

name="Ian"

echo Hello $name
echo Goodbye $name

When we run this script the output is:

finlaysoni@myvm:~$ hello.sh 
Hello Ian
Goodbye Ian

When a command contains a variable reference, the shell will substitute it before executing the command. So the shell replaces "Hello $name" with "Hello Ian" and passes that string to the echo command.

Notice that when a variable is created, there is no $ sign in its name, but referencing a variable does begin with the $ sign.

Also note that, unlike most programming languages, we cannot put spaces around the = sign in a variable assignment. This line:


name = "Ian"

produces this error:

finlaysoni@myvm:~$ hello.sh
hello.sh: line 3: name: command not found

Shell scripts are somewhat more syntactically picky than most other languages.

Sometimes we may want a variable to have text immediately after it. Suppose we are writing a script which backs up a file, and the original filename is stored in a variable:


#!/bin/bash

filename="data.txt"

echo Backing up $filename
cp $filename $filenamebackup

This script attempts to copy the file name it is given to the same file name with "backup" appended to it. When we run this, however, we will get this error:

finlaysoni@myvm:~$ backup.sh 
Backing up data.txt
cp: missing destination file operand after 'data.txt'
Try 'cp --help' for more information.

The problem is that the shell sees the variable name as "filenamebackup" which is not defined. Rather than give an error if an undefined variable is used, the shell simply uses an empty string for its value. This means that it passes only the argument "data.txt" to the cp command, hence the error about a missing argument.

To fix this, we can use another form of variable reference, demonstrated below:


#!/bin/bash

filename="data.txt"

echo Backing up $filename
cp $filename ${filename}backup

By wrapping the name of the variable in curly braces, it's clear to the shell that the variable name is "filename" which is expanded correctly.


On Quotes

The examples above use double-quotes for variable assignment. This is not really necessary in these cases, and the scripts will run the same without quotes.

However, if the values being assigned contain spaces, we cannot forgo the quotes:


#!/bin/bash

name=Ian Finlayson

echo Hello $name
echo Goodbye $name
finlaysoni@myvm:~$ hello.sh
hello.sh: line 3: Finlayson: command not found
Hello
Goodbye

Here, we need quotes around the name:


#!/bin/bash

name="Ian Finlayson"

echo Hello $name
echo Goodbye $name
finlaysoni@myvm:~$ hello.sh 
Hello Ian Finlayson
Goodbye Ian Finlayson

I generally use the quotes even when not necessary.

We could also use single quotes in the example above:


name='Ian Finlayson'

In this case, the behavior is exactly the same. However, there is an important difference between single and double quotes.

Single quoted strings are not expanded at all by the shell prior to use. This means they do not have variables replaced with their values.

Suppose we are writing a line in a script to tell the user to set their EDITOR environment variable. Here, we may want to output the actual text '$EDITOR' and not the value of the variable.

The following commands demonstrate this difference:

finlaysoni@myvm:~$ echo 'Please set your $EDITOR variable'
Please set your $EDITOR variable
finlaysoni@myvm:~$ echo "Please set your $EDITOR variable"
Please set your vim variable

See that the single-quoted version did not get its included variable expanded, but the double-quoted version did!

This is also sometimes needed on the shell if you want to suppress a wild card expansion. For instance, if you want to pass a program the '*' character, you would need to wrap it in single quotes:

finlaysoni@myvm:~$ echo *
backup backup.sh bin config downloads hello.sh projects
finlaysoni@myvm:~$ echo '*'
*

Here, the first command had the * expanded by the shell so that echo is passed a list of all the files. In the second command the single quotes cause an actual * character to be passed directly to echo, without expansion.


Saving the Results of Commands

There is another kind of quote which you will sometimes see in shell scripts which is the back-tick quotes ``. These behave quite differently than the others. They run whatever is between them as a command and evaluate to the output of that command. This allows us to capture the result of a command and save it in a variable or use in some other command:


#!/bin/bash

now=`date`
echo The current time is ${now}.

When run this will produce:

finlaysoni@myvm:~$ ./now.sh
The current time is Wed Jul 15 11:50:54 EDT 2018.

However, this method of capturing the result of a command is deprecated (though still common). Instead, its better to wrap the command as $(date):


#!/bin/bash

now=$(date)
echo The current time is ${now}.
finlaysoni@myvm:~$ ./now.sh
The current time is Wed Jul 15 11:54:04 EDT 2018.

These are preferred because they are more easily nested.


Example: A Backup Script

Suppose we want to write a script which will backup our projects directory into a compressed tar file which has the current date as a part of the name.

The first thing we could start with is getting the current date in a nice format. The output of date contains spaces which are not nice to put in file names.

Luckily date is a very flexible program and supports passing a "format string" which describes how the date should be formatted. The man page has all the details, but we can get a simple "year-month-day" output like:

finlaysoni@myvm:bin$ date +%Y-%m-%d
2018-07-15

We can go ahead and make a variable for the file name which contains this formatted date like so:


filename="backup-$(date +%Y-%m-%d).tar.gz"

Here we are making the filename variable equal to the text "backup" concatenated with the result of running the above date command, then concatenating the extension ".tar.gz"

Next we can make a variable which refers to the directories and/or files to backup:


files="projects"

This way we can add to the list of directories to back up simply by changing that line.

The whole script might look like this:


#!/bin/bash

# the files to backup
files="projects"

# the filename has the date in it
filename="backup-$(date +%Y-%m-%d).tar.gz"

# the actual work
echo "Backing up..."
tar -czvf $filename $files
echo "All done backing up!"

Note that this script has comments in it which begin with the pound character.

We can run it as:

finlaysoni@myvm:~$ backup.sh
Backing up...
projects/
projects/input.py
projects/output.py
projects/main.py
All done backing up!

Notice that the lines beginning with "projects/" get printed by tar itself, because we passed it the "-v", verbose flag.


Taking Arguments

Instead of putting the list of files directly in the shell script above, we may want to pass them to our script as arguments.

When we run a script, the shell populates some special variables that allow us to see our arguments:

VariableMeaning
$#The number of arguments passed to the script.
$0The name of the script itself as it was written on the command line. This is rarely useful.
$1, $2, ...The first, second, etc. argument to the script.
$*All of the arguments to the script written together.

The following script shows how these are populated:


#!/bin/bash

echo '$#' is $#
echo '$0' is $0
echo '$1' is $1
echo '$2' is $2
echo '$*' is $*

Below shows an example run of this script:

finlaysoni@myvm:~$ args.sh have some arguments
$# is 3
$0 is args.sh
$1 is have
$2 is some
$* is have some arguments

If we do not supply arguments, they are simply blank:

finlaysoni@myvm:~$ args.sh
$# is 0
$0 is args.sh
$1 is
$2 is
$* is

We can now make our backup script take the files as parameters, and use $* to reference them:


#!/bin/bash

filename="backup-$(date +%Y-%m-%d).tar.gz"

echo "Backing up..."
tar -czvf $filename $*
echo "All done backing up!"

We can now pass any files we like:

finlaysoni@myvm:~$ backup.sh data.txt projects file.txt
Backing up...
data.txt
projects/
projects/input.py
projects/output.py
projects/main.py
file.txt
All done backing up!

If we pass nothing, tar will complain. We'll see how to detect errors in the next lesson.


User Input

In addition to having user interaction via passed arguments, we can also have scripts get user input directly with the read command.

read gets user input and stores it in a variable. It takes an optional prompt which is passed after a "-p" flag.

The following script demonstrates this:


#!/bin/bash

read -p "Enter your name: " name
echo Hello $name!

Running this, it will ask for our name:

finlaysoni@myvm:~$ input.sh 
Enter your name: Ian Finlayson
Hello Ian Finlayson!

read also accepts a "-s" silent option in which it will not print back our input:


#!/bin/bash

read -p "Enter your user name: " name
read -s -p "Enter your password: " password

This script uses the -s flag to avoid your password showing up in the terminal:

finlaysoni@myvm:~$ input.sh
Enter your user name: finlaysoni
Enter your password:

Doing Math

To perform mathematical calculations, in the shell, we can place expressions between $(( and )) delimiters. That will evaluate to the value of the mathematical expression:

finlaysoni@myvm:~$ echo $((3 + 4))
7
finlaysoni@myvm:~$ echo $((3 + 4 * 5))
23
finlaysoni@myvm:~$ echo $((10 / 5))
2

This can allow us to do things like perform arithmetic in shell scripts. For instance, the following backup script uses an incrementing number to version its backups instead of the current date:


#!/bin/bash

# read the current backup number from a hidden file
current=$(cat ~/.backup-number)

# increment the current backup number
next=$(($current + 1))

# overwrite the file with the new backup number
echo $next > ~/.backup-number

# use current as a version
filename="backup-${current}.tar.gz"

echo "Backing up..."
tar -czvf $filename $*
echo "All done backing up file $filename."

This script uses a configuration file called "~/.backup-number" which stores the number to use in the backup. Ideally the script should create this file if it does not exist, but we'll see how to that next time. Currently, it assumes the file is there and reads its contents into a variable called "current". It then calculates "current + 1" into the variable "next". Note that the variable $current is expanded inside of the $(()) expression. It then overwrites the configuration file with the new value by redirecting an echo into the file.

Our script now automatically increments the backup number:

finlaysoni@myvm:~$ echo 0 > ~/.backup-number
finlaysoni@myvm:~$ backup.sh projects
Backing up...
projects/
projects/input.py
projects/output.py
projects/main.py
All done backing up file backup-1.tar.gz.
finlaysoni@myvm:~$ backup.sh projects
Backing up...
projects/
projects/input.py
projects/output.py
projects/main.py
All done backing up file backup-2.tar.gz.
finlaysoni@myvm:~$ backup.sh projects
Backing up...
projects/
projects/input.py
projects/output.py
projects/main.py
All done backing up file backup-3.tar.gz.

Note that shell scripts do not have any concept of variable types. All variables are just text. If the variable text happens to contain a number, it will be used in calculations contained in $(()), but there is nothing like type checking or declaration. Non-numerical variables just have the value 0 when used in expressions.


Conclusion

Shell scripts provide a powerful means of saving commands in a file where they can easily be run all at once. Writing shell scripts can be tricky, but once written, they can be run every time we need them.

In the next lesson, we will see the shell actually supports programming language features such as functions, loops and conditions.

Copyright © 2018 Ian Finlayson | Licensed under a Creative Commons Attribution 4.0 International License.