BASH Frequently Asked Questions

1. How can I read a file line-by-line?

    while read line
    do
        echo "$line"
    done < "$file"          # or   <<< "$var"    to iterate over a variable 

If you want to operate on individual fields within each line, you may supply additional variables to read:

    # Input file has 3 columns separated by white space.
    while read first_name last_name phone; do
      ...
    done < "$file"

If the field delimiters are not whitespace, you can set IFS (input field separator):

    while IFS=: read user pass uid gid gecos home shell; do
      ...
    done < /etc/passwd

Also, please note that you do not necessarily need to know how many fields each line of input contains. If you supply more variables than there are fields, the extra variables will be empty. If you supply fewer, the last variable gets "all the rest" of the fields after the preceding ones are satisfied. For example,

    while read first_name last_name junk; do
      ...
    done <<< 'Bob Smith 123 Main Street Elk Grove Iowa 123-555-6789'
    # Inside the loop, first_name will contain "Bob", and
    # last_name will contain "Smith".  The variable "junk" holds
    # everything else.

The read command modifies each line read, e.g. by default it removes all leading whitespace characters (blanks, tab characters, ... -- basically any leading characters present in IFS). If that is not desired, the IFS variable has to be cleared:

    while IFS= read line
    do
        echo "$line"
    done < "$file"

As a feature, the read command concatenates lines that end with a backslash '\' character to one single line. To disable this feature, KornShell and BASH, as well as the POSIX standard for the Bourne shell, have read -r:

    while IFS= read -r line
    do
        echo "$line"
    done < "$file"

Note that reading a file line by line this way is very slow for large files. Consider using e.g. AWK instead if you get performance problems.

One may also read from a command instead of a regular file:

    some command | while read line; do
       other commands
    done

This method is especially useful for processing the output of find with a block of commands:

    find . -print0 | while read -d $'\0' file; do
        mv "$file" "${file// /_}"
    done

This command reads one filename at a time from the file command and renames the file so that its spaces are replaced by underscores.

Note the usage of -print0 in the find command, which uses NUL bytes as filename delimiters, and -d $'\0' in the read command to instruct it to read all text into the file variable until it finds a NUL byte. By default, find and read delimit their input with newlines; however, since filenames can potentially contain newlines themselves, this default behaviour will split those filenames with newlines up and cause the command block to fail. See FAQ #20 for more details.

Using a pipe to send find's output into a while loop places the loop in a subshell and may therefore cause problems later on if the commands inside the body of the loop attempt to set variables which need to be used outside the loop; in that case, see FAQ 24, or use process substitution like:

    while read line; do
        other commands
    done < <(some command)

Sometimes it's useful to read a file into an array, one array element per line. You can do that with the following example:

    O=$IFS IFS=$'\n' arr=($(< myfile)) IFS=$O

This temporarily changes the Input Field Separator to a newline, so that each line will be considered one field by read. Then it populates the array arr with the fields. Then it sets the IFS back to what it was before.

This same trick works on a stream of data as well as a file:

    O=$IFS IFS=$'\n' arr=($(find . -type f)) IFS=$O

Of course, this will blow up in your face if the filenames contain newlines; see FAQ 20 for hints on dealing with such filenames.

On the other hand, if the file lacks a trailing newline (such as /proc/$$/cmdline on Linux), the line will not be printed by a while read ... loop, as read returns a failure that aborts the while loop, thus failing to print the ultimate line:

    # This does not work:
    echo -en 'line 1\ntruncated line 2' | while read line; do echo $line; done

    # This does not work either:
    echo -en 'line 1\ntruncated line 2' | while read line; do echo "$line"; done; echo "$line"

    # This works:
    echo -en 'line 1\ntruncated line 2' | (while read line; do echo "$line"; done; echo "$line")

For a discussion of why the second example above does not work as expected, see FAQ #24.

2. How can I store the return value of a command in a variable?

Well, that depends on exactly what you mean by that question. Some people want to store the command's output (either stdout, or stdout + stderr); and others want to store the command's exit status (0 to 255, with 0 typically meaning "success").

If you want to capture the output, you use command substitution:

    var=$(command)      # stdout only; stderr remains uncaptured
    var=$(command 2>&1) # both stdout and stderr will be captured

If you want the exit status:

    command
    var=$?

If you want both:

    var1=$(command)
    var2=$?

The assignment to var1 has no effect on command's exit status, which is still in $?.

If you don't actually want the exit status, but simply want to take an action upon success or failure:

    if command
    then
        echo "it succeeded"
    else
        echo "it failed"
    fi

Or (shorter):

    command && echo "it succeeded" || echo "it failed"

What if you want the exit status of a command in a few that are piped to each other? Use the PIPESTATUS array (BASH only). Say you want the exit status of grep in the following:

    grep foo somelogfile | head -5
    result=${PIPESTATUS[0]}

Now, some trickier stuff. Let's say you want only the stderr, but not stdout. Well, then first you have to decide where you do want stdout to go:

    var=$(command 2>&1 >/dev/null)  # Save stderr, discard stdout.
    var=$(command 2>&1 >/dev/tty)   # Save stderr, send stdout to the terminal.
    var=$(command 3>&2 2>&1 1>&3-)  # Save stderr, send stdout to stderr

It's possible, although considerably harder, to let stdout "fall through" to wherever it would've gone if there hadn't been any redirection. This involves "saving" the current value of stdout, so that it can be used inside the command substitution:

    exec 3>&1                    # Save the place that stdout (1) points to.
    var=$(command 2>&1 1>&3)     # Run command.  stderr is captured.
    exec 3>&-                    # Close FD #3.

    # Or this alternative:
    { var=$(command 2>&1 1>&3-) ;} 3>&1 # Capture stderr, let stdout through.

In the last example above, note that 1>&3- duplicates FD 3 and stores a copy in FD 1, and then closes FD 3.

What you cannot do is capture stdout in one variable, and stderr in another, using only FD redirections. You must use a temporary file to achieve that one.

Well, you can use a horrible hack like:

   result=$( { stdout=$(cmd) ; } 2>&1; echo "this line is the separator"; echo "$stdout")
   var_out=${result#*this line is the separator$'\n'}
   var_err=${result%$'\n'this line is the separator*}

Obviously, this is not robust, because either the standard output or the standard error of the command could contain whatever separator string you employ.

3. How can I insert a blank character after each character?

    sed 's/./& /g'

Example:

    $ echo "testing" | sed 's/./& /g'
    t e s t i n g

For more examples of sed 1-liners, see sed 1-liners or the sed FAQ.

4. How can I check whether a directory is empty or not? How do I check for any *.mpg files?

• I just deleted three completely wrong answers from this question. Please, people, make sure that when you add to the FAQ, your answers

  • answer the question that was asked, and

  • actually work

  Thanks. -- GreyCat

Most modern systems have an "ls -A" which explicitly omits "." and ".." from the directory listing:

    if [ -n "$(ls -A somedir)" ]
    then
        echo directory is non-empty
    fi

This can be shortened to:

    if [ "$(ls -A somedir)" ]
    then
        echo directory is non-empty
    fi

Another way, using Bash features, involves setting a special shell option which changes the behavior of globbing. Some people prefer to avoid this approach, because it's so drastically different and could severely alter the behavior of scripts.

Nevertheless, if you're willing to use this approach, it does greatly simplify this particular task:

    shopt -s nullglob
    if [[ -z $(echo *) ]]; then
        echo directory is empty
    fi
    shopt -u nullglob

This can also be combined with Bash's arrays. The major advantage here is that you probably wanted to do something with all the files, so having them loaded into an array is something that will help you with the overall task:

    shopt -s nullglob
    files=(*)
    (( ${#files[*]} )) || echo directory is empty
    shopt -u nullglob 

nullglob also simplifies various other operations:

    shopt -s nullglob
    for i in *.zip; do
        blah blah "$i"  # No need to check $i is a file.
    done
    shopt -u nullglob

Without the nullglob, that would have to be:

    for i in *.zip; do
        [[ -f $i ]] || continue  # If no .zip files, i becomes *.zip
        blah blah "$i"
    done

(You may want to use the latter anyway, if there's a possibility that the glob may match directories in addition to files.)

Finally, you may wish to avoid the direct question altogether. Usually people want to know whether a directory is empty... because they want to do something involving the files therein, etc. Look to the larger question. For example, one of these find-based examples may be an appropriate solution:

   find "$somedir" -type f -exec echo Found unexpected file {} \;
   find "$somedir" -maxdepth 0 -empty -exec echo {} is empty. \;  # GNU/BSD
   find "$somedir" -type d -empty -exec cp /my/configfile {} \;   # GNU/BSD

It's all a matter of addressing the program's actual requirements.

5. How can I use array variables?

BASH and KornShell have one-dimensional arrays indexed by a numerical expression, e.g.

•  # Bash
   host[0]="micky"
   host[1]="minnie"
   host[2]="goofy"
   i=0
   while (( $i < ${#host[@]} ))
   do
       echo "host number $i is ${host[i++]}"
   done

The indexing always begins with 0.

The awkward expression ${#host[@]} returns the number of elements for the array host. Also noteworthy for BASH is the fact that inside the square brackets, i++ works as a C programmer would expect. The square brackets in an array reference force an ArithmeticExpression. (That shortcut does not work in ksh88.)

It's possible to assign multiple values to an array at once, but the syntax differs across shells.

•  # Bash
   array=(one two three four)
  
   # Korn
   set -A array -- one two three four

Bash also lets you initialize an array using a glob:

•  # Bash
   oggs=(*.ogg)

Using array elements en masse is one of the key features. In exactly the same way that "$@" is expanded for positional parameters, "${arr[@]}" is expanded to a list of words, one array element per word. For example,

•  # Korn/Bash
   for x in "${arr[@]}"; do
     echo "next element is '$x'"
   done

This works even if the elements contain whitespace. You always end up with the same number of words as you have array elements.

If one simply wants to dump the full array, "${arr[*]}" will cause the elements to be concatenated together, with the first character of IFS (or a space if IFS isn't set) between them. As it happens, "$*" is expanded the same way for positional parameters.

•  # Bash
   arr=(x y z)
   IFS=/; echo "${arr[*]}"; unset IFS
   # prints x/y/z

BASH and Korn shell arrays are also sparse. Elements may be added and deleted out of sequence.

•  # Bash
   arr=(0 1 2 3)
   arr[42]="what was the question?"
   unset arr[2]
   echo "${arr[*]}"
   # prints 0 1 3 what was the question?

BASH 3.0 added the ability to retrieve the list of index values in an array, rather than just iterating over the elements:

•  # Bash 3.0 or higher
   echo ${!arr[*]}
   # using the previous array, prints 0 1 3 42

Bash's Parameter Expansions may be performed on array elements en masse as well:

•  # Bash
   arr=(abc def ghi jkl)
   echo "${arr[@]#?}"          # prints bc ef hi kl
   echo "${arr[@]/[aeiou]/}"   # prints bc df gh jkl

Parameter Expansion can also be used to extract elements from an array:

•  # Bash
   echo "${arr[@]:1:3}"        # three elements starting at #1 (second element)
   echo "${arr[@]:(-2)}"       # last two elements
   echo "${@:(-1)}"            # last positional parameter
   echo "${@:(-2):1}"          # second-to-last positional parameter

The @ array (the array of positional parameters) can be used just like any regularly named array.

For examples of loading data into arrays, see FAQ #1. For examples of using arrays to hold complex shell commands, see FAQ #50 and FAQ #40.

6. How can I use variable variables (indirect variables, pointers, references) or associative arrays?

6.1. Evaluating indirect/reference variables

BASH allows you to expand a parameter indirectly -- that is, one variable may contain the name of another variable:

•   # Bash
    realvariable=contents
    ref=realvariable
    echo "${!ref}"   # prints the contents of the real variable

KornShell (ksh93) has a completely different, more powerful syntax -- the nameref command (also known as typeset -n):

•   # ksh93
    realvariable=contents
    nameref ref=realvariable
    echo "$ref"      # prints the contents of the real variable

ksh93's nameref allows us to work with references to arrays, as well as regular scalar variables. For example,

•   # ksh93
    myfunc() {
      nameref ref=$1
      echo "array $1 has ${#ref[*]} elements"
    }
    realarray=(...)
    myfunc realarray

We are not aware of any trick that can duplicate that functionality in Bash, POSIX or Bourne shells (short of using eval, which is extremely difficult to do securely).

Unfortunately, for shells other than Bash and ksh93, there is no syntax for evaluating a referenced variable. You would have to use eval, which means you would have to undergo extreme measures to sanitize your data to avoid catastrophe.

6.2. Assigning indirect/reference variables

Assigning a value "through" a reference (or pointer, or indirect variable, or whatever you want to call it -- I'm going to use "ref" from now on) is more widely possible, but the means of doing so are extremely shell-specific.

In ksh93, we can just use nameref again:

•   # ksh93
    nameref ref=realvariable
    ref="contents"
    # realvariable now contains the string "contents"

In Bash, we can use read and Bash's here string syntax:

•   # Bash
    ref=realvariable
    read $ref <<< "contents"
    # realvariable now contains the string "contents"

This works equally well with Bash array variables too:

•   # Bash
    aref=realarray
    read -a $aref <<< "words go into array elements"
    echo "${realarray[1]}"   # prints "go"

Another trick is to use Bash's printf -v (only available in recent versions):

•   # Bash 3.1 or higher
    ref=realvariable
    printf -v $ref "contents"

The printf -v trick is handy if your contents aren't a constant string, but rather, something dynamically generated. You can use all of printf's formatting capabilities.

Yet another trick is Korn shell's typeset or Bash's declare. These are roughly equivalent to each other. Both of them cause a variable to become locally scoped to a function, if used inside a function; but if used outside a function, they can operate on global variables.

•   # Korn shell (all versions):
    typeset $ref="contents"
  
    # Bash:
    declare $ref="contents"

The advantage of using typeset or declare over eval is that the right hand side of the assignment is not parsed by the shell. If you used eval here, you would have to sanitize/escape the entire right hand side first.

If you aren't using Bash or Korn shell, you can still do assignments to referenced variables using here document syntax:

•   # Bourne
    ref=realvariable
    read $ref <<EOF
    contents
    EOF

Remember that, when using a here document, if the sentinel word (EOF in our example) is unquoted, then parameter expansions will be performed inside the body. If the sentinel is quoted, then parameter expansions are not performed. Use whichever is more convenient for your task.

6.3. Associative Arrays

Sometimes it's convenient to have associative arrays, arrays indexed by a string. Awk has associative arrays. Perl calls them "hashes", while Tcl simply calls them "arrays". ksh93 supports this kind of array:

•  # ksh93
   typeset -A homedir             # Declare ksh93 associative array
   homedir[jim]=/home/jim
   homedir[silvia]=/home/silvia
   homedir[alex]=/home/alex
   
   for user in ${!homedir[@]}     # Enumerate all indices (user names)
   do
       echo "Home directory of user $user is ${homedir[$user]}"
   done

BASH (including version 3.x) does not support them, unfortunately. Consider switching to awk, perl, ksh93, tcl, etc. if you need this type of data structure to solve your problem.

Before you think of using eval to mimic this behavior in a shell (probably by creating a set of variable names like homedir_alex), try to think of a simpler approach that you could use instead. If this hack still seems to be the best thing to do, have a look at the following disadvantages:

1. It's hard to read and to maintain.

2. The variable names must match the RegularExpression ^[a-zA-Z_][a-zA-Z_0-9]* -- i.e., a variable name cannot contain arbitrary characters but only letters, digits, and underscores. We cannot have a variable's name contain Unix usernames, for instance -- consider a user named hong-hu. A dash '-' cannot be part of a variable name, so the entire attempt to make a variable named homedir_hong-hu is doomed from the start.

3. Quoting is hard to get right. If content strings (not variable name) can contain whitespace characters and quotes, it's hard to quote it right to preserve it through both shell parsings. And that's just for constants, known at the time you write the program.

4. If the program handles unsanitized user input, it can be VERY dangerous!

7. Is there a function to return the length of a string?

The fastest way, not requiring external programs (but usable only with BASH and KornShell):

${#varname}

or

expr "$varname" : '.*'

(expr prints the number of characters matching the pattern .*, which is the length of the string)

or

expr length "$varname"

(for a BSD/GNU version of expr. Do not use this, because it is not POSIX).

8. How can I recursively search all files for a string?

90% of the time, all you need is one of these:

# Recurse and print matching lines (GNU grep):
grep -r "$search" .

# Recurse and print only the filenames (GNU grep):
grep -r -l "$search" .

You can use find if your grep lacks a -r option:

find . -type f -exec grep -l "$search" {} \;

The {} characters will be replaced with the current file name.

This command is slower than it needs to be, because find will call grep with only one file name, resulting in many grep invocations (one per file). Since grep accepts multiple file names on the command line, find can be instructed to call it with several file names at once:

find . -type f -exec grep -l "$search" {} +

The trailing '+' character instructs find to call grep with as many file names as possible, saving processes and resulting in faster execution. This example works for POSIX find, e.g. with Solaris, as well as very recent GNU find.

Traditional Unix has a helper program called xargs for the same purpose:

find . -type f | xargs grep -l "$seach"

However, if your filenames contain spaces or other metacharacters, you'll need to use the BSD/GNU -print0 option:

find . -type f -print0 | xargs -0 grep -l "$search"

The -print0 / -0 options ensure that any file name can be processed, even one containing blanks, TAB characters, or newlines.

9. My command line produces no output: tail -f logfile | grep 'foo bar'

Most standard Unix commands buffer their output if used non-interactively. This means, that they don't write each character (or even each line) as they are ready, but collect a larger number (e.g. 4 kilobytes) before printing it. In the case above, the tail command buffers its output, and therefore grep only gets its input in e.g. 4K blocks.

Unfortunately there's no easy solution to this, because the behaviour of the standard programs would need to be changed. *See bottom of section before taking 'no easy solution' to heart* Some programs provide special command line options for this purpose, e.g.

The expect package (http://expect.nist.gov/) has an unbuffer example program, which can help here. It disables buffering for the output of a program. Example usage:

    unbuffer tail -f logfile | grep 'foo bar'

There is another option when you have more control over the creation of the log file. If you would like to grep the real-time log of a text interface program which does buffered session logging by default (or you were using script to make a session log), then try this instead:

   $ program | tee -a program.log

   In another window:
   $ tail -f program.log | grep whatever

Apparently this works because tee produces unbuffered output. This has only been tested on GNU tee, YMMV.

If you simply wanted to highlight the search term, rather than filter out non-matching lines, you can use the 'less' program instead of Bash:

   $ less program.log

Inside less, start a search with the '/' command (similar to searching in vi). This should highlight any instances of the search term. Now put less into "follow" mode, which by default is bound to shift+f. You should get an unfiltered tail of the specified file, with the search term highlighted.

10. How can I recreate a directory structure, without the files?

With the cpio program:

•  cd "$srcdir"
   find . -type d -print | cpio -pdumv "$dstdir"

or with GNU tar, and more verbose syntax:

•  cd "$srcdir"
   find . -type d -print | tar c --files-from - --no-recursion |
     tar x --directory "$dstdir"

This creates a list of directory names with find, non-recursively adds just the directories to an archive, and pipes it to a second tar instance to extract it at the target location.

11. How can I print the n'th line of a file?

The dirty (but not quick) way would be:

    sed -n ${n}p "$file"

but this reads the whole input file, even if you only wanted the third line.

This one avoids that problem:

    sed -n "$n{p;q;}" "$file"

At line $n the command "p" is run, printing it, with a "q" afterwards: quit the program.

Another way, more obvious to some, is to grab the last line from a listing of the first n lines:

   head -n $n $file | tail -n 1 

Another approach, using AWK:

   awk "NR==$n{print;exit}" file

If you want more than one line, it's pretty easy to adapt any of the previous methods:

   x=3 y=4;
   sed -n "$x,${y}p;${y}q;" "$file"                # Print lines $x to $y; quit after $y.
   head -n $y "$file" | tail -n $(($y - $x + 1))   # Same
   awk "NR>=$x{print} NR==$y{exit}" "$file"        # Same

12. A program (e.g. a file manager) lets me define an external command that an argument will be appended to - but i need that argument somewhere in the middle...

    sh -c 'echo "$1"' -- hello

13. How can I concatenate two variables? How do I append a string to a variable?

There is no (explicit) concatenation operator for strings (either literal or variable dereferences) in the shell, *BUT* one can concat the values of variables by sequentially naming eaching variable to be joined together. The variables whose values are to be concatenated are written one after the other:

    var=$var1$var2

If the right-hand side contains whitespace characters, it needs to be quoted:

    var="$var1 - $var2"

If you're appending a string that doesn't "look like" part of a variable name, you just smoosh it all together:

    var=$var1/.-

Otherwise, braces or quotes may be used to disambiguate the right-hand side:

    var=${var1}xyzzy
    # Without braces, var1xyzzy would be interpreted as a variable name

    var="$var1"xyzzy
    # Alternative syntax

CommandSubstitution can be used as well. The following line creates a log file name logname containing the current date, resulting in names like e.g. log.2004-07-26:

    logname="log.$(date +%Y-%m-%d)"

There's no difference when the variable name is reused, either: A variable's value (the string it holds) may be reassigned at will:

    string="$string more data here"

Bash 3.1 has a new += operator that you may see from time to time:

    string+=" more data here"     # EXTREMELY non-portable!

It's generally best to use the portable syntax.

14. How can I redirect the output of multiple commands at once?

Redirecting the standard output of a single command is as easy as

    date > file

To redirect standard error:

    date 2> file

To redirect both:

    date > file 2>&1

In a loop or other larger code structure:

    for i in $list; do
        echo "Now processing $i"
        # more stuff here...
    done > file 2>&1

However, this can become tedious if the output of many programs should be redirected. If all output of a script should go into a file (e.g. a log file), the exec command can be used:

    # redirect both standard output and standard error to "log.txt"
    exec > log.txt 2>&1
    # all output including stderr now goes into "log.txt"

Otherwise command grouping helps:

    {
        date
        # some other command
        echo done
    } > messages.log 2>&1

In this example, the output of all commands within the curly braces is redirected to the file messages.log.

15. How can I run a command on all files with the extension .gz?

Often a command already accepts several files as arguments, e.g.

    zcat *.gz

(One some systems, you would use gzcat instead of zcat. If neither is available, or if you don't care to play guessing games, just use gzip -dc instead.) If an explicit loop is desired, or if your command does not accept multiple filename arguments in one invocation, the for loop can be used:

    # Bourne
    for file in *.gz
    do
        echo "$file"
        # do something with "$file"
    done

To do it recursively, you should use a loop, plus the find command:

    # Bash
    while read file; do
        echo "$file"
        # do something with "$file"
    done < <(find . -name '*.gz' -print)

For more hints in this direction, see FAQ #20. To see why the find command comes after the loop instead of before it, see FAQ #24.

16. How can I use a logical AND in a shell pattern (glob)?

"Globs" are simple patterns that can be used to match filenames or strings. They're generally not very powerful. If you need more power, you can use extended globs. In BASH, you'll need the extglob option to be set. It can be checked with:

$ shopt extglob

and set with:

$ shopt -s extglob

To warm up, we'll move all files starting with foo AND not ending with .d to directory foo_thursday.d:

$ mv foo!(*.d) foo_thursday.d

A more complex example -- delete all files containing Pink_Floyd AND not containing The_Final_Cut:

$ rm !(!(*Pink_Floyd*)|*The_Final_Cut*)

By the way: these kind of patterns can be used with the KornShell, too. They don't have to be enabled there, but are the default patterns.

For a more thorough explanation of extended globs, see glob.

17. How can I group expressions, e.g. (A AND B) OR C?

The TestCommand [ uses parentheses () for expression grouping. Given that "AND" is "-a", and "OR" is "-o", the following expression

    (0<n AND n<=10) OR n=-1

can be written as follows:

    if [ \( $n -gt 0 -a $n -le 10 \) -o $n -eq -1 ]
    then
        echo "0 < $n <= 10, or $n=-1"
    else
        echo "invalid number: $n"
    fi

Note that the parentheses have to be quoted: \(, '(' or "(".

BASH and KornShell have different, more powerful comparison commands with slightly different (easier) quoting:

• ArithmeticExpression for arithmetic expressions, and

• NewTestCommand for string (and file) expressions.

Examples:

    if (( (n>0 && n<10) || n == -1 ))
    then echo "0 < $n < 10, or n==-1"
    fi

or

    if [[ ( -f $localconfig && -f $globalconfig ) || -n $noconfig ]]
    then echo "configuration ok (or not used)"
    fi

Note that the distinction between numeric and string comparisons is strict. Consider the following example:

    n=3
    if [[ n>0 && n<10 ]]
    then echo "$n is between 0 and 10"
    else echo "ERROR: invalid number: $n"
    fi

The output will be "ERROR: ....", because in a string comparision "3" is bigger than "10", because "3" already comes after "1", and the next character "0" is not considered. Changing the square brackets to double parentheses (( makes the example work as expected.

18. How can I use numbers with leading zeros in a loop, e.g. 01, 02?

As always, there are different ways to solve the problem, each with its own advantages and disadvantages.

If there are not many numbers, BraceExpansion can be used:

    # Bash
    for i in 0{1,2,3,4,5,6,7,8,9} 10
    do
        echo $i
    done

Output:

   00
   01
   02
   03
   [...]

This gets tedious for large sequences, but there are other ways, too. If you have the printf command (which is a Bash builtin, and is also POSIX standard), it can be used to format a number:

    # Bash
    for ((i=1; i<=10; i++))
    do
        printf "%02d " "$i"
    done

In Bash 3, you can use ranges inside brace expansion. Also, since printf will implicitly loop if given more arguments than format specifiers, you can simplify this enormously:

   # Bash 3
   printf "%03d\n" {1..300}

If you don't know in advance what the starting and ending values are:

   # Bash 3
   # START and END are variables containing integers
   eval printf '"%03d\n"' {$START..$END}

The eval is needed here because you cannot have variables in a brace expansion -- only constants. The extra quotes are required by the eval so that our \n isn't changed to an n. Given how messy that eval solution is, please give serious thought to using the for loop instead.

The KornShell has the typeset command to specify the number of leading zeros:

    # Korn
    $ typeset -Z3 i=4
    $ echo $i
    004

If the command seq(1) is available (it's part of GNU sh-utils/coreutils), you can use it as follows:

    seq -w 1 10

or, for arbitrary numbers of leading zeros (here: 3):

    seq -f "%03g" 1 10

Combining printf with seq(1), you can do things like this:

   # POSIX
   printf "%03d\n" $(seq 300)

(That may be helpful if your version of seq(1) lacks printf-style format specifiers. Since it's a nonstandard external tool, it's good to keep your options open.)

Be warned however that seq might be considered bad style; it's even mentioned in Don't Ever Do These.

Finally, the following example works with any BourneShell derived shell (which also has expr and sed) to zero-pad each line to three bytes:

   # Bourne
   i=0
   while test $i -le 10
   do
       echo "00$i"
       i=`expr $i + 1`
   done |
       sed 's/.*\(...\)$/\1/g'

In this example, the number of '.' inside the parentheses in the sed command determines how many total bytes from the echo command (at the end of each line) will be kept and printed.

But if you're going to rely on an external Unix command, you might as well just do the whole thing in awk in the first place:

   # Bourne
   # COUNT variable contains an integer
   awk "BEGIN {for (i=1;i<$COUNT;i++) {printf(\"%03d\n\",i)} }"

Now, since the number one reason this question is asked is for downloading images in bulk, you can use the examples above with xargs(1) and wget(1) to fetch files:

   almost any example above | xargs -i% wget $LOCATION/%

The xargs -i% will read a line of input at a time, and replace the % at the end of the command with the input.

Or, a simpler example using a for loop:

   # Bash
   for i in {1..100}; do
      wget "$prefix$(printf %03d $i).jpg"
      # other commands
   done

19. How can I split a file into line ranges, e.g. lines 1-10, 11-20, 21-30?

Some Unix systems provide the split utility for this purpose:

    split --lines 10 --numeric-suffixes input.txt output-

For more flexibility you can use sed. The sed command can print e.g. the line number range 1-10:

    sed -n -e '1,10p' -e '10q'

This stops sed from printing each line (-n). Instead it only processes the lines in the range 1-10 ("1,10"), and prints them ("p"). The command will quit after reading line 10 ("10q").

We can now use this to print an arbitrary range of a file (specified by line number):

# POSIX shell
file=/etc/passwd
range=10
cur=1
last=$(wc -l < "$file") # count number of lines
chunk=1
while [ $cur -lt $last ]
do
    endofchunk=$(($cur + $range - 1))
    sed -n -e "$cur,${endofchunk}p" -e "${endofchunk}q" "$file" > chunk.$(printf %04d $chunk)
    chunk=$(($chunk + 1))
    cur=$(($cur + $range))
done

The previous example uses POSIX arithmetic, which older Bourne shells do not have. In that case the following example should be used instead:

# legacy Bourne shell; assume no printf either
file=/etc/passwd
range=10
cur=1
last=`wc -l < "$file"` # count number of lines
chunk=1
while test $cur -lt $last
do
    endofchunk=`expr $cur + $range - 1`
    sed -n -e "$cur,${endofchunk}p" -e "${endofchunk}q" "$file" > chunk.$chunk
    chunk=`expr $chunk + 1`
    cur=`expr $cur + $range`
done

Awk can also be used to produce a more or less equivalent result:

 awk -v range=10 '{print > FILENAME "." (int((NR -1)/ range)+1)}' file

20. How can I find and deal with file names containing newlines, spaces or both?

The preferred method is still to use find(1):

    find ... -exec command {} \;

or, if you need to handle filenames en masse, with GNU and recent BSD tools:

    find ... -print0 | xargs -0 command

or with POSIX find:

    find ... -exec command {} +

Use that unless you really can't.

Another way to deal with files with spaces in their names is to use the shell's filename expansion (globbing). This has the disadvantage of not working recursively (except with zsh's extensions), but if you just need to process all the files in a single directory, it works fantastically well.

This example changes all the *.mp3 files in the current directory to use underscores in their names instead of spaces. It uses Parameter Expansions that will not work in the original BourneShell or POSIX shell, but should be good in KornShell and BASH.

for file in *.mp3; do
    mv "$file" "${file// /_}"
done

Remember, you need to quote all your Parameter Expansions using double quotes. If you don't, the expansion will undergo WordSplitting (see also BashGuide/TheBasics/ArgumentSplitting and BashPitfalls).

You could do the same thing for all files with spaces in their names (regardless of extension) by using

for file in *\ *; do

instead of *.mp3.

Another way to handle filenames recursively involves using the -print0 option of find (a GNU/BSD extension), together with bash's -d option for read:

# Bash
unset a i
while read -d $'\0' file; do
  a[i++]="$file"        # or however you want to process each file
done < <(find /tmp -type f -print0)

The preceding example reads all the files under /tmp (recursively) into an array, even if they have newlines or other whitespace in their names, by forcing read to use the NUL byte (\0) as its line delimiter. Since NUL is not a valid byte in Unix filenames, this is the safest approach besides using find -exec.

21. How can I replace a string with another string in all files?

sed is a good command to replace strings, e.g.

    sed 's/olddomain\.com/newdomain.com/g' input > output

To replace a string in all files of the current directory:

    for i in *; do
        sed 's/old/new/g' "$i" > atempfile && mv atempfile "$i"
    done

GNU sed 4.x has a special -i flag which makes the loop and temp file unnecessary:

      sed -i 's/old/new/g' *

On some (but not all) BSD systems, sed has a -i flag as well, but it takes a mandatory argument. The above example then becomes

      sed -i '' 's/old/new/g' *

which in turn does not work with GNU sed. Effectively, whenever portability matters, sed -i should be avoided.

Those of you who have perl 5 can accomplish the same thing using this code:

    perl -pi -e 's/old/new/g' *

Recursively (requires GNU or BSD find):

    find . -type f -print0 | xargs -0 perl -pi -e 's/old/new/g'

If you want to delete lines instead of making substitutions:

    perl -ni -e 'print unless /foo/' *
    # Deletes any line containing the perl regex foo

To replace for example all "unsigned" with "unsigned long", if it is not "unsigned int" or "unsigned long" ...: