Tuesday Tiny Techie Tip

Command-line looping

One of the coolest things about the UNIX shells is that they are full-fledged programming languages (unlike the popular bitty-box user interfaces), so you can do stuff at the command line using the shell that would have to be hard coded in every program on a PC.

For example, in DOS, you can change the extension on a bunch of files at once with one command:

P:\> REN *.FOO *.BAR

In UNIX, the rename command (mv(1)) doesn't know about parallel renames like this, so if you try to do the similar thing you'll probably get an error:

% mv *.foo *.bar
usage: mv [-if] f1 f2 or mv [-if] f1 ... fn d1 (`fn' is a file or directory)

You get this error because unlike in DOS where the wildcards are passed on unchanged to the program which has to know how to expand them, the shell expands the wildcards before it calls mv. So mv gets called as "mv this.foo that.foo other.foo old.bar new.bar junk.bar" which just doesn't make sense. And, even worse, if there are only two files with a .foo extension, and no .bar files, then mv takes you at your word that "mv one.foo two.foo" is really what you wanted to do. ("one.foo" gets renamed to "two.foo" which get's destroyed! oops.)

Brief aside: If the shell is doing something which you find incomprehensible, you can see what command it's really trying to call by setting the verbosity level so that the command is printed after the shell is finished messing with it, but before it is called. To do this in csh give the command "set echo", in sh do "set -x".

Since the shell is a programming language, you can do the bulk rename by writing a loop at the shell prompt:

% ls
althea.foo      fire.foo        kcj.foo         playin.foo
% foreach file (*.foo)
? echo $file
? mv $file $file:r.bar
? end
althea.foo
fire.foo
kcj.foo
playin.foo
% ls
althea.bar      fire.bar        kcj.bar         playin.bar

So what this does, is take the list provided inside the parentheses on the foreach line, and run through the following commands up to end once for each value in the list, setting the variable file to that value.

I often start these loops off with an "echo $file" so I can see as each loop is completed. The mv command above uses the csh(1) variable modifier to return the filename without its extension (as described in the TTTT from December 17th 1996) then adds the new extension.

If you use a Bourne shell-derived shell (like sh(1), bash(1), or ksh(1)), the syntax is slightly different:

% sh
$ ls
althea.bar      fire.bar        kcj.bar         playin.bar
% for file in *.bar ; do
> echo $file
> mv $file `echo $file | sed 's/\.bar$/.foo/'`
> done
althea.bar
fire.bar
kcj.bar
playin.bar
% ls
althea.foo      fire.foo        kcj.foo         playin.foo

Since the Bourne shells don't support the variable modifiers like csh(1), we have to get a little more creative with our rename operation. Here we're using command substitution as described in last week's tip to take the existing name, and transform it into the new name using sed(1).

Just to clarify, here's the general syntax of for-style loops in each shell:

`csh`-like shells:

foreach var ( list of values for var )
commands using $var to give current value
end

`sh`-like shells:

for var in list of values for var ; do
commands using $var to give current value
done

Here's another example of using a loop at the shell prompt. This time I'll use tcsh(1) so you can see what the default prompts look like in that shell:

% foreach user ( `who | awk '{print $1}' | sort -u` )
foreach? echo -n "Here is how many processes $user has running:"
foreach? ps aux | grep ^$user | wc -l
foreach? end
Here is how many processes jeffy has running:      54
Here is how many processes vobadm has running:      33

That one is kind of complicated so if it gave me information that I wanted to get more than once or twice a year I'd probably want to write a permanent script to do it so I wouldn't have to re-type it (and mis-type it) at the command line every time. This is why so many people start off writing scripts in the same shell they use interactively, despite the relative shortcomings of csh for script writing.

So if it's longer than you want to type and you want to make a script use the Bourne shell form as detailed above and you'll be much happier when it comes time to write scripts longer than three lines.

Since we've jumped from UNIX usage to shell programming, this tip may be rather baffling to some of you. If you found any of it baffling, send me some email (click on my name below) so I can either go over the baffling bits in a future tip, or tailor future tips to be less baffling.

Tuesday Tiny Techie Tip -- 21 January 1997
Forward to (01/28/97)
Back to (01/14/97)
Written by Jeff Youngstrom

Up to the TTTT index

Tuesday Tiny Techie Tip

Command-line looping

csh-like shells:

sh-like shells:

`csh`-like shells:

`sh`-like shells: