July
28, 2003 -
Eliminating
Duplicate Lines - The uniq Command
|
Consider
this week's contents of unixfile, our
frequently-used sample data file:
|
unix commands
shell script
command prompt
unix commands
unix system administration
shell script
unix commands
|
Since there
are only a few number of lines in unixfile,
duplicate lines can be quickly identified
with a visual inspection and then removed
if desired. This manual process is
not a viable option for files containing
hundreds or thousands of lines. The
tool for a job of this magnitude is the
uniq (unique) command:
|
uniq
[option] [input-file] [output-file]
|
uniq can be used
to display, count, or delete adjacent
duplicate lines from a file or standard
input (stdin). If duplicate lines in a
file are not adjacent to one another, uniq
will not treat them as duplicates:
|
$ uniq unixfile
unix commands
shell script
command prompt
unix commands
unix system administration
shell script
unix commands
|
|
For this
reason, uniq is often combined with the sort
command to group duplicate lines prior to
performing an action on them:
|
$ sort unixfile | uniq
command prompt
shell script
unix commands
unix system administration
|
To compress this
combination of commands, the -u option can
be used with sort to produce the same
results:
|
$ sort -u unixfile
command prompt
shell script
unix commands
unix system administration
|
Two useful
options for the uniq command are -d and
-c. I will leave it to you to discover
what these options will do to your
data. Make sure to sort your data
prior to using them...
|
|
|
Learn
more...
If you are new to the UNIX or Linux
operating system and would like to learn
more, you may want to consider
registering for LiveFire Labs' UNIX
and Linux Operating System Fundamentals
online training course.
If you already have a solid grasp of the
fundamentals but would like to learn more
about the Korn shell and basic and
advanced shell scripting, taking our Korn
Shell Scripting course will be
beneficial to you.
Our innovative hands-on training model
allows you to learn
UNIX by completing hands-on
exercises on real servers in our Internet
Lab.
More
Tips...
· Popular
UNIX Tips from the Past
|
|
