LiveFire Labs: Online UNIX Training with Hands-on Internet Lab


"Taking a LiveFire Labs' course is an excellent way to learn Linux/Unix. The lessons are well thought out, the material is explained thoroughly, and you get to perform exercises on a real Linux/Unix box. It was money well spent."

Ray S.
Pembrook Pines, Florida

LiveFire Labs' UNIX and Linux Operating System Fundamentals course was very enjoyable. Although I regularly used UNIX systems for 16 years, I haven't done so since 2000. This course was a great refresher. The exercises were fun and helped me gain a real feel for working with UNIX/Linux OS. Thanks very much!"

Ming Sabourin
Senior Technical Writer
Nuance Communications, Inc.
Montréal, Canada

Read more student testimonials...



Receive UNIX Tips, Tricks, and Shell Scripts by Email






Custom Search



LiveFire Labs' UNIX Tip, Trick, or Shell Script of the Week

Basic UNIX Commands for Beginners: Eliminating Duplicate Lines using the uniq Command

Consider this week's contents of unixfile, our frequently-used sample data file:

unix commands
shell script
command prompt
unix commands
unix system administration
shell script
unix commands

Since there are only a few number of lines in unixfile, duplicate lines can be quickly identified with a visual inspection and then removed if desired.  This manual process is not a viable option for files containing hundreds or thousands of lines.  The tool for a job of this magnitude is the UNIX uniq command (uniq is short for unique):

uniq [option] [input-file] [output-file]


[ If you are new to UNIX and need an overview of important UNIX commands and concepts, check out our Basic UNIX Commands and Concepts Tutorial for Beginners ]

uniq can be used to display, count, or delete adjacent duplicate lines from a file or standard input (stdin).  If duplicate lines in a file are not adjacent to one another, uniq will not treat them as duplicates:

$ uniq unixfile
unix commands
shell script
command prompt
unix commands
unix system administration
shell script
unix commands

For this reason, uniq is often combined with the sort command to group duplicate lines prior to performing an action on them:

$ sort unixfile | uniq
command prompt
shell script
unix commands
unix system administration

To compress this combination of commands, the -u option can be used with sort to produce the same results:

$ sort -u unixfile
command prompt
shell script
unix commands
unix system administration

Two useful options for the uniq command are -d and -c.  I will leave it to you to discover what these options will do to your data.  Make sure to sort your data prior to using them...