MC, 2025

Mastering the Command Linux Uniq: Your Guide to Filtering Duplicates

Linux, with its vast array of commands and tools, offers powerful utilities to help users manage files, process data, and automate tasks with ease. One of the lesser-known yet incredibly useful commands in the Linux toolbox is the `uniq` command. This command is used to filter out duplicate lines in a file, leaving you with only unique entries. In this blog post, we’ll take a deep dive into the `uniq` command, how it works, and some common examples of its usage. Let’s get started!

What is the Command Linux Uniq?

The `uniq` command in Linux is a simple yet effective tool used to remove duplicate lines from a file or input stream. It stands for "unique," which is exactly what it does—it filters out repeated lines and leaves only the first occurrence of each unique line. By using `uniq`, you can clean up data and make it more manageable for further analysis or processing.

The basic syntax for the `uniq` command is:

uniq [OPTION]... [INPUT [OUTPUT]]

Where:

INPUT is the file or stream containing the data you want to filter.
OUTPUT is the file where the unique lines will be saved (optional).
OPTION is a set of flags or parameters that modify how the `uniq` command behaves.

Why Use the Command Linux Uniq?

You might be wondering, why would you need to filter duplicate lines from a file? There are a few scenarios where this command really shines:

Data Cleanup: When working with log files, CSVs, or other large datasets, duplicates may accumulate over time. Using `uniq`, you can easily clean up the data.
Streamlining Output: If you’re running commands or processes that generate repetitive output, `uniq` helps you focus on the unique results.
Analyzing Unique Entries: Whether you’re counting occurrences or simply extracting distinct values, the `uniq` command is perfect for isolating unique items from a list.

Basic Usage of the Command Linux Uniq

Let’s begin by looking at some of the most common ways to use the `uniq` command. These examples will show you how you can filter out duplicate lines in different scenarios.

1. Removing Duplicate Lines from a File

The most basic usage of `uniq` is simply removing duplicates from a file. For example, imagine you have a file called `data.txt` with the following content:

apple
apple
banana
cherry
banana
apple

If you run the following command:

uniq data.txt

It will output:

apple
banana
cherry

The duplicates have been removed, and only the unique lines remain. Simple, right?

2. Displaying Duplicate Lines Only

Sometimes, you might be interested in seeing only the duplicate lines. To achieve this, you can use the `-d` flag. Let’s look at an example using the same `data.txt` file:

uniq -d data.txt

The output will be:

apple
banana

Here, the `-d` option filters and shows only the lines that appear more than once. This can be very useful for identifying redundancy in your data!

3. Counting Occurrences of Each Line

Another incredibly helpful feature of the `uniq` command is the ability to count how many times each unique line appears in the input. This can be done with the `-c` flag. For example:

uniq -c data.txt

The output will look like this:

      3 apple
      2 banana
      1 cherry

As you can see, `uniq` has prefixed each unique line with a count of how many times it appears in the file. This feature is often used in conjunction with other commands like `sort` to generate frequency distributions of data.

4. Ignoring Case Sensitivity

By default, the `uniq` command is case-sensitive, meaning that `apple` and `Apple` would be treated as distinct lines. If you want to make it case-insensitive, you can use the `-i` flag. Here’s an example:

uniq -i data.txt

If `data.txt` contains:

apple
Apple
banana

The output will be:

apple
banana

Now, both "apple" and "Apple" are treated as the same line and only one of them appears in the output.

5. Combining Multiple Options

You can also combine multiple options to perform more complex operations. For example, if you want to count the occurrences of lines, ignore case, and show only duplicates, you can combine the `-c`, `-i`, and `-d` flags like so:

uniq -c -i -d data.txt

The output will give you the count of duplicate lines, ignoring case sensitivity. Combining options like this makes `uniq` an extremely flexible tool for various tasks!

Practical Examples of Using the Command Linux Uniq

Let’s take a look at a few practical scenarios where the `uniq` command can be especially useful:

1. Cleaning Up Log Files

If you’re working with server logs, you might encounter files with thousands of entries, many of which may be duplicates. The `uniq` command can help you quickly filter out the repetitive entries, leaving only unique log entries to analyze.

uniq /var/log/syslog > unique_syslog.txt

This command will take the `syslog` file and output a cleaned version with only unique entries into a new file called `unique_syslog.txt`.

2. Data Processing for Reports

If you have a CSV or text file containing lists of products, transactions, or customer IDs, the `uniq` command can help you extract the distinct values from the file. This can be handy for generating unique lists, summarizing data, or preparing for further analysis.

uniq transactions.csv > unique_transactions.csv

This will take the `transactions.csv` file and output only the unique transaction entries into a new file called `unique_transactions.csv`.

3. Analyzing Data for Duplicates

Another common use case is identifying duplicates in datasets. Whether you’re cleaning up a list of emails, usernames, or inventory items, `uniq` makes it easy to extract only the unique entries while leaving out the duplicates.