If you've sorted files using the handy-dandy sort command, you might end up with results that have duplicates in them. Heck, you might have files with duplicates anyway. At any rate, here's how to find and work with them. As Code Listing 6.18 shows, you can get rid of duplicate lines by using the uniq command (short for "unique") in conjunction with sort.
To eliminate duplicates with uniq:
sort address.book | uniq
Tip
uniq only finds identical, adjacent (sorted) lines. For example, if you have both Jones and jones in your address book, uniq won't identify either entry because they differ in capitalization.
Tip
You can also use the -d flag to specify that you want to see only the duplicate lines. For example, say you want to see all of the people who are in both your carpool file and your nightout file. You'd just use sort carpool nightout | uniq -d.
[ejr@hobbes manipulate]$ more long.address.book Schmidt, Sven, 1 Circle Drive, Denver, CO, 80221, 555-555-8382 Feldman, Fester, RR1, Billings, MT 62832, 285-555-0281 Brown, John, 1453 South Street, Tulsa, OK, 74114, 918-555-1234 Smith, Sally, 452 Center Ave., Salt Lake City, UT, 84000, 801-555-8982 Jones, Kelly, 14 Main Street, Santa Clara, CA, 95051, 408-555-7253 Schmidt, Swen, 1 Circle Drive, Denver, CO, 80221, 555-555-8382 Feldman, Fester, RR1, Billings, MT 62832, 285-555-0281 Brown, Jonathon, 1453 South Street, Tulsa, OK, 74114, 918-555-1234 Smith, Sally, 452 Center Ave., Salt Lake City, UT, 84000, 801-555-8982 Jones, Kelly, 14 Main Street, Santa Clara, CA, 95051, 408-555-7253 [ejr@hobbes manipulate]$ sort long.address.book | uniq Brown, John, 1453 South Street, Tulsa, OK, 74114, 918-555-1234 Brown, Jonathon, 1453 South Street, Tulsa, OK, 74114, 918-555-1234 Feldman, Fester, RR1, Billings, MT 62832, 285-555-0281 Jones, Kelly, 14 Main Street, Santa Clara, CA, 95051, 408-555-7253 Schmidt, Sven, 1 Circle Drive, Denver, CO, 80221, 555-555-8382 Schmidt, Swen, 1 Circle Drive, Denver, CO, 80221, 555-555-8382 Smith, Sally, 452 Center Ave., Salt Lake City, UT, 84000, 801-555-8982 [ejr@hobbes manipulate]$ |
52.14.130.13