remove all duplicates from a text file without sort

simply put, I have a file with lines of text that are unknown to me, something like

abaa
dddd
bbbb
cccc
abaa
aaaa
abaa

the result I’d like to get is

dddd
bbbb
cccc
aaaa

where all the duplicates are completely removed. if one line of text is duplicated anywhere else, remove this line and any repetitions that follow. is this possible to do? all of my searches show output with duplicates removed, leaving one instance, which I would like removed.

Here is Solutions:

We have many solutions to this problem, But we recommend you to use the first solution because it is tested & true solution that will 100% work for you.

Solution 1

this approach takes 2 passes through the file: one to count the number of times each line occurs, one to print the lines that only appear once:

awk 'NR == FNR {count[$0]++; next}; count[$0] == 1' file file

Note: Use and implement solution 1 because this method fully tested our system.
Thank you 🙂

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0

Leave a Reply