How to make list into set with same sequence? (python)

Question

I have a file which I am using for brute-force attacks so I want to get rid of the duplicates to save some time and copy it into another file without ruining sequence. I can make another list and append it and then check if it'= is in there but file is large so I do not think it is best way to do this. What I tried is:

with open('original_file.txt','r') as file:
   words = file.readlines()

with open('file_without_duplicates','w') as file:
   for i in set(words): file.write(i)

I have most common words at the beginning so sequence is important but this example ruins it.

question from:https://stackoverflow.com/questions/65598253/how-to-make-list-into-set-with-same-sequence-python

Answer 1

You can still use a set, but in a different way.

Make a new empty set and use that to keep track of words that have been written to the file. For each word in the original list, if it's not already in the set, then write it to the file and add it to the set.

written_words = set()
with open('file_without_duplicates','w') as file:
    for i in words:
        if i not in written_words:
            file.write(i)
            written_words.add(i)