How to remove duplicates in list Python

In this article, we will be going through different ways to remove duplicates in list. Sometimes we want to generate a list with unique values, or remove the duplicated elements, this article will be helpful in solving all those problems.

Method 1

In this method, we are creating a new empty list and appending the unique elements from the original list to the new list using list

org_list = [1, 2, 4, 2, 1, 4, 2]
print ("The original list: " + str(org_list))
new_list=[] #creating an empty list
for i in org_list:
  if i not in new_list:
      new_list.append(i)

print ("After removing the duplicates :" + str(new_list))

Output:

The original list: [1, 2, 4, 2, 1, 4, 2]
After removing the duplicates :[1, 2, 4]

Method 2: Using List Comprehension

This method is similar to above method, but the difference is that this is a one-liner code with the help of list comprehension

org_list = [1, 2, 4, 2, 1, 4, 2]
print ("The original list: " + str(org_list))

new_list=[] 
[new_list.append(i) for i in org_list if i not in new_list]

print ("After removing the duplicates :" + str(new_list))

Output:

The original list: [1, 2, 4, 2, 1, 4, 2]
After removing the duplicates :[1, 2, 4]

Method 3 : Using Dictionary

Dictionary is a very powerful datatype in Python. List duplicates can be removed using dictionaries also. The idea used here is that dictionary cannot have duplicate keys. Create a dictionary using the list elements as keys, this will automatically remove the duplicates if any present.

org_list = [1, 2, 4, 2, 1, 4, 2]
print ("The original list: " + str(org_list))
org_list = list(dict.fromkeys(org_list))
print ( "After removing the duplicates :" + str(org_list) )

Output:

The original list: [1, 2, 4, 2, 1, 4, 2]
After removing the duplicates :[1, 2, 4]

Method 4: Using set()

Set elements are unique, duplicate elements are not allowed in sets. This is the property used here. One thing to note is that the ordering of elements is lost in this method. The list is converted to a set first, this will remove the duplicate elements from the list. Then the set containing unique elements are converted to list again

org_list = [1, 4, 4, 2, 1, 4, 2]
print ("The original list: " + str(org_list))
org_list = list(set(org_list))
print ( "After removing the duplicates :" + str(org_list) )

Output:

The original list: [1, 4, 4, 2, 1, 4, 2]
After removing the duplicates :[1, 2, 4]