# Get unique values from a list in python

## Get unique values from a list in python

### Question

I want to get the unique values from the following list:

```
['nowplaying', 'PBS', 'PBS', 'nowplaying', 'job', 'debate', 'thenandnow']
```

The output which I require is:

```
['nowplaying', 'PBS', 'job', 'debate', 'thenandnow']
```

This code works:

```
output = []
for x in trends:
if x not in output:
output.append(x)
print(output)
```

is there a better solution I should use?

### Popular Answer

First declare your list properly, separated by commas. You can get the unique values by converting the list to a set.

```
mylist = ['nowplaying', 'PBS', 'PBS', 'nowplaying', 'job', 'debate', 'thenandnow']
myset = set(mylist)
print(myset)
```

If you use it further as a list, you should convert it back to a list by doing:

```
mynewlist = list(myset)
```

Another possibility, probably faster would be to use a set from the beginning, instead of a list. Then your code should be:

```
output = set()
for x in trends:
output.add(x)
print(output)
```

As it has been pointed out, sets do not maintain the original order. If you need that, you should look for an ordered set implementation (see this question for more).

Read more... Read less...

If we need to keep the elements order, how about this:

```
used = set()
mylist = [u'nowplaying', u'PBS', u'PBS', u'nowplaying', u'job', u'debate', u'thenandnow']
unique = [x for x in mylist if x not in used and (used.add(x) or True)]
```

And one more solution using `reduce`

and without the temporary `used`

var.

```
mylist = [u'nowplaying', u'PBS', u'PBS', u'nowplaying', u'job', u'debate', u'thenandnow']
unique = reduce(lambda l, x: l.append(x) or l if x not in l else l, mylist, [])
```

**UPDATE - March, 2019**

And a 3rd solution, which is a neat one, but kind of slow since `.index`

is O(n).

```
mylist = [u'nowplaying', u'PBS', u'PBS', u'nowplaying', u'job', u'debate', u'thenandnow']
unique = [x for i, x in enumerate(mylist) if i == mylist.index(x)]
```

**UPDATE - Oct, 2016**

Another solution with `reduce`

, but this time without `.append`

which makes it more human readable and easier to understand.

```
mylist = [u'nowplaying', u'PBS', u'PBS', u'nowplaying', u'job', u'debate', u'thenandnow']
unique = reduce(lambda l, x: l+[x] if x not in l else l, mylist, [])
#which can also be writed as:
unique = reduce(lambda l, x: l if x in l else l+[x], mylist, [])
```

**NOTE:** Have in mind that more human-readable we get, more unperformant the script is.

```
import timeit
setup = "mylist = [u'nowplaying', u'PBS', u'PBS', u'nowplaying', u'job', u'debate', u'thenandnow']"
#10x to Michael for pointing out that we can get faster with set()
timeit.timeit('[x for x in mylist if x not in used and (used.add(x) or True)]', setup='used = set();'+setup)
0.4188511371612549
timeit.timeit('[x for x in mylist if x not in used and (used.append(x) or True)]', setup='used = [];'+setup)
0.6157128810882568
timeit.timeit('reduce(lambda l, x: l.append(x) or l if x not in l else l, mylist, [])', setup=setup)
1.8778090476989746
timeit.timeit('reduce(lambda l, x: l+[x] if x not in l else l, mylist, [])', setup=setup)
2.13108491897583
timeit.timeit('reduce(lambda l, x: l if x in l else l+[x], mylist, [])', setup=setup)
2.207760810852051
timeit.timeit('[x for i, x in enumerate(mylist) if i == mylist.index(x)]', setup=setup)
2.3621110916137695
```

**ANSWERING COMMENTS**

Because **@monica** asked a good question about "how is this working?". For everyone having problems figuring it out. I will try to give a more deep explanation about how this works and what sorcery is happening here ;)

So she first asked:

I try to understand why

`unique = [used.append(x) for x in mylist if x not in used]`

is not working.

Well it's actually working

```
>>> used = []
>>> mylist = [u'nowplaying', u'PBS', u'PBS', u'nowplaying', u'job', u'debate', u'thenandnow']
>>> unique = [used.append(x) for x in mylist if x not in used]
>>> print used
[u'nowplaying', u'PBS', u'job', u'debate', u'thenandnow']
>>> print unique
[None, None, None, None, None]
```

The problem is that we are just not getting the desired results inside the `unique`

variable, but only inside the `used`

variable. This is because during the list comprehension `.append`

modifies the `used`

variable and returns `None`

.

So in order to get the results into the `unique`

variable, and still use the same logic with `.append(x) if x not in used`

, we need to move this `.append`

call on the right side of the list comprehension and just return `x`

on the left side.

But if we are too naive and just go with:

```
>>> unique = [x for x in mylist if x not in used and used.append(x)]
>>> print unique
[]
```

We will get nothing in return.

Again, this is because the `.append`

method returns `None`

, and it this gives on our logical expression the following look:

```
x not in used and None
```

This will basically always:

- evaluates to
`False`

when`x`

is in`used`

, - evaluates to
`None`

when`x`

is not in`used`

.

And in both cases (`False`

/`None`

), this will be treated as `falsy`

value and we will get an empty list as a result.

But why this evaluates to `None`

when `x`

is not in `used`

? Someone may ask.

Well it's because this is how Python's short-circuit operators works.

The expression

`x and y`

first evaluates x; if x is false, its value is returned; otherwise, y is evaluated and the resulting value is returned.

So when `x`

is not in used *(i.e. when its True)* the next part or the expression will be evaluated

*(*and its value

`used.append(x)`

)*(*will be returned.

`None`

)But that's what we want in order to get the unique elements from a list with duplicates, we want to `.append`

them into a new list only when we they came across for a fist time.

So we really want to evaluate `used.append(x)`

only when `x`

is not in `used`

, maybe if there is a way to turn this `None`

value into a `truthy`

one we will be fine, right?

Well, yes and here is where the 2nd type of `short-circuit`

operators come to play.

The expression

`x or y`

first evaluates x; if x is true, its value is returned; otherwise, y is evaluated and the resulting value is returned.

We know that `.append(x)`

will always be `falsy`

, so if we just add one `or`

next to him, we will always get the next part. That's why we write:

```
x not in used and (used.append(x) or True)
```

so we can **evaluate** `used.append(x)`

and get `True`

as a result, **only when** the first part of the expression * (x not in used)* is

`True`

.Similar fashion can be seen in the 2nd approach with the `reduce`

method.

```
(l.append(x) or l) if x not in l else l
#similar as the above, but maybe more readable
#we return l unchanged when x is in l
#we append x to l and return l when x is not in l
l if x in l else (l.append(x) or l)
```

where we:

- Append
`x`

to`l`

and return that`l`

when`x`

is not in`l`

. Thanks to the`or`

statement`.append`

is evaluated and`l`

is returned after that. - Return
`l`

untouched when`x`

is in`l`

A Python list:

```
>>> a = ['a', 'b', 'c', 'd', 'b']
```

To get unique items, just transform it into a set (which you can transform back again into a list if required):

```
>>> b = set(a)
>>> print(b)
{'b', 'c', 'd', 'a'}
```

What type is your output variable?

Python sets are what you need. Declare output like this:

```
output = set() # initialize an empty set
```

and you're ready to go adding elements with `output.add(elem)`

and be sure they're unique.

Warning: sets DO NOT preserve the original order of the list.

Maintaining order:

```
# oneliners
# slow -> . --- 14.417 seconds ---
[x for i, x in enumerate(array) if x not in array[0:i]]
# fast -> . --- 0.0378 seconds ---
[x for i, x in enumerate(array) if array.index(x) == i]
# multiple lines
# fastest -> --- 0.012 seconds ---
uniq = []
[uniq.append(x) for x in array if x not in uniq]
uniq
```

Order doesn't matter:

```
# fastest-est -> --- 0.0035 seconds ---
list(set(array))
```