UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 13: ordinal not in range(128)
I'm using NLTK to perform kmeans clustering on my text file in which each line is considered as a document. So for example, my text file is something like this:
belong finger death punch <br> hasty <br> mike hasty walls jericho <br> jägermeister rules <br> rules bands follow performing jägermeister stage <br> approach
Now the demo code I'm trying to run is this:
import sys import numpy from nltk.cluster import KMeansClusterer, GAAClusterer, euclidean_distance import nltk.corpus from nltk import decorators import nltk.stem stemmer_func = nltk.stem.EnglishStemmer().stem stopwords = set(nltk.corpus.stopwords.words('english')) @decorators.memoize def normalize_word(word): return stemmer_func(word.lower()) def get_words(titles): words = set() for title in job_titles: for word in title.split(): words.add(normalize_word(word)) return list(words) @decorators.memoize def vectorspaced(title): title_components = [normalize_word(word) for word in title.split()] return numpy.array([ word in title_components and not word in stopwords for word in words], numpy.short) if __name__ == '__main__': filename = 'example.txt' if len(sys.argv) == 2: filename = sys.argv with open(filename) as title_file: job_titles = [line.strip() for line in title_file.readlines()] words = get_words(job_titles) # cluster = KMeansClusterer(5, euclidean_distance) cluster = GAAClusterer(5) cluster.cluster([vectorspaced(title) for title in job_titles if title]) # NOTE: This is inefficient, cluster.classify should really just be # called when you are classifying previously unseen examples! classified_examples = [ cluster.classify(vectorspaced(title)) for title in job_titles ] for cluster_id, title in sorted(zip(classified_examples, job_titles)): print cluster_id, title
(which can also be found here)
The error I receive is this:
Traceback (most recent call last): File "cluster_example.py", line 40, in words = get_words(job_titles) File "cluster_example.py", line 20, in get_words words.add(normalize_word(word)) File "", line 1, in File "/usr/local/lib/python2.7/dist-packages/nltk/decorators.py", line 183, in memoize result = func(*args) File "cluster_example.py", line 14, in normalize_word return stemmer_func(word.lower()) File "/usr/local/lib/python2.7/dist-packages/nltk/stem/snowball.py", line 694, in stem word = (word.replace(u"\u2019", u"\x27") UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 13: ordinal not in range(128)
What is happening here?
The file is being read as a bunch of
strs, but it should be
unicodes. Python tries to implicitly convert, but fails. Change:
job_titles = [line.strip() for line in title_file.readlines()]
to explicitly decode the
unicode (here assuming UTF-8):
job_titles = [line.decode('utf-8').strip() for line in title_file.readlines()]
Read more... Read less...
This works fine for me.
f = open(file_path, 'r+', encoding="utf-8")
You can add a third parameter encoding to ensure the encoding type is 'utf-8'
Note: this method works fine in Python3, I did not try it in Python2.7.
For me there was a problem with the terminal encoding. Adding UTF-8 to .bashrc solved the problem:
Don't forget to reload .bashrc afterwards:
You can try this also:
import sys reload(sys) sys.setdefaultencoding('utf8')
When on Ubuntu 18.04 using Python3.6 I have solved the problem doing both:
with open(filename, encoding="utf-8") as lines:
and if you are running the tool as command line:
Note that if you are in Python2.7 you have do to handle this differently. First you have to set the default encoding:
import sys reload(sys) sys.setdefaultencoding('utf-8')
and then to load the file you must use
io.open to set the encoding:
import io with io.open(filename, 'r', encoding='utf-8') as lines:
You still need to export the env
I got this error when trying to install a python package in a Docker container. For me, the issue was that the docker image did not have a
locale configured. Adding the following code to the Dockerfile solved the problem for me.
# Avoid ascii errors when reading files in Python RUN apt-get install -y locales && locale-gen en_US.UTF-8 ENV LANG='en_US.UTF-8' LANGUAGE='en_US:en' LC_ALL='en_US.UTF-8'