Develop Reference

Python UnicodeError trying to download Image

Im trying to download a picture from the internet but once I use string formatting it gives me an Error:
"UnicodeEncodeError: 'ascii' codec can't encode character '\xdf' in position 161: ordinal not in range(128)"
If I remove string formatting the picture is downloadable.
I have already tried to encode and decode but nothings works.
def get_location(self):
self.key = self.key_entry.get()
self.location = self.location_entry.get()
self.zoom = self.zoom_entry.get()
self.type = self.type_entry.get()
self.url = "https://www.mapquestapi.com/staticmap/v4/getplacemap?key=%s&size=600,600&type=%s&imagetype=png&zoom=%s&scalebar=false&traffic=false&location=%s" % (self.key, self.type, self.zoom, self.location)
webbrowser.open_new_tab('%s' % self.url)
urllib.request.urlretrieve(self.url, "location.png")

try using utf-8
webbrowser.open_new_tab('%s' % self.url, 'utf-8')

UnicodeEncodeError: trying to print win32_process (wmi module)

i'm trying to print all the process ( like task mannager), so i used
this code :
from ctypes import *
import wmi
c = wmi.WMI()
kernel32 = windll.kernel32
i = 0
txt = ""
for process in c.Win32_Process():
try:
txt = txt + str(process)
except:
print process
most of the process (about 144/146) pass the try and then i print all together but there are about 2 process that fail the try, and the except and raise this error while i'm trying to print them:
UnicodeEncodeError: 'ascii' codec can't encode characters in position 178-181: ordinal not in range(128)
someone know what to do?

YouTube API search list_next() throws UnicodeEncodeError

When I feed a non-English string into the YouTube API library's
search, it only works during the initial search. If I call list_next(),
it throws a UnicodeEncodeError.
When I use a simple ascii string, everything works correctly.
Any suggestions about what I should do?
Here's a simplified code of what I'm doing:
# -*- coding: utf-8 -*-
import apiclient.discovery
def test(query):
youtube = apiclient.discovery.build('youtube', 'v3', developerKey='xxx')
ys = youtube.search()
req = ys.list(
q=query.encode('utf-8'),
type='video',
part='id,snippet',
maxResults=50
)
while (req):
res = req.execute()
for i in res['items']:
print(i['id']['videoId'])
req = ys.list_next(req, res)
test(u'한글')
test(u'日本語')
test(u'\uD55C\uAE00')
test(u'\u65E5\u672C\u8A9E')
Error message:
Traceback (most recent call last):
File "E:\prj\scripts\yt\search.py", line 316, in _search
req = ys.list_next(req, res)
File "D:\Apps\Python\lib\site-packages\googleapiclient\discovery.py", line 966, in methodNext
parsed[4] = urlencode(newq)
File "D:\Apps\Python\lib\urllib.py", line 1343, in urlencode
v = quote_plus(str(v))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-9: ordinal not in range(128)
Versions:
google-api-python-client (1.6.2)
Python 2.7.13 (Win32)
EDIT: I posted a workaround below.

If anyone else is interested, here's one workaround that works for me:
googleapiclient/discovery.py:
(old) q = parse_qsl(parsed[4])
(new) q = parse_qsl(parsed[4].encode('ascii'))
Explanation
In discovery.py, list_next() parses and unescapes the previous url, then makes a new url from it:
pageToken = previous_response['nextPageToken']
parsed = list(urlparse(request.uri))
q = parse_qsl(parsed[4])
# Find and remove old 'pageToken' value from URI
newq = [(key, value) for (key, value) in q if key != 'pageToken']
newq.append(('pageToken', pageToken))
parsed[4] = urlencode(newq)
uri = urlunparse(parsed)
It seems the problem is when parse_qsl unescapes the unicode parsed[4], it
returns the utf-8 encoded value in a unicode type. urlencode does not like
this:
q = urlparse.parse_qsl(u'q=%ED%95%9C%EA%B8%80')
[(u'q', u'\xed\x95\x9c\xea\xb8\x80')]
urllib.urlencode(q)
UnicodeEncodeError
If parse_qsl is given a plain ascii string, it returns a plain utf-8 encoded string which urlencode likes:
q = urlparse.parse_qsl(u'q=%ED%95%9C%EA%B8%80'.encode('ascii'))
[('q', '\xed\x95\x9c\xea\xb8\x80')]
urllib.urlencode(q)
'q=%ED%95%9C%EA%B8%80'

UnicodeEncodeError when running Json

Im getting a Unicode error when running this test script.
import urllib
import json
movieTitle = "Bee Movie"
title = movieTitle.replace(" ", "+")
year = ""
omdbAPI = "http://www.omdbapi.com/?t={}&y={}&plot=short&r=json".format(
title, year)
print (omdbAPI)
response = urllib.urlopen(omdbAPI)
data = json.loads(response.read())
valid_data = data["Response"]
print ("This data is: " + valid_data)
if valid_data == "True":
print data["Title"]
print data["Year"]
print data["Plot"]
print data["Rated"]
print data["Released"]
print data["Runtime"]
print data["Genre"]
print data["Director"]
print data["Writer"]
print data["Actors"]
print data["Language"]
print data["Country"]
print data["Awards"]
print data["Poster"]
print data["Metascore"]
print data["imdbRating"]
print data["imdbVotes"]
print data["imdbID"]
print data["Type"]
print data["Response"]
elif valid_data == "False":
print ("This data is: " + valid_data)
else:
raise ValueError("The information was not found")
Error :
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 19: ordinal not in range(128)
I guess its because one off the actors seams to have a é character.
I figured out that I could put .encode('utf8') after print data["Actors"] but it don't seams like the smartest thing to do.
I mean a random letter could occur on more places then on actor. And seams odd to go put .encode('utf8') after every instance
UPDATE :
Traceback (most recent call last):
File "/Volumes/postergren_projectDrive/Projekt/programmingSandbox/python/courses/udacity/Programming Foundations with Python/moveis/Advance/media.py", line 25, in <module>
print data["Actors"]
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 19: ordinal not in range(128)
[Finished in 0.1s with exit code 1]
[shell_cmd: "python" -u "/Volumes/postergren_projectDrive/Projekt/programmingSandbox/python/courses/udacity/Programming Foundations with Python/moveis/Advance/media.py"]
[dir: /Volumes/postergren_projectDrive/Projekt/programmingSandbox/python/courses/udacity/Programming Foundations with Python/moveis/Advance]
[path: /usr/bin:/bin:/usr/sbin:/sbin]

Try this at the begining of your code:
import sys
reload(sys)
sys.setdefaultencoding('utf-8')

You can do this:
for key in data.keys()
data[key] = data[key].encode('utf8')

unicode error displayed on the server on running app (django)

my views.py code:
#!/usr/bin/python
from django.template import loader, RequestContext
from django.http import HttpResponse
#from skey import find_root_tags, count, sorting_list
from search.models import Keywords
from django.shortcuts import render_to_response as rr
def front_page(request):
if request.method == 'POST' :
from skey import find_root_tags, count, sorting_list
str1 = request.POST['word']
fo = open("/home/pooja/Desktop/xml.txt","r")
for i in range(count.__len__()):
file = fo.readline()
file = file.rstrip('\n')
find_root_tags(file,str1,i)
list.append((file,count[i]))
sorting_list(list)
for name, count1 in list:
s = Keywords(file_name=name,frequency_count=count1)
s.save()
fo.close()
list1 = Keywords.objects.all()
t = loader.get_template('search/results.html')
c = RequestContext({'list1':list1,
})
return HttpResponse(t.render(c))
else :
str1 = ''
list = []
template = loader.get_template('search/front_page.html')
c = RequestContext(request)
response = template.render(c)
return HttpResponse(response)
skey.py has another function called within from find_root_tags():
def find_text(file,str1,i):
str1 = str1.lower()
exp = re.compile(r'<.*?>')
with open(file) as f:
lines = ''.join(line for line in f.readlines())
text_only = exp.sub('',lines).strip()
text_only = text_only.lower()
k = text_only.count(str1) #**line 34**
count[i] = count[i]+k
when I ran my app on server it gave me this error:
UnicodeDecodeError at /search/
'ascii' codec can't decode byte 0xef in position 0: ordinal not in range(128)
Request Method: POST
Request URL: http://127.0.0.1:8000/search/
Django Version: 1.4
Exception Type: UnicodeDecodeError
Exception Value:
'ascii' codec can't decode byte 0xef in position 0: ordinal not in range(128)
Exception Location: /home/pooja/Desktop/mysite/search/skey.py in find_text, line 34
Python Executable: /usr/bin/python
Python Version: 2.6.5
Python Path: ['/home/pooja/Desktop/mysite',
'/usr/lib/python2.6',
'/usr/lib/python2.6/plat-linux2',
'/usr/lib/python2.6/lib-tk',
'/usr/lib/python2.6/lib-old',
'/usr/lib/python2.6/lib-dynload',
'/usr/lib/python2.6/dist-packages',
'/usr/lib/python2.6/dist-packages/PIL',
'/usr/lib/python2.6/dist-packages/gst-0.10',
'/usr/lib/pymodules/python2.6',
'/usr/lib/python2.6/dist-packages/gtk-2.0',
'/usr/lib/pymodules/python2.6/gtk-2.0',
'/usr/local/lib/python2.6/dist-packages'] error :
Can anyone tell me why is it showing this error?How can I remove this error
Please help.

You're mixing Unicode strings and bytestrings. str1 = request.POST['word'] is probably a Unicode string and text_only is a bytestring. Python fails to convert the later to Unicode. You could use codecs.open() to specify the character encoding of the file. See Pragmatic Unicode and The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!).

Probable your str1 is in unicode, but text_only is not (on line 34). The next is not a panacea but if this corrects your problem then I am right.
k = u"{0}".format( text_only ).count(str1)