Develop Reference

Python - variable not defined but I think it is

I am pulling images from the Internet Archive as a test of some python code and I am incorporating the requests module. My code is as follows: (note, not the entire code, just the relevant section)
image_results = []
image_hashes = []
session = requests.Session()
for image in image_list:
if txtUrl not in image:
continue
try:
self.rslBox.AppendText("[v] Downloading %s" % image + "\n")
self.rslBox.Refresh()
response = session.get(image)
except:
self.rslBox.AppendText("[!] Failed to download: %s" % image + "\n")
self.rslBox.Refresh()
# continue
if "image" in response.headers['content-type']:
sha1 = hashlib.sha1(response.content).hexadigest()
if sha1 not in image_hashes:
image_hashes.append(sha1)
image_path = "WayBackImages/%s-%s" % (sha1.image.split("/")[-1])
with open(image_path, "wb") as fd:
fd.write(response.content)
self.rslBox.AppendText("[*] Saved %s" % images + "\n")
self.rslBox.Refresh()
info = pyexifinfo.get_json(image_path)
info[0]['ImageHash'] = sha1
image_results.append(info[0])
image_results_json = json.dumps(image_results)
data_frame = pandas.read_json(image_results_json)
csv = data_frame.to_csv('results.csv')
self.rslBox.AppendText("[*] Finished writing CSV to results.csv" + '\n')
self.rslBox.Refresh()
return
When I run my code, I get the following message:
Traceback (most recent call last):
File "C:\eclipse-workspace\test\tabbedPage.py", line 136, in OnSearch
if "image" in response.headers['content-type']:
NameError: name 'response' is not defined
But response is defined in the try statement - or so I would think. It only complains on the if "image" section - why??
I am new to python and I am using python3.6 and pydev with Eclipse.
Thanks!

Something inside your try failed. Your except caught it handle the error but since there is no raise in it, it continues execution, but response is not set.

It's because you are declaring response in the try block. If the exception gets thrown then response is not declared.
A work around for this would be putting the code that relies on response being declared into that try block.

Python: error with file op operations

I am loop through json and writing to file and passing that to another funtion to upload using a curl command.
Here is the error message i am getting:
Traceback (most recent call last):
File "D:\python_ws\test.py", line 33, in <module>
main((sys.argv[1:]))
File "D:\python_ws\test.py", line 30, in main
upload_file_bams(out_file)
File "D:\python_ws\test.py", line 7, in upload_file
file = open(rel_file).read()
TypeError: coercing to Unicode: need string or buffer, file found
I tried different ways but seems like i am missing some basics here. any help is appreciated.
Here is the my code:
#!/usr/bin/env python
import urllib2,json,sys,requests,subprocess,os
def upload_file(rel_file):
url = "https://localhost:8080/artifactory/list/default.generic.local/ct/releases/"
user = "john.smith"
file = open(rel_file).read()
cmd = 'curl --verbose --user %s --upload-file %s %s' %(user,file,url)
print 'trying to execute %s' % cmd
x = subprocess.Popen('cmd', shell=True)
#subprocess.call('cmd', shell=True)
retval = x.wait()
def main(argv):
#environment
env = sys.argv[1]
rel_name=sys.argv[2]
consul_url=("http://localhost:9090") % (env)
list_services = "/v1/catalog/services"
services_url = consul_url+list_services
list_of_services = urllib2.urlopen(services_url).read()
each_service = json.loads(list_of_services)
#to remove keys with blank values
newdict = dict([(vkey,vdata) for vkey, vdata in each_service.iteritems() if(vdata) ])
try:
out_file = open(rel_name,"wb")
json.dump(newdict,out_file, indent=4)
finally:
out_file.close()
#uploading release json file to BAMS
upload_file(out_file)
if __name__ == "__main__":
main((sys.argv[1:]))

When you call upload_file(), you pass it out_file which is of type file instead of string (file name). The function open() takes for the first argument the file name that you want to open.

How can I get around this try except block?

I wrote a try except block that I now realize was a bad idea because it keep throwing 'blind' exceptions that are hard to debug. The problem is that I do not know how to go about writing it another way besides going through each of the methods that are called and manually reading all the exceptions and making a case for each.
How would you structure this code?
def get_wiktionary_audio(self):
'''function for adding audio path to a definition, this is meant to be run before trying to get a specific URL'''
#this path is where the audio will be saved, only added the kwarg for testing with a different path
path="study_audio/%s/words" % (self.word.language.name)
try:
wiktionary_url = "http://%s.wiktionary.org/wiki/FILE:en-us-%s.ogg" % (self.word.language.wiktionary_prefix, self.word.name)
wiktionary_page = urllib2.urlopen(wiktionary_url)
wiktionary_page = fromstring(wiktionary_page.read())
file_URL = wiktionary_page.xpath("//*[contains(concat(' ', #class, ' '), ' fullMedia ')]/a/#href")[0]
file_number = len(self.search_existing_audio())
relative_path = '%s/%s%s.ogg' % (path, self.word.name, file_number)
full_path = '%s/%s' % (settings.MEDIA_ROOT, relative_path)
os.popen("wget -q -O %s 'http:%s'" % (full_path, file_URL))
except:
return False
WordAudio.objects.create(word=self.word, audio=relative_path, source=wiktionary_url)
return True

Often, exceptions come with error strings which can be used to pinpoint the problem. You can access this value like so:
try:
# code block
except Exception as e:
print str(e)
You can also print what class of exception it is along with any error messages by using the repr method:
try:
# code block
except Exception as e:
print repr(e)

One way I like to go about it is configure Python logging and log the output. This gives you a lot of flexibility in what you do with the log output. The below example logs the exception traceback.
import traceback
import logging
logger = logging.getLogger(__name__)
try:
...
except Exception as e:
logger.exception(traceback.format_exc()) # the traceback
logger.exception(e) # just the exception message

First your code is un-pythonic. You are using 'self' for a function. "self" is usually reserved for a class. So in reading your code, it feels unnatural. Second, my style is to line up "=" signs for readability. My advice is to start over -- Use standard pythonic conventions. You can get this by going through python tutorials.
Throw exception early and often -ONLY when the code stops running. You could also move some of the naming outside the try/except block.
def get_wiktionary_audio(self):
'''function for adding audio path to a definition, this is meant to be run before trying to get a specific URL'''
#this path is where the audio will be saved, only added the kwarg for testing with a different path
path = "study_audio/%s/words" % (self.word.language.name)
try:
wiktionary_url = "http://%s.wiktionary.org/wiki/FILE:en-us-%s.ogg" % (self.word.language.wiktionary_prefix, self.word.name)
wiktionary_page = urllib2.urlopen(wiktionary_url)
wiktionary_page = fromstring(wiktionary_page.read())
file_URL = wiktionary_page.xpath("//*[contains(concat(' ', #class, ' '), ' fullMedia ')]/a/#href")[0]
file_number = len(self.search_existing_audio())
relative_path = '%s/%s%s.ogg' % (path, self.word.name, file_number)
full_path = '%s/%s' % (settings.MEDIA_ROOT, relative_path)
os.popen("wget -q -O %s 'http:%s'" % (full_path, file_URL))
except Exception as e : print e
WordAudio.objects.create(word=self.word, audio=relative_path, source=wiktionary_url)
return True

Python 2.4.4 incessant type error

Im using python 2.4.4 and for some reason it keeps throwing a type error when i try to open a log file for writing... Here is the function in question...
import os
def write(inlog, outlog):
# parse the logs and save them to our own...
parsed = NTttcpParse(inlog)
debug("PARSED")
debug(parsed)
if os.path.exists(outlog):
fh = os.open(outlog, os.O_WRONLY | os.O_APPEND)
debug("Opened '%s' for writing (%d)" % (outlog, fh))
else:
fh = os.open(outlog, os.O_WRONLY | os.O_CREAT)
debug("Created '%s' for writing (%d)" % (outlog, fh))
debug("type(fh) = %s" % type(fh))
os.write(fh, LOGFORMAT % parsed)
os.close(fh)
And here is the maddening error...
TypeError: int argument required
Please hel... and thanks in advance :P

You are doing file I/O in a strange way. Here is the way to do it:
f = open(outlog, "w")
f.write("some data written to file\n")
f.close()
If you want to append, use open(outlog, "a") instead. If you want to read, use open(outlog, "r"). Also read the Python tutorial, which explains basic file I/O operations like this.
Note that in Python 2.5 and up, you can use the with statement:
with open(outlog, "w") as f:
f.write("some data written to file\n")
(I originally posted this as the main answer before I noticed you said you were using 2.4.)

Yikes, i made a simple error-- i wasnt entering enough items in the parsed tuple ^_^ Thanks for your answers though!

What is the equivalent of Perl's (<>) in Python? fileinput doesn't work as expected

In Perl one uses:
while (<>) {
# process files given as command line arguments
}
In Python I found:
import fileinput
for line in fileinput.input():
process(line)
But, what happens when the file given in the command line does NOT exist?
python test.py test1.txt test2.txt filenotexist1.txt filenotexist2.txt test3.txt was given as the argument.
I tried various ways of using try: except: nextfile, but I couldn't seem to make it work.
For the above commandline, the script should run for test1-3.txt but just go to next file silent when the file is NOT found.
Perl does this very well. I have searched this all over the net, but I couldn't find the answer to this one anywhere.

import sys
import os
for f in sys.argv[1:]:
if os.path.exists(f):
for line in open(f).readlines():
process(line)

Something like this;
import sys
for f in sys.argv[1:]:
try:
data = open(f).readlines()
process(data)
except IOError:
continue

Turning #Brian's answer into a generator, and catching IOError rather than testing for existence which is more Pythonic and then printing a warning to stderr on failure:
import sys
def read_files(files = None):
if not files:
files = sys.argv[1:]
for file in files:
try:
for line in open(file):
yield line
except IOError, e:
print >>sys.stderr, 'Warning:', e
for line in read_files():
print line,
Output (the file baz does not exist):
$ python read_lines.py foo bar baz
line 1 of foo
line 2 of foo
line 1 of bar
line 2 of bar
Warning: [Errno 2] No such file or directory: 'baz'
You might want to put in a little effort tidying up the error message, but it might not be worth the effort.

You can solve your problem with fileinput module as follows:
import fileinput
input = fileinput.input()
while True:
try:
process(input.next())
except IOError:
input.nextfile()
except StopIteration:
break
Unfortunately you can't use for loop because the IOException breaks it.

I tried to implement #VGE's suggestion, but my attempt turned out not to be too elegant. I'd appreciate any suggestions for how to improve this.
import sys, fileinput, errno, os
class nosuchfile:
def readlines(foo, bar):
return []
def close(arg):
pass
EXITCODE=0
def skip_on_error (filename, mode):
"""Function to pass in as fileinput.input(openhook=...) hook function.
Instead of give up on the first error, skip the rest of the file and
continue with the next file in the input list.
In case of an error from open() an error message is printed to standard
error and the global variable EXITCODE gets overwritten by a nonzero
value.
"""
global EXITCODE
try:
return open(filename, mode)
except IOError, e:
sys.stderr.write ("%s: %s: %s\n" % (sys.argv[0], filename, os.strerror(e.errno)))
EXITCODE = 1
return nosuchfile()
def main ():
do_stuff(fileinput.input(openhook=skip_on_error))
return EXITCODE
Both the placeholder dummy filehandle class nosuchfile and the global variable EXITCODE are pretty serious warts. I tried to figure out how to pass in a reference to a locally scoped exitcode variable, but gave up.
This also fails to handle errors which happen while reading, but the majority of error cases seem to happen in open anyway.

Simple, explicit, and silent:
import fileinput
from os.path import exists
import sys
for line in fileinput.input(files=filter(exists, sys.argv[1:])):
process(line)

Maybe You can play with the openhook parameter to control not existing file.