Develop Reference

Restart script when error happens and continue with right line in csv file

hi guys its my first time to ask help here i hope you can help me
i have this code that i write it
def my_function():
try :
with open('file.csv', 'r') as f:
data = list(csv.reader(f, delimiter=','))
i = 1
while i <= 10:
i += 1
fname = data[i][0]
lname = data[i][1]
options = Options()
driver = webdriver.Chrome(options=options)
driver.get("https://www.test.net/")
#Do staff
except Exception as e:
print(e)
driver.quit()
time.sleep(1)
print('******RESTART******')
my_function()
my_function()
well i'm trying to make this script run without stop .. the problem that i'm facing is when it stops for example in line number 8 (i = 8) and restart again ,it starts from first line (i = 1) .
i want the script to restart from line 8 and continue to 9 , 10 ...
can you please guide me to the right solution .. thank you

Your code is making this far more difficult than it needs to be.
First, you almost certainly don't want to wrap this entire block of code in a "catch all" exception handler. You want your exception handling to be sufficiently specific (limited) that you can do something meaningful with the exception. For example:
#!python
# Assumes Python version 3 or later
import sys, csv
filename='myfile.csv'
with open(filename as f:
try:
reader = csv.reader(f)
for record in reader:
if len(record) != 2:
# log error and continue
print('Malformed records in {}: {}'.format(filename, reader.line_num), file=sys.stderr)
continue
# do stuff with this record, knowing it has exactly two fields:
fname = record[0]
lname = record[1]
# etc ...
except csv.Error as e:
print('Error handling {} at line {}: {}'.format(filename, reader.line_num, e), file=sys.stderr)
Note that your errors probably weren't specifically in the csv module. It's pretty tolerant of malformed lines. But I'm showing how to wrap the reader and processing code within exception handling just for that. Your error was probably an IndexError (trying to access an item past the number of items in a list ... outside of its valid indexing range. It's better to just check the length of each record rather than use exception handling for that ... though it's possible either way.
There's a quite reasonable example (very similar code) in the documentation for the standard libraries: https://docs.python.org/3/library/csv.html
Also, stylistically, I'd suggest that a named tuple or a lightweight class (using __slots__) for managing these records. This would allow you to use dot notation to access the .fname and .lname of each rather than using [x] and numeric indexing. (Numeric indexing gets progressively more cumbersome and error prone as your code complexity increases).

You can set i to a key word argument with a default of 1 then on each exception pass the current i when you restart your function so it picks up from there.
This is a simplified example of what I'm recommending following the same general method you are using in your question (but with fake data so I can run it without having your CSV file).
def my_function(i=1):
try:
if i == 4: # to prevent forever recursion
return
else:
print(i) # keep track of loops
i += 1
x = int('te') # causes an error
except ValueError:
my_function(i) # send current i back through
my_function(i=0)

thank you for your quick response .. i tried the solution provided by –Kevin Welch and –Selcuk it works fine for me thx
here is the solution
def my_function():
try :
with open('file.csv', 'r') as f:
data = list(csv.reader(f, delimiter=','))
i = 1
while i <= 10:
i += 1
try :
fname = data[i][0]
lname = data[i][1]
options = Options()
driver = webdriver.Chrome(options=options)
driver.get("https://www.test.net/")
# Do staff
except Exception as e:
print(e)
driver.quit()
time.sleep(1)
print('******RESTAR******')
continue
my_function()

How to skip one part of a single loop iteration in Python

I am creating about 200 variables within a single iteration of a python loop (extracting fields from excel documents and pushing them to a SQL database) and I am trying to figure something out.
Let's say that a single iteration is a single Excel workbook that I am looping through in a directory. I am extracting around 200 fields from each workbook.
If one of these fields I extract (lets say field #56 out of 200) and it isn't in proper format (lets say the date was filled out wrong ie. 9/31/2015 which isnt a real date) and it errors out with the operation I am performing.
I want the loop to skip that variable and proceed to creating variable #57. I don't want the loop to completely go to the next iteration or workbook, I just want it to ignore that error on that variable and continue with the rest of the variables for that single loop iteration.
How would I go about doing something like this?
In this sample code I would like to continue extracting "PolicyState" even if ExpirationDate has an error.
Some sample code:
import datetime as dt
import os as os
import xlrd as rd
files = os.listdir(path)
for file in files: #Loop through all files in path directory
filename = os.fsdecode(file)
if filename.startswith('~'):
continue
elif filename.endswith( ('.xlsx', '.xlsm') ):
try:
book = rd.open_workbook(os.path.join(path,file))
except KeyError:
print ("Error opening file for "+ file)
continue
SoldModelInfo=book.sheet_by_name("SoldModelInfo")
AccountName=str(SoldModelInfo.cell(1,5).value)
ExpirationDate=dt.datetime.strftime(xldate_to_datetime(SoldModelInfo.cell(1,7).value),'%Y-%m-%d')
PolicyState=str(SoldModelInfo.cell(1,6).value)
print("Insert data of " + file +" was successful")
else:
continue

Use multiple try blocks. Wrap each decode operation that might go wrong in its own try block to catch the exception, do something, and carry on with the next one.
try:
book = rd.open_workbook(os.path.join(path,file))
except KeyError:
print ("Error opening file for "+ file)
continue
errors = []
SoldModelInfo=book.sheet_by_name("SoldModelInfo")
AccountName=str(SoldModelInfo.cell(1,5).value)
try:
ExpirationDate=dt.datetime.strftime(xldate_to_datetime(SoldModelInfo.cell(1,7).value),'%Y-%m-%d')
except WhateverError as e:
# do something, maybe set a default date?
ExpirationDate = default_date
# and/or record that it went wrong?
errors.append( [ "ExpirationDate", e ])
PolicyState=str(SoldModelInfo.cell(1,6).value)
...
# at the end
if not errors:
print("Insert data of " + file +" was successful")
else:
# things went wrong somewhere above.
# the contents of errors will let you work out what

As suggested you could use multiple try blocks on each of your extract variable, or you could streamline it with your own custom function that handles the try for you:
from functools import reduce, partial
def try_funcs(cell, default, funcs):
try:
return reduce(lambda val, func: func(val), funcs, cell)
except Exception as e:
# do something with your Exception if necessary, like logging.
return default
# Usage:
AccountName = try_funcs(SoldModelInfo.cell(1,5).value, "some default str value", str)
ExpirationDate = try_funcs(SoldModelInfo.cell(1,7).value), "some default date", [xldate_to_datetime, partial(dt.datetime.strftime, '%Y-%m-%d')])
PolicyState = try_funcs(SoldModelInfo.cell(1,6).value, "some default str value", str)
Here we use reduce to repeat multiple functions, and pass partial as a frozen function with arguments.
This can help your code look tidy without cluttering up with lots of try blocks. But the better, more explicit way is just handle the fields you anticipate might error out individually.

So, basically you need to wrap your xldate_to_datetime() call into try ... except
import datetime as dt
v = SoldModelInfo.cell(1,7).value
try:
d = dt.datetime.strftime(xldate_to_datetime(v), '%Y-%m-%d')
except TypeError as e:
print('Could not parse "{}": {}'.format(v, e)

How do I run python '__main__' program file from bash prompt in Windows10?

I am trying to run a python3 program file and am getting some unexpected behaviors.
I'll start off first with my PATH and env setup configuration. When I run:
which Python
I get:
/c/Program Files/Python36/python
From there, I cd into the directory where my python program is located to prepare to run the program.
Roughly speaking this is how my python program is set up:
import modulesNeeded
print('1st debug statement to show program execution')
# variables declared as needed
def aFunctionNeeded():
print('2nd debug statement to show fxn exe, never prints')
... function logic...
if __name__ == '__main__':
aFunctionNeeded() # Never gets called
Here is a link to the repository with the code I am working with in case you would like more details as to the implementation. Keep in mind that API keys are not published, but API keys are in local file correctly:
https://github.com/lopezdp/API.Mashups
My question revolves around why my 1st debug statements inside the files are printing to the terminal, but not the 2nd debug statements inside the functions?
This is happening in both of the findRestaurant.py file and the geocode.py file.
I know I have written my if __name__ == '__main__': program entry point correctly as this is the same exact way I have done it for other programs, but in this case I may be missing something that I am not noticing.
If this is my output when I run my program in my bash terminal:
$ python findRestaurant.py
inside geo
inside find
then, why does it appear that my aFunctionNeeded() method shown in my pseudo code is not being called from the main?
Why do both programs seem to fail immediately after the first debug statements are printed to the terminal?
findRestaurant.py File that can also be found in link above
from geocode import getGeocodeLocation
import json
import httplib2
import sys
import codecs
print('inside find')
sys.stdout = codecs.getwriter('utf8')(sys.stdout)
sys.stderr = codecs.getwriter('utf8')(sys.stderr)
foursquare_client_id = "..."
foursquare_client_secret = "..."
def findARestaurant(mealType,location):
print('inside findFxn')
#1. Use getGeocodeLocation to get the latitude and longitude coordinates of the location string.
latitude, longitude = getGeocodeLocation(location)
#2. Use foursquare API to find a nearby restaurant with the latitude, longitude, and mealType strings.
#HINT: format for url will be something like https://api.foursquare.com/v2/venues/search?client_id=CLIENT_ID&client_secret=CLIENT_SECRET&v=20130815&ll=40.7,-74&query=sushi
url = ('https://api.foursquare.com/v2/venues/search?client_id=%s&client_secret=%s&v=20130815&ll=%s,%s&query=%s' % (foursquare_client_id, foursquare_client_secret,latitude,longitude,mealType))
h = httplib2.Http()
result = json.loads(h.request(url,'GET')[1])
if result['response']['venues']:
#3. Grab the first restaurant
restaurant = result['response']['venues'][0]
venue_id = restaurant['id']
restaurant_name = restaurant['name']
restaurant_address = restaurant['location']['formattedAddress']
address = ""
for i in restaurant_address:
address += i + " "
restaurant_address = address
#4. Get a 300x300 picture of the restaurant using the venue_id (you can change this by altering the 300x300 value in the URL or replacing it with 'orginal' to get the original picture
url = ('https://api.foursquare.com/v2/venues/%s/photos?client_id=%s&v=20150603&client_secret=%s' % ((venue_id,foursquare_client_id,foursquare_client_secret)))
result = json.loads(h.request(url, 'GET')[1])
#5. Grab the first image
if result['response']['photos']['items']:
firstpic = result['response']['photos']['items'][0]
prefix = firstpic['prefix']
suffix = firstpic['suffix']
imageURL = prefix + "300x300" + suffix
else:
#6. if no image available, insert default image url
imageURL = "http://pixabay.com/get/8926af5eb597ca51ca4c/1433440765/cheeseburger-34314_1280.png?direct"
#7. return a dictionary containing the restaurant name, address, and image url
restaurantInfo = {'name':restaurant_name, 'address':restaurant_address, 'image':imageURL}
print ("Restaurant Name: %s" % restaurantInfo['name'])
print ("Restaurant Address: %s" % restaurantInfo['address'])
print ("Image: %s \n" % restaurantInfo['image'])
return restaurantInfo
else:
print ("No Restaurants Found for %s" % location)
return "No Restaurants Found"
if __name__ == '__main__':
findARestaurant("Pizza", "Tokyo, Japan")
geocode.py File that can also be found in link above
import httplib2
import json
print('inside geo')
def getGeocodeLocation(inputString):
print('inside of geoFxn')
# Use Google Maps to convert a location into Latitute/Longitute coordinates
# FORMAT: https://maps.googleapis.com/maps/api/geocode/json?address=1600+Amphitheatre+Parkway,+Mountain+View,+CA&key=API_KEY
google_api_key = "..."
locationString = inputString.replace(" ", "+")
url = ('https://maps.googleapis.com/maps/api/geocode/json?address=%s&key=%s' % (locationString, google_api_key))
h = httplib2.Http()
result = json.loads(h.request(url,'GET')[1])
latitude = result['results'][0]['geometry']['location']['lat']
longitude = result['results'][0]['geometry']['location']['lng']
return (latitude,longitude)

The reason you're not seeing the output of the later parts of your code is that you've rebound the standard output and error streams with these lines:
sys.stdout = codecs.getwriter('utf8')(sys.stdout)
sys.stderr = codecs.getwriter('utf8')(sys.stderr)
I'm not exactly sure why those lines are breaking things for you, perhaps your console does not expect utf8 encoded output... But because they don't work as intended, you're not seeing anything from the rest of your code, including error messages, since you rebound the stderr stream along with the stdout stream.

Searching for text in python without re module

Im creating a definition in python that use urllib2 to download a status page and loop until a condition is met.
The status page looks like the one below:
reportId:327686
reportName:report2
status:Running
percent_done:0
I need to;
Parse reportId and create a variable with this value
Loop until status is different than "Running"
Can I accomplish this without using re module? At the end I will need to convert this to an exe using pyinstaller so wanted to avoid loading a lot of modules to keep the program small.

This should do it:
import urllib2
def parse_data(raw_data): # Name this better
parsed_data = dict(line.split(':') for line in raw_data.splitlines())
parsed_data['reportId'] = int(parsed_data['reportId'])
parsed_data['percent_done'] = int(parsed_data['percent_done'])
return parsed_data
def get_parsed_data_from_url(url): # Name this better
raw_data = urllib2.urlopen(url).read()
parsed_data = parse_data(raw_data)
return parsed_data
parsed_data = get_parsed_data_from_url('http://example.com')
# And to loop until status != 'Running', you could do this..
while get_parsed_data_from_url('http://example.com')['status'] == 'Running':
do_some_stuff()

If that is your result and you don't have more HTML as That1Guy commented, you can probably use startswith and endswith. Something along the lines of (I've skipped a lot of checking and default values here!) ...
if line.startswith("reportId:"):
report_id = line.split(":")[1]
if line.startswith("status:"):
if line.endswith("Running") == false:
# abort processing

Since you have stable patterns to match to, it's pretty simple:
reportIds=[]
# not written here: load report into variable s
for line in s.splitlines():
if 'reportId' in line:
reportIds.append(line.split(':')[1])
if 'status' in line:
if not line.split(':')[1] == 'Running':
break
# not written here: pause for some period of time

Python: Why will this string print but not write to a file?

I am new to Python and working on a utility that changes an XML file into an HTML. The XML comes from a call to request = urllib2.Request(url), where I generate the custom url earlier in the code, and then set response = urllib2.urlopen(request) and, finally, xml_response = response.read(). This works okay, as far as I can tell.
My trouble is with parsing the response. For starters, here is a partial example of the XML structure I get back:
I tried adapting the slideshow example in the minidom tutorial here to parse my XML (which is ebay search results, by the way): http://docs.python.org/2/library/xml.dom.minidom.html
My code so far looks like this, with try blocks as an attempt to diagnose issues:
doc = minidom.parseString(xml_response)
#Extract relevant information and prepare it for HTML formatting.
try:
handleDocument(doc)
except:
print "Failed to handle document!"
def getText(nodelist): #taken straight from slideshow example
rc = []
for node in nodelist:
if node.nodeType == node.TEXT_NODE:
print "A TEXT NODE!"
rc.append(node.data)
return ''.join(rc) #this is a string, right?
def handleDocument(doc):
outputFile = open("EbaySearchResults.html", "w")
outputFile.write("<html>\n")
outputFile.write("<body>\n")
try:
items = doc.getElementsByTagName("item")
except:
"Failed to get elements by tag name."
handleItems(items)
outputFile.write("</html>\n")
outputFile.write("</body>\n")
def handleItems(items):
for item in items:
title = item.getElementsByTagName("title")[0] #there should be only one title
print "<h2>%s</h2>" % getText(title.childNodes) #this works fine!
try: #none of these things work!
outputFile.write("<h2>%s</h2>" % getText(title.childNodes))
#outputFile.write("<h2>" + getText(title.childNodes) + "</h2>")
#str = getText(title.childNodes)
#outputFIle.write(string(str))
#outputFile.write(getText(title.childNodes))
except:
print "FAIL"
I do not understand why the correct title text does print to the console but throws an exception and does not work for the output file. Writing plain strings like this works fine: outputFile.write("<html>\n") What is going on with my string construction? As far as I can tell, the getText method I am using from the minidom example returns a string--which is just the sort of thing you can write to a file..?

If I print the actual stack trace...
...
except:
print "Exception when trying to write to file:"
print '-'*60
traceback.print_exc(file=sys.stdout)
print '-'*60
traceback.print_tb(sys.last_traceback)
...
...I will instantly see the problem:
------------------------------------------------------------
Traceback (most recent call last):
File "tohtml.py", line 85, in handleItems
outputFile.write(getText(title.childNodes))
NameError: global name 'outputFile' is not defined
------------------------------------------------------------
Looks like something has gone out of scope!
Fellow beginners, take note.