Develop Reference

Accessible variables at the root of a python script

I've declared a number of variables at the start of my script, as I'm using them in a number of different methods ("Functions" in python?). When I try to access them, I can't seem to get their value = or set them to another value for that matter. For example:
baseFile = open('C:/Users/<redacted>/Documents/python dev/ATM/Data.ICSF', 'a+')
secFile = open('C:/Users/<redacted>/Documents/python dev/ATM/security.ICSF', 'a+')
def usrInput(raw_input):
if raw_input == "99999":
self.close(True)
else:
identity = raw_input
def splitValues(source, string):
if source == "ident":
usrTitle = string.split('>')[1]
usrFN = string.split('>')[2]
usrLN = string.split('>')[3]
x = string.split('>')[4]
usrBal = Decimal(x)
usrBalDisplay = str(locale.currency(usrBal))
elif source == "sec":
usrPIN = string.split('>')[1]
pinAttempts = string.split('>')[2]
def openAccount(identity):
#read all the file first. it's f***ing heavy but it'll do here.
plString = baseFile.read()
xList = plString.split('|')
parm = str(identity)
for i in xList:
substr = i[0:4]
if parm == substr:
print "success"
usrString = str(i)
else:
lNumFunds = lNumFunds + 1
splitValues("ident", usrString)
When I place baseFile and secFile in the openAccount method, I can access the respective files as normal. However, when I place them at the root of the script, as in the example above, I can no longer access the file - although I can still "see" the variable.
Is there a reason to this? For reference, I am using Python 2.7.

methods ("Functions" in python?)
"function" when they "stand free"; "methods" when they are members of a class. So, functions in your case.
What you describe does definitely work in python. Hence, my diagnosis is that you already read something from the file elsewhere before you call openAccount, so that the read pointer is not at the beginning of the file.

Pyqt and general python, can this be considered a correct approach for coding?

I have a dialog window containing check-boxes, when each of them is checked a particular class needs to be instantiated and a run a a task on a separated thread (one for each check box). I have 14 check-boxes to check the .isChecked() property and is comprehensible checking the returned Boolean for each of them is not efficient and requires a lot more coding.
Hence I decided to get all the children items corresponding to check-box element, get just those that are checked, appending their names to list and loop through them matching their name to d dictionary which key is the name of the check box and the value is the corresponding class to instantiate.
EXAMPLE:
# class dictionary
self.summary_runnables = {'dupStreetCheckBox': [DupStreetDesc(),0],
'notStreetEsuCheckBox': [StreetsNoEsuDesc(),1],
'notType3CheckBox': [Type3Desc(False),2],
'incFootPathCheckBox': [Type3Desc(True),2],
'dupEsuRefCheckBox': [DupEsuRef(True),3],
'notEsuStreetCheckBox': [NoLinkEsuStreets(),4],
'invCrossRefCheckBox': [InvalidCrossReferences()],
'startEndCheckBox': [CheckStartEnd(tol=10),8],
'tinyEsuCheckBox': [CheckTinyEsus("esu",1)],
'notMaintReinsCheckBox': [CheckMaintReins()],
'asdStartEndCheckBox': [CheckAsdCoords()],
'notMaintPolysCheckBox': [MaintNoPoly(),16],
'notPolysMaintCheckBox': [PolyNoMaint()],
'tinyPolysCheckBox': [CheckTinyEsus("rd_poly",1)]}
# looping through list
self.long_task = QThreadPool(None).globalInstance()
self.long_task.setMaxThreadCount(1)
start_report = StartReport(val_file_path)
end_report = EndReport()
# start_report.setAutoDelete(False)
# end_report.setAutoDelete(False)
end_report.signals.result.connect(self.log_progress)
end_report.signals.finished.connect(self.show_finished)
# end_report.setAutoDelete(False)
start_report.signals.result.connect(self.log_progress)
self.long_task.start(start_report)
# print str(self.check_boxes_names)
for check_box_name in self.check_boxes_names:
run_class = self.summary_runnables[check_box_name]
if run_class[0].__class__.__name__ is 'CheckStartEnd':
run_class[0].tolerance = tolerance
runnable = run_class[0]()
runnable.signals.result.connect(self.log_progress)
self.long_task.start(runnable)
self.long_task.start(end_report)
example of a runnable (even if some of them use different global functions)
I can't post the global functions that write content to file as they are too many and not all 14 tasks execute the same type function. arguments of these functions are int keys to other dictionaries that contain the report static content and the SQL queries to return report main dynamic contents.
class StartReport(QRunnable):
def __init__(self, file_path):
super(StartReport,self).__init__()
# open the db connection in thread
db.open()
self.signals = GeneralSignals()
# self.simple_signal = SimpleSignal()
# print self.signals.result
self.file_path = file_path
self.task = "Starting Report"
self.progress = 1
self.org_name = org_name
self.user = user
self.report_title = "Validation Report"
print "instantiation of start report "
def run(self):
self.signals.result.emit(self.task, self.progress)
if self.file_path is None:
print "I started and found file none "
return
else:
global report_file
# create the file and prints the header
report_file = open(self.file_path, 'wb')
report_file.write(str(self.report_title) + ' for {0} \n'.format(self.org_name))
report_file.write('Created on : {0} at {1} By : {2} \n'.format(datetime.today().strftime("%d/%m/%Y"),
datetime.now().strftime("%H:%M"),
str(self.user)))
report_file.write(
"------------------------------------------------------------------------------------------ \n \n \n \n")
report_file.flush()
os.fsync(report_file.fileno())
class EndReport(QRunnable):
def __init__(self):
super(EndReport,self).__init__()
self.signals = GeneralSignals()
self.task = "Finishing report"
self.progress = 100
def run(self):
self.signals.result.emit(self.task, self.progress)
if report_file is not None:
# write footer and close file
report_file.write("\n \n \n")
report_file.write("---------- End of Report -----------")
report_file.flush()
os.fsync(report_file.fileno())
report_file.close()
self.signals.finished.emit()
# TODO: checking whether opening a db connection in thread might affect the db on the GUI
# if db.isOpen():
# db.close()
else:
return
class DupStreetDesc(QRunnable):
"""
duplicate street description report section creation
:return: void if the report is to text
list[string] if the report is to screen
"""
def __init__(self):
super(DupStreetDesc,self).__init__()
self.signals = GeneralSignals()
self.task = "Checking duplicate street descriptions..."
self.progress = 16.6
def run(self):
self.signals.result.emit(self.task,self.progress)
if report_file is None:
print "report file is none "
# items_list = write_content(0, 0, 0, 0)
# for item in items_list:
# self.signals.list.emit(item)
else:
write_content(0, 0, 0, 0)
Now, I used this approach before and it has always worked fine without using multiprocessing. In this case it works good to some extent, I can run the tasks the first time but if I try to run for the second time I get the following Python Error :
self.long_task.start(run_class[0])
RuntimeError: wrapped C/C++ object of type DupStreetDesc has been deleted
I tried to use run_class[0].setAutoDelete(False) before running them in the loop but pyQt crashes with a minidump error (I am running the code in QGIS) and I the programs exists with few chances to understand what has happened.
On the other hand, if I run my classes separately, checking with an if else statement each check-box, then it works fine, I can run the tasks again and the C++ classes are not deleted, but it isn't a nice coding approach, at least from my very little experience.
Is there anyone else out there who can advise a different approach in order to make this run smoothly without using too many lines of code? Or knows whether there is a more efficient pattern to handle this problem, which I think must be quite common?

It seems that you should create a new instance of each runnable, and allow Qt to automatically delete it. So your dictionary entries could look like this:
'dupStreetCheckBox': [lambda: DupStreetDesc(), 0],
and then you can do:
for check_box_name in self.check_boxes_names:
run_class = self.summary_runnables[check_box_name]
runnable = run_class[0]()
runnable.signals.result.connect(self.log_progress)
self.long_task.start(runnable)
I don't know why setAutoDelete does not work (assuming you are calling it before starting the threadpool). I suppose there might be a bug, but it's impossible to be sure without having a fully-working example to test.

"Backspace" over last character written to file

I have a Python application which outputs an SQL file:
sql_string = "('" + name + "', " + age + "'),"
output_files['sql'].write(os.linesep + sql_string)
output_files['sql'].flush()
This is not done in a for loop, it is written as data becomes available. Is there any way to 'backspace' over the last comma character when the application is done running, and to replace it with a semicolon? I'm sure that I could invent some workaround by outputting the comma before the newline, and using a global Bool to determine if any particular 'write' is the first write. However, I think that the application would be much cleaner if I could just 'backspace' over it. Of course, being Python maybe there is such an easier way!
Note that having each insert value line in a list and then imploding the list is not a viable solution in this use case.

Use seek to move your cursor one byte (character) backwards, then write the new character:
f.seek(-1, os.SEEK_CUR)
f.write(";")
This is the easiest change, maintaining your current code ("working code" beats "ideal code") but it would be better to avoid the situation.

How about adding the commas before adding the new line?
first_line = True
...
sql_string = "('" + name + "', " + age + "')"
if not first_line:
output_files['sql'].write(",")
first_line = False
output_files['sql'].write(os.linesep + sql_string)
output_files['sql'].flush()
...
output_files['sql'].write(";")
output_files['sql'].flush()
You did mention this in your question - I think this is a much clearer to a maintainer than seeking commas and overwriting them.
EDIT: Since the above solution would require a global boolean in your code (which is not desirable) you could instead wrap the file writing behaviour into a helper class:
class SqlFileWriter:
first_line = True
def __init__(self, file_name):
self.f = open(file_name)
def write(self, sql_string):
if not self.first_line:
self.f.write(",")
self.first_line = False
self.f.write(os.linesep + sql_string)
self.f.flush()
def close(self):
self.f.write(";")
self.f.close()
output_files['sql'] = SqlFileWriter("myfile.sql")
output_files['sql'].write("('" + name + "', '" + age + "')")
This encapsulates all the SQL notation logic into a single class, keeping the code readable and at the same time simplifying the caller code.

Try opening the file to write as binary: 'wb' instead of 'w'.

Use generators, e.g.:
def with_separator(data, sep):
first = True:
for datum in data:
if first:
first = False
else:
yield sep
yield datum
with open("sdfasdfas", "w") as outf:
for x in with_separator(sql_get_rows(), ",\n"):
outf.write(x)
# flush if needed
For hardcore iterator use, this should get you started:
In [11]: list( itertools.imap("".join, itertools.izip(itertools.chain([""], itertools.repeat(",\n")), "abc")) )
Out[11]: ['a', ',\nb', ',\nc']
If your data uses imperative API, that is not iterable, send() your data to generator:
def write_with_separator(filename, sep):
with file(filename, "w"):
first = True
yield None
while True:
datum = yield None
if first:
first = False
else:
fout.write(sep)
fout.write(datum)
# flush if needed
writer = write_with_separator("somefilename", ",\n")
writer.next() # can't send to just-started generator
# to be called when you get data
for row in sql_get_rows():
writer.send(row)

Saving dict with nested class instances in Python 2.7

I'm trying to keep this as simple as possible. Basically I want the data to be saved to a file, and the retrieve it so that questor.py works and can "remember" everything it was ever taught on your machine. The original code is available on the web at http://www.strout.net/info/coding/python/questor.py
If I'm reading the code right, you end up with an object that looks something like {key:{key:{key:class instance},class instance},class instance} . (rough estimate)
Please ignore the unfished method Save, I'm working on that as soon as I figure out how to pickle the dictionary without losing any of the imbedded instances.
The following is my attempt at trying to save the dict via pickler. Some of the code is unfinished, but you should be able to get an idea of what I was trying to do. So far all I am able to do is retrieve the last question/answer set. Either my pickle isn't saving the imbedded instances, or they're not actually there when I save the pickle. I've followed the spaghetti lines as much as possible, but can't seem to figure out how to set up a way to save to file without losing anything.
Also my file doesn't have to be .txt originally I was going to use .data for the pickle.
# questor.py
# define some constants for future use
kQuestion = 'question'
kGuess = 'guess'
questfile = 'questfile.txt'
## Added
import cPickle as p
# create a file for questor
def questor_file():
try:
questor = open(questfile,'rb')
try:
q = p.Unpickler(questor)
quest = q.load()
questor.close()
return quest
except:
print 'P.load failed'
except:
print 'File did not open'
questor = open('questfile.data', 'wb')
questor.close()
return Qnode('python')
# define a function for asking yes/no questions
def yesno(prompt):
ans = raw_input(prompt)
return (ans[0]=='y' or ans[0]=='Y')
# define a node in the question tree (either question or guess)
class Qnode:
# initialization method
def __init__(self,guess):
self.nodetype = kGuess
self.desc = guess
##Added
## Not sure where I found this, but was going to attempt to use this as a retreival method
## haven't gotten this to work yet
def Load(self):
f = open(self.questfile,'rb')
tmp_dict = cPickle.load(f)
f.close()
self.__dict__.update(tmp_dict)
##Added
# was going to use this as a save method, and call it each time I added a new question/answer
def Save(self,node):
f = open(self.questfile,'wb')
quest = p.pickler(f)
# get the question to ask
def query(self):
if (self.nodetype == kQuestion):
return self.desc + " "
elif (self.nodetype == kGuess):
return "Is it a " + self.desc + "? "
else:
return "Error: invalid node type!"
# return new node, given a boolean response
def nextnode(self,answer):
return self.nodes[answer]
# turn a guess node into a question node and add new item
# give a question, the new item, and the answer for that item
def makeQuest( self, question, newitem, newanswer ):
# create new nodes for the new answer and old answer
newAnsNode = (Qnode(newitem))
oldAnsNode = (Qnode(self.desc))
# turn this node into a question node
self.nodetype = kQuestion
self.desc = question
# assign the yes and no nodes appropriately
self.nodes = {newanswer:newAnsNode, not newanswer:oldAnsNode}
self.save(self.nodes)
def traverse(fromNode):
# ask the question
yes = yesno( fromNode.query() )
# if this is a guess node, then did we get it right?
if (fromNode.nodetype == kGuess):
if (yes):
print "I'm a genius!!!"
return
# if we didn't get it right, return the node
return fromNode
# if it's a question node, then ask another question
return traverse( fromNode.nextnode(yes) )
def run():
# start with a single guess node
# This was supposed to assign the data from the file
topNode = questor_file()
done = 0
while not done:
# ask questions till we get to the end
result = traverse( topNode )
# if result is a node, we need to add a question
if (result):
item = raw_input("OK, what were you thinking of? ")
print "Enter a question that distinguishes a",
print item, "from a", result.desc + ":"
q = raw_input()
ans = yesno("What is the answer for " + item + "? ")
result.makeQuest( q, item, ans )
print "Got it."
# repeat until done
print
done = not yesno("Do another? ")
# Added
# give me the dictionary
return result
# immediate-mode commands, for drag-and-drop or execfile() execution
if __name__ == '__main__':
print "Let's play a game."
print 'Think of something, just one thing.'
print 'It can be anything, and I will try to guess what it is.'
raw_input('Press Enter when ready.')
print
questdata = run()
print
# Added
# Save the dictionary
questor = open(questfile,'wb')
q = p.Pickler(questor)
q.dump(questdata)
questor.close()
raw_input("press Return>")
else:
print "Module questor imported."
print "To run, type: questor.run()"
print "To reload after changes to the source, type: reload(questor)"
# end of questor.py

one way that comes to mind is creating a list of all the nodes and saving that ... they should keep their internal pointers on their own.
declare a list of nodes at the top of your file (and use pickle... just cause Im more familiar with that)
import pickle
kQuestion = 'question'
kGuess = 'guess'
questfile = 'questfile.txt'
nodes = []
....
change your load method to something like
def questor_file():
global nodes
try:
questor = open(questfile,'rb')
try:
nodes= pickle.load(questor)
quest = nodes[0]
questor.close()
return quest
except:
print 'P.load failed'
nodes = []
except:
print 'File did not open'
nodes = []
return Qnode('python')
change your class constructor so that it adds each node to nodes
class Qnode:
# initialization method
def __init__(self,guess):
self.nodetype = kGuess
self.desc = guess
nodes.append(self)
at the end where it says #added save dictionary , save your list of nodes
questor = open(questfile,'wb')
q = pickle.dump(nodes,questor)
make sure you exit the program by typing no when prompted ...
you could also save it to a database or whatever but you would still have to store each node and it might be more complicated... this method should really be fine I think , (although there may be a more natural way to save a tree structure) ...

Dictionary in python

I'm creating a server-client communication, and I want to store some information in a dictionary, so I created a Global dictionary
global commandList
commandList = {}
And when a client is connection to the server I'm trying to store some information in the following way
self.clientname = str( self.client_address )
commandList[self.clientname]['lastcommand'] = GET_SETUP
but I'm getting the following error
commandList[self.clientname]['isready'] = False
KeyError: "('134.106.74.22', 49194)"
UPDATED:
This is a part of the code.
class MCRequestHandler( SocketServer.BaseRequestHandler ):
global clientsLock, postbox, rxQueue, disconnect, isRunning, commandList
postbox = {}
rxQueue = Queue.Queue()
disconnect = {}
commandList = {}
clientsLock = threading.RLock()
isRunning = {}
def setup( self ):
clientsLock.acquire()
if len( postbox ) == 0:
self.clientname = 'MasterClient'
postbox['MasterClient'] = Queue.Queue()
mess = str( self.client_address );
postbox['MasterClient'].put( self.createMessage( MASTER_CLIENT_CONNECTED, mess ) )
print "Client name: %s" % str( self.clientname )
self.request.settimeout( 0.1 )
else:
#Send message to the master client
self.clientname = str( self.client_address )
print "Client name:%s" % self.clientname
postbox[self.clientname] = Queue.Queue()
#Setting up the last command
if not commandList.has_key( self.clientname ):
commandList[self.clientname] = {}
commandList[self.clientname]['lastcommand'] = GET_SETUP
commandList[self.clientname]['isready'] = False
self.request.settimeout( COMMANDTABLE[commandList[self.clientname]['lastcommand']]['timeout'] )
self.transNr = 0;
self.start = True;
isRunning[self.clientname] = True;
disconnect[self.clientname] = True
clientsLock.release()

Other people have already pointed out how to fix this problem, but perhaps not explained why.
What you are trying to do is create values in a nested dictionary, or, in other words, a dictionary of dictionaries.
Thus your first dictionary is called commandList with keys of self.clientname. Each of the values in this dictionary is in fact a dictionary itself- or it should be.
You are never actually telling python that these values should be dictionaries, however, and so you are getting an error. As has been pointed out, from the code you've given, this should occur in the second snippet as well.
There are lots of ways to solve this, but the easiest would be to do something like:
if not commandlist.has_key(self.clientname):
commandlist[self.clientname] = {} # Create an empty dictionary.
commandlist[self.clientname]['secondlevelkey'] = 'value'
You should perhaps be a little concerned that you are using global variables. I see that you are doing this as thread synchronisation, which is a bad idea because you are not doing any variable locking or access control to prevent dead/live locks and other synchronisation errors. Without knowing how you're using this command list, it is impossible to say how you should go about solving it.
If you give more information about the problem, we can probably advise how to solve it better.

You're trying to read value that's not in the dict. Use defaultdict, or create commandList[self.clientname] first. Also try not to use global variables.
import collections
commandList = collections.defaultdict(dict)

I guess you have to do a two-step initialization
self.clientname = str( self.client_address )
commandList[self.clientname] = {}
commandList[self.clientname]['lastcommand'] = GET_SETUP
Try now.

Are trying to insert a value into a nested directory without checking if the key for the first level exists...
if not self.clientname in commandList:
commandList[self.clientname] = dict()

To solve the problem with KeyError, you have to create a nested dictionary first. There're two simple options:
import collections
commandList = collections.defaultdict(dict)
or wiothout importing anything:
commandList.setdefault(self.clientname, {})['lastcommand'] = GET_SETUP
Depenending if your application is multithreaded or not, you could consider wrapping your dictionary with some class, so you pass the instance to threads instead of using global variable and probably adding some locking as you shouldn't relay on GIL. You could try to pass just a subdictionary to the clients as well.