Develop Reference

multiprocessing a function with parameters that are iterated through

I'm trying to improve the speed of my program and I decided to use multiprocessing!
the problem is I can't seem to find any way to use the pool function (i think this is what i need) to use my function
here is the code that i am dealing with:
def dataLoading(output):
name = ""
link = ""
upCheck = ""
isSuccess = ""
for i in os.listdir():
with open(i) as currentFile:
data = json.loads(currentFile.read())
try:
name = data["name"]
link = data["link"]
upCheck = data["upCheck"]
isSuccess = data["isSuccess"]
except:
print("error in loading data from config: improper naming or formating used")
output[name] = [link, upCheck, isSuccess]
#working
def userCheck(link, user, isSuccess):
link = link.replace("<USERNAME>", user)
isSuccess = isSuccess.replace("<USERNAME>", user)
html = requests.get(link, headers=headers)
page_source = html.text
count = page_source.count(isSuccess)
if count > 0:
return True
else:
return False
I have a parent function to run these two together but I don't think i need to show the whole thing, just the part that gets the data iteratively:
for i in configData:
data = configData[i]
link = data[0]
print(link)
upCheck = data[1] #just for future use
isSuccess = data[2]
if userCheck(link, username, isSuccess) == True:
good.append(i)
you can see how I enter all of the data in there, how would I be able to use multiprocessing to do this when I am iterating through the dictionary to collect multiple parameters?

I like to use mp.Pool().map. I think it is easiest and most straight forward and handles most multiprocessing cases. So how does map work? For starts, we have to keep in mind that mp creates workers, each worker receives a copy of the namespace (ya the whole thing), then each worker works on what they are assigned and returns. Hence, doing something like "updating a global variable" while they work, doesn't work; since they are each going to receive a copy of the global variable and none of the workers are going to be communicating. (If you want communicating workers you need to use mp.Queue's and such, it gets complicated). Anyway, here is using map:
from multiprocessing import Pool
t = 'abcd'
def func(s):
return t[int(s)]
results = Pool().map(func,range(4))
Each worker received a copy of t, func, and the portion of range(4) they were assigned. They are then automatically tracked and everything is cleaned up in the end by Pool.
Something like your dataLoading won't work very well, we need to modify it. I also cleaned the code a little.
def loadfromfile(file):
data = json.loads(open(file).read())
items = [data.get(k,"") for k in ['name','link','upCheck','isSuccess']]
return items[0],items[1:]
output = dict(Pool().map(loadfromfile,os.listdir()))

python structs for dict of files

This may be a newb question, but I'm a bit of newb with python, so here's what I'm trying to do...
I am using python2.7
I would like to assign a file path as a string into a dict in functionA, and then call this dict in functionB.
I looked at C-like structures in Python to try and use structs with no luck, possibly from a lack of understanding... The below sample is an excerpt from the link.
I also took a look at What are metaclasses in Python?, but I'm not sure if I understand metaclasses either.
So, how would I call assigned parameters in functionaA, within frunctionB such as:
class cstruct:
path1 = ""
path2 = ""
path3 = ""
def functionA():
path_to_a_file1 = os.path.join("/some/path/", "filename1.txt")
path_to_a_file2 = os.path.join("/some/path/", "filename2.txt")
path_to_a_file3 = os.path.join("/some/path/", "filename3.txt")
obj = cstruct()
obj.path1 = path_to_a_file1
obj.path2 = path_to_a_file2
obj.path3 = path_to_a_file3
print("testing string here: ", obj.path1)
# returns the path correctly here
# this is where things fall apart and the print doesn't return the string that I've tested with print(type(obj.path))
def functionB():
obj = cstructs()
print(obj.path1)
print(obj.path2)
print(obj.path3)
print(type(obj.path))
# returns <type 'str'>, which is what i want, but no path
Am I passing the parameters properly for the paths? If not, could someone please let me know what would be the right way to pass the string to be consumed?
Thanks!

You need to do something like this:
class Paths:
def __init__(self, path1, path2, path3):
self.path1 = path1
self.path2 = path2
self.path3 = path3
def functionA():
path_to_a_file1 = os.path.join("/some/path/", "filename1.txt")
path_to_a_file2 = os.path.join("/some/path/", "filename2.txt")
path_to_a_file3 = os.path.join("/some/path/", "filename3.txt")
obj = Paths(path_to_a_file1, path_to_a_file2, path_to_a_file3)
return obj
def functionB(paths): # should take a parameter
# obj = cstructs() don't do this! This would create a *new empty object*
print(paths.path1)
print(paths.path2)
print(paths.path3)
print(type(paths.path))
paths = functionA()
functionB(paths) # pass the argument
In any case, you really should take the time to read the official tutorial on classes. And you really should be using Python 3, Python 2 is passed its end of life.

Search for a string inside a text document

I'm new to python and was wondering what am i missing in my code.
I want to build a class that receives 3 letter airport destination and origin, and prints out if it's in the text file
I appreciate your help !
class departure:
def __init__(self, destfrom, destto):
self.destfrom = destfrom
self.destto = destto
def verdest(self,dest):
flag = 0
destinations = ["JFK","AMS"]
for i in destinations:
if i == dest:
flag = i
return flag
if verdest() in open('airportlist.txt').read():
print("true")

There are a few changes you need to make. if i == dest: is checking if JFK is equal to the file contents, you probably mean in. Then you have a class but you never initialize it.
class departure:
def __init__(self, destfrom, destto):
self.destfrom = destfrom
self.destto = destto
def verdest(self,dest):
flag = 0
destinations = ["JFK","AMS"]
for i in destinations:
if i in dest: # change to in
flag = i
return flag
d = departure(['BWI'],['AMS'])
f = open('airportlist.txt','r')
flag = d.verdest(f.read()) #find last airport that was in file, could modify this to return list
if flag:
print("true" + flag)
else:
print('false')
f.close() #close the file

read reads the lines of a file into a single string.
If you use readlines instead you will get a list of lines in the file.
Then you can see if an individual code is in these lines.
Without a class, like this:
def verdest(self, dest):
flag = 0 # note - not used!
destinations = open('airportlist.txt').readlines()
return dest in destinations
if verdest("LGW"):
print("true")
If you want to store the two airport names in the class and look them up in a file later one, save the three letter codes as you do, but pass the filename contents to the checking function?
class departure:
def __init__(self, destfrom, destto):
self.destfrom = destfrom
self.destto = destto
def verdest(self, destinations):
return self.destfrom in destinations and self.destto in destinations
Then make a class and use it:
places = departure("JFK","AMS")
#This makes your class, and remembers the variables in member variables
if places.verdest(open('airportlist.txt').readlines()):
#In this member function call, places remembers the member variable set up above
print("true")
Now, you could read the file in the __init__ method of the class, rather than every time you want to check.

You are missing an argument in verdest() function call.

Python function call within a property of a class

I am trying to use the "setx" function of a Property in a Class to do some processing of date information that I get from excel. I have a few of my own functions that do the data processing which I tested outside the class, and they worked just fine. But when I move them into the class they suddenly become invisible unless I use the self. instance first. When I use the self.My_xldate_as_tuple() method I get an error:
My_xldate_as_tuple() takes 1 positional argument but 2 were given
Even though the code is EXACTLY what i used outside the class before and it worked.
Before moving into the Property Set block, I was doing the processing of date data outside of the class and setting the variables from outside of the class. That gets clunky when I have about 15 different operations that are all based on when the NumDates Property change. I'm showing shortened versions of both the working set of code and the non-working set of code. What is going on with the self. call that changes how the function takes inputs?
Broken Code:
class XLDataClass(object):
_NumDates = []
TupDates = []
def getNumDates(self): return self._NumDates
def setNumDates(self, value):
self._NumDates = value
self.TupDates = list(map(self.My_xldate_as_tuple,value)) #Error here
#This version doesn't work either, since it can't find My_xldate_as_tuple anymore
self.TupDates = list(map(My_xldate_as_tuple,value))
def delNumDates(self):del self._NumDates
NumDates = property(getNumDates,setNumDates,delNumDates,"Ordinal Dates")
#exact copy of the My_xldate_as_tuple function that works outside the class
def My_xldate_as_tuple(Date):
return xlrd.xldate_as_tuple(Date,1)
#Other code and functions here
#end XlDataClass
def GetExcelData(filename,rowNum,titleCol):
csv = np.genfromtxt(filename, delimiter= ",")
NumDates = deque(csv[rowNum,:])
if titleCol == True:
NumDates.popleft()
return NumDates
#Setup
filedir = "C:/Users/blahblahblah"
filename = filedir + "/SamplePandL.csv"
xlData = XLDataClass()
#Put csv data into xlData object
xlData.NumDates= GetExcelData(filename,0,1)
Working Code:
class XLDataClass(object):
NumDates = []
TupDates = []
#Other code and functions here
#end XlDataClass
#exact copy of the same function outside of the class, which works here
def My_xldate_as_tuple(Date):
return xlrd.xldate_as_tuple(Date,1)
def GetExcelData(filename,rowNum,titleCol):
csv = np.genfromtxt(filename, delimiter= ",")
NumDates = deque(csv[rowNum,:])
if titleCol == True:
NumDates.popleft()
return NumDates
#Setup
filedir = "C:/Users/blahblahblah"
filename = filedir + "/SamplePandL.csv"
xlData = XLDataClass()
#Put csv data into xlData object
xlData.NumDates = GetExcelData(filename,0,1)
#same call to the function that was inside the Setx Property of the class, but it works here.
xlData.TupDates = list(map(self.My_xldate_as_tuple,value))

Instance methods in Python require an explicit self in the argument list. Inside the class, you need to write your method definition like:
def My_xldate_as_tuple(self, Date):

Why does my program add ('', ' to the name of my file?

Here is my code (sorry for the messy code):
def main():
pass
if __name__ == '__main__':
main()
from easygui import *
import time
import os
import random
import sys
##multenterbox(msg='Fill in values for the fields.', title=' ', fields=(), values=())
msg = "Enter your personal information"
title = "Credit Card Application"
fieldNames = ["First name",'Last name','email',"Street Address","City","State","ZipCode",'phone','phone 2)']
fieldValues = [] # we start with blanks for the values
fieldValues = multenterbox(msg,title, fieldNames)
# make sure that none of the fields was left blank
def make(x):
xys = x,".acc"
xyzasd = str(xys)
tf = open(xyzasd,'a+')
tf.writelines(lifes)
tf.writelines("\n")
tf.writelines("credits = 0")
tf.close
def add(x):
nl = "\n"
acc = ".acc"
xy = x + acc
exyz = xy
xyz = exyz
xxx = str(xyz)
tf = open('accounts.dat',"a+")
tf.writelines(nl)
tf.writelines(xxx)
tf.close
while 1:
if fieldValues == None: break
errmsg = ""
for i in range(len(fieldNames)-1):
if fieldValues[i].strip() == "":
errmsg += ('"%s" is a required field.\n\n' % fieldNames[i])
if errmsg == "":
break # no problems found
fieldValues = multenterbox(errmsg, title, fieldNames, fieldValues)
names = enterbox(msg= ('confirm FIRST name and the FIRST LETTER of the persons LAST name'))
##txt = "acc"
##na = str(name)
##name = (names)
life = ( str(fieldValues))
lifes = life,'\n'
herro = ("Reply was: %s" % str(fieldValues))
correct = buttonbox(msg=(herro,'\n is that correct'),choices = ('yes','no','cancel'))
if correct == "yes":
make(names)
add(names)
elif correct == "no":
os.system('openacc.py')
time.sleep(0.5)
sys.exit()
else:
os.system('cellocakes-main.py')
sys.exit()
os.system('cellocakes-main.py')
I don't know what the problem is also I am sorry about how sloppy it was programmed I have a white board to help me out still new to programming (I'm only 13) sorry. Personally I think the issue is in the def add area's syntax but because I am still new I don't see the issue personally I am hoping to have a more experienced programmer help me out.

This is an answer not directly answering your question.
Alas, comment fields are STILL not capable to hold formatted code, so I choose this way.
def main():
pass
if __name__ == '__main__':
main()
This is a nice coding pattern, but used by you in a useless way.
It is supposed to prevent executing of the stuff if it is imported as a module and not executed as a script.
Nevertheless, it is not bad to use it always, but then put your code inside the main() function instead of adding it below.
fieldNames = ["First name",'Last name','email',"Street Address","City","State","ZipCode",'phone','phone 2)']
There is a ) too much.
fieldValues = [] # we start with blanks for the values
fieldValues = multenterbox(msg,title, fieldNames)
The second line makes the first one useless, as you don't use fieldValues in-between.
It would be different if you expected multenterbox() to fail and would want [] as a default value.
def make(x):
xys = x,".acc"
xyzasd = str(xys)
tf = open(xyzasd,'a+')
tf.writelines(lifes)
tf.writelines("\n")
tf.writelines("credits = 0")
tf.close
You was already told about this: x, ".acc" creates a tuple, not a string. To create a string, use x + ".acc".
Besides, your close call is no call, because it is missing the (). This one just references the function and ignores the value.
A better way to write this would be (please name your variables appropriately)
with open(xyzs, 'a+') as tf:
tf.writelines(lifes)
tf.writelines("\n")
tf.writelines("credits = 0")
The with statement automatically closes the file, even if an error occurs.
Besides, you use writelines() wrong: it is supposed to take a sequence of strings and write each element to the file. As it doesn't add newlines in-between, the result looks the same,. but in your case, it writes each byte separately, making it a little bit more inefficient.
Additionally, you access the global variable lifes from within the function. You should only do such things if it is absolutely necessary.
def add(x):
Here the same remarks hold as above, plus
xy = x + acc
exyz = xy
xyz = exyz
xxx = str(xyz)
why that? Just use xy; the two assignments do nothing useful and the str() call is useless as well, as you already have a string.
for i in range(len(fieldNames)-1):
if fieldValues[i].strip() == "":
errmsg += ('"%s" is a required field.\n\n' % fieldNames[i])
Better:
for name, value in zip(fieldNames, fieldValues):
if not value.strip(): # means: empty
errmsg += '"%s" is a required field.\n\n' % name
Then:
life = ( str(fieldValues))
makes a string from a list.
lifes = life,'\n'
makes a tuple from these 2 strings.
os.system('openacc.py')
os.system('cellocakes-main.py')
Please don't use os.system(); it is deprecated. Better use the subprocess module.

The problem of the question is here:
# assign the tuple (x, ".acc") to xys
xys = x,".acc"
# now xyzasd is the tuple converted to a string, thus
# making the name of your file into '("content of x", ".acc")'
xyzasd = str(xys)
# and open file named thus
tf = open(xyzasd,'a+')
What you wanted to do is:
# use proper variable and function names!
def make_account(account):
filename = account + '.acc'
the_file = open(filename, 'a+')
....
On the other hand there are other problems with your code, for example the
def main():
pass
if __name__ == '__main__':
main()
is utterly useless.