I know that this question looks exactly like so many other on here, because I just read them all and they all say to do what I already tried, and it hasn't worked (or I'm missing a subtle difference with my situation). Here is my situation:
I am writing a scraper using Scrapy and Python 2.7.11, and my code looks like this (this is a copy and paste with irrelevant lines omitted, but I can re-add them upon request):
class LbcSubtopicSpider(scrapy.Spider):
...omitted...
rawTranscripts = []
rawTranslations = []
def parse(self, response):
#global rawTranscripts, rawTranslations
rawTitles = []
rawVideos = []
for sel in response.xpath('//ul[1]'): #only scrape the first list
...omitted...
index = 0
for sub in sel.xpath('li/ul/li/a'): #scrape the sublist items
index += 1
if index%2!=0: #odd numbered entries are the transcripts
transcriptLink = sub.xpath('#href').extract()
#url = response.urljoin(transcriptLink[0])
#yield scrapy.Request(url, callback=self.parse_transcript)
else: #even numbered entries are the translations
translationLink = sub.xpath('#href').extract()
url = response.urljoin(translationLink[0])
yield scrapy.Request(url, callback=self.parse_translation)
print rawTitles
print rawVideos
print rawTranslations
def parse_translation(self, response):
global rawTranslations
for sel in response.xpath('//p[not(#class)]'):
rawTranslation = sel.xpath('text()').extract()
rawTranslations.append(rawTranslation)
This will return an error any time either "print rawTranslations" or "rawTranslations.append(rawTranslation)" is called because the global "rawTranslations" is not defined.
As I said before, I have looked into this pretty extensively and pretty much everyone on the internet says to just add a "global (name)" line to the beginning of any function you'd use/modify it in (although I'm not assigning to it ever, so I shouldn't even need this). I get the same result whether or not my global lines are commented out. This behavior seems to defy everything I've read about how globals work in Python, so I suspect this might be a Scrapy quirk related to how parse functions are called through scrapy.Request(....).
Apologies for posting what appears to be the same question you've seen so much yet again, but it seems to be a bit twisted this time around and hopefully someone can get to the bottom of it. Thanks.
In your case the variable you want to access is not global, it is in the scope of the class.
global_var = "global"
class Example:
class_var = "class"
def __init__(self):
self.instance_var = "instance"
def check(self):
print(instance_var) # error
print(self.instance_var) # works
print(class_var) # error
print(self.class_var) # works, lookup goes "up" to the class
print(global_var) # works
print(self.global_var) # works not
You only need the global keyword if you want to write to a global variable. Hint: Don't do that because global variables that are written to bring nothing but pain and despair. Only use global variables as (config) constants.
global_var = "global"
class Example:
def ex1(self):
global_var = "local" # creates a new local variable named "global_var"
def ex2(self):
global global_var
global_var = "local" # changes the global variable
Example().ex1()
print(global_var) # will still be "global"
Example().ex2()
print(global_var) # willnow be "local"
if you want to use variable in class, you can use self.xxx
class A:
... var = []
... def test(self):
... self.var.append(10)
... print self.var
Related
My question is tied to Dan Bader's schedule package. From what I understand, you schedule function calls. That's pretty simple if you are defining the function in your script. However, what about functions that are built dynamically using exec()? Is there any way to make these callable? I keep getting errors when trying to schedule these functions. I recognize that this is likely not the best idea (maybe not even a good idea), but this is just for a POC and I'm still curious if this can be done.
def buildJob(lang, repType, name, file='', recipient='', server = '', db='', path=''):
today = datetime.datetime.strftime(datetime.datetime.today(), '%m%d%Y%H%M%S')
filePath = f"{c.path}{name}-{today}".replace('\\', '/')
filename = f'{name}-{today}.xlsx'
funcText = f"""
def {name}():
sql = ("{file}")
filePath = ("{filePath}")
engine = sa.create_engine(conString)
dat = pd.read_sql_query(sql, engine)
engine.dispose()
del engine
buildSpreadsheet(dat, filePath)
sendSpreadsheet("{recipient}", "{filePath}.xlsx", "{filename}")
"""
I then have a function to grab the funcText and exec() it. However, when I pass this into schedule, it says that the argument must be callable.
Any help would be greatly appreciated!
You can retrieve defined functions with the locals() and globals() dicts.
# in global scope
as_str = f"""
def foo():
print('foo')
"""
exec(as_str)
foo = globals()['foo']
foo()
# in function scope
def bar():
as_str = f"""
def baz():
print('baz')
"""
exec(as_str)
return locals()['baz']
baz = bar()
baz()
Someone may correct me but dynamic function creation with exec seems like a bad idea. Especially if the input isn't being sanitized.
I am new to python and test complete just trying to automate a sample webpage but stuck with below error.
I have searched and know that I am surely making mistakes in defining the variables.
have recorded the below script and it runs fine if I keep it one function but when I keep it in separate functions(to login into page have kept the code in Login() function and calling it in Test1() ) it fails though it login into the page .
global browser,page
def Login():
Browsers.Item[btChrome].Navigate("http://secure.smartbearsoftware.com/samples/testcomplete11/WebOrders/login.aspx")
browser=Aliases.browser
page = browser.pageWebOrdersLogin
form = page.formAspnetform
form.textboxUsername.Keys("[Enter]")
page.Wait()
textbox = form.textboxUsername2
textbox.SetText("Tester")
textbox.Keys("[Tab]")
passwordBox = form.passwordboxPassword
passwordBox.SetText(Project.Variables.Password1)
passwordBox.Keys("[Enter]")
page = browser.pageDefault
page.Wait()
def Test1():
global page
Login()
page.formAspnetform.link.Click()
page = browser.pageProcess
page.Wait()
form = page.formAspnetform
form.selectProduct.ClickItem("FamilyAlbum")
textbox = form.textboxQuantity
textbox.SetText("40")
form.submitbuttonCalculate.ClickButton()
textbox = form.textboxCustomerName
textbox.SetText("nitin")
textbox.Keys("[Tab]")
textbox = form.textboxStreet
textbox.SetText("marvel")
textbox.Keys("[Tab]")
textbox = form.textboxCity
textbox.SetText("pune")
textbox.Keys("[Tab]")
textbox = form.textboxState
textbox.SetText("maharashta")
textbox.Keys("[Tab]")
form.textboxZip.SetText("411014")
cell = form.cell
cell.radiobuttonVisa.ClickButton()
textbox = form.textboxCardNr
textbox.SetText("411882781991")
textbox = form.textboxExpireDateMmYy
textbox.SetText("01/23")
form.linkInsertbutton.Click()
page.Wait()
textNode = page.textnode
aqObject.CheckProperty(textNode, "contentText", cmpEqual, "New order has been successfully added.")
page.link.Click()
browser.pageDefault2.Wait()
Error:
Python runtime error.
NameError: name 'page' is not defined
Error location:
Unit: "WebTesting\WebTesting\Script\WebTest"
Line: 22 Column: 1.
The global declaration for page needs to be repeated inside the def Login; but a much better design is to pass non-global variables between these functions. Either
def login():
browser = ...
page = ...
...
return browser, page
def test1():
browser, page = login()
...
or perhaps conversely have the caller define and pass these in;
def login(browser, page):
...
def test1()
browser = ...
page = ...
login(browser, page)
...
Your current design calls for the former, but both patterns are common, and the second is perhaps more logical. Generally, try to define variables in the context where they are going to be used, and then pass them down into other functions as necessary. As a rule of thumb, try to make your variables as narrowly scoped and short-lived as possible.
Notice also how we usually do not capitalize function names in Python.
My app has a simple process, the user uploads a gif file and then the gif is conveted to frames that are saved as objects. For the upload_to part of my gif, I run a function content_file_name() that uses uuid to create a folder path. I want the image frames to be saved to the same folder as the gif. Problem is, I can't set the path as a variable as much as I try. The variable needs to be defined first, but if I define it, it doesn't change no matter what I do. Here's what I got now:
currentFilePath = '' # Defining the variable
def content_file_name(instance, filename):
ext = ''.join(filename.split())[:-4]
foldername = "%s/%s" % (uuid.uuid4(), ext)
store_random_path('/'.join(['documents', str(foldername)])) # Running a function to change the variable
return '/'.join(['documents', str(foldername), filename])
def store_random_path(path):
currentFilePath = str(path) # The variable should be changed here
class Document(models.Model):
docfile = models.ImageField(upload_to=content_file_name)
def create_documentfiles(self):
gif = Image.open(self.docfile.path)
frames = [frame.copy() for frame in ImageSequence.Iterator(gif)]
basename, _ext = os.path.splitext(self.docfile.name)
for index, frame in enumerate(frames):
buffer = BytesIO()
frame.convert('RGB').save(fp=buffer, format='JPEG')
destname = "{}{}.png".format(basename, index)
finalImage = InMemoryUploadedFile(buffer, None, destname, 'image/jpeg', frame.tell, None)
imageToSave = DocumentImage(imagefile=finalImage)
imageToSave.save()
class DocumentImage(models.Model):
imagefile = models.ImageField(upload_to=currentFilePath) # Trying to save using the variable as path
image = models.ForeignKey(Document, related_name='Image', null=True, on_delete=models.CASCADE)
So, when the DocumentImage gets saved, it should have seen the variable as path, but it doesn't. Instead it just saves to the initially declared '' which is the root of my media file. Not sure if what I'm trying is possible in Python/Django. I've just started learing Python a month ago. Thank you for your time.
You are assigning to a global variable inside a function. You need to use the global keyword for that or you could refactor your code to not use any global variable (see Why are global variables evil?). This is not specific to Django.
The wrong way
global_var = ''
def function():
global_var = 'hello' # Local variable created due to assignment
Here global_var is still ''
The Correct way
global_var = ''
def function():
global global_var
global_var = "hello" # Assignment to the global variable
Here global_var is "Hello"
But all you've done is to set a local variable inside store_random_path. You haven't even returned that variable, let alone changed the global variable with the same name.
But you must not use global variables like this anyway. That would cause serious bugs as any future request would get the same value of the global variable. Don't do this.
I'm not sure why you are trying to set this variable in the first place. You have access to the Document, which contains the docfile which has its path correctly set. You should use that to calculate the path of the frames.
My code keeps giving me the above syntax warning. My code includes, but is not limited to:
class SocketHandler(WebSocketHandler):
def on_message(self, message):
ball = json.loads(message)
user = User(ball['identifier'])
if ball['command'] == 'search':
global our_mongo
search_results = our_mongo.find({'$text':{'$search':ball['search_term']}},{'score':{'$meta':"textScore"}})
self.write_message({
'command': 'search-results',
'results': list(search_results.sort([('score', {'$meta': 'textScore'})]).limit(10)),
})
elif ball['command'] == 'save-node': # hopefully this can handle both new nodes and changes to nodes
node_dict = ball['node_dict']
if 'importance' in node_dict.keys():
node_dict['importance'] = int(node_dict['importance'])
try:
node_obj = node.create_appropriate_node(node_dict)
print('node made. looks like: '+str(node_obj)+'. Now time to put it into the DB...')
global our_mongo
# take a look at the dependencies now
previous_dependency_ids = [node.reduce_string(dependency) for dependency in list(our_mongo.find({"_id": node_obj.id}))[0]["_dependencies"]] # if this works, try using set() instead of list and elimination the set()s below
print('prev deps are: '+str(previous_dependency_ids))
our_mongo.upsert({ "_id": node_obj.id }, node_obj.__dict__)
# take a look at the current dependencies
current_dependency_ids = [node.reduce_string(dependency) for dependency in list(our_mongo.find({"_id": node_obj.id}))[0]["_dependencies"]] # if this works, try using set() instead of list and elimination the set()s below
print('curr deps are: '+str(current_dependency_ids))
update_our_DAG()
# send an update of the graph to the user if there is a new dependency:
for new_dependency in set(current_dependency_ids) - set(previous_dependency_ids):
self.request_node(new_dependency, ball, user)
# OR IF THE SAVED NODE IS A BRAND NEW NODE, we have to include all the deps
except Exception as error:
# stuff didn't work, send error back to user
print('ERROR: '+str(error))
self.write_message({
'command': 'display-error',
'message': str(error),
})
def update_our_DAG():
# 1. grab nodes and edges from database
all_node_dicts = list(Mongo("math", "nodes").find())
# 2. create a networkx graph with the info...
global our_DAG
our_DAG = nx.DAG()
for node_dict in all_node_dicts:
node = create_appropriate_node(strip_underscores(node_dict))
our_DAG.add_n(node)
def make_app():
return Application(
[
url('/', RedirectHandler, {"url": "index.html"}, name="rooth"),
url('/websocket', SocketHandler),
url('/json', JSONHandler, name="jsonh"),
url(r'/index(?:\.html)?', IndexHandler, name="indexh"),
# captures anything at all, and serves it as a static file. simple!
url(r'/(.*)', StaticHandler, {"path": "../www/"}),
],
# settings:
debug=True,
)
def make_app_and_start_listening():
# enable_pretty_logging()
application = make_app()
# by listening on the http port (default for all browsers that i know of),
# user will not have to type "http://" or ":80" in the URL
application.listen(80)
# other stuff
IOLoop.current().start()
if __name__ == "__main__":
# 0. create a global mongo object for later use (for upserts in the future)
our_mongo = Mongo("math", "nodes")
# 1 and 2
update_our_DAG()
axioms = [our_DAG.n(node_id) for node_id in ['set', 'multiset', 'vertex']]
# 3. launch!
make_app_and_start_listening()
I've tried everything and it's still an issue. WHY is this a syntax error? What is the best way to fix this?
It is not a SyntaxError but a SyntaxWarning. Python warns that you're using our_mongo variable before a global our_mongo within the function.
Syntactically it does not actually matter on which line of the function the global statement is; but the idiomatic way would be to use global before the first access.
Another issue altogether is that you have multiple global our_mongo statements, where single would do, and also that you do not even need the global at all - it is only for the case when you want to assign a new value to the global variable; i.e.
def foo():
global bar # this line is required for the next line
# to change the global variable instead of
# the local variable
bar = 42
Thus, just remove all global statements altogether from your on_message, they're unnecessary.
The only reason a name is declared global in a function is so that it will be bound in global scope instead of local scope; when accessing a name the global scope is always searched regardless. Since your code never binds our_mongo within any function there is no reason to declare it global in the first place.
def get(self):
if self.request.get('fmt')=='json':
KeyofQuestion = self.request.path[1:]
QuestionText = Question.get_by_key_name(KeyofQuestion).question
AnswersQuery = Question.get_by_key_name(KeyofQuestion).answers_collection
a=[]
Jsonobject = {'Question':QuestionText}
for each in AnswersQuery:
a = a.append(each.answer)
Hey, i am just confused that when I run the codes above, I got an error that says, Nonetype variable:a doesnt have method append, but I declared the a as a list before I called and they are inside the same function "get", so I assumed they are all treated as local variables. How come it cant map it? Thank you
You are assigning None to a. Change this:
a = a.append(each.answer)
to:
a.append(each.answer)