I write a Python script to manage my account on a webpage automatically.
Code Description:
The script has a while loop and at the end of the loop, it waits 12 hours before starting again.
Before the while loop starts, it's logging in to my account, and when entering the while loop, it checks if I'm still logged in. If not, it's logging in to my account again.
Problem:
After re-entering the while loop (first time everything goes fine), the script does only work, when print("Name is:") and print(name) is at the very beginning. I tested it several times and maybe it is just a bug/glitch, which was just unlucky to be caused only when the print statements aren't there, but this is very confusing me right now, how those print statements fixed my issue. I would like to know, what is or could causing the issue and how do I have to solve it properly?
Some side info:
The webpage is saving the login credentials through session cookies with a lifetime of ~6 hours. So after re-entering the script loop again, I'm not logged in for sure. If I'm reducing the wait time to 30 minutes instead of 12 hours, the script works also without the print statements.
General notes:
The script is running through nohup on my Raspberry Pi 3
Python version is 3.7.3
Code related notes:
I'm using the post method from requests to log in to my account
For checking, if I'm still logged in, I'm using beautifulSoup4
The following code is abbreviated and in a very basic shape.
"account" is an instance of a self-made class. When instantiating, it is log in itself with arguments, if given
This is the core code:
import time
import requests
from account import Account # costum made class
from bs4 import BeautifulSoup
# login credentials
name = "lol" # I replaced them with placeholders
pw = "lol"
account = Account(name, pw) # instantiating an account class and log in itself with given arguments
while True: # script loop
print("name is:") # Without those both print statements,
print(name) # the code won't work
if not account.stillAlive(): # if not signed in anymore ...
account.login(name, pw) # ... sign in again
account.doStuff() # Do the automating stuff
time.sleep(43200) # Wait 12 hours, before entering the while loop again
This is the doStuff() method from the Account class:
def doStuff(self):
html = requests.get("example.com").text # Note: example.com is only for demonstration purpose only
crawler = BeautifulSoup(html, "lxml")
lol = crawler.find("input", attrs={"name": "Submit2"}).get("value")
# ...
Error message:
So, if I'm executing the program without the print statements, I'm getting this error:
lol = crawler.find("input", attrs={"name": "Submit2"}).get("value")
AttributeError: 'NoneType' object has no attribute 'get'
This does not occur when executing with the print statements. With the print statements, the code runs fine.
My guess
My guess is, what the memory management of Python is deleting the name variable. When entering the script loop in the first time, I'm already logged in and therefore it is skipping the account.login(name, pw) part. Since this is the only part, where name is continued to be, maybe Python is interpreting this as dead code after too many time has passed without the line to be executed, and don't see the reason to keep the name/pw variable and deletes them. Still, I'm just an amateur and I don't have any expertise in this segment.
Side notes:
This is my first question I'm submitting, if I forgot or did something wrong, pls tell me.
I already searched for this problem, but I didn't find anything similar. Maybe I just searched badly, but I searched for a few hours now. If so, I apologize. (I had to wait for every test 12 hours and since I tried it several times, you can tell, I had some time available to search)
Related
I have this code in a test helper class
#staticmethod
def setup_with_new_displayname():
""" Sets the application up for tests. 'execute_script' functions had
to be called as a workaround because issues with the webdriver ones
"""
time.sleep(5)
# the next line is where I have the problem:
TestHelper.driver.execute_script(
"document.querySelector('#displayname-input').value = 'gecó '")
TestHelper.driver.find_element_by_id("displayname-input").send_keys(
Keys.BACKSPACE) #Workaround for the field to notice input
# clicks the "Use this name" button
time.sleep(1) # otherwise the button is not always found
TestHelper.driver.execute_script(
"document.querySelector('#display-name-ok-btn').click()")
And I have this code that uses the one above:
#classmethod
def setUpClass(cls):
""" Get the display name prompt out of the way
(it is tested in test_display_name_dialog.py)
"""
TestHelper.setup_with_new_displayname()
time.sleep(3)
I use this VERY SAME seccond snippet at two places, and it works in the first case, but doesn't in the second (WTF?). The second time it opens up the page and the dialog, the focus is even on the text input, the cursor in it, but the text is not written there. (As you can see in my comments, I had problems before with the "no-execute_script" approach too).
The bash I run this from does not show any test results as it should, but is ready to take the next command, which tells me the test script is finished (well, at least it thinks so.) I even tried to print from the code as a way of debugging - no result.
Do you have any idea why this happens? And generally speaking, do you have any idea for a stabler approach for testing in Chrome? I feel I am ready to dump Webdriver, since it caused (almost) only frustration, and it's hard to see what I gained with it now.
The files are to be found here, if for some reason you would like to see more of them:
https://github.com/thenakedthunder/flack/blob/reactify/flack/src/test/selenium/test_helper.py
https://github.com/thenakedthunder/flack/blob/reactify/flack/src/test/selenium/test_channel_view.py
Im working off of the Azure Function HTTP Python Template that has this code:
import os
import json
postreqdata = json.loads(open(os.environ['req']).read())
response = open(os.environ['res'], 'w')
response.write("hello world from "+postreqdata['name'])
response.close()
This all works great. But when trying to implement it in my python script package, the response never gets sent back.
Heres how my code looks:
import os
import json
postreqdata = json.loads(open(os.environ['req']).read())
while True:
mode = 0
if mode == 0:
response = open(os.environ['res'], 'w')
response.write("Mode selected is 0, testing has begun.")
response.close()
test.testing()
As you can see my python script test.testing() is running in its own loop and runs successfully, but I never get the response back. Even if I put the "response" code last.
I simply want to call HTTP POST, which executes the script and get an OUTPUT of the
"Mode selected is 0, testing has begun."
message only once, and let the test.testing() script work its magic while in a loop.
Im pretty sure im not getting the logic right, let me know if anyone can point me in the right direction. Python current version is 2.7 and dont want to upgrade to 3 for this.
Functions are supposed to be short-lived, they shouldn't run long loops, and especially eternal loops.
The response is only sent when your function completed which, in my my understanding, never happens with the code above.
Get rid of While or add a break, and make sure the whole function execution time is below 5 minutes (or whatever your max expected response time is).
Here is the goal: a parser that reunites some information from some domains and organize them into one place.
I am a newbie with Python, having chosen to do this job with this language because of learning curve and things.
For the matter, I am doing the parsing with BeautifulSoup lib and that works like a charm. The routine is triggered via crontab in a CentOS 6, Python 2.7.
However, one of my parsing scripts sent me a log with memory error, what was causing the py file to quit without complete its job. Google here and there and found out that some very long html Python parsing would be doing my server ran out of memory. It would be better close, decompose and even garbage collect everything script would not be using there anymore.
Implemented the three things, no more memory errors in the crontab task. However, every time the script runs, I receive an email from crontab with the log of the parsing, what means that something went wrong there. Checking the databank, all the information was recorded alright, script also completed the entire task, still some error occurred, or crontab would not email me with a log.
In fact, when I run the script directly at the terminal on server, same occurs: the script won’t conclude, unless I ctrl+c it, it will be frozen in the screen. However, again, looking at the bank, all the tasks where completed without a error.
I tried work only with gc, tried only close() and only release(). Any of these three resources would freeze the screen/generate a log error (however without a error explicitaly in it).
Here is a simple version of what I am doing to better understanding:\
class GrabCategories():
def __init__(self):
target = 'http://provider-site.com/info.html'
try:
page = urllib2.urlopen(target)
if page.getcode() == 404:
print 'Page not found', target
return False
soup = BeautifulSoup(page.read())
page.close() #not using this anymore, may I close it?
except:
print 'Could not open', target
return
content = soup.find('div', {'id': 'box-content'})
soup.decompose() #not using this anymore, may I decompose it?
c=0
for link in content.findAll('a'):
#define some vars
try:
catPage = urllib2.urlopen(link['a'])
if catPage.getcode() == 404:
print 'Page not found', catPage
return False
catSoup = BeautifulSoup(catPage.read())
catPage.close() #not using this anymore, may I close it?
except:
print 'Could no open', target
continue
#do some things with the page content etc
catSoup.decompose() #not using this anymore, may I decompose it?
if(c%10):
gc.collect()
c=c+1
I'm trying to connect to a TeamSpeak server using the QueryServer to make a bot. I've taken advice from this thread, however I still need help.
This is The TeamSpeak API that I'm using.
Before the edits, this was the summary of what actually happened in my script (1 connection):
It connects.
It checks for channel ID (and it's own client ID)
It joins the channel and starts reading everything
If someone says an specific command, it executes the command and then it disconnects.
How can I make it so it doesn't disconnect? How can I make the script stay in a "waiting" state so it can keep reading after the command is executed?
I am using Python 3.4.1.
I tried learning Threading but either I'm dumb or it doesn't work the way I thought it would. There's another "bug", once waiting for events, if I don't trigger anything with a command, it disconnects after 60 seconds.
#Librerias
import ts3
import threading
import datetime
from random import choice, sample
# Data needed #
USER = "thisisafakename"
PASS = "something"
HOST = "111.111.111.111"
PORT = 10011
SID = 1
class BotPrincipal:
def __init__(self, manejador=False):
self.ts3conn = ts3.query.TS3Connection(HOST, PORT)
self.ts3conn.login(client_login_name=USER, client_login_password=PASS)
self.ts3conn.use(sid=SID)
channelToJoin = Bot.GettingChannelID("TestingBot")
try: #Login with a client that is ok
self.ts3conn.clientupdate(client_nickname="The Reader Bot")
self.MyData = self.GettingMyData()
self.MoveUserToChannel(ChannelToJoin, Bot.MyData["client_id"])
self.suscribirEvento("textchannel", ChannelToJoin)
self.ts3conn.on_event = self.manejadorDeEventos
self.ts3conn.recv_in_thread()
except ts3.query.TS3QueryError: #Name already exists, 2nd client connect with this info
self.ts3conn.clientupdate(client_nickname="The Writer Bot")
self.MyData = self.GettingMyData()
self.MoveUserToChannel(ChannelToJoin, Bot.MyData["client_id"])
def __del__(self):
self.ts3conn.close()
def GettingMyData(self):
respuesta = self.ts3conn.whoami()
return respuesta.parsed[0]
def GettingChannelID(self, nombre):
respuesta = self.ts3conn.channelfind(pattern=ts3.escape.TS3Escape.unescape(nombre))
return respuesta.parsed[0]["cid"]
def MoveUserToChannel(self, idCanal, idUsuario, passCanal=None):
self.ts3conn.clientmove(cid=idCanal, clid=idUsuario, cpw=passCanal)
def suscribirEvento(self, tipoEvento, idCanal):
self.ts3conn.servernotifyregister(event=tipoEvento, id_=idCanal)
def SendTextToChannel(self, idCanal, mensajito="Error"):
self.ts3conn.sendtextmessage(targetmode=2, target=idCanal, msg=mensajito) #This works
print("test") #PROBLEM HERE This doesn't work. Why? the line above did work
def manejadorDeEventos(sender, event):
message = event.parsed[0]['msg']
if "test" in message: #This works
Bot.SendTextToChannel(ChannelToJoin, "This is a test") #This works
if __name__ == "__main__":
Bot = BotPrincipal()
threadprincipal = threading.Thread(target=Bot.__init__)
threadprincipal.start()
Prior to using 2 bots, I tested to launch the SendTextToChannel when it connects and it works perfectly, allowing me to do anything that I want after it sends the text to the channel. The bug that made entire python code stop only happens if it's triggered by the manejadorDeEventos
Edit 1 - Experimenting with threading.
I messed it up big time with threading, getting to the result where 2 clients connect at same time. Somehow i think 1 of them is reading the events and the other one is answering. The script doesn't close itself anymore and that's a win, but having a clone connection doesn't looks good.
Edit 2 - Updated code and actual state of the problem.
I managed to make the double connection works more or less "fine", but it disconnects if nothing happens in the room for 60 seconds. Tried using Threading.timer but I'm unable to make it works. The entire question code has been updated for it.
I would like an answer that helps me to do both reading from the channel and answering to it without the need of connect a second bot for it (like it's actually doing...) And I would give extra points if the answer also helps me to understand an easy way to make a query to the server each 50 seconds so it doesn't disconnects.
From looking at the source, recv_in_thread doesn't create a thread that loops around receiving messages until quit time, it creates a thread that receives a single message and then exits:
def recv_in_thread(self):
"""
Calls :meth:`recv` in a thread. This is useful,
if you used ``servernotifyregister`` and you expect to receive events.
"""
thread = threading.Thread(target=self.recv, args=(True,))
thread.start()
return None
That implies that you have to repeatedly call recv_in_thread, not just call it once.
I'm not sure exactly where to do so from reading the docs, but presumably it's at the end of whatever callback gets triggered by a received event; I think that's your manejadorDeEventos method? (Or maybe it's something related to the servernotifyregister method? I'm not sure what servernotifyregister is for and what on_event is for…)
That manejadorDeEventos brings up two side points:
You've declared manejadorDeEventos wrong. Every method has to take self as its first parameter. When you pass a bound method, like self.manejadorDeEventos, that bound self object is going to be passed as the first argument, before any arguments that the caller passes. (There are exceptions to this for classmethods and staticmethods, but those don't apply here.) Also, within that method, you should almost certainly be accessing self, not a global variable Bot that happens to be the same object as self.
If manejadorDeEventos is actually the callback for recv_in_thread, you've got a race condition here: if the first message comes in before your main threads finishes the on_event assignment, the recv_on_thread won't be able to call your event handler. (This is exactly the kind of bug that often shows up one time in a million, making it a huge pain to debug when you discover it months after deploying or publishing your code.) So, reverse those two lines.
One last thing: a brief glimpse at this library's code is a bit worrisome. It doesn't look like it's written by someone who really knows what they're doing. The method I copied above only has 3 lines of code, but it includes a useless return None and a leaked Thread that can never be joined, not to mention that the whole design of making you call this method (and spawn a new thread) after each event received is weird, and even more so given that it's not really explained. If this is the standard client library for a service you have to use, then you really don't have much choice in the matter, but if it's not, I'd consider looking for a different library.
I'm creating an instant messenger program for my school's common drive. I have everything working except for on small detail. In the code below it checks for a new message from a friend and prints the last message they sent. If there are no messages it says so. The problem is when it moves to the next step of the code it waits for the user to put in an input. Until you give an input it won't let you receive any more messages because the program stops reading and searching the while loop and gets caught on the input statement. I want to know if there is anyway to make an input statement optional. To say that it doesn't require an input but if there is an input it will send it and do it's thing. I just can't seem to figure out a way to make the input statement optional. Any ideas or working code would be greatly appreciated. If you need the entire code I don't have a problem with sending it to you or posting it. This is the only bit of code that should really matter for this problem though.
LastMessage = ""
while Message:
Path = "Message"+SendTo+"-"+UserName+".txt"
if path.isfile(Path):
LoadMessage = open(Path, "rb")
NewMessage = pickle.load(LoadMessage)
LoadMessage.close()
else:
NewMessage = "Sorry, No messages found"
if LastMessage != NewMessage:
LastMessage = NewMessage
print(NewMessage)
print("")
SendMessage = raw_input() #--- This is where it gets caught up! ---
Save = open("Message"+UserName+"-"+SendTo+".txt", "wb")
pickle.dump(SendMessage, Save)
Save.close()
You have two main options as I see it:
Simultaneous input and checking (various options, search for e.g. threading or multiprocessing from the standard library); or
Input with timeout and loop (see e.g. How to set time limit on raw_input).
So it sounds like you want to do two separate things at the same time - look for input from a user and poll for new messages from other users. Jonrsharpe gives threading as his first option to solve this and I agree its the most straightforward. What you need to do is something like this:
import threading
class InputMessageThread(threading.Thread):
def run(self):
SendMessage = raw_input() # This thread will hang here for input but thats
# OK as original thread will keep going
Save = open("Message"+UserName+"-"+SendTo+".txt", "wb")
pickle.dump(SendMessage, Save)
Save.close()
inputthread = InputMessageThread()
inputthread.start()
# rest of your code here
While you are at it though you might want to look at some other issues. For example if I understand what you are trying to do correctly you are going to have a file containing a message from a source user to a destination user. But if the source user sends a second message before this file gets processed then the first message will be overwritten. In practice you may never see this but some sort of handshaking to make sure the message has actually been sent before you allow the next to be written would be a more robust approach.