I've seen a few instances of this question, but I was not sure how to apply the changes to my particular situation. I have code that monitors a webpage for changes and refreshes every 30 seconds, as follows:
import sys
import ctypes
from time import sleep
from Checker import Checker
USERNAME = sys.argv[1]
PASSWORD = sys.argv[2]
def main():
crawler = Checker()
crawler.login(USERNAME, PASSWORD)
crawler.click_data()
crawler.view_page()
while crawler.check_page():
crawler.wait_for_table()
crawler.refresh()
ctypes.windll.user32.MessageBoxW(0, "A change has been made!", "Attention", 1)
if __name__ == "__main__":
main()
The problem is that Selenium will always show an error stating it is unable to locate the element after the first refresh has been made. The element in question, I suspect, is a table from which I retrieve data using the following function:
def get_data_cells(self):
contents = []
table_id = "table.datadisplaytable:nth-child(4)"
table = self.driver.find_element(By.CSS_SELECTOR, table_id)
cells = table.find_elements_by_tag_name('td')
for cell in cells:
contents.append(cell.text)
return contents
I can't tell if the issue is in the above function or in the main(). What's an easy way to get Selenium to refresh the page without returning such an error?
Update:
I've added a wait function and adjusted the main() function accordinly:
def wait_for_table(self):
table_selector = "table.datadisplaytable:nth-child(4)"
delay = 60
try:
wait = ui.WebDriverWait(self.driver, delay)
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, table_selector)))
except TimeoutError:
print("Operation timeout! The requested element never loaded.")
Since the same error is still occurring, either my timing function is not working properly or it is not a timing issue.
I've run into the same issue while doing web scraping before and found that re-sending the GET request (instead of refreshing) seemed to eliminate it.
It's not very elegant, but it worked for me.
I appear to have fixed my own problem.
My refresh() function was written as follows:
def refresh():
self.driver.refresh()
All I did was switch frames right after the refresh() call. That is:
def refresh():
self.driver.refresh()
self.driver.switch_to.frame("content")
This took care of it. I can see that the page is now refreshing without issues.
Related
The code I wrote is very basic and it simply works. It takes a value on the Selenium-related site, writes it to a txt, then reads the necessary part of the value and should sound an alarm according to this value. The code terminates before reaching the alarm part or it does not see the alarm part. The problem here may be related to the value I got from the txt, but I could not solve the problem despite my attempts. How can I solve this?
note:There is no problem with the vlc library, it works when used separately and in this example the value in the txt is 12 feb 2022 and it only reads the first character
from selenium import webdriver
import time
import vlc
driver = webdriver.Chrome()
driver.get("https://demoqa.com/automation-practice-form")
driver.maximize_window()
print("Site Title:",driver.title)
#####################################################
nameElement =driver.find_element_by_id("dateOfBirthInput")
nameElement.click()
time.sleep(5)
taleptAttribute = nameElement.get_attribute('value')
print(taleptAttribute)
#print("Talep Sayısı: " + nameElement.get_attribute('value'))
################################################################
talep_satırı = open("talep_satiri.txt", "w")
talep_satırı.write(taleptAttribute)
talep_satırı = open("talep_satiri.txt","r")
talepsayisi=talep_satırı.read(1)
print(talepsayisi)
alarm = vlc.MediaPlayer("path")
if (talepsayisi == 1 ):
alarm.play()
time.sleep(10)
alarm.stop()
else:
alarm.play()
I want to make a script where there are a few functions, the first is the add_cart which by name attempts to add an item to cart using proper cookies/headers because I can get a response which I get that ["error"] to print the log which will say retrying cart but the script suddenly stops even if I put the add_cart() function on the bottom but I also want to use the datetime module to time.sleep(2) before running the add_cart() function so confused how I could get this all up and running, I attached images below with my code it currently gets a response because it's printing in the terminal but I want to find out what I said above. Thanks!
Image (All code with headers, cookies, and payload minimized)
https://i.imgur.com/2jGwAeA.png
Please inform if something is wrong or any way I can fix my formatting? Again all headers, cookies, payload, requests url and responses is right trying to fix my other errors though
this is the full code since the bot said add it:
import json
from datetime import datetime
import time
import os
cookies = {
}
headers = {
}
atcPayload = {
}
def add_cart(cookies, headers, atcPayload):
response = requests.post('apiurl', headers=headers, cookies=cookies, data=atcPayload)
data = response.json()
print("adding")
if data["error"] == 'true':
print("retrying cart")
#(cookies/stuff hidden because it's a private project but that isn't the issue anyways)
Seems to be another issue where it won't run now in the visual studio code terminal either :(
If I understand you problem correctly, you would like to implement a loop, where you call your function, then wait 2 seconds, and do it indefinetly.
*edit
I hope I get it right now :)
Modified the code based on your comments, now it wait until "ok" returns from your function.
import time
import sys
from datetime import datetime,timedelta
Start_Time = datetime.now()
# Your function
def add_cart():
# This is for make some timing response
global Start_Time
Return_Value = "retry cart"
Current_Time = datetime.now()
# Just print out the current time
print(Current_Time)
# Get 10 second delay on status change
if Current_Time>(Start_Time+timedelta(seconds=10)):
Return_Value = "ok"
return Return_Value
# The main function
def main():
# For the response from the function
Response = None
# This will make an infinite loop
while True:
# Call your function, it will return "rety cart" until 10 seconds not pass, then it returns "ok"
Response = add_cart()
print(Response)
# Wait 2 second
time.sleep(2)
# Check the response and break out from the while loop
if ("ok" == Response):
break
# This will run if you run your file, and not run if you import it (for later use)
if __name__ == "__main__":
try:
# Run the main function defined above
main()
# If you want to interrupt the script press CTRL+c and the below part will catch it
except KeyboardInterrupt:
print("Interrupted")
sys.exit(0)
I want to create a program, that can read all the messages from my whatsApp and print them to the screen using python.
In order to do that I tried using the whatsapp-web library https://pypi.org/project/whatsapp-web/.
But when i tried to run their code example I got a timeout error
this is the code
import time
from selenium import webdriver
from simon.accounts.pages import LoginPage
from simon.header.pages import HeaderPage
from simon.pages import BasePage
# Creating the driver (browser)
driver = webdriver.Firefox()
driver.maximize_window()
login_page = LoginPage(driver)
login_page.load()
login_page.remember_me = False
time.sleep(7)
base_page = BasePage(driver)
base_page.is_welcome_page_available()
base_page.is_nav_bar_page_available()
base_page.is_search_page_available()
base_page.is_pane_page_available()
base_page.is_chat_page_available()
# 3. Logout
header_page = HeaderPage(driver)
header_page.logout()
# Close the browser
driver.quit()
and this is the error
base_page.is_welcome_page_available()
File "D:\zoom\venv\lib\site-packages\simon\pages.py", line 18, in wrapper
return func(*args, **kwargs)
File "D:\zoom\venv\lib\site-packages\simon\pages.py", line 51, in is_welcome_page_available
if self._find_element(WelcomeLocators.WELCOME):
File "D:\zoom\venv\lib\site-packages\simon\pages.py", line 77, in _find_element
lambda driver: self.driver.find_element(*locator))
File "D:\zoom\venv\lib\site-packages\selenium\webdriver\support\wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
Your code is not completed to find error or correct.
Because the error is coming from the imported page.
Try by increasing the time limit to load the page,
use time.sleep(15)
You can try to automate the WhatsApp web by yourself without using pip of WhatsApp-web. because I guess that is not updated.
The code is giving error because of WhatsApp web has changed its elements classes names. Because of that the code section is not able to find welcome page in given time limit. So the program execution breaks
I have done the same without using WhatsApp-web pip.
I hope it will work for you.
A complete code reference Example :
from selenium import webdriver
import time
# You can use any web-browser which supported by selenium and which can run WhatsApp web.
# For using GoogleChrome
web_driver = webdriver.Chrome("Chrome_Driver_Path/chromedriver.exe")
web_driver.get("https://web.whatsapp.com/")
# For using Firefox
# web_driver = webdriver.Firefox(executable_path=r"C:/Users/Pascal/Desktop/geckodriver.exe")
# web_driver.get("https://web.whatsapp.com/")
time.sleep(25) # For scan the qr code
# Plese make sure that you have done the qr code scan successful.
confirm = int(input("Press 1 to proceed if sucessfully login or press 0 for retry : "))
if confirm == 1:
print("Continuing...")
elif confirm == 0:
web_driver.close()
exit()
else:
print("Sorry Please Try again")
web_driver.close()
exit()
while True:
unread_chats = web_driver.find_elements_by_xpath("// span[#class='_38M1B']")
# In the above line Change the xpath's class name from the current time class name by inspecting span element
# which containing the number of unread message showing the contact card inside a green circle before opening the chat room.
# Open each chat using loop and read message.
for chat in unread_chats:
chat.click()
time.sleep(2)
# For getting message to perform action
message = web_driver.find_elements_by_xpath("//span[#class='_3-8er selectable-text copyable-text']")
# In the above line Change the xpath's class name from the current time class name by inspecting span element
# which containing received text message of any chat room.
for i in message:
try:
print("Message received : " + str(i.text))
# Here you can use you code to perform action according to your need
except:
pass
Please make sure that the indentation is equal in code blocks if you are copying it.
Can following link for more info about WhatsApp web using python.
https://stackoverflow.com/a/68288416/15284163
I am developing WhatsApp bot using python.
For contribution you can contact at : anurag.cse016#gmail.com
Please give a star on my https://github.com/4NUR46 If this Answer helps you.
I'm working on a web scraper that needs to open several thousand pages and get some data.
Since one of the data fields I need the most is only loaded after all javascripts of the site have been loaded, I'm using html-requests to render the page and then get the data I need.
I want to know, what's the best way to do this?
1- Open a session at the start of the script, do my whole scraping and then close the session when the script finishes after thousands of "clicks" and several hours?
2- Or should I open a session everytime I open a link, render the page, get the data, and then close the session, and repeat n times in a cycle?
Currently I'm doing the 2nd option, but I'm getting a problem. This is the code I'm using:
def getSellerName(listingItems):
for item in listingItems:
builtURL = item['href']
try:
session = HTMLSession()
r = session.get(builtURL,timeout=5)
r.html.render()
sleep(1)
sellerInfo = r.html.search("<ul class=\"seller_name\"></ul></div><a href=\"{user}\" target=")["user"]
##
##Do some stuff with sellerinfo
##
session.close()
except requests.exceptions.Timeout:
log.exception("TimeOut Ex: ")
continue
except:
log.exception("Gen Ex")
continue
finally:
session.close()
break
This works pretty well and is quite fast. However, after about 1.5 or 2 hours, I start getting OS exception like this one:
OSError: [Errno 24] Too many open files
And then that's it, I just get this exception over and over again, until I kill the script.
I'm guessing I need to close something else after every get and render, but I'm not sure what or if I'm doing it correctly.
Any help and/or suggestions, please?
Thanks!
You should create a session object outside the loop
def getSellerName(listingItems):
session = HTMLSession()
for item in listingItems:
//code
I've compiled this code to perform an iteration of downloads from a webpage which has multiple download links. Once the download link is clicked, the webpage produces a webform which has to be filled and submitted for the download to start. I've tried running the code and face issue in 'try'& 'except' block code (Error: Too broad exception clause) and towards the end there is an error associated with the 'submit' (Error: method submit maybe static) both of these subsequently result in 'SyntaxError: invalid syntax '. Any suggestions / help will be much appreciated. Thank you.
import os
from selenium import webdriver
fp = webdriver.FirefoxProfile()
fp.set_preference("browser.download.folderList",2)
fp.set_preference("browser.download.manager.showWhenStarting",False)
fp.set_preference("browser.download.dir", os.getcwd())
fp.set_preference("browser.helperApps.neverAsk.saveToDisk", "application/x-msdos-program")
driver = webdriver.Firefox(firefox_profile=fp)
driver.get('http://def.com/catalog/attribute')
#This is to find the download links in the webpage one by one
i=0
while i<1:
try:
driver.find_element_by_xpath('//*[#title="xml (Open in a new window)"]').click()
except:
i=1
#Once the download link is clicked this has to fill the form for submission which fill download the file
class FormPage(object):
def fill_form(self, data):
driver.find_element_by_xpath('//input[#type = "radio" and #value = "Non-commercial"]').click()
driver.find_element_by_xpath('//input[#type = "checkbox" and #value = "R&D"]').click()
driver.find_element_by_xpath('//input[#name = "name_d"]').send_keys(data['name_d'])
driver.find_element_by_xpath('//input[#name = "mail_d"]').send_keys(data['mail_d'])
return self
def submit(self):
driver.find_element_by_xpath('//input[#value = "Submit"]').click()
data = {
'name_d': 'abc',
'mail_d': 'xyz#gmail.com',
}
FormPage().fill_form(data).submit()
driver.quit()
Actually you have two warnings and a error:
1 - "Too broad exception" this is a warning telling you that you should except espefic errors, not all of them. In your "except" line should be something like except [TheExceptionYouAreTreating]: an example would be except ValueError:. However this should not stop your code from running
2 - "Error: method submit maybe static" this is warning telling you that the method submit is a static method (bassically is a method that don't use the self attribute) to supress this warning you can use the decorator #staticmethod like this
#staticmethod
def submit():
...
3 - "SyntaxError: invalid syntax" this is what is stopping your code from running. This is a error telling you that something is written wrong in your code. I think that may be the indentation on your class. Try this:
i=0
while i<1:
try:
driver.find_element_by_xpath('//*[#title="xml (Open in a new window)"]').click()
except:
i=1
#Once the download link is clicked this has to fill the form for submission which fill download the file
class FormPage(object):
def fill_form(self, data):
driver.find_element_by_xpath('//input[#type = "radio" and #value = "Non-commercial"]').click()
driver.find_element_by_xpath('//input[#type = "checkbox" and #value = "R&D"]').click()
driver.find_element_by_xpath('//input[#name = "name_d"]').send_keys(data['name_d'])
driver.find_element_by_xpath('//input[#name = "mail_d"]').send_keys(data['mail_d'])
return self
def submit(self):
driver.find_element_by_xpath('//input[#value = "Submit"]').click()
data = {
'name_d': 'abc',
'mail_d': 'xyz#gmail.com',
}
FormPage().fill_form(data).submit()
driver.quit()
One more thing. Those are really simple errors and warnings, you should be able to fix them by yourself by carefully reading what the error has to say. I also reccomend you reading about Exceptions