I am running a script with Selenium that loops over a number of URLs. However, some URLs give a 503 service error. For example the following code gives me such an error.
import requests
import selenium
from selenium import webdriver
url = 'https://destyy.com/q3P5W7'
driver = webdriver.Chrome()
driver.get(url)
response = requests.get(driver.current_url).status_code
I tried refreshing the browser in the case of an error, but then the error persists.
if response == 503:
driver.refresh()
I noticed that if I manually select the url in the address bar of my driver and press enter, the page loads to the desired location. I tried to write this into code but it fails:
from selenium.webdriver.common.keys import Keys
address_bar = driver.find_element_by_xpath('/html/body').send_keys(Keys.CONTROL, "l")
address_bar.sendKeys(Keys.ENTER)
Please help me out. How can I load the page? Thanks in advance,
Related
I'm trying to scrape a website with Selenium, but I think it's blocking this access in many ways.
The error message shown is: "selenium.common.exceptions.NoSuchWindowException: Message: Browsing context has been discarded" but sometimes is shown an error saying that time for loading page had expired
Furthermore, Firefox is consuming a huge percent of CPU and Memory when loading this page.
I've already tried to change user-agent, or run it headlessly, but no results.
Below is the code:
from selenium import webdriver
browser = webdriver.Firefox()
browser.get('https://www.bet365.com/#/HO/')
matches = browser.find_elements_by_class_name('him-Fixture')
browser.quit()
Any tips to bypass it ?
Sometimes browsers loading late . So you add time.sleep() function in you code.
Example :
from selenium import webdriver
import time
browser = webdriver.Firefox()
browser.get('https://www.bet365.com/#/HO/')
time.sleep(5)
matches = browser.find_elements_by_class_name('him-Fixture')
browser.quit()
I am trying to automate logging into my salesforce account. When I use my code:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
browser= webdriver.Firefox()
browser.get("https://xxxxx.my.salesforce.com/?
SAMLRequest=&startURL=%2Fidp%2Flogin%3Fapp%
3D0sp0g000000Gmhj&un=xxxxx.xxxxx%40xxxxx.com")
elem=browser.find_element_by_id("username")
elem.send_keys("xxxxx.xxxx#xxxxx.com")
elem_pass=browser.find_element_by_id("password")
elem_pass.send_keys("xxxxxxx")
rem_me=browser.find_element_by_id("rememberUn")
rem_me.click()
elem.send_keys(Keys.ENTER)
As you can see, I pass the link to the url, pass the usname, password and remember me.
When I run this with Selenium, it goes to a email 2FA authentication page.
But when I do it manually:
Copy the url mentioned above.
Paste it into the address bar of firefox browser.
The uname and pass show already populated.
When I hit enter, it logs me in. (No 2FA).
Is Salesforce somehow detecting that request is from selenium?
And is there a way to get around it?
Could this be related to this?
Different results when using Selenium + Python
Yup, I got it resolved. I had to import cookies from Firefox, and use them with Selenium. from selenium import webdriver from selenium.webdriver.common.keys import Keys import os os.chdir("C:\Users\tsingh\Desktop\Cookies") ffprofile = webdriver.FirefoxProfile('C:\Users\tsingh\Desktop\Cookies') browser = webdriver.Firefox(firefox_profile=ffprofile)
I am using selenium webdriver to open a page in my website using python.
The page link is : "www.example.com/App/Details.aspx/I=5aM%+8KbCv1o=&T=M30Lr7RtcdR=&H=fRFKse5fKA=="
Since I cannot open this page directly, I have to first login on
www.example.com, which I am doing this way:
from selenium.common.exceptions import TimeoutException
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
try:
driver.set_page_load_timeout(10)
driver.get('http://www.example.com')
except Exception:
print ('time out')
#driver.find_element_by_id("Default").send_keys(Keys.CONTROL +'Escape')
username = driver.find_element_by_id("usrid")
password = driver.find_element_by_id("pswd")
username.send_keys("Sample_user")
password.send_keys("samplepass")
driver.find_element_by_id("submitbtn").click()
After this I am am able to log in successfully,
but when I redirect to the above link using
driver.get("www.example.com/App/Details.aspx/I=5aM%+8KbCv1o=&T=M30Lr7RtcdR=&H=fRFKse5fKA==")
I get error saying please login first. Please suggest a solution for the same.
You sent the data via
password.send_keys
but after that you were to also send enter command using
password.send_keys((Keys.ENTER)
After this you have to have a
sleep(1)
or
WebDriverWait Until(expect...
before you open the next page
I am using selenuim webdriver to get a page with https scheme. if i go to the page using chrome i get the 'privacy error' and the response is empty.
If i use webdriver to get the page the webdriver stay on the last page it was in. How can i detect the webdriver did not move to the new page ?
I thought about checking if the driver.current_url is same as the url i want but it doesnt feel like the right solution.
Code:
from selenium import webdriver
driver = webdriver.PhantomJS()
driver.get('http://doman.com/1.html') # this return 200 status and content
driver.get('https://domain2.com/2.html') # this return the privacy error
# so the driver is still at the first url
I think i have found a solution after reading support desired_capabilities: acceptSslCerts=True
from selenium import webdriver
service_args = ['--ignore-ssl-errors=true']
driver = webdriver.PhantomJS(service_args=service_args)
driver.get('http://doman.com/1.html')
driver.get('https://domain2.com/2.html') # now it returns 404 as it should
I am having a strange issue with PhantomJS or may be I am newbie. I am trying to login on NewEgg.com via Selenium by using PhantomJS. I am using Python for it. Issue is, when I use Firefox as a driver it works well but as soon as I set PhantomJS as a driver it does not go to next page hence give message:
Exception Message: u'{"errorMessage":"Unable to find element with id \'UserName\'","request":{"headers":{"Accept":"application/json","Accept-Encoding":"identity","Connection":"close","Content-Length":"89","Content-Type":"application/json;charset=UTF-8","Host":"127.0.0.1:55372","User-Agent":"Python-urllib/2.7"},"httpVersion":"1.1","method":"POST","post":"{\\"using\\": \\"id\\", \\"sessionId\\": \\"aaff4c40-6aaa-11e4-9cb1-7b8841e74090\\", \\"value\\": \\"UserName\\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/aaff4c40-6aaa-11e4-9cb1-7b8841e74090/element"}}' ; Screenshot: available via screen
The reason I found after taking screenshot that phantom could not navigate the page and script got finished. How do I sort this out? Code Snippet I tried given below:
import requests
from bs4 import BeautifulSoup
from time import sleep
from selenium import webdriver
import datetime
my_username = "user#mail.com"
my_password = "password"
driver = webdriver.PhantomJS('/Setups/phantomjs-1.9.7-macosx/bin/phantomjs')
firefox_profile = webdriver.FirefoxProfile()
#firefox_profile.set_preference('permissions.default.stylesheet', 2)
firefox_profile.set_preference('permissions.default.image', 2)
firefox_profile.set_preference('dom.ipc.plugins.enabled.libflashplayer.so', 'false')
#driver = webdriver.Firefox(firefox_profile)
driver.set_window_size(1120, 550)
driver.get('http://newegg.com')
driver.find_element_by_link_text('Log in or Register').click()
driver.save_screenshot('screen.png')
I even put sleep but it is not making any difference.
I experienced this with PhantomJS when the content type of the second page is not correct. A normal browser would just interpret the content dynamically, but Phantom just dies, silently.