Intro: I am trying to scrape a page without reloading it and I need many user use the same page again without launching new webdriver sessions.
I was following this Re-using existing browser session in selenium
But when I used the following code I am not able to create new sessions (driver = webdriver.Chrome(chromedriverPath)) anymore because the seision_id always the same
from selenium.webdriver.remote.webdriver import WebDriver as RemoteWebDriver
# Save the original function, so we can revert our patch
org_command_execute = RemoteWebDriver.execute
def new_command_execute(self, command, params=None):
if command == "newSession":
# Mock the response
return {'success': 0, 'value': None, 'sessionId': session_id}
else:
return org_command_execute(self, command, params)
# Patch the function before creating the driver object
RemoteWebDriver.execute = new_command_execute
Goals in my website I have too many users, so I want to prevent using too many requests as much as possible. Hence, I am trying to lunch 3 or 4 seasons then I use one of them.
Now, I have too many other things I may need to explain.
For example to prevent the possibility of two users clicking the same button the same time I created a data_base where it has the driver session_id and a field called is_used and a toggle it False/True.
Also, when I user need data from the website I filter the database that I have then like this DriverSesions.objects.filter(is_used=False)
question
How can I use this article without losing the ability to launch new session?
Are their alternatives?
Related
I am using selenium webdriver with chrome webdriver. In script1 I get a URL from the driver.get(" ...") and do some stuff and web scraping( for example clicking some buttoms, getting some informations and loging into the site).
When my script runs and finishes, I want to run another script(script2) that continues the last opened window( so in that case I don't have to spend a lot of time login to that site, clicking some buttons until I reach where I want to be).
for example imagine you want to login to your Gmail account and click some buttons to reach your mailbox and your script finishes right here. and then you want to run another script to open your emails one by one.
# script1
driver = webdriver.Chrome()
driver.get("https://gmail.google.com/inbox/")
inbox_button = driver.find_element_by_xpath("//*[#id=":5a"]/div/div[2]/span/a']")
inbox_button.click()
# the code finishes successfully right here
# script2
from script1 import driver
emails = driver.find_elements_by_xpath("path to emails']").find_element_by_tag_name("button")
print('email_button: ', emails)
for email in emails
emails.click()
I do not want to open a new chrome driver and run my code line by line again. I expect something that refers to the current chrome driver.
You need to save your session cookies to be able to return to the previous state.
You can either do this by
# Manually login to the website and then print the cookie
time.sleep(60)
print(driver.get_cookies())
# Then add_cookie() to add the cookie
driver.add_cookie({'domain': ''})
But this solution is not very elegant. You can instead use pickle to store and load the cookies
1. You would need to install pickle using pip - https://pypi.org/project/pickle5/
2. Add cookies after logging in to gmail
pickle.dump(driver.get_cookies(),open("cookies.pkl","wb"))
3. In the last part, you need to load the cookies and add it to your driver again when opening the browser for the second test
cookies = pickle.load(open("cookies.pkl","rb"))
for cookie in cookies:
driver.add_cookies(cookie)
I am trying to automate some processes on the site. At first I tried to use queries, but a captcha came in response. Now I'm using selenium queries, and here's the problem: when I log in using selenium tools only, everything works fine, but I can't add coupons on the site and confirm them.
from seleniumrequests import Firefox
driver = Firefox()
user = '000000'
password = '000000'
driver_1x.get("https://1xstavka.ru/")
driver.find_element_by_id('curLoginForm').click()
driver.find_element_by_id('auth_id_email').send_keys(user)
driver.find_element_by_id('auth-form-password').send_keys(password)
driver.find_element_by_class_name('auth-button__text').click()
But if you use:
from seleniumrequests import Firefox
driver = Firefox()
driver.request('GET', 'https://1xstavka.ru')
The window opens for a second and immediately closes, a 200 response is received, but there are no cookies. It's the same with publishing requests, with which I'm trying to automate the process. After the request for publication, the response is 200, but nothing happens on the site.
driver.request('POST', 'https://1xstavka.ru/user/auth', json=json)
please tell me what is wrong or how you can solve this problem
I am unable to access the URL specified in the query; but for captcha/coupons, I created a loop with an interim stop function. This gives me the chance to input it manually and then continue the loop.
So I am using geckodriver.exe (for Firefox), and I use the following code to acces whatsapp web:
from selenium import webdriver
browser = None
def init():
browser = webdriver.Firefox(executable_path=r"C:/Users/Pascal/Desktop/geckodriver.exe")
browser.get("https://web.whatsapp.com/")
init()
But everytime I rerun the code, the QR-Code from whatsappweb has to be scanned again and I dont want that. In my normal chrome browser I dont have to scan the QR-Code everytime. How can I fix this ?
Since every time you close your selenium driver/browser, the cookies that attached with the session will also be deleted. So to restore the cookies you haved saved, you can retrieve it after the end of the session and restore it in the beginning of the next.
For getting the cookies,
# Go to the correct domain, i.e. your Whatsapp web
browser.get("https://www.example.com")
# get all the cookies from this domain
cookies = browser.get_cookies()
# store it somewhere, maybe a text file
For restoring the cookies
# Go to the correct domain, i.e. your Whatsapp web
browser.get("https://www.example.com")
# get back the cookies
cookies = {‘name’ : ‘foo’, ‘value’ : ‘bar’}
browser.add_cookies(cookies)
What you could do is define a profile in Firefox. Then open firefox with that profile and open web.whatsapp.com. You will be prompted with the QR code. You link that instance. From there you can use the newly created profile in Python.
Creating a new profile can be done by typing about:profiles in the url section of Firefox:
Then open the browser by clicking 'Launch profile in new browser':
In your Python code you create a reference to this profile:
options.add_argument('-profile')
options.add_argument('/home/odroid/Documents/PythonProfile')
A step by step guide can also be found here.
I am currently using selenium to automate the input of data in to a website. The website never changes, and the fields are always the same with obviously the data differing.
How I want it to work is for the user to already be logged in to the website, they run a script and a new tab opens in their current browser session with the relevant fields having the data in them.
At the moment it opens a new Chrome session (ignoring the login from the previous session), has to log-in to the site, open a new tab, go to the data input page and push the keys from there. This can be a time consuming activity, and I don't like how it has to login each time. Snippet of my code below.
req = request.get_json()
jsonify(req)
url1 = "www.loginpage.com"
driver = webdriver.Chrome(executable_path=r'chromedriver.exe')
driver.get(url1)
u = driver.find_element_by_id('username')
u.send_keys("username")
u = driver.find_element_by_id('password')
u.send_keys("password")
u = driver.find_element_by_id('loginButton').submit()
driver.execute_script('''window.open("www.datainputpage.com","_blank");''')
driver.switch_to_window(driver.window_handles[1])
driver.find_element_by_id('Field1').send_keys(req[0])
driver.find_element_by_id('Field2').send_keys(req[1])
driver.find_element_by_id('Field3').send_keys(req[2])
driver.find_element_by_id('Field4').send_keys(req[3])
Is there a way using python I can automate it as mentioned? Opens new tab in current session - fills in fields?
You can use profiles in Chrome. You specify the directory of your profile and all cookies and stuff will be saved in there. So the next time you run it, it should load those same cookies from your previous session and stay logged in.
chrome_options = Options()
chrome_options.add_argument("user-data-dir=selenium")
driver = webdriver.Chrome(executable_path=r'chromedriver.exe', chrome_options=chrome_options)
Another possible option is saving the cookies to a json file, then on the next run, load them and set them in the browser.
Selenium Cookies
Reading & Writing JSON
I am trying to run tests against a website that requires four different sets of basic auth. There is all sorts of content embedded in the site requiring separate auth. For example:
http://store.domain.com
http://content.domain.com
https://content.domain.com
https://store.domain.com
Another way to say this is that when I hit one URL, I get four different sets of auth pop ups.
I'm not sure how to tackle this via Selenium. I currently have a working test written in Python / Selenium Web Driver, but when I run it I am forced to manually keep entering in creds each time, which is not a viable solution as this is going to be run from a TC build on a remote server.
Since it seems like Selenium isn't very flexible with auth stuff, I thought maybe as I workaround I could just hit each of the URLs requesting auth, pass them creds (this would happen during setup), and then start the actual tests. I need help with this for multiple reasons.
class newSmokeTest(unittest.TestCase):
def setUp(self):
self.driver = webdriver.Firefox()
self.driver.implicitly_wait(30)
self.base_url = "http://domain.com"
self.verificationErrors = []
self.accept_next_alert = True
driver = self.driver
# sending auth credentials early to avoid prompts while running test suite
driver.get("http://username:pass#store.domain.com")
wait = WebDriverWait(driver, 10)
driver.get("https://username:pass#store.domain.com")
wait = WebDriverWait(driver, 10)
driver.get("http://username:pass#content.domain.com")
wait = WebDriverWait(driver, 10)
driver.get("https://username:pass#content.domain.com")
def test1(self):
#do stuff
def test2(self):
#do stuff
One, I'm not even sure if that is a viable solution, would manually passing creds to each URL in the setup of my unittest create a session in Firefox, how do I find that out?
Two, when I run the test, I actually don't visually see the browser hitting those URL's. Is putting those steps in the setUp for some reason preventing those steps from running?
I'm kind of at a dead end with this, would appreciate any help. Using a Firefox profile is not the answer I'm looking for as this will be ran on a remote server by another department and I'm not even sure that they would allow the installation of a Firefox profile on there. Hoping I can do this in a more Pythonic way.