Python Multithreading Selenium - python

I am writing a code in Tkinter with a button that starts 6 selenium chrome instances, each with a different url. The goal is : the drivers have to be initiated in the fastest possible way and every driver instance has to reenter its specific url (refresh) every 4 seconds. Every driver is attached to a class, that contains the wanted url. I tried this with threads:
import tkinter as tk
import threading
import os
import time
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
browsers = []
class Browser:
def __init__(self, url, session_file):
self.url = url
self.session_file = session_file
def manipulate_browser(browser):
browser.driver = webdriver.Chrome(ChromeDriverManager().install())
while True:
browser.driver.get(browser.url)
time.sleep(4)
def start_browsers():
for browser in browsers:
browser.thread = threading.Thread(target=manipulate_browser, args=(browser,))
browser.thread.start()
if __name__=='__main__':
lock = threading.Lock()
threads = []
urls = 'https://google.com', 'https://facebook.com', 'https://instagram.com', 'https://snapchat.com', 'https://stackoverflow.com', 'https://amazon.com', 'https://microsoft.com'#, 'https://stackoverflow.com', 'https://youtube.com', 'https://yahoo.com'
for url in urls:
session_file = 'session_' + ''.join(random.choice(string.ascii_letters + string.digits) for _ in range(10))
newBrowser = Browser(url, session_file)
browsers.append(newBrowser)
root = tk.Tk()
button_start_browsers = tk.Button(root, command=start_browsers, width=50, height=4, bg='red', text='Start Browsers')
button_start_browsers.pack()
And this works just fine, but I want to add some options and capabilities to the driver, in the manipulate_browser function . Like so:
import tkinter as tk
import threading
import os
import time
import random, string
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from multiprocessing.pool import ThreadPool
browsers = []
class Browser:
def __init__(self, url, session_file):
self.url = url
self.session_file = session_file
def manipulate_browser(browser):
global lock
with lock:
caps = DesiredCapabilities().CHROME
caps["pageLoadStrategy"] = "none"
chrome_options = webdriver.ChromeOptions();
current_dir = os.path.dirname(os.path.abspath(__file__))
os.mkdir(current_dir + '\\' + browser.session_file)
chrome_options.add_argument(r'--user-data-dir=' + current_dir + '\\' + browser.session_file + '\\selenium')
chrome_options.add_argument("--window-size=750,750")
browser.driver = webdriver.Chrome(ChromeDriverManager().install(), options=chrome_options, desired_capabilities=caps)
while True:
browser.driver.get(browser.url)
time.sleep(4)
def start_browsers():
for browser in browsers:
browser.thread = threading.Thread(target=manipulate_browser, args=(browser,))
browser.thread.start()
if __name__=='__main__':
lock = threading.Lock()
threads = []
urls = 'https://google.com', 'https://facebook.com', 'https://instagram.com', 'https://snapchat.com', 'https://stackoverflow.com', 'https://amazon.com', 'https://microsoft.com'#, 'https://stackoverflow.com', 'https://youtube.com', 'https://yahoo.com'
for url in urls:
session_file = 'session_' + ''.join(random.choice(string.ascii_letters + string.digits) for _ in range(10))
newBrowser = Browser(url, session_file)
browsers.append(newBrowser)
root = tk.Tk()
button_start_browsers = tk.Button(root, command=start_browsers, width=50, height=4, bg='red', text='Start Browsers')
button_start_browsers.pack()
After implementing this, it doesn't work anymore : meaning : sometimes, I get this error selenium.common.exceptions.WebDriverException: Message: unknown error: cannot parse internal JSON template: Line: 1, column: 1, Unexpected token. And all the threads seem to work on just one driver (the last one). I believe this is because the threads mix with each other so I have to use a lock, but putting this line browser.driver = webdriver.Chrome(ChromeDriverManager().install(), options=chrome_options, desired_capabilities=caps) under the lock will affect drastically my speed, that I need. I also understand that I have to use ThreadPool, for avoiding thread mixing and I tried it, but it freezed the GUI app. Another option that others suggest is implementing queue with threads, which I am not very familiar with. I also thought of trying multiprocessing, but the maximum numbers of processes depends on every machine characteristics, from my understanding, and maybe I want to start more processes than that. How can I workaround this situation to achieve my goal? What is the best way?
(Note : I've done plenty of research regarding my issue, but I still couldn't figure it out. Any help is welcomed !)
EDIT :
Due to multiple testing, I figured out that the line chrome_options.add_argument(r'--user-data-dir=' + current_dir + '\\' + browser.session_file + '\\selenium') may cause all the problems. This line keeps the chrome sessions all in one icon and I really like this feature and I wouldn't like getting rid of it. How can I overcome the issue, knowing this information? Any help would be appreciated !

Related

(Python Selenium) Even though I didn't write the function ".close()" on Selenium, The browser is being closed by itself

from selenium import webdriver
from selenium.webdriver.common.by import By as by
import time
import os
import idpas
os.chdir("C:/Users/ben/Desktop/Yeni klasör")
class browser:
def __init__(self,link):
self.link = link
self.browser = webdriver.Chrome()
browser.goInstagram(self)
def goInstagram(self):
time.sleep(3.5)
self.browser.get(self.link)
time.sleep(3.5)
browser.login(self)
def login(self):
id = self.browser.find_element(by.NAME,"username")
password = self.browser.find_element(by.NAME,"password")
log_in = self.browser.find_element(by.XPATH,"//*[#id='loginForm']/div/div[3]")
id.send_keys(idpas.id)
password.send_keys(idpas.password)
log_in.click()
time.sleep(1)
def go_to_profile(self):
goProfile = self.browser.find_element(by.XPATH,"//*[#id='f2079fe6736dc34']/div/div/a")
goProfile.click()
browser.getFollowers(self)
def getFollowers(self):
getTheFollowers = self.browser.find_element(by.CSS_SELECTOR,"_aacl _aaco _aacw _aacx _aad7 _aade")
with open("C:/Users/ben/Desktop/Yeni klasör/followers.txt","w") as f:
for i in getTheFollowers:
f.write("\n")
f.write(i.lower())
browser("https://www.instagram.com")
time.sleep(100000)
Hello everybody. As I wrote in the title, I am having trouble with Selenium in Python.
If I don't write those codes that I wrote above in a class, The browser isn't closed by itself.
But, I wanted to write those codes in a class. I am really about to cry due to my anger.
I am trying to fix it for 2 hours, but my effort could do nothing to those codes.
Please help me about this.
My codes are above.

Problem with Filedialog in Python, Selenium 'NameError'

I work on this Projekt where I try to get Links from a Website and I got it to work after 2-3 Weeks and now my Task is to build an GUI with TKinter. The Problem is that I ask the User to select the Driver that is needed for the Selenium Operation, that grabs the Links from the Website
So this is my Code and probably the Problem is easy but I'm just stuck at the part from coderetter() to code() where I give the Filepath to the Driver it always says: "NameError: name 'save' is not defined" and I tried everything
The Problem occurs on the last line by executable_path that where i want to let the User choose the path.
Has anyone got any similar kind of problem or does someone maybe see the problem here?
import csv
from tkinter import *
from tkinter import filedialog
from multiprocessing import Process
def Fenster():
root = Tk()
root.title("MS-Search Grabber")
w = Label(root, text='W!')
driverButt = Button(text='Chrome-Driver', command=coderetter)
FileSave = Button(text='Save CSV', command=saveFile)
StartB = Button(text='Start Process', command=proSt)
w.pack()
driverButt.pack()
FileSave.pack()
StartB.pack()
root.mainloop()
def coderetter():
file = filedialog.askopenfilename()
def saveFile():
save = filedialog.asksaveasfile()
return save
def proSt():
p2 = Process(target=code)
p2.start()
def code():
why = saveFile(save)
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('headless')
chrome_options.add_argument('window-size=1920x1080')
chrome_options.add_argument('disable-gpu')
driver = webdriver.Chrome(executable_path=why, options=chrome_options)
if __name__ == "__main__":
file = ''
driver= ''
save = ''
p1 = Process(target=Fenster)
p1.start()
The use of global will enable file and save strings to be available to other parts of your code.
By using global in saveFile, you do not need a return
This means Filesave button will also function correctly since buttons do not accept returned information.
def coderetter():
global file
file = filedialog.askopenfilename()
def saveFile():
global save
save = filedialog.asksaveasfile()
def proSt():
p2 = Process(target=code)
p2.start()
def code():
saveFile()

How to run multiple instances of Chrome parallely with webdriver in python?

Hi I try to open parallely multiple instances of Chrome in python using Webdriver and Multiprocessing.
After running processes, instances are opening smoothly, but they are not sent to my "instance" array and I can't access instances after that. Please help me, there is my code:
from selenium import webdriver
from multiprocessing import Process
import time
num = 3
process = [None] * num
instance = [None] * num
def get():
for i in range(num):
try:
instance[i].get("https://www.youtube.com")
except:
print("Can't connect to the driver")
time.sleep(1)
get()
def create_instance(i):
instance[i] = webdriver.Chrome()
if __name__ == '__main__':
for i in range(num):
process[i] = Process(target = create_instance, args = [i])
process[i].start()
for i in range(num):
process[i].join()
get()
when the multiprocessing try to pickle the webdriver object, it'll occur some weird error, so instead of passing the object, we can pass the class and build the object inside the new process.
BUT, in that kind of situation, you can not access the driver instances anymore, maybe you can try to send signals to the process.
from selenium import webdriver
from multiprocessing import Process
import time
num = 3
process = [None] * num
def get(id, Driver):
driver = Driver()
driver.get(f"https://www.google.com?id={id}")
time.sleep(10)
driver.close()
if __name__ == '__main__':
for i in range(num):
process[i] = Process(target=get, args = [i, webdriver.Chrome])
process[i].start()
for i in range(num):
process[i].join()

How to prevent the mobile application closing and re-opening each time a test case is running?

I am performing mobile application automation using Appium with Python. I am also in need of creating HTML reports. I am wanting to create multiple test suites too. And all these works, except for one problem.
My problem is that the application closes and re-opens in every test case. How can I fix this? Thanks in advance.
(Please note that this is a sample code I'm putting on here.)
from adb.client import Client as AdbClient
import HtmlTestRunner
import datetime
import os, sys
import glob
import unittest
from appium import webdriver
from time import sleep
from appium.webdriver.common.touch_action import TouchAction
PLATFORM_VERSION = '8.1.0'
class Q_suite1_01(unittest.TestCase):
def setUp(self):
desired_caps = {}
desired_caps['platformName'] = 'Android'
desired_caps['platformVersion'] = '8.1.0'
desired_caps['deviceName'] = 'Samsung Galaxy J7 Max'
devices = AdbClient(host= "127.0.0.1", port= 5037).devices()
for device in devices:
desired_caps['udid'] = device.serial
desired_caps['appPackage'] = 'com.testapp'
desired_caps['appActivity'] = 'com.testapp.MainActivity'
url = "http://localhost:{}/wd/hub".format(4723)
self.driver = webdriver.Remote(url, desired_caps)
def install(self):
print 'ABDC!'
def run_app(self):
try:
x = self.driver.is_app_installed('com.quallogi')
if x is True:
print 'App is already installed.'
else:
print 'App is not installed.'
except:
print 'App not installed'
def signin(self):
sleep(5)
self.driver.find_element_by_xpath('//*[contains(#text,"Login") and contains(#class, "android.widget.TextView")]').click()
print 'Sign'
def testcase_Install_app(self):
self.install()
def testcase_Run_app(self):
self.run_app()
def testcase_SignIn(self):
self.signin()
# def testcase_Install_app(self):
# self.install()
# self.run_app()
# self.signin()
#
def tearDown(self):
self.driver.quit()
if __name__ == '__main__':
result = []
suite1= unittest.TestLoader().loadTestsFromTestCase(Q_suite1_01)
result.append(HtmlTestRunner.HTMLTestRunner(output='./HTML Reports/'
+ str(datetime.date.today())).run(suite1))
print(result)
At the first I want to recomend you to look at the Appium capability "noReset" - "Don't reset app state before this session." (true, false).
If i right understand you question. What do you meant "application closes and re-opens in every test case"? Can you describe it more detail?

How do I import selenium webdriver tests from one Python file to another?

I'm incredibly new to separating modules. I have this long Python script that I want to separate into different files by class and run them collectively in the same browser instance/window. The reason for this is all the tests are reliant on being logged into the same session. I'd like to do a universal setUp, then login, and then pull the different tests in one after another.
Folder structure is:
ContentCreator
- main.py
- _init_.py
- Features
- login.py
- pytest.py
- _init_.py
Here is my code:
login.py
import unittest
from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException
import time
import json
driver = webdriver.Chrome()
class logIn(unittest.TestCase):
#classmethod
def test_login(self):
"""Login"""
driver.get("sign_in_url")
# load username and pw through a json file
with open('path/to/file.json', 'r') as f:
config = json.load(f)
# login
driver.find_element_by_id("email").click()
driver.find_element_by_id("email").clear()
driver.find_element_by_id("email").send_keys(config['user']['name'])
driver.find_element_by_id("password").click()
driver.find_element_by_id("password").clear()
driver.find_element_by_id("password").send_keys(config['user']['password'])
driver.find_element_by_id("submit").click()
time.sleep(3)
print("You are Logged In!")
pytest.py
import time
import unittest
from datetime import datetime
from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.common.exceptions import NoSuchElementException
from displays import DISPLAY_TYPES, DISPLAY_NAMES
driver = webdriver.Chrome()
#driver.get("url")
class createContent(unittest.TestCase):
#classmethod
def test_add_required(self):
"""Test adding all the required fields across all sites:"""
for i in range(1):
"""This is the number of each type of article that will be created."""
for i in range(1):
"""This is how many different article types that will be created."""
print("create new content")
time.sleep(1)
driver.find_element_by_link_text("Content").click()
time.sleep(1)
driver.find_element_by_link_text("Create New").click()
print("select a display type:")
display = DISPLAY_TYPES
display_type = driver.find_element_by_id(display[i])
display_type.click()
names = (DISPLAY_NAMES[i])
print(names), (" created and saved successfully!")
#classmethod
def tearDownClass(cls):
# close the browser window
driver.quit()
def is_element_present(self, how, what):
"""
Helper method to confirm the presence of an element on page
:params how: By locator type
:params what: locator value
"""
try:
driver.find_element(by=how, value=what)
except NoSuchElementException:
return False
return True
main.py
import unittest
from HtmlTestRunner import HTMLTestRunner
from features.login import logIn
from features.pytest import createContent
login_script = unittest.TestLoader().loadTestsFromTestCase(logIn)
add_pytest = unittest.TestLoader().loadTestsFromTestCase(createContent)
# create a test suite combining all tests
test_suite = unittest.TestSuite([login, add_pytest])
# create output
runner = HTMLTestRunner(output='Test Results')
# run the suite
runner.run(test_suite)
When running the above code it opens two browser sessions, and only the login script get executed. The test fails do to not finding the elements outlined in the next script.
EDIT:
Alfonso Jimenez or anyone else, here's what I have so far...
Folder structure:
- Files
- singleton.py
- singleton2.py
New Singleton code...
singleton.py:
from robot.api import logger
from robot.utils import asserts
from selenium import webdriver
class Singleton(object):
instance = None
def __new__(cls, base_url, browser='chrome'):
if cls.instance is None:
i = object.__new__(cls)
cls.instance = i
cls.base_url = base_url
cls.browser = browser
if browser == "chrome":
# Create a new instance of the Chrome driver
cls.driver = webdriver.Chrome()
else:
# Sorry, we can't help you right now.
asserts.fail("Support for Chrome only!")
else:
i = cls.instance
return i
singleton2.py:
import time
import json
from datetime import datetime
from singleton import Singleton
driver = Singleton('base_url')
def teardown_module(module):
driver.quit()
class logIn(object):
def test_login(self):
"""Login"""
driver.get("url.com")
# load username and pw through a json file
with open('file.json', 'r') as f:
config = json.load(f)
# login
driver.find_element_by_id("email").click()
driver.find_element_by_id("email").clear()
driver.find_element_by_id("email").send_keys(config['user']['name'])
driver.find_element_by_id("password").click()
driver.find_element_by_id("password").clear()
driver.find_element_by_id("password").send_keys(config['user']['password'])
driver.find_element_by_id("submit").click()
time.sleep(3)
print("You are Logged In!")
# take screenshot
driver.save_screenshot('path/screenshot_{}.png'.format(datetime.now()))
The result is that an instance of Chrome kicks off, but nothing happens. The base_url (or any other URL defined in my test) doesn't come up. The blank window is all I get. Any insights on what I'm doing wrong?
You're instantiating two times the selenium driver.
If you want to keep the same session opened you should pass the same object to both scripts, or import it, what it could be valid, however it would be a more dirty solution.
The best thing to do is create a singleton class to initiate the driver. Once you have done this, every time you create an object from this class you will get the a unique object of webdriver.
You can get an example from this answer.
You can also check more about singleton instances, they're a very common and very useful. You can check here.
I dont understand what you mean with robot, perhaps the testing framework?
You can write the singleton class wherever you want to. You will have to import the class from that place and then instantiate the object. Ex:
lib/singleton_web_driver.py
from robot.api import logger
from robot.utils import asserts
from selenium import webdriver
class Singleton(object):
instance = None
def __new__(cls, base_url, browser='firefox'):
if cls.instance is None:
i = object.__new__(cls)
cls.instance = i
cls.base_url = base_url
cls.browser = browser
if browser == "firefox":
# Create a new instance of the Firefox driver
cls.driver = webdriver.Firefox()
elif browser == "remote":
# Create a new instance of the Chrome driver
cls.driver = webdriver.Remote("http://localhost:4444/wd/hub", webdriver.DesiredCapabilities.HTMLUNITWITHJS)
else:
# Sorry, we can't help you right now.
asserts.fail("Support for Firefox or Remote only!")
else:
i = cls.instance
return i
and then in every script youre going to need the webdriver:
test_script_file.py
from lib.singleton_web_driver import Singleton
driver = Singleton('base_url')
This just a dummy code, I dindnt test it. The important point is to create the class with the _new_ method where you can check if the class has already been called. The import is just like any other class import, you write the class in a folder and then import it in the scripts youre going to use.
I had a similar problem. My solution was just to initiate the driver in the main file and then to use this driver within the imported files and functions. Like in this example to change createContent(unittest.TestCase) to createContent(unittest.TestCase, driver)

Categories

Resources