On an ubuntu system I am trying to use a 'browsernmob-proxy' with python-selenium tests. Following the documentation here I installed 'browsermob-proxy' and I tried the following python code:
from selenium import webdriver
from browsermobproxy import Server
server = Server("/home/adietz/Projects/Venv/nosetests/lib/python2.7/site-packages/browsermobproxy/server.py")
server.start()
but this immediately failed with the following error:
======================================================================
ERROR: test_example2.TestSuite.test_network
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/adietz/Projects/Venv/nosetests/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/home/adietz/Projects/Jenkins/bsp-usecase-tests/selenium/test_example2.py", line 41, in test_network
server.start()
File "/home/adietz/Projects/Venv/nosetests/local/lib/python2.7/site-packages/browsermobproxy/server.py", line 113, in start
stderr=subprocess.STDOUT)
File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
raise child_exception
OSError: [Errno 13] Permission denied
Any idea how to fix this?
I also tried to use
server = Server("/home/adietz/Projects/Venv/nosetests/lib/python2.7/site-packages/browsermobproxy")
instead, but this failed with the following error:
======================================================================
ERROR: test_example2.TestSuite.test_network
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/adietz/Projects/Venv/nosetests/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/home/adietz/Projects/Jenkins/bsp-usecase-tests/selenium/test_example2.py", line 40, in test_network
server = Server("/home/adietz/Projects/Venv/nosetests/lib/python2.7/site-packages/browsermobproxy")
File "/home/adietz/Projects/Venv/nosetests/local/lib/python2.7/site-packages/browsermobproxy/server.py", line 81, in __init__
"in path provided: %s" % path)
ProxyServerError: Browsermob-Proxy binary couldn't be found in path provided: /home/adietz/Projects/Venv/nosetests/lib/python2.7/site-packages/browsermobproxy
Addendum
I probably managed to get the browsermob-proxy to tun at port 8088 or 8089 (not exactly sure), but the example code still won't run ....
I guess, you are giving wrong path for the proxy server. Your code is pointing to the browsermob proxy python bindings instead of the server. Please follow the below steps.
Download the browsermob proxy from the location https://github.com/lightbody/browsermob-proxy/releases and unzip the file to particular directory. I have extracted/unzipped to directory C:\\Projects\\BrowserMobProxy
Install the browsermob proxy bindings if not already installed.
pip install browsermob-proxy
Then you can point the server path in the coding as follows. I am using windows. as given below.
from browsermobproxy import Server
server = Server("C:\\Projects\\BrowserMobProxy\\bin\\browsermob-proxy")
server.start()
proxy = server.create_proxy()
from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.set_proxy(proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)
proxy.new_har("google")
driver.get("http://www.google.co.uk")
proxy.har # returns a HAR JSON blob
server.stop()
driver.quit()
Please change the proxy path accordingly(where you have downloaded or extracted).
Related
I have tried running below code in python IDE interpreter as well as in PyCharm installing the module but still getting an error,( I have installed selenium module),
from selenium import webdriver
browser = webdriver.Firefox()
type(browser)
browser.get('http://inventwithpython.com')
------------------------------------------------------------------------------------------
Output:
Traceback (most recent call last):
File "demo.py", line 2, in <module>
browser = webdriver.Firefox()
File "/usr/lib/python3/dist-packages/selenium/webdriver/firefox/webdriver.py", line 164, in __init__
self.service.start()
File "/usr/lib/python3/dist-packages/selenium/webdriver/common/service.py", line 81, in start
raise WebDriverException(
selenium.common.exceptions.WebDriverException: Message: 'geckodriver' executable needs to be in PATH.
First look at this and install webdriver for your system.
Then add it to your system path (i don't know what is your OS so you have to find the tutorial of this by your self and it's not hard) or for temporary usage add this to your code:
browser=webdriver.Chrome(executable_path=r"C:\path\to\chromedriver.exe")
# Change Chrome to your desired browser and if you are not on Windows it doesn't have .exe
Tried adding the drivers to the right path. mentioned the path in the code and yet getting runtime errors with chromedriver. Cannot see what is wrong with the code. whats wrong?
Tried adding the path to the code. imported selenium and webdriver. placed the drivers in the python directory.
from selenium import webdriver
#to open the page in Chrome.firefox
driver = webdriver.Chrome("executable_path=D:\Python\Crawler\chromedriver_win32")
driver.get("http://econpy.pythonanywhere.com/ex/001.html")
given Below are the errors:
D:\Python\Crawler\venv\Scripts\python.exe D:/Python/Crawler/crawler_sel.py
Traceback (most recent call last):
File "D:\Python\Crawler\venv\lib\site-packages\selenium\webdriver\common\service.py", line 76, in start
stdin=PIPE)
File "C:\Users\vidwa\AppData\Local\Programs\Python\Python37-32\lib\subprocess.py", line 775, in __init__
restore_signals, start_new_session)
File "C:\Users\vidwa\AppData\Local\Programs\Python\Python37-32\lib\subprocess.py", line 1178, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:/Python/Crawler/crawler_sel.py", line 5, in <module>
driver = webdriver.Chrome("executable_path=D:\Python\Crawler\chromedriver_win32")
File "D:\Python\Crawler\venv\lib\site-packages\selenium\webdriver\chrome\webdriver.py", line 73, in __init__
self.service.start()
File "D:\Python\Crawler\venv\lib\site-packages\selenium\webdriver\common\service.py", line 83, in start
os.path.basename(self.path), self.start_error_message)
selenium.common.exceptions.WebDriverException: Message: 'chromedriver_win32' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home
Process finished with exit code 1
Try the below this worked for me
from selenium import webdriver
#to open the page in Chrome.firefox
def set_up(self):
self.driver = webdriver.Chrome("D:/Python/Crawler/chromedriver_win32.exe")
Your syntax here is off. It should be:
driver = webdriver.Chrome(executable_path='D:/Python/Crawler/chromedriver_win32.exe')
I've taken quotation marks off executable_path, and added .exe extension to your file path.
Alternative solution -- If you do not want to use executable_path, you can try adding chromedriver_win32.exe to your Path variable:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get("http://econpy.pythonanywhere.com/ex/001.html")
Then, in your environment variable for Path, you have something like:
D:\Python\Crawler\chromedriver_win32.exe
This guide may help you with the Path part: https://developers.refinitiv.com/sites/default/files/How%20To%20Add%20ChromeDriver%20To%20System%20Variables_0.pdf
I am trying to learn python web scraping, but I cannot get selenium to work with either browser.
from selenium import webdriver
browser = webdriver.Firefox()
This is all the code I have and I get this for a error.
Traceback (most recent call last):
File "C:\Users\tjhall\AppData\Local\Programs\Python\Python35-32\lib\site-packages\selenium\webdriver\common\service.py", line 64, in start
stdout=self.log_file, stderr=self.log_file)
File "C:\Users\tjhall\AppData\Local\Programs\Python\Python35-32\lib\subprocess.py", line 950, in __init__
restore_signals, start_new_session)
File "C:\Users\tjhall\AppData\Local\Programs\Python\Python35-32\lib\subprocess.py", line 1220, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "H:\codingpractice\python\python challenge.com.py", line 2, in <module>
browser = webdriver.Firefox()
File "C:\Users\tjhall\AppData\Local\Programs\Python\Python35-32\lib\site-packages\selenium\webdriver\firefox\webdriver.py", line 135, in __init__
self.service.start()
File "C:\Users\tjhall\AppData\Local\Programs\Python\Python35-32\lib\site-packages\selenium\webdriver\common\service.py", line 71, in start
os.path.basename(self.path), self.start_error_message)
selenium.common.exceptions.WebDriverException: Message: 'geckodriver' executable needs to be in PATH.
Exception ignored in: <bound method Service.__del__ of <selenium.webdriver.firefox.service.Service object at 0x00A11350>>
Traceback (most recent call last):
File "C:\Users\tjhall\AppData\Local\Programs\Python\Python35-32\lib\site-packages\selenium\webdriver\common\service.py", line 163, in __del__
self.stop()
File "C:\Users\tjhall\AppData\Local\Programs\Python\Python35-32\lib\site-packages\selenium\webdriver\common\service.py", line 135, in stop
if self.process is None:
AttributeError: 'Service' object has no attribute 'process'
I have tried everything I can find on the internet from adding the path to the code
from selenium import webdriver
browser = webdriver.Firefox("C:\Program Files (x86)\Mozilla Firefox\firefox.exe")
to adding it the the PATH in my environment variables. I can't seem to figure this out...
For both Firefox and Chrome you now need to download geckodriver / chromedriver.These drivers are necessary to communicate between your installed browser and selenium. So you need:
Install selenium for python (pip install selenium)
Download drivers for the browser you want to use (chromedriver, geckodriver, operadriver etc)
Install the browser you want to use on your system (probably already have this)
Now you can add the geckodriver to your path as metioned in this anwser. Or you can set it up directly in your code like this:
Chome:
driver = webdriver.Chrome(executable_path='/path/to/chromedriver.exe')
Firefox:
driver = webdriver.Firefox(executable_path='/opt/geckoDriver/geckodriver.exe')
According to the line in your message
selenium.common.exceptions.WebDriverException: Message: 'geckodriver' executable needs to be in PATH.
you do not have geckodriver.exe. You need to download it from enter link description here, put exe in the directory where Python script is located and try code below:
Please try this code:
# -*- coding: utf-8 -*-
from selenium import webdriver
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
gecko = os.path.normpath(os.path.join(os.path.dirname(__file__), 'geckodriver'))
binary = FirefoxBinary(r'C:\Program Files (x86)\Mozilla Firefox\firefox.exe')
browser = webdriver.Firefox(firefox_binary=binary,executable_path=gecko+'.exe')
browser.get('http:///www.google.com')
browser.close()
Trying to learn how to use selenium, I managed to overcome first error which involved chrome driver not being in the path name but it has thrown up another error.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome('/Users/williamneal/Scratch/Titanic/chromedriver')
driver.get("http://www.bbc.com")
The error:
Traceback (most recent call last):
File "<ipython-input-1-84256e62b8db>", line 5, in <module>
driver = webdriver.Chrome('/Users/williamneal/Scratch/Titanic/chromedriver')
File "/Users/williamneal/anaconda/lib/python3.5/site-packages/selenium/webdriver/chrome/webdriver.py", line 62, in __init__
self.service.start()
File "/Users/williamneal/anaconda/lib/python3.5/site-packages/selenium/webdriver/common/service.py", line 64, in start
stdout=self.log_file, stderr=self.log_file)
File "/Users/williamneal/anaconda/lib/python3.5/subprocess.py", line 950, in __init__
restore_signals, start_new_session)
File "/Users/williamneal/anaconda/lib/python3.5/subprocess.py", line 1544, in _execute_child
raise child_exception_type(errno_num, err_msg)
OSError: [Errno 8] Exec format error
There is a potential solution here, which involves installing Chrome Drivers via Home Brew but that option is not available to me.
Any ideas?
Looks like this is complaining about the format of chromedriver binary.
It might be because of platform and chromedriver format mismatch. For example windows requires chromedriver.exe while there are different formats for linux and mac.
If you don't want to install through package manager, just download chromedriver from https://sites.google.com/a/chromium.org/chromedriver/downloads
Note : Choose file as per your os
Then place it anywhere on the os and pass that path as an argument. You can also set webdriver.chrome.driver environment variable if you don't want to pass the location every time.
FYI you could also encounter this issue if you did not unzip the chromedriver before adding it to your PATH.
I"m trying to get web2py running on an Ubuntu machine. All the docs seem to indicate that to run it on a *nix system, you download the source and do:
python web2py.py
I grabbed the source (stable source, not the trunk, version 1.99.4) and tried the above, but after entering a password for the server I get (in the terminal):
$ python web2py.py
web2py Web Framework
Created by Massimo Di Pierro, Copyright 2007-2011
Version 1.99.4 (2011-12-14 14:46:14) stable
Database drivers available: google
Starting hardcron...
WARNING:web2py.cron:WEB2PY CRON: Disabled because no file locking
please visit:
http://127.0.0.1:8000
starting browser...
failed to create drawable
DEBUG: connect attempt 0, connection error:
Traceback (most recent call last):
File "/home/aparkin/Downloads/web2py/gluon/dal.py", line 4736, in __init__
self._adapter = ADAPTERS[self._dbname](*args)
File "/home/aparkin/Downloads/web2py/gluon/dal.py", line 1634, in __init__
raise RuntimeError, "Unable to import driver"
RuntimeError: Unable to import driver
WARNING:web2py.cron:WEB2PY CRON: Disabled because no file locking
DEBUG: connect attempt 1, connection error:
Traceback (most recent call last):
File "/home/aparkin/Downloads/web2py/gluon/dal.py", line 4736, in __init__
self._adapter = ADAPTERS[self._dbname](*args)
File "/home/aparkin/Downloads/web2py/gluon/dal.py", line 1634, in __init__
raise RuntimeError, "Unable to import driver"
RuntimeError: Unable to import driver
DEBUG: connect attempt 2, connection error:
Traceback (most recent call last):
File "/home/aparkin/Downloads/web2py/gluon/dal.py", line 4736, in __init__
self._adapter = ADAPTERS[self._dbname](*args)
File "/home/aparkin/Downloads/web2py/gluon/dal.py", line 1634, in __init__
raise RuntimeError, "Unable to import driver"
RuntimeError: Unable to import driver
DEBUG: connect attempt 3, connection error:
Traceback (most recent call last):
File "/home/aparkin/Downloads/web2py/gluon/dal.py", line 4736, in __init__
self._adapter = ADAPTERS[self._dbname](*args)
File "/home/aparkin/Downloads/web2py/gluon/dal.py", line 1634, in __init__
raise RuntimeError, "Unable to import driver"
RuntimeError: Unable to import driver
DEBUG: connect attempt 4, connection error:
Traceback (most recent call last):
File "/home/aparkin/Downloads/web2py/gluon/dal.py", line 4736, in __init__
self._adapter = ADAPTERS[self._dbname](*args)
File "/home/aparkin/Downloads/web2py/gluon/dal.py", line 1634, in __init__
raise RuntimeError, "Unable to import driver"
RuntimeError: Unable to import driver
And then in the browser I get:
Internal error
Ticket issued: welcome/127.0.0.1.2012-.....[abbreviated]
Any suggestions? I'm running Python 2.7.1+. I've read elsewhere that you also need pyschopg2 installed for web2py to run, and I've confirmed that this is in fact installed as well (I did a sudo apt-get install python-psycopg2 and got "python-psycopg2 is already the newest version").
I just downloaded and web2py runs ok in ubuntu, python 2.7.2+
bruce#vaiubuntu:~/Downloads/web2py$ python web2py.py
web2py Web Framework
Created by Massimo Di Pierro, Copyright 2007-2011
Version 1.99.4 (2011-12-14 14:46:14) stable
Database drivers available: SQLite3, pymysql, PostgreSQL
Starting hardcron...
Are you trying to run web2py on appengine environment?
Do you have SQLITE module working ok in your Python?
A possibility is permission issue, your user have permission to write to web2py/* folder?
It looks like a bug in web2py 1.99.4. I was having the same problem using Postgres and psycopg2, but in Windows.
I tried with 1.99.2 and it worked fine.
Actually, the bug seems to be related to the cryptic error message - 1.99.2 gave me the error message that the database doesn't exist, which was easily fixed. 1.99.4 does not mention the missing database. However, when I created the database and tried with 1.99.4, it worked fine.