when I can't delete FF cookies from webdriver. When I use the .delete_all_cookies method, it returns None. And when I try to get_cookies, I get the following error:
webdriver_common.exceptions.ErrorInResponseException: Error occurred when processing
packet:Content-Length: 120
{"elementId": "null", "context": "{9b44672f-d547-43a8-a01e-a504e617cfc1}", "parameters": [], "commandName": "getCookie"}
response:Length: 266
{"commandName":"getCookie","isError":true,"response":{"lineNumber":576,"message":"Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIDOMLocation.host]","name":"NS_ERROR_FAILURE"},"elementId":"null","context":"{9b44672f-d547-43a8-a01e-a504e617cfc1} "}
How can I fix it?
Update:
This happens with clean installation of webdriver with no modifications. The changes I've mentioned in another post were made later than this post being posted (I was trying to fix the issue myself).
Hmm, I actually haven't worked with Webdriver so this may be of no help at all... but in your other post you mention that you're experimenting with modifying the delete cookie webdriver js function. Did get_cookies fail before you were modifying the delete function? What happens when you get cookies before deleting them? I would guess that the modification you're making to the delete function in webdriver-read-only\firefox\src\extension\components\firefoxDriver.js could break the delete function. Are you doing it just for debugging or do you actually want the browser itself to show a pop up when the driver tells it to delete cookies? It wouldn't surprise me if this modification broke.
My real advice though would be actually to start using Selenium instead of Webdriver since it's being discontinued in it's current incarnation, or morphed into Selenium. Selenium is more actively developed and has pretty active and responsive forms. It will continue to be developed and stable while the merge is happening, while I take it Webdriver might not have as many bugfixes going forward. I've had success using the Selenium commands that control cookies. They seem to be revamping their documentation and for some reason there isn't any link to the Python API, but if you download selenium rc, you can find the Python API doc in selenium-client-driver-python, you'll see there are a good 5 or so useful methods for controlling cookies, which you use in your own custom Python methods if you want to, say, delete all the cookies with a name matching a certain regexp. If for some reason you do want the browser to alert() some info about the deleted cookies too, you could do that by getting the cookie names/values from the python method, and then passing them to selenium's getEval() statement which will execute arbitrary js you feed it (like "alert()"). ... If you do go the selenium route feel free to contact me if you get a blocker, I might be able to assist.
Related
I would like to launch multiple selenium sessions and only one of them make visible. An user can interact with this webdriver window, and I want to retranslate all his actions on other sessions.
How can I do that on python?
What you could do is that you set up the driver to inject javascript on each page load. There are solutions for this here in the answer section of this question.
You will need a javascript which is capable is detecting the actions you want to 'copy'. If you are interested only in click actions, a very simple js could do the job of capturing click events, there are a few examples in the answer section here. If you can ensure that the window sizes are the same, the X,Y coordinates from the event could be just enough. In the project where I did something similar I calculated the XPATH of the clicked element, and grabbed the value of all input fields. It is not a trivial task but a quite possible one. When you have the data to replicate the events, you need to send that back to python. See the answers to this question how to post data via javascript.
You will need a flask/bottle or something similar framework (or re-invent the wheel) to receive the data and send it to the other selenium instances. There are some good examples how to get the JSON data from the browser to flask in the answers of this question.
You may need to handle CORS problems when sending the data from the browser to flask. See the answers of this question how you can handle that.
If you have the data in flask, all you need to do is replicate the events in the other driver instances. If you work with coordinates, you can issue a click on the page via ActionChains, see answers of this question. Or if you have element id-s or xpath or something, you can find the element and click it. Writing a parser which translates the events into selenium actions is not the most difficult part of a project like this. :)
Well, simple as that. I'd recommend to try it, because it would improve your skills a lot. I don't have copy-paste ready code for this, but I gave all the ingredients you need to start.
No.
Selenium can control only those sessions, which have been initialized by its own.
You cannot retranslate test actions for more that 1 session.
The only similar thing you can do is to create few (as much as you need) tests, which will contain the same copy-pasted steps inside itself and run each of them in a separate window.
I've been working on an automation script with selenium, after developing it locally I was able to make it consistently work on my local browser, however, I would like to set up the file to run on pythonanywhere.com. So I uploaded the code and ran it from there. However, when I run the code I come with an error Message: Element is disabled and so may not be used for actions. The element must be correct and interactable with since it worked perfectly locally on my firefox browser, it may be because of version differences in the firefox but I don't think so. I've seen answers for this on Stack but they say that the element is wrong, which shouldn't (but may) apply in my case. I am able to successfully grab the title from the website.
I won't be able to share the code due to the sensitivity of the process, but I can share that's it's a google form and the full error message, any help will be gratefully appreciated.
Thank you
Message: Element is disabled and so may not be used for actions
Stacktrace:
at fxdriver.preconditions.enabled (file:///tmp/tmpi99ptn1x/extensions/fxdriver#googlecode.com/components/command-processor.js:10098)
at DelayedCommand.checkPreconditions_ (file:///tmp/tmpi99ptn1x/extensions/fxdriver#googlecode.com/components/command-processor.js:126
44)
at DelayedCommand.executeInternal_/h (file:///tmp/tmpi99ptn1x/extensions/fxdriver#googlecode.com/components/command-processor.js:1266
1)
at DelayedCommand.executeInternal_ (file:///tmp/tmpi99ptn1x/extensions/fxdriver#googlecode.com/components/command-processor.js:12666)
at DelayedCommand.execute/< (file:///tmp/tmpi99ptn1x/extensions/fxdriver#googlecode.com/components/command-processor.js:12608)
This error message...
Element is disabled and so may not be used for actions
...implies that the element with which you are trying to interact is disabled and hence cannot be used for actions.
Deep Dive
The relevant HTML and your code trials would have helped us to construct a cannonical answer. However, as you mentioned you were able to make it consistently work on my local browser but doesn't pythonanywhere.com at this junction it is worth to mention that Selenium tends to mock the User Actions and some among them are:
Sending Text : sendKeys("LiamHarries")
Special Keyboard Characters : sendKeys(Keys.ENTER) and sendKeys(Keys.RETURN)
Mouse Hover : moveToElement(element).perform()
Click : click()
All these User Actions are independent of the underlying os and Hardware configuration. Hence if they run the same application they will behave the same.
You can find a detailed discussion in Chrome & Firefox on Windows vs Linux (selenium)
This error
This issue have been discussed earlier in the thread Selenium webdriver :org.openqa.selenium.InvalidElementStateException: Element is disabled and so may not be used for actions where it was pretty much apparent that OP was trying to probe if the element driver.findElement(By.xpath("//input[#id='gs_htif0']")) was enabled.
Definitely, the xpath based on the value of id attribute gs_htif0 wasn't a static one and hence it is bound to change periodically and may be available/unavailable in indigenous systems.
Solution
In these cases the solution is to construct the Locator Strategy based on static values of attributes.
References
You can find a couple of relevant discussions in:
InvalidElementStateException invalid element state: Element must be user-editable in order to clear it" error while sending text with Selenium Python
Invalid element state: Element must be user-editable in order to clear it error trying to click and insert a date on a dropdown-toggle using Selenium
Selenium : How to solve org.openqa.selenium.InvalidElementStateException: invalid element state
InvalidElementStateException when attempting to clear text through Selenium
As a beginner I've been heavily warned to avoid resource heavy browsers for web scraping such as Selenium.
Then I looked at this site: Intcomex Webstore
My idea was to make an alert program to tell me the price and if the item was low in quantity.
I can't for the life of me figure out how one would even attempt to get any of this information, whether through the CSV/EXML files or directly.
I'd possibly use requests however it only returns the javascript function as a link: href="javascript:PriceListExportCSV('/en-XUS/Products/Csv','query');
In Developer Tools after I've clicked the CSV link I see a GET request to http://store.intcomex.com/en-XUS/Products/Csv
However if I use requests I get status_code = 404.
Any help to point me in the right direction is greatly appreciated.
After taking the advice of many helpful commenters, I've come to the conclusion that I indeed need to use a browser such as Selenium.
While it may not be the ideal solution, it appears to be only viable one at the moment.
from selenium import webdriver
browser = webdriver.Chrome()
browser.get('http://store.intcomex.com/en-XUS/Products/ByCategory/cpt.allone?r=True')
browser.execute_script("javascript:PriceListExportCSV('/en-XUS/Products/Csv','query');")
I'll have to figure it out from here...
I'm trying to load one web page and get some elements from it. So the first thing I do is to check the page using "inspect element". When I search for the tags I'm looking for, I can see them (in Chrome).
But when I try to do driver.get(url) and then driver.find_element_by_..., it doesn't find those elements because they aren't in the source code.
I think that it is probably because it doesn't load the whole page but only a part.
Here is an example:
I'm trying to find ads on the web page.
PREPARED_TABOOLA_BLOCK = """//div[contains(#id,'taboola') and not(ancestor::div[contains(#id,'taboola')])]"""
driver = webdriver.PhantomJS(service_args=["--load-images=false"])
# driver = webdriver.Chrome()
driver.maximize_window()
def find_taboola_blocks_selenium(url):
driver.get(url)
taboola_blocks = driver.find_elements_by_xpath(PREPARED_TABOOLA_BLOCK)
return taboola_blocks
print len(find_taboola_blocks_selenium('http://www.breastfeeding-problems.com/breastfeeding-a-sick-baby.html'))
driver.get('http://www.breastfeeding-problems.com/breastfeeding-a-sick-baby.html')
print len(driver.page_source)
OUTPUTS:
Using PhantomJS:
0
85103
Using ChromeDriver:
3
420869
Do you know how to make PhantomJS to load as much Html as possible or any other way to solve this?
Can you compare the request that ChromeDriver is making versus the request you are making in PhantomJS? Since you are only doing GET for the specified url, you may not be including other request parameters that are needed to get the advertisements.
The open() method may give you a better representation of what you are looking for here: http://phantomjs.org/api/webpage/method/open.html
The reason for this is because PhantomJS, by default, renders in a really small window, which makes it load the mobile version of the site. And with the PhantomJSDriver, calling maximizeWindow() (or maximize_window() in python) does absolutely nothing, since there is no rendered window to maximize. You will have to explicitly set the window's render size with:
edit: Below is the Java solution. I'm not entirely sure what the Python solution would be when setting the window size, but it should be similar.
driver.manage().window().setSize(new Dimension(1920, 1200));
edit again: Found the python version:
driver.set_window_size(1920, 1200)
Hope that helps!
PhantomJS 1.x is a really old browser. It only uses SSLv3 (now disabled on most sites) by default and doesn't implement most cutting edge functionality.
Advertisement scripts are usually delivered over HTTPS (SSLv3/TLS) and usually use some obscure feature of JavaScript which is not well tested or simply not implemented in PhantomJS.
If you use PhantomJS < v1.9.8 then you should use those commandline options (service_args): --ignore-ssl-errors=true --ssl-protocol=any.
If iframes or strange cross-domain requests are necessary for the page/ads to work, then add --web-security=false to the service_args.
If this still doesn't solve the problem, then try upgrading to PhantomJS 2.0.0. You might need to compile it yourself on Linux.
My coding experience is in Python. Is there a simple way to execute a python code in firefox that would detect a particular address, say nytimes.com, load the page, then delete the end of the address following html (this allows bypassing the 20 pageviews/month limit) and reload?
Your best bet is to use selenium as proposed before. Here's a small example how you could do it. Basically the code checks if the limit has been reached and if it has it deletes cookies and refreshes the page letting you to continue reading. Deleting cookies lets you read another 10 articles without continuously editing the address. Thats the technical part, you have to consider the legal implications yourself.
from selenium import webdriver
browser=webdriver.Firefox()
browser.get('http://www.nytimes.com')
if browser.find_element_by_xpath('.//*[contains(.,"You’ve reached the limit of 10 free articles a month.")]'):
browser.delete_all_cookies()
browser.refresh()
you can use selenium it lets you easily fully control firefox and other web browsers with python. it would only be a few lines of code to acheive this. this answer How to integrate Selenium and Python has a working example