I'm fairly new to programing and have been trying my best to get a grasp of different concepts. I've made a few games and twitter bots, and am looking at trying some web scraping/automation.
I have been following along with this tutorial https://www.youtube.com/watch?v=f7LEWxX4AVI
and have gotten stuck trying to enter text into the search bar. I'm using chrome Version 81.0.4044.129 and have the right Chromedriver to go with it.
This is my code from the tutorial -
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://youtube.com')
searchbox = driver.find_element_by_xpath('//*[#id="search"]')
searchbox.send_keys('Type Type Type')
searchButton = driver.find_element_by_xpath('//*[#id="search-icon-legacy"]')
searchButton.click()
It opens up chrome but does not enter anything into the search bar
📷
https://gyazo.com/68da5e2e2340265ae50125558ed8ed11
I get these errors that open in a new window
and I get these in my python shell
📷
https://gyazo.com/e9d702ef1ec4e4bc422ae537a3b59fbe
any advice would be very much appreciated.
Related
I'm trying to translate webpage to English using python but not able to convert. can anyone please help me
options.add_argument("−−lang=es")
# # Create a new instance of the Chrome driver
driver = webdriver.Chrome(chrome_options=options)
I added the options but not working
Selenium newb here. I've had a very simple python script to fill out a questionnaire working for about 3 weeks. I just noticed it was getting stuck on the 3rd question today, and when I cleared cache and rebooted, Selenium seemingly can't find any element I try using Full or short XPATH on that page any longer.
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element:
I spent some time trying to match up ChromeDriver and Chrome Browser versions assuming that it auto-updated. However, I tested the same previously working script on a different website and it's working fine. So not sure if this small version difference is really the problem
Driver = 94.0.4606.61
Chrome = 94.0.4606.81
I checked for iframes, but can't seem to locate one under source or inspect. Here is the web form:
https://form.typeform.com/to/eUIsSKGd
I've been using hard.coded sleep since the beginning and it was working fine. I even extended the sleeps to exaggerate them, but still can't get past the "Start Button".
Here's the first snippet of the script that was working. Yes, I tried grabbing a fresh new XPATH
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome()
driver.get('https://form.typeform.com/to/eUIsSKGd')
time.sleep(5)
#StartButton
startbutton = driver.find_element_by_xpath('/html/body/div[3]/main/div[1]/div[2]/div/section/div[2]/div[1]/div')
startbutton.click()
Please help
I am trying to scrape information from a website. So far, I've been able to access the webpage, log in with a username and password, and then print that landing page's page source into a separate .html/.txt file as needed.
Here's where the problems arise: on that "landing page," there's a table that I want to scrape the data from. If I were to manually right-click on any integer on that table, and select "inspect," I'd find the integer with no problem. However, when looking at the page source as a whole, I don't see the integers- just variable/parameter names. This leads me to believe it is a dynamic website.
How can I scrape the data?
I've been to hell and back trying to scrape this website, and so far, here's how the available technology has worked for me:
Firefox, IE, and Opera do not render the table. My guess is that this is a problem on the website's end. Only Chrome seems to work if I log in manually.
Selenium's Chromium package has been failing on me repeatedly (on my Windows 7 laptop) and I have even posted a question about the matter here. For now I'll assume it's just a lost cause, but I'm willing to graciously accept anyone's benevolent help.
Spynner's description looked promising, but that setup has frustrated me for quite some time- and the lack of a clear introduction only compounds its cumbersome nature to a novice like myself.
I prefer to code in Python, as it is the language I am most comfortable with. I have a pending company request to have the company install Visual Studio on my computer (to try doing this in C#), but I'm not holding my breath...
If my code can be of any use, so far, here's how I'm using mechanize:
# Headless Browsing Using PhantomJS and Selenium
#
# PhantomJS is installed in current directory
#
from selenium import webdriver
import time
browser = webdriver.PhantomJS()
browser.set_window_size(1120, 550) # need a fake browser size to fetch elements
def login_entry(username, password):
login_email = browser.find_element_by_id('UserName')
login_email.send_keys(username)
login_password = browser.find_element_by_id('Password')
login_password.send_keys(password)
submit_elem = browser.find_element_by_xpath("//button[contains(text(), 'Log in')]")
submit_elem.click()
browser.get("https://www.example.com")
login_entry('usr_name', 'pwd')
time.sleep(10)
test_output = open('phantomjs_test_source_output.html', 'w')
test_output.write(repr(browser.page_source))
test_output.close()
browser.quit()
p.s.- if anyone thinks I should be tagging javascript to this question, let me know. I personally don't know javascript but I'm sensing that it might be part of the problem/solution.
Try something like this. Sometimes with dynamic pages you need to wait for the data to load.
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
WebDriverWait(my_driver, my_time).until(EC.presence_of_all_elements_located(my_expected_element))
http://selenium-python.readthedocs.io/waits.html
https://seleniumhq.github.io/selenium/docs/api/py/webdriver_support/selenium.webdriver.support.expected_conditions.html
I'm making a bot with selenium and IE to automatize some tasks. I'm now facing a problem that I don't know how to solve : there is a Java Applet and I need to click in a button that is inside this Applet, how could I do that ?
Try this example for python , i have removed the Java one:
from selenium import webdriver
wd = webdriver.Firefox()
wd.get("http://localhost")
wd.execute_script("function()")
Thank you for answering my previous question but as one is solved another is found apparently.
Interacting with the flash game itself is now the problem. I have tried researching how to do it in Selenium but it can't be done. I've seen FlashSelenium, Sikuli, and AutoIT.
I can only find documentation for FlashSelenium in Java, It's easier for me to use AutoIT rather than Sikuli as I'd have to learn to use Jpython to create the kind of script I want to, which I am not straying away from learning just trying to finish this as fast as possible. As for AutoIT, the only problem with it is that I don't undertsand how to use it with seleium
from selenium import webdriver
import autoit
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
driver.get("http://na58.evony.com/s.html?loginid=747970653D74727947616D65&adv=index")
driver.maximize_window()
assert "Evony - Free forever " in driver.title
So far I have this and It's doing what is suppose to do which is create a new account using that "driver.get" but when I reach to the page, it is all flash and I can not interact with anything in the webpage so I have to use AutoIT but I don't know how to get it to "pick-up" from where selenium left off. I want it to interact with a button on the webpage and from viewing a previous post on stackoverflow I can use a (x,y) to specify the location but unfortunately that post didn't explain beyond that. Any and all information would be great, thanks.
Yes, you can use any number of scraping libraries (scrapy and beautiful soup are both easy to use and very powerful). Personally though, I like Selenium and its python bindings because they're the most flexible. Your final script would look something like this:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
driver.get("http://xx.yy.zz")
# Click the "New comer, click here to play!" button
elem = driver. find_element_by_partial_link_text("Click here to play")
elem.send_keys(Keys.RETURN)
Can you post what the source of the page looks like (maybe using a Pastebin)?
Edit: updated to show you how to click the "Click here to play" button.