Basically exactly as the question says, I'm trying to get the background colour out from a website.
At the moment I'm using BeautifulSoup to get the HTML, but it's proving a difficult way of the getting the CSS. Any help would be great!
This is not something you can reliably solve with BeautifulSoup. You need a real browser.
The simplest option would be to use selenium browser automation tool:
from selenium import webdriver
driver = webdriver.Firefox()
driver.get('url')
element = driver.find_element_by_id('myid')
print(element.value_of_css_property('background-color'))
value_of_css_property() documentation.
Related
I'm really new to web scraping. Is there anyone that could tell me how to search on google.com with Selenium in Python?
In order to search something on google, try this:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://www.google.com/')
textbox = driver.find_element_by_xpath('//input[#name=\"q\"]')
textbox.send_keys('who invented python\n')
Selenium probably isn't the best. other libraries/tools would work better. BeautifulSoup is the first one that comes to mind
Don't use Selenium, there are other tools more suitable for that. Be aware that Google does not look upon favorably on scraping their search results:
Google does not take legal action against scraping, likely for
self-protective reasons. However, Google is using a range of defensive
methods that makes scraping their results a challenging task.
Source: https://en.wikipedia.org/wiki/Search_engine_scraping
I am trying to get the value from webpage using selenium base on python. However, I do not get the point about how to make it. The value I want to catch is in the red mark side of below picture. Please help me to figure out this. Thank you very much!
Please click here to check the picture
There are few ways (By css,by xpath...) I recommend you reading this page: http://selenium-python.readthedocs.io/locating-elements.html
You also need some understanding about html, css and the DOM but the simplest way to get the value you want is by xpath (But its the slowest)
Open your web browser lets say chrome, right click on the element you want and inspect. It will open developer tools and you will have the html line selected already, right click it again-> copy -> copy xpath
from selenium import webdriver
driver = webdriver.Chrome('/path/to/chromedriver')
driver.get("http://www.web.com")
driver.find_element_by_xpath(the_xpath_you_copied_before_from_dev_tools).text
I'm trying to scroll down a specific div in a webpage (ticker box in facebook) using selenium (with python).
I can find the element, but when i try to scroll it down using:
header = driver.find_element_by_class_name("tickerActivityStories")
driver.execute_script('arguments[0].scrollTop = arguments[0].scrollHeight', header)
Nothing happens.
I've found a lot of answers, but mostly are for selenium in java, so i would like to know if it's possible to do it from python.
I've found a lot of answers, but mostly are for selenium in java
There is no difference, selenium has the same api in java and python, you just need to find the same function in python selenium.
First of all check if your JS works in the browser(if not, try scroll parent element).
Also you can try :
from selenium.webdriver.common.keys import Keys
header.send_keys(Keys.PAGE_DOWN)
I have a big amount of HTML files which I want to process using BeautifulSoup and generate some statistics. Although, I came across the problem that the HTML files contain scripts that may generate more HTML code which is not being processed. Therefore, I need to render all Javascript into static HTML before proceeding.
I have seen some options such as using Selenium, but it doesn't seem to fit since I don't want to launch a browser (it should be done in background).
Can someone please suggest an appropriate approach to this?
Thanks in advance!
Since you need a Javascript engine, using a headless browser is the way to go.
Using Selenium web driver with the PhantomJS headless browser is probably your best option:
driver = webdriver.PhantomJS()
driver.get("...")
bs = BeautifulSoup(driver.page_source)
Thank you for answering my previous question but as one is solved another is found apparently.
Interacting with the flash game itself is now the problem. I have tried researching how to do it in Selenium but it can't be done. I've seen FlashSelenium, Sikuli, and AutoIT.
I can only find documentation for FlashSelenium in Java, It's easier for me to use AutoIT rather than Sikuli as I'd have to learn to use Jpython to create the kind of script I want to, which I am not straying away from learning just trying to finish this as fast as possible. As for AutoIT, the only problem with it is that I don't undertsand how to use it with seleium
from selenium import webdriver
import autoit
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
driver.get("http://na58.evony.com/s.html?loginid=747970653D74727947616D65&adv=index")
driver.maximize_window()
assert "Evony - Free forever " in driver.title
So far I have this and It's doing what is suppose to do which is create a new account using that "driver.get" but when I reach to the page, it is all flash and I can not interact with anything in the webpage so I have to use AutoIT but I don't know how to get it to "pick-up" from where selenium left off. I want it to interact with a button on the webpage and from viewing a previous post on stackoverflow I can use a (x,y) to specify the location but unfortunately that post didn't explain beyond that. Any and all information would be great, thanks.
Yes, you can use any number of scraping libraries (scrapy and beautiful soup are both easy to use and very powerful). Personally though, I like Selenium and its python bindings because they're the most flexible. Your final script would look something like this:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
driver.get("http://xx.yy.zz")
# Click the "New comer, click here to play!" button
elem = driver. find_element_by_partial_link_text("Click here to play")
elem.send_keys(Keys.RETURN)
Can you post what the source of the page looks like (maybe using a Pastebin)?
Edit: updated to show you how to click the "Click here to play" button.