Scraping Google Play Store BeautifulSoup / Selenium - python

I'm scraping data from the Android Store (Google Play Store) and I want to automate the downloading process on a connected phone. I'm trying to automate the click on the 'Install' button in an app page with Selenium but I can't click on it
Here is my python code :
from selenium import webdriver
driver=webdriver.Safari()
driver.get("https://play.google.com/store/apps/details?
id=com.playdemic.golf.android")
dr_button = driver.find_element_by_xpath("//*[#id='fcxH9b']/div[4]/c-wiz/div/div[2]/div/div[1]/div/c-wiz[1]/c-wiz[1]/div/div[2]/div/div[2]/div/div[2]/div[2]/c-wiz/div/span/button")
dr_button.click()

The problem is Google uses element obfuscation to prevent automation against their site in a malicious manner. You are on the right path with using XPATH, but you're going to have to manually create the XPATH... path.. which will help simplify your code, anyway. You could so something such as:
dr_button = driver.find_element_by_xpath("//button[#aria-label='Install']")
EDIT: To clarify on element obfuscation, you can see all the class names, as well as other element attributes are a seemingly random 6 character alpha numeric string. These strings can and will change intermittently. Most element-finding is used my element id's and classes.

EDIT
I solved 90% of the problem with Safari, I had first to log in here is my code.
driver = webdriver.Safari()
driver.get("https://play.google.com/store/apps/details?
id=com.king.candycrushsaga")
connexionbutton= "//*[#id='gb_70']"
emailfield= "identifierId"
GoogleAccUser="*****#gmail.com"
GoogleAccPassword="*****"
passwordfield="//*[#id='password']/div[1]/div/div[1]/input"
nextButton = "//*[#id='identifierNext']/content/span"
nextButtonTwo = "//*[#id='passwordNext']/content/span"
appsTabW = "//*[#id='wrapper']/div[1]/div/ul/li[2]/a/span/span[2]"
appsTab = "//*[#id='wrapper']/div[1]/div/ul/li[2]/a"
installButton = "//*[#id='fcxH9b']/div[4]/c-
wiz/div/div[2]/div/div[1]/div/c-wiz[1]/c-
wiz[1]/div/div[2]/div/div[2]/div/div[2]/div[2]/c-wiz/div/span/button"
confirmInstallButton = "//*[#id='purchase-ok-button']/span"
xx = "//*[#id='purchase-cancel-button']"
loginbuttonElement = WebDriverWait(driver,10).until(lambda driver:
driver.find_element_by_xpath(xpathbutton))
loginbuttonElement.click()
emailFieldElement = WebDriverWait(driver,10).until(lambda driver:
driver.find_element_by_id(emailfield))
emailFieldElement.clear()
emailFieldElement.send_keys(GoogleAccUser)
nextButtonElement = WebDriverWait(driver,10).until(lambda driver:
driver.find_element_by_xpath(nextButton))
nextButtonElement.click()
passwordFieldElement = WebDriverWait(driver,10).until(lambda driver:
driver.find_element_by_xpath(passwordfield))
passwordFieldElement.clear()
passwordFieldElement.send_keys(GoogleAccPassword)
nextButtonElementTwo = WebDriverWait(driver,10).until(lambda driver:
driver.find_element_by_xpath(nextButtonTwo))
nextButtonElementTwo.click()
installButtonElement = WebDriverWait(driver,10).until(lambda driver:
driver.find_element_by_xpath(installButton))
installButtonElement.click()
confirmInstallButtonElement = WebDriverWait(driver,50).until(lambda
driver: driver.find_element_by_xpath(xx))
confirmInstallButtonElement.click()
Now the problem is the Install confirmation which is on a popup window and I can't detect the button to click on it and download the app

I have to log in, but i can click on "install"! :) Try other browser?
from selenium import webdriver
driver=webdriver.Firefox()
driver.get("https://play.google.com/store/apps/details?id=com.playdemic.golf.android")
dr_button = driver.find_element_by_xpath("//*[#id='fcxH9b']/div[4]/c-wiz/div/div[2]/div/div[1]/div/c-wiz[1]/c-wiz[1]/div/div[2]/div/div[2]/div/div[2]/div[2]/c-wiz/div/span/button")
dr_button.click()

Related

Problem with mobile auto-test python+appium+selenium

I'am new to mobile automation. I thought I got the gist, but I ran into a problem.
An error occurs when using the command send_keys()
selenium.common.exceptions.InvalidElementStateException: Message: Cannot set the element to '321785214'. Did you interact with the correct element?
I have a code
driver = webdriver.Remote("http://localhost:4723/wd/hub", desired_cap)
# Skip fingerprint
driver.find_element_by_xpath("//android.widget.Button[#bounds='[413,1802][668,1877]']").click()
# Click Continue
driver.find_element_by_xpath("//android.widget.Button[#content-desc='Continue']").click()
# Click to input phone number
input_phone_number = driver.find_element_by_xpath("//android.widget.EditText").click()
input_phone_number_edit = driver.find_element_by_class_name("android.widget.EditText")
# Enter phone number
input_phone_number_edit.send_keys("321785214")
And I have code from other mobile app and this code is working
driver = webdriver.Remote("http://localhost:4723/wd/hub", desired_cap)
# click to Set up later button
set_up_later_btn = driver.find_element_by_xpath('//android.widget.Button[#content-desc="Set up later"]').click()
input_phone_number = driver.find_element_by_xpath('//android.view.View[#content-desc="Phone number"]').click()
input_phone_number_edit = driver.find_element_by_class_name('android.widget.EditText')
driver.implicitly_wait(50)
input_phone_number_edit.send_keys('178546128')
driver.find_element_by_accessibility_id("Continue").click()
I don't know why first code isn't working. I need help. Thanks
Instead of using the default send_keys method, you can use the send_keys method provided by the ActionChains class
from selenium.webdriver import ActionChains
actions = ActionChains(self.driver)
actions.send_keys("321785214")
actions.perform()

can not switch to a new page using python selenium webdriver?

I need to scrape the CSV result of the website (https://requestmap.webperf.tools/). But I can’t.
This is the process I need to do:
1- load the website (https://requestmap.webperf.tools/)
2- enter a new website as an input (for example, https://stackoverflow.com/)
3- click submit
4- in the new page which opens after the submit, download the csv file at the end of the page
But I think the driver has the main page and doesn’t switch to the new page. That's why I can’t download the CSV file.
Would you please tell me how to do it?
here is my code:
options = webdriver.ChromeOptions()
options.add_argument('-headless')
options.add_argument('-no-sandbox')
options.add_argument('-disable-dev-shm-usage')
driver = webdriver.Chrome('chromedriver',options=options)
driver.get("https://requestmap.webperf.tools/")
driver.find_element_by_id("url").send_keys("https://stackoverflow.com/")
driver.find_element_by_id("submitter").click()
I have tested different ways to solve my issue:
First of all:
I got this code from here :
But it doesn't work.
window_after = driver.window_handles[1]
driver.switch_to.window(window_after)
I also tried this or this, but they are not working as well.
# wait to make sure there are two windows open
# it is not working
WebDriverWait(driver, 30).until(lambda d: len(d.window_handles) == 2)
# switch windows
# it is not working
driver.switch_to_window(driver.window_handles[1])
content = driver.page_source
soup = BeautifulSoup(content)
driver.find_elements_by_name("Download CSV")
So How can I get this issue solved?
Is there any other way in python to do so and switch to a new windows?

How to click the icon using selenium

I am trying to automate my certain activities using selenium. I was launching a webpage and a security windows popup appears for the login credentials.
I have automated that using Autoit. Now after login I need to click on the option and I have tried it based on the find_element_by_text and find_element_by_id.
I was getting an error as Unable to find the element with CSS selector and I have seen some other post in the StackOverflow with the same issue but I could not able to fix it by myself.
Here is how my HTML looks like. Could you please guide me on this and also please share any document for further checking. Thanks.
driver = webdriver.Ie(r"C:\\IEDriverServer\\IEDriverServer.exe")
driver.get('URL of the page')
#driver.implicitly_wait(10) # seconds
autoit.win_wait("Windows Security")
# now make that crap active so we can do stuff to it
autoit.win_activate("Windows Security")
# type in the username
autoit.send('username')
# tab to the password field
autoit.send("{TAB}")
# type in the password
autoit.send('password')
# kill!
autoit.send("{ENTER}")
driver.maximize_window()
driver.implicitly_wait(120) # seconds
#submit_button_incidents = driver.find_element_by_link_text("3-Normal Incidents")
submit_button_incidents= driver.find_element_by_id("nodeImgs13pm")
submit_button_incidents.click()
driver.implicitly_wait(10)
++ updating the info
I have tried to copy the whole HTML but the page was restricted so I cant able to view the full HTML page other than the basic templates. Adding some more screenshots of the developer tools.
Here how my webpage looks like.
try with this code :
submit_button_incidents = WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.XPATH, "//span[contains(text(),'My Group'])")))
submit_button_incidents.click()
and do not use implicit wait too many times in code.
Implicit wait is set for life time of web driver instance.
Reference :
Selenium official document for wiats
Xpath tutorial
cssSelector tutorial
UPDATE :
As OP has shared the HTML code. As per the requirement you can go ahead with this code.
As there are two elements with text as My Groups's queue.
For first one you can write XPATH as :
//a[#class='scbdtext']/span[contains(text(),'My Group')]
Code:
submit_button_incidents = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//a[#class='scbdtext']/span[contains(text(),'My Group')]")))
submit_button_incidents.click()
For second one you can write XPATH as :
//a[#class='scbdtextfoc']/span[contains(text(),'My Group')]
Code :
submit_button_incidents = WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.XPATH, "//a[#class='scbdtextfoc']/span[contains(text(),'My Group')]")))
submit_button_incidents.click()
Hope this will help.
Use ActionChains with double click for this to work in Python.
from selenium.webdriver import ActionChains
# Get the element however you want
element = driver.find_element_by_xpath("//a[#class='scbdtextfoc']/span[contains(text(),'My Group')]")
ActionChains(driver).double_click(settings_icon).perform()

send files to a website through a 'browse files' on pc

I'm browsing through a website using dryscrape in python and i need to upload a file to this site. But there is only one way of doing it, that is clicking in a button and browse into my files and select the one i want. How can i do it with python? i would appreciate if someone could help me using dryscrape too, but i'm accepting all answers.
heres the example image:
You can use Selenium. I tested this code and it works.
from selenium import webdriver
url = "https://example.com/"
driver = webdriver.Chrome("./chromedriver")
driver.get(url)
input_element = driver.find_element_by_css_selector("input[type=\"file\"]")
# absolute path to file
abs_file_path = "/Users/foo/Downloads/bar.png"
input_element.send_keys(abs_file_path)
sleep(5)
driver.quit()
Resources
Selenium python
Chrome driver download
For those who are searching for the answer in dryscrape i translated the selenium code to dryscrape:
element = sessin.at_xpath("xpath...") # Session is a dryscrape session
# The xpath is from the button like the one in the image "Browse..."
element.set("fullpath")
just as simple as it is.

Can we Zoom the browser window in python selenium webdriver?

I am trying to ZOOM IN and ZOOM OUT the Chrome( selenium webdriver) only using keyboard. I have tried --
from selenium.webdriver.common.keys import Keys
driver.find_element_by_tag_name("body").send_keys(Keys.CONTROL,Keys.SUBTRACT).
but it is not working. Need answer in python.
I was just struggling with this. I managed to find something that works for me, hopefully it works for you:
driver.execute_script("document.body.style.zoom='zoom %'")
Have 'zoom%' = whatever zoom level you want. (e.g. '67%')
Environment:
Selenium 3.6.0
chromedriver 2.33
Chrome version 62.0.3202.75 (Official Build) (64-bit)
macOS Sierra 10.12.6
I tried the ways (without use the CSS) that people suggested in other questions in the past. For example, the answers in this question: Selenium webdriver zoom in/out page content.
Or this: Test zoom levels of page on browsers
without success.
So, I thought: if not with the shortcuts, what could be a different way to do that?
The idea is to use the "chrome://settings/" page in order to change the zoom:
Ok I know, for example from Going through Chrome://settings by Selenium, that every settings should be set in the ChromeOptions.
From this question I noticed that in the list of preferences the only paramater (I think) could be:
// Double that indicates the default zoom level.
const char kPartitionDefaultZoomLevel[] = "partition.default_zoom_level";
I tried, without success.
I want to repeat that I know it isn't the correct approach (and that will be different with different browser versions), but it works and, at least, was useful for me to understand how to go inside a shadow root element with selenium.
The following method return the elements inside a shadow root:
def expand_shadow_element(element):
shadow_root = driver.execute_script('return arguments[0].shadowRoot', element)
return shadow_root
It is useful because in the chrome://settings/ page there are shadow root elements.
In order to do that in my browser, this is the path:
root1=driver.find_element_by_xpath("*//settings-ui")
shadow_root1 = expand_shadow_element(root1)
container= shadow_root1.find_element_by_id("container")
root2= container.find_element_by_css_selector("settings-main")
shadow_root2 = expand_shadow_element(root2)
root3=shadow_root2.find_element_by_css_selector("settings-basic-page")
shadow_root3 = expand_shadow_element(root3)
basic_page = shadow_root3.find_element_by_id("basicPage")
settings_section= basic_page.find_element_by_xpath(".//settings-section[#section='appearance']")
root4= settings_section.find_element_by_css_selector("settings-appearance-page")
shadow_root4=expand_shadow_element(root4)
and finally:
settings_animated_pages= shadow_root4.find_element_by_id("pages")
neon_animatable=settings_animated_pages.find_element_by_css_selector("neon-animatable")
zoomLevel= neon_animatable.find_element_by_xpath(".//select[#id='zoomLevel']/option[#value='0.5']")
zoomLevel.click()
The entire code:
driver = webdriver.Chrome(executable_path=r'/pathTo/chromedriver')
def expand_shadow_element(element):
shadow_root = driver.execute_script('return arguments[0].shadowRoot', element)
return shadow_root
driver.get('chrome://settings/')
root1=driver.find_element_by_xpath("*//settings-ui")
shadow_root1 = expand_shadow_element(root1)
container= shadow_root1.find_element_by_id("container")
root2= container.find_element_by_css_selector("settings-main")
shadow_root2 = expand_shadow_element(root2)
root3=shadow_root2.find_element_by_css_selector("settings-basic-page")
shadow_root3 = expand_shadow_element(root3)
basic_page = shadow_root3.find_element_by_id("basicPage")
settings_section= basic_page.find_element_by_xpath(".//settings-section[#section='appearance']")
root4= settings_section.find_element_by_css_selector("settings-appearance-page")
shadow_root4=expand_shadow_element(root4)
settings_animated_pages= shadow_root4.find_element_by_id("pages")
neon_animatable=settings_animated_pages.find_element_by_css_selector("neon-animatable")
zoomLevel= neon_animatable.find_element_by_xpath(".//select[#id='zoomLevel']/option[#value='0.5']")
zoomLevel.click()
driver.get("https://www.google.co.uk/")
EDIT
As suggested by #Florent B in the comments, we can obtain the same result simple with:
driver.get('chrome://settings/')
driver.execute_script('chrome.settingsPrivate.setDefaultZoom(1.5);')
driver.get("https://www.google.co.uk/")
firefox solution for me,
Zoom body browser
zoom is a non-standard property, use transform instead (demo):
driver.execute_script("document.body.style.transform = 'scale(0.8)'")
https://github.com/SeleniumHQ/selenium/issues/4244
driver.execute_script('document.body.style.MozTransform = "scale(0.50)";')
driver.execute_script('document.body.style.MozTransformOrigin = "0 0";')
Yes, you can invoke the Chrome driver to zoom without having to use CSS. There are methods packaged into the Chrome DevTools Protocol Viewer, one being Input.synthesizePinchGesture aka zoom in/out.
For ease of use, with regards to the DevTools Protocol API, we will use a class called MyChromeDriver with webdriver.Chrome as a metaclass and a new method for sending these commands to Chrome:
# selenium
from selenium import webdriver
# json
import json
class MyChromeDriver(webdriver.Chrome):
def send_cmd(self, cmd, params):
resource = "/session/%s/chromium/send_command_and_get_result" % self.session_id
url = self.command_executor._url + resource
body = json.dumps({'cmd':cmd, 'params': params})
response = self.command_executor._request('POST', url, body)
return response.get('value')
1. Setup our webdriver and get some page:
webdriver = MyChromeDriver()
webdriver.get("https://google.com")
2. Send Chrome the Input.synthesizePinchGesture command along with its parameters via our new method send_cmd:
webdriver.send_cmd('Input.synthesizePinchGesture', {
'x': 0,
'y': 0,
'scaleFactor': 2,
'relativeSpeed': 800, # optional
'gestureSourceType': 'default' # optional
})
3. Walla! Chrome's zoom is invoked:
As a side note, there are tons of other commands you could use with send_cmd. Find them here: https://chromedevtools.github.io/devtools-protocol/
Based off this answer: Take full page screen shot in Chrome with Selenium
As you mentioned that Need it to work in Chrome. The current solutions are only for Firefox, here are a few updates and options :
Zoom the CSS :
driver.execute_script("document.body.style.zoom='150%'")
This option did work for me. But it zooms the CSS, not the Chrome Browser. So probably you are not looking at that.
Zoom In & Zoom Out the Chrome Browser :
After 4131, 4133 and 1621 the fullscreen() mode got supported to Zoom In through Selenium-Java Clients but it's not yet publicly released to PyPI.
I can see it's implemented but not pushed. Selenium 3.7 (Python) will be out soon. The push to sync version numbers will include that.
Configure the webdriver to open the Browser :
If your requirement is to execute the Test Suite in Full Screen mode, you can always use the Options Class and configure the webdriver instance with --kiosk argument as follows:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("--kiosk")
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get('https://www.google.co.in')
# zoom in firefox browser
driver.get("about:preferences")
driver.execute_script("arguments[0].click();", driver.find_element(By.XPATH, "// [#id='defaultZoom']"))
ActionChains(driver).click(driver.find_element(By.XPATH, "//*[#value='50']")).perform()

Categories

Resources