I am using anticaptcha to help out with bypassing recaptcha on a webpage I'm crawling.
I have managed to work out the api part of this solution. It's quite straightforward.
The part I am struggling with is the injection of the token received from anti-captcha into the webpage.
Haven't found too many resources on this. I am using Selenium and Python alongside the anticaptchaofficial module.
The script I am executing does change the innerHtml of the textarea with id g-recaptcha-response but the webpage does nothing and the checkbox doesn't load the spinner or get verified.
Here's my code:
from anticaptchaofficial.recaptchav2proxyless import recaptchaV2Proxyless
from selenium import webdriver
import os
import time
driver = webdriver.Chrome(os.path.normpath(os.getcwd()+"\\chromedriver.exe"))
driver.get("https://www.google.com/recaptcha/api2/demo")
time.sleep(1)
data_sitekey = driver.find_element_by_class_name('g-recaptcha').get_attribute('data-sitekey')
solver = recaptchaV2Proxyless()
solver.set_verbose(1)
solver.set_key("<--my-key-->")
solver.set_website_url("https://www.google.com/recaptcha/api2/demo")
solver.set_website_key(data_sitekey)
g_response = solver.solve_and_return_solution()
driver.execute_script('document.getElementById("g-recaptcha-response").innerHTML = "{}";'.format(g_response)) # target textarea that is supposed to be injected with the token, I found upon some research
driver.execute_script("onSuccess('{}')".format(g_response))
time.sleep(1)
Turns out I was under the assumption that the recaptcha frame would show visible feedback on injection of the token (or some other equivalent action) but it turns out just the line:
driver.execute_script('document.getElementById("g-recaptcha-response").innerHTML = "{}";'.format(g_response))
which updates the textarea's innerHtml is enough. So you would basically need to continue with your task ie: click submit, if it is a recaptcha on form or reload the page if it is just randomly triggered
from anticaptchaofficial.recaptchav2proxyless import recaptchaV2Proxyless
from selenium import webdriver
import os
import time
driver = webdriver.Chrome(os.path.normpath(os.getcwd()+"\\chromedriver.exe"))
driver.get("https://www.google.com/recaptcha/api2/demo")
time.sleep(1)
data_sitekey = driver.find_element_by_class_name('g-recaptcha').get_attribute('data-sitekey')
solver = recaptchaV2Proxyless()
solver.set_verbose(1)
solver.set_key("<--my-key-->")
solver.set_website_url("https://www.google.com/recaptcha/api2/demo")
solver.set_website_key(data_sitekey)
g_response = solver.solve_and_return_solution()
driver.execute_script('document.getElementById("g-recaptcha-response").innerHTML = "{}";'.format(g_response))
time.sleep(1)
# whatever the next step is. Could be clicking on a submit button
driver.refresh()
Related
I am trying to make a bot that automatically fills out this form automatically. I am using selenium with python, and I can get the script to populate the fields of the form correctly, but when I click on the search button, it doesn't bring me to the next page - it seems to just refresh the current page. I have tried filling out the form with the script and hitting the search button manually, but the same behavior is produced. I have also tried using .click() and .submit() - no dice. Apologies if some of my code is messy as this is one of my first python scripts. Thanks!
# Tee time booker
# This script books a tee time at ponemah greens
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
import time
site = webdriver.Chrome(ChromeDriverManager().install())
# user inputs
play_date="07/11/2021"
num_players=2
site.get("https://amherstpp.ezlinksgolf.com/index.html#/preSearch")
time.sleep(1)
title = site.title
assert "Amherst CC Prepaid - Online tee times made EZ" in title
dateField=site.find_element_by_xpath('//*[#id="dateInput"]')
dateField.clear()
dateField.send_keys(play_date)
time.sleep(1)
playerField=site.find_element_by_xpath('//*[#id="pc"]')
playerField.send_keys(num_players)
time.sleep(1)
courseSelector=site.find_element_by_xpath('/html/body/div[3]/div[2]/div[2]/ui-view/div/div/div/div[2]/div[2]/form/div/ul/li[4]/div/ul/li[2]/div/div[1]/input')
courseSelector.click()
time.sleep(1)
searchButton=site.find_element_by_xpath('/html/body/div[3]/div[2]/div[2]/ui-view/div/div/div/div[2]/div[2]/form/div/div/div/button')
searchButton.send_keys("\n")
print("Button Click")
I am coding a Selenium bot (with Python) that uploads a picture to instagram from a queue in a directory. For now, I have successfully logged in on Instagram and I am now trying to interact somehow with the upload button.
I have tried to click() on it, but then a window pops up where I would normally browse my computer to find the image I want to upload. I've found that I need import autoit, but I can't understand how it works and the documentation doesn't help either, so I'd rather avoid using this.
This is what I have for now:
import os
from selenium import webdriver
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
class InstaBot():
# A COOKIES POP UP ALWAYS APPEARS UPON OPENING INSTAGRAM, SO INIT ALSO CLOSES IT
# TO UPLOAD ON INSTAGRAM, THE MOBILE VERSION IS NEEDED, WE TRY TO EMULATE A GALAXY S5
def __init__(self):
mobile_emulation = {"deviceName": "Galaxy S5"}
chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option("mobileEmulation", mobile_emulation)
# experimental options for mobile emulation added
self.driver = webdriver.Chrome(ChromeDriverManager().install(), chrome_options = chrome_options)
self.driver.get("https://www.instagram.com/")
# note that this accepts all cookies
cooki = self.driver.find_element_by_xpath('/html/body/div[2]/div/div/div/div[2]/button[1]')
cooki.click()
# FINDS THE USERNAME AND PASSWORD AND TYPES 2 INPUTS ACCORDINGLY
def loginfun(self):
entrar = WebDriverWait(self.driver, 10).until(EC.presence_of_element_located((By.XPATH, '//*[#id="react-root"]/section/main/article/div/div/div/div[2]/button')))
entrar.click()
usbar = pasbar = self.driver.find_element_by_xpath('//*[#id="loginForm"]/div[1]/div[3]/div/label/input')
usbar.send_keys(input('Username: '))
# once usbar is found, the rest will be as well
pasbar = self.driver.find_element_by_xpath('//*[#id="loginForm"]/div[1]/div[4]/div/label/input')
pasbar.send_keys(input('Password: '))
logbtn = self.driver.find_element_by_xpath('//*[#id="loginForm"]/div[1]/div[6]/button')
logbtn.click()
# CLOSES PASSWORD SAVING AND NOTIFICATION MESSAGES IN CASE THEY APPEAR
def tryclose(self):
try:
nopass = WebDriverWait(self.driver, 10).until(EC.presence_of_element_located((By.XPATH, '//*[#id="react-root"]/section/main/div/div/div/button')))
nopass.click()
except Exception:
pass
try:
nonot = WebDriverWait(self.driver, 10).until(EC.presence_of_element_located((By.XPATH, '/html/body/div[4]/div/div/div/div[3]/button[2]')))
nonot.click()
except Exception:
pass
# bla bla bla
# Basically I emulate a mobile device and log into my account,
# then I have at the bottom center of the page the upload button that
# looks like this [+]. Here is what I try:
# SELECTS THE FIRST IMAGE FROM THE PENDING DIR. USES THE UPLOAD BUTTON VIA SEND_KEYS.
# AFTER THAT, IT MOVES SAID IMAGE TO THE 'DONE' DIR.
def upload(self):
# first pending image
pend_img = (os.listdir('C:/path to my queue dir')[0])
# finds the upload button and send_keys the image to it
upbtn = self.driver.find_element_by_xpath('//*[#id="react-root"]/section/nav[2]/div/div/form/input')
upbtn.send_keys('C:/path to image in queue'+pend_img)
# moves the image to the 'done' directory
os.rename('C:/path to image in queue dir'+pend_img , \
'C:/path to image in done dir'+pend_img)
After this process, this code is able to find the image in the 'pending' (queue) directory and move it to the 'done' directory, BUT it does not interact with instagram whatsoever. So send_keys() is not working. I am a newbie in this, but I recon that the HTML path to the button upbtn may be wrong, although I cannot find any other input path or anything.
NOTE: to clarify, no errors are shown, the problem is just that send_keys does not interact with the upload button with this code.
Anyone has a fool-proof solution or an intuitive way to upload to Instagram?
Following advice from the comments, I looked for all the input tags in the HTML of the page:
After my search, these are all the XPaths to the input tags I could find, all of them have type="file":
//*[#id="react-root"]/form/input
//*[#id="react-root"]/section/main/div[1]/form/input
//*[#id="react-root"]/section/nav[1]/div/div/form/input # this one is for stories i think
//*[#id="react-root"]/section/nav[2]/div/div/form/input # it should be this one
I have tried send_keys() to all of them, yet none seems to work.
If you don't mind, I recommend a package that might help and works so good!
Instabot is capable to upload photos/videos & stories emulating a mobile device.
Installation:
pip install instabot
Implementation:
#Call bot
from instabot import Bot
Then only need a few more steps:
bot = Bot()
bot.login(username = 'user', password = 'pass')
bot.upload_photo(image_path, caption = 'Hello world')
I recommend this option due to is clean, fast and reliable
More info, visit: https://pypi.org/project/instabot/
I'm trying to automate a duolingo login with Selenium with the code posted below.
While everything seems to work as expected at first, I always get an "Wrong password" message on the website after the login button is clicked.
I have checked the password time and time again and even changed it to one without special characters, but still the login fails.
I have seen in other examples that there is sometimes an additional password input field, however I cannot find one while inspecting the html.
What could I be missing ?
(Side note: I'm also open to a completely different solution without a webdriver since I really only want to get to the duolingo.com/learn page to scrape some data, but as of yet I haven't found an alternative way to login)
The code used:
from selenium import webdriver
from time import sleep
url = "https://www.duolingo.com/"
def login():
driver = webdriver.Chrome()
driver.get(url)
sleep(2)
hve_acnt_btn = driver.find_element_by_xpath("/html/body/div/div/div/span[1]/div/div[1]/div[2]/div/div[2]/a")
hve_acnt_btn.click()
sleep(2)
email_input = driver.find_element_by_xpath("/html/body/div[1]/div[3]/div[2]/form/div[1]/div/label[1]/div/input")
email_input.send_keys("email#email.com")
sleep(2)
pwd_input = driver.find_element_by_css_selector("input[type=password]")
pwd_input.clear()
pwd_input.send_keys("password")
sleep(2)
login_btn = driver.find_element_by_xpath("/html/body/div[1]/div[3]/div[2]/form/div[1]/button")
login_btn.click()
sleep(5)
login()
I couldn't post the website's html because of the character limit, so here is the link to the duolingo page: Duolingo
Switch to Firefox or a browser which does not tell the page that you are visiting it automated. See my earlier answer for a very similar issue here: https://stackoverflow.com/a/57778034/8375783
Long story short: When you start Chrome it will run with navigator.webdriver=true. You can check it in console. Pages can detect that flag and block login or other actions, hence the invalid login. This is a read-only flag set by the browser during startup.
With Chrome I couldn't log in to Duolingo either. After I switched the driver to Firefox, the very same code just worked.
Also if I may recommend, try to use Xpath with attributes.
Instead of this:
hve_acnt_btn = driver.find_element_by_xpath("/html/body/div/div/div/span[1]/div/div[1]/div[2]/div/div[2]/a")
You can use:
hve_acnt_btn = driver.find_element_by_xpath('//*[#data-test="have-account"]')
Same goes for:
email_input = driver.find_element_by_xpath("/html/body/div[1]/div[3]/div[2]/form/div[1]/div/label[1]/div/input")
vs:
email_input = driver.find_element_by_xpath('//input[#data-test="email-input"]')
I've tried everything, but can't seem to select this element within the Google Analytics page. I'm trying to automate pulling metrics every week. Any suggestions would help, thank you! Below is a copy of my code (username and pw retracted of course!) ^_^
from selenium import webdriver
from bs4 import BeautifulSoup
import time
import request
from xlsxwriter import Workbook
from fake_useragent import UserAgent
import lxml
# Get to the metrics page on Google Analytics
browser = webdriver.Chrome()
browser.get("https://analytics.google.com/analytics/web/?
authuser=1#/report-home/a127318841w186198941p183230211")
email_field =
browser.find_element_by_css_selector("input[name='identifier']")
email_field.send_keys('asdf')
email_btn = browser.find_element_by_class_name('CwaK9').click()
time.sleep(1)
password_field =
browser.find_element_by_css_selector("input[name='password']")
password_field.send_keys('asdf')
password_btn = browser.find_element_by_class_name('CwaK9').click()
time.sleep(15)
behavior_btn = browser.find_element_by_link_text('Behavior').click()
time.sleep(1)
behavior_btn = browser.find_element_by_link_text('Site Content').click()
time.sleep(1)
behavior_btn = browser.find_element_by_link_text('All Pages').click()
time.sleep(10)
unique_page_views = browser.find_element_by_xpath("//*[#id='ID-
rowTable']/thead/tr[2]/td[4]/div[1]/div/p[3]")
print(unique_page_views)
time.sleep(5)
browser.close()
The webpage with target element highlighted
The problem in here is you are trying to access the content from an iframe. You cannot traverse through <iframe>'s until switching to them.
You need to switch driver context via
driver.switch_to.frame(iframeIdentifier)
Also, to discover that I had to log-in to analytics and find that particular object myself. Please be more considerate when asking questions and provide all the details you can.
Windows 10 Home 64 Bit
Python 2.7 (also tried in 3.3)
Pycharm Community 2006.3.1
Very new to Python so bear with me.
I want to write a script that will go to Google, enter a Search Phrase, click the Search button, look through the search results for a URL (or any string), if there is no result on that page, click the Next button and repeat on subsequent pages until it finds the URL, stops and Prints what page the result was found on.
I honestly don't care if it just runs in the background and gives me the result. At first I was trying to have it litterally open the browser, find the browser objects (search field and search button) via Xpath and execute that was.
You can see the modules I've installed and tried. And I have tried almost every code example I've found on StackOverflow for 2 days so listing everything I've tried would be quite wordy.
If anyone just tell me the modules that would work best and any other direction would be very much appreciated!
Specific modules I've tried for this were Selenim, clipboard, MechanicalSoup, BeautifulSoup, webbrowser, urllib, enter image description hereunittest and Popen.
Thank you in advance!
Chantz
import clipboard
import json as m_json
import mechanicalsoup
import random
import sys
import os
import mechanize
import re
import selenium
from selenium import webdriver
from selenium.webdriver.common.by import By
import time
import unittest
import webbrowser
from mechanize import Browser
from bs4 import BeautifulSoup
from subprocess import Popen
######################################################
######################################################
# Xpath Google Search Box
# //*[#id="lst-ib"]
# Xpath Google Search Button
# //*[#id="tsf"]/div[2]/div[3]/center/input[1]
######################################################
######################################################
webbrowser.open('http://www.google.com')
time.sleep(3)
clipboard.copy("abc") # now the clipboard content will be string "abc"
driver = webdriver.Firefox()
driver.get('http://www.google.com/')
driver.find_element_by_id('//*[#id="lst-ib"]')
text = clipboard.paste("abc") # text will have the content of clipboard
print('text')
# browser = mechanize.Browser()
# url = raw_input("http://www.google.com")
# username = driver.find_element_by_xpath("//form[input/#name='username']")
# username = driver.find_element_by_xpath("//form[#id='loginForm']/input[1]")
# username = driver.find_element_by_xpath("//*[#id="lst-ib"]")
# elements = driver.find_elements_by_xpath("//*[#id="lst-ib"]")
# username = driver.find_element_by_xpath("//input[#name='username']")
# CLICK BUTTON ON PAGE
# http://stackoverflow.com/questions/27869225/python-clicking-a-button-on-a-webpage
Selenium would actually be a straightforward/good module to use for this script; you don't need anything else in this case. The easiest way to reach your goal is probably something like this:
from selenium import webdriver
import time
driver = webdriver.Firefox()
url = 'https://www.google.nl/'
linkList = []
driver.get(url)
string ='search phrase'
text = driver.find_element_by_xpath('//*[#id="lst-ib"]')
text.send_keys(string)
time.sleep(2)
linkBox = driver.find_element_by_xpath('//*[#id="nav"]/tbody/tr')
links = linkBox.find_elements_by_css_selector('a')
for link in links:
linkList.append(link.get_attribute('href'))
print linkList
This code will open your browser, enter your search phrase and then gets the links for the different page numbers. From here you only need to write a loop that enters every link in your browser and looks whether the search phrase is there.
I hope this helps; if you have further questions let me know.