Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
Is there a way to scrape data from a popup? I'd like to import data from the site tennisinsight.com.
For example, http://tennisinsight.com/match-preview/?matchid=191551201
This is a sample data extraction link. When clicking "overview" there is a button with "Match Stats", I'd like to be able to import those data from many links in a text or CSV file.
What's the best way to accomplish this? Is Scrapy able to do this? Is there software able to do this?
You want to open the network analyzer in your browser (e.g. in Web Developer in Firefox) to see what requests are sent when you click the "match stats" button in order to replicate them using python.
When I do it, a POST request is sent to http://tennisinsight.com/wp-admin/admin-ajax.php with action and matchID parameters.
You presumably already know the match ID (see URL you posted above), so you just need to set up a POST request for each matchID you have.
import requests
r = requests.post('http://tennisinsight.com/wp-admin/admin-ajax.php', data={'action':'showMatchStats', 'matchID':'191551201'})
print r.text #this is your content of interest
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
I am trying to scrape this website with scrapy and I have had to search for each link extracting the information from each one, I would like to know if there is an API of the site that I can use (I don't know how to find it).
I would also like to know how I can obtain the latitude and longitude? Currently the map is shown but I do not know how to obtain the numbers
I appreciate any suggestions
The website may be loading the data dynamically using Javascript. Use your browser dev tools and look at the networking tab, look for any XHR calls which may be accessing an API. Then you can scrape from that directly.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
So, I wanted to make something to automatically sign me into my twitter account. How would I use python, or HTML, to go to twitter.com/login and automatically put in my username and password, and then submits it?
You don't have to go through browser automation to access data in twitter. What you really need is a Twitter API. There are multiple Python clients, like tweepy or python-twitter.
What you want to do is called Web automation. In Python, I recommend Selenium. It's pretty much all you need. The documentation is quite good and you can find examples online.
Note: use passwd for your password! Keep it safe 👊
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 7 years ago.
Improve this question
I'm trying to make a simple post method with the requests module, like this :
s=requests.Session()
s.post(link,data=payload)
In order to do it properly, the payload is an id from the page itself, and it's generated in every access to the page.
So I need to POST without reloading the entire webpage so i can get the right id .
Is this code in a view?
If yes, I'd advise you to create an Ajax view receiving this POST and send the payload from the page using javascript as Ajax.
This way on success, you can get the right id in the response of that view without reloading the page.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I want to scan about 1000 pdf files using "wepawet" which is an online scanner but it takes one file at a time how could I scan the whole 1000 files, could I do that using python ?
https://wepawet.iseclab.org/
could any one help me please?
thank you in advance for helping
You can automate the process by using python tools like selenium, mechanize or urllib(I'm not sure about urllib). Fill the form using mechanize (a simple example of filling a form and submitting)
response = br.open(url)
print response.read()
response1 = br.response()
print response1.read()
br.select_form("form1")
br.form = list(br.forms())[0]
response = br.submit()
print response.read()
and submit it as in the code. For more info on mechanize, visit http://www.pythonforbeginners.com/cheatsheet/python-mechanize-cheat-sheet. Hope it works.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 9 years ago.
Improve this question
I want to create a little program with following features.
Use proxy in format proxy:port:username:password
Choose a proxy sequentialially from list
Open http://example.com
Fill Details choosing data from data.txt ( CSV )
Export Cookie,username,password,email address --> cookie.txt
Delete Cookies
Log into associated email account and confirm account by visiting
link sent to that email address.
Then cycle through Step1 again.
I read several similar question on stackoverflow.
I planned to use Selenium for this program, but reading comment here How to save and restore all cookies with Selenium RC?
the get_cookie method doesn't provide the path, domain, and expiry
date for each cookie, so it isn't possible to fully restore those
parameters with create_cookie. any other ideas
And i won't be able to manipulate cookies using method as describe here http://hub.tutsplus.com/tutorials/how-to-build-a-python-bot-that-can-play-web-games--active-11117
I want to know easiest way to tackle this problem. I plan to run single threaded application.
I don't know selenium, but why not use mechanize and requests ? Both are awesome.