I am working with ChromeDriver via selenium Library in python. After loading certain page it gives table, that refreshes itself every 2 seconds via sending response to run php script.
https://someurl.com/getTasks.php?tasks_type=1&from_date=01.06.2021&to_date=01.06.2021
This is screenshot of requests
To work with content of this table i need to block php request immediately. how can i perform this action in python?
Related
So I am trying to have program that uses browser automation. I will be using the Selenium package from Python. This will all be in a Flask route in system A.
Essentially I will be doing browser automated commands through Selenium, then sending an output to system B via a get request. I then want to wait for system B to verify that the request sent is valid. Then system B will send a request back to the same endpoint on system A to finish the Selenium process.
Remember I CANNOT use redirects because I have to stay within the same Flask route and Selenium process. I cannot simply just call another function because system A is waiting on system B to send back a request to finish the Selenium process.
Is there a way to do this?
I was just making a simple Python program that would update me about my Instagram likes and followers. I don't want to open the browser every time it runs so I'm using requests and Beautifulsoup to get the page data. But the get method gives me a script that is totally different from the actual page script(the script in the browser). There's no class so I can't use BeautifulSoup.find() method on it. It seems like I'm receiving a different script because it's a robot accessing the page.
Is there any other module or method I can use?
I will need to test few things from my linux server. I wonder if it is possible to manipulate some actions on web without access to browser. I have only access to linux server via command line. I know only selenium webdriver to do that actions but but for that I need browser.
What I want to do:
1) Input text to textbox on webpage using python script which is placed on linux server
2) Click button on webpage
Generally it is possible to manipulate actions on webpage from linux using python scripts?
If you input text to fom on webpage and submit the form, the browser will send POST or GET request to the server with contained information. The server then proceses (for example saves ) the information. You dont need a browser to send http request, you can send them directly from python.
AN example can be found here: How to simulate HTTP post request using Python Requests module?
If requests module isn't enough already, try using selenium with PhantomJS. PhantomJS is a headless WebKit scriptable with a JavaScript API.
PhantomJS : http://phantomjs.org
A great tutorial : https://realpython.com/headless-selenium-testing-with-python-and-phantomjs/
I am using Python with Selenium WebDriver to browse through website.
Now I have the problem that I want to monitor the XHR AJAX calls, thrown on the current site.
For example: I am on a specific page and the python selenium script clicks on a button on this site. This website button calls an AJAX request within this site.
Is it possible to monitor this XHR AJAX request and get it in my python script to handle the called AJAX URL?
Thanks!
UPDATE: I exactly want to do sth like this (but in python obviously)
https://dmitripavlutin.com/catch-the-xmlhttp-request-in-plain-javascript/
You can use browser.execute_script to catch the calls as explained in the link that you mentioned. In addition, start a fake website using Django on a separate thread. And in the JavaScript handler (sendReplacement), replace the url with the one of your django server.
In that server you will receive the AJAX call and be able to examine it.
You may be able to implement a simpler solution without the django server by simply monitor the calls directly from the JavaScript snippet and make it return the value you want directly. But if you need to monitor many calls and perform more complex examination is the requests, the former solution is more powerful.
I am novice in python (c++ developer), I am trying to do some hands-on on web scraping on windows IE.
The problem which I am facing is, when I open a URL using "requests" library the server sends me a login page always. I figured out the problem. Its actually doing it because it presumes you are coming through IE tries to execute on function which uses some information from the SSO ( single signup object ) which is there executing on the background in Windows on the first login to the web server ( consider this as some weird setup.)
On observing this I changed my strategy & started using webbrowser lib.
Now, when I try to do a webbrowser.open("url"), the browser is opening the page properly which is great thing!!!
But, my problems now are :
1) I do not want that the browser page opened should be visible to the user ( some way that the browser is opened in background ). I tried to used this :
ie = webbrowser.BackgroundBrowser(webbrowser.iexplore)
ie.Visible = 0
ie.open('url')
but no success.
It opens the page which is visible to the user.
2) [This is main activity] I want to scrape the page which is opened in the web browser's IE page opened above. how to do?
I tried to dig into this link but did not find any APIs for getting the data.
Kindly help.
PS : I tried to use beautiful soup for scraping on some other web pages using requests. It was successful & I go the data I wanted. But not in this case.
The webbrowser module doesn't allow to do that. The get function you mentioned is to retrieve registered web browsers not to scrap a HTTP GET request.
I don't know what is triggering the behavior you described with IE, have you tried to change your User-Agent with IE ones? You can check this post for more details: Sending "User-agent" using Requests library in Python