Refresh a local web page using Python - python

I'm using Python to gather some information, construct a very simple html page, save it locally and display the page in my browser using webbrowser.open('file:///c:/testfile.html'). I check for new information every minute. If the information changes, I rewrite the local html file and would like to reload the displayed page.
The problem is that webbrowser.open opens a new tab in my browser every time I run it. How do I refresh the page rather than reopen it? I tried new=0, new=1 and new=2, but all do the same thing. Using controller() doesn't work any better.
I suppose I could add something like < META HTTP-EQUIV="refresh" CONTENT="60" > to the < head > section of the html page to trigger a refresh every minute whether or not the content changed, but would prefer finding a better way.
Exact time interval is not important.
Python 2.7.2, chrome 26.0.1410.64 m, Windows 7 64.


If you're going to need a refresh on the same tab, you'll need selenium webdriver.
After installing selenium using pip, you can use the following code :
from selenium import webdriver
import time
import urllib
import urllib2
x = raw_input("Enter the URL")
refreshrate = raw_input("Enter the number of seconds")
refreshrate = int(refreshrate)
driver = webdriver.Firefox()
driver.get("http://"+x)
while True:
time.sleep(refreshrate)
driver.refresh()
This will open the URL and refresh the tab every refreshrate seconds


I use pyautogui module to refresh the browser page. It's one liner:
import pyautogui
pyautogui.hotkey('f5') #Simulates F5 key press = page refresh


Keep it very short, as simple as:
from selenium import webdriver
import time
driver = webdriver.Firefox()
driver.get('URL')
while True:
time.sleep(20)
driver.refresh()
driver.quit()


It looks like several people have asked this in the past but here is a link that sums it up.
Python refresh HTML document
But webbrowser.open( url, new=0 ) should open the page in the current window and not initialize a new one.


The LivePage extension for Chrome. You can write to a file, then LivePage will monitor it for you. You can also optionally refresh on imported content like CSS. Chrome will require that you grant permissions on local file:// urls.
(I'm unaffiliated with the project.)


Or, you can use the easy auto refresh extension in chrome :D
It allows you to set your own refresh time, fastest being one second!


I have had same issues as you for some reason new=1 , new=0 doesn't work. U may try the code below. It simply open the url, waits for 10 sec then kill the process n repeat.
from subprocess import Popen
import time
while True:
link = ('https://www.google.com/')
Popen(['start', 'chrome', link], shell=True)
time.sleep(10)
Popen('taskkill /F /IM chrome.exe', shell=True)


After using pip install to install pyautogui use the following:
import pyautogui
import time
for i in range(number of times you want to refresh):
time.sleep(refreshrate in seconds)
pyautogui.hotkey('f5')

Related

My function click() dont work with href of this site, how i can fix and make work?

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
#Service("local do diretorio do chromeDriver")
driver = webdriver.Firefox('/home/arch/Downloads/bot/')
driver.get("https://blaze.com/pt/?modal=auth&tab=login")
element=driver.find_element_by_name("username").send_keys("email")
driver.find_element_by_name("password").send_keys("pass")
#link da url desejada
driver.find_element_by_xpath('/html/body/div[1]/main/div[3]/div/div[2]/div[2]/form/div[4]/button').click()
driver.find_element_by_xpath('/html/body/div[1]/main/div[1]/div[4]/div[2]/div[1]/div/div/div[4]/div[1]/div/div[3]/div/a')
I have this code in python that opens in blazer.com and makes login but when i try to click o game mines which in this href:
<a href="/pt/games/mines"/>
and doesnt work justs stops , someone could help with this error

You probably forgot to click the "mines" button. Don't forget the .click() at the end of your code, like this:
driver.find_element_by_xpath('/html/body/div[1]/main/div[1]/div[4]/div[2]/div[1]/div/div/div[4]/div[1]/div/div[3]/div/a').click()
And you also might want to give the site some time to load by importing time and using time.sleep(x) which sleeps the code for x seconds.
import time
time.sleep(2) # Use this between the .click() lines to give the site time to load after clicking a button

Automation on the site using seleniumrequests

I am trying to automate some processes on the site. At first I tried to use queries, but a captcha came in response. Now I'm using selenium queries, and here's the problem: when I log in using selenium tools only, everything works fine, but I can't add coupons on the site and confirm them.
from seleniumrequests import Firefox
driver = Firefox()
user = '000000'
password = '000000'
driver_1x.get("https://1xstavka.ru/")
driver.find_element_by_id('curLoginForm').click()
driver.find_element_by_id('auth_id_email').send_keys(user)
driver.find_element_by_id('auth-form-password').send_keys(password)
driver.find_element_by_class_name('auth-button__text').click()
But if you use:
from seleniumrequests import Firefox
driver = Firefox()
driver.request('GET', 'https://1xstavka.ru')
The window opens for a second and immediately closes, a 200 response is received, but there are no cookies. It's the same with publishing requests, with which I'm trying to automate the process. After the request for publication, the response is 200, but nothing happens on the site.
driver.request('POST', 'https://1xstavka.ru/user/auth', json=json)
please tell me what is wrong or how you can solve this problem

I am unable to access the URL specified in the query; but for captcha/coupons, I created a loop with an interim stop function. This gives me the chance to input it manually and then continue the loop.

Python webdriver connect to already webpage (selenium)

I need to open multiple links in separate tabs or sessions...
I already know how to do it, so what i would like to know is if it's possible to connect to an already open webpage instead of open every links every time i run the script.
What i used now in Python is:
from selenium import webdriver
driver.get(link)
The purpose would be once i run the first script (to load multiple links), the second should connect to the webpages, refresh them and continue with the code.
Is it possible? Anyone know how to do it?
Thanks a lot for the help!!!!

Connecting to the previously opened window is easy:
driver = webdriver.Firefox()
url = driver.command_executor._url
session_id = driver.session_id
driver2 = webdriver.Remote(command_executor=url,desired_capabilities={})
driver2.session_id = session_id
#You're all set to do whatever with the previously opened browser
driver2.get("http://www.stackoverflow.com")

Python selenium get redirected url with Phantomjs

Here is my problem: I'm trying to use selenium to access a webpage and the special about this page is it is an auto redirecting page (you open that page and after few seconds, it automatically redirect to another page). When i use driver = webdriver.Firefox(), my IDM catched that link just perfectly after few seconds.
And because i don't want the browser to come up so i use Phantomjs instead, ut it not working. My application just can get the loading page url (bitdl-1336...) but not the redirected link. Please help!
This is my code:
link = 'http://torrent.ajee.sh/hash.php?hash=' + self.global_hash_code
driver = webdriver.PhantomJS('phantomjs.exe')
driver.get(str(link))
element = driver.find_element_by_link_text('Download Zip')
element.click()
time.sleep(10)
msg = QMessageBox.information(self, QString('Thành công'),QString(driver.current_url))
And this is the result:
Please help!
Sorry about my english

Not exactly an answer to your PhantomJS-specific question, but a workaround to the problem.
And because i don't want the browser to come up so i use Phantomjs instead
You can continue using Firefox, but start it in a Virtual Display, see more information at:
How do I run Selenium in Xvfb?
You may also need to let the browser automatically save the archive in a specified directory, see:
How do I automatically download files from a pop up dialog using selenium-python
Access to file download dialog in Firefox

Can't capture HAR using Python Selenium Script with BrowserMob-Proxy

Goal:
I want to run a Selenium Python script through BrowserMob-Proxy, which will capture and output a HAR file capture.
Problem:
I have a functional (very basic) Python script (shown below). When it is altered to utilize BrowserMob-Proxy to capture HAR however, it fails. Below I provide two different scripts that both fail, but for differing reasons (details provided after code snippets).
BrowserMob-Proxy Explanation:
As mentioned before, I am using both 0.6.0 AND 2.0-beta-8. The reasoning for this is that A) LightBody (lead designer of BMP) recently indicated that his most current release (2.0-beta-9) is not functional and advises users to use 2.0-beta-8 instead and B) from what I can tell from reading various site/stackoverflow information is that 0.6.0 (acquired through PIP) is used to make calls to the Client.py/Server.py, whereas 2.0-beta-8 is used to initiate the Server. To be honest, this confuses me. When importing BMP's Server however, it requires a batch (.bat) file to initiate the server, which is not provided in 0.6.0, but is with 2.0-beta-8...if anyone can shed some light on this area of confusion (I suspect it is the root of my problems described below), then I'd be most appreciative.
Software Specs:
Operating System: Windows 7 (64x) -- running in VirtualBox
Browser: FireFox (32.0.2)
Script Language: Python (2.7.8)
Automated Web Browser: Selenium (2.43.0) -- installed via PIP
BrowserMob-Proxy: 0.6.0 AND 2.0-beta-8 -- see explanation below
Selenium Script (this script works):
"""This script utilizes Selenium to obtain the Google homepage"""
from selenium import webdriver
driver = webdriver.Firefox() # Opens FireFox browser.
driver.get('https://google.com/') # Gets google.com and loads page in browser.
driver.quit() # Closes Firefox browser
This script succeeds in running and does not produce any errors. It is provided for illustrative purposes to indicate it works before adding BMP logic.
Script ALPHA with BMP (does not work):
"""Using the same functional Selenium script, produce ALPHA_HAR.har output"""
from browsermobproxy import Server
server = Server('C:\Users\Matt\Desktop\\browsermob-proxy-2.0-beta-8\\bin\\browsermob-proxy')
server.start()
proxy = server.create_proxy()
from selenium import webdriver
driver = webdriver.Firefox() # Opens FireFox browser.
proxy.new_har("ALPHA_HAR") # Creates a new HAR
driver.get("https://www.google.com/") # Gets google.com and loads page in browser.
proxy.har # Returns a HAR JSON blob
server.stop()
This code will succeed in running the script and will not produce any errors. However, when searching the entirety of my hard drive, I never succeed in locating ALPHA_HAR.har.
Script BETA with BMP (does not work):
"""Using the same functional Selenium script, produce BETA_HAR.har output"""
from browsermobproxy import Server
server = Server("C:\Users\Matt\Desktop\\browsermob-proxy-2.0-beta-8\\bin\\browsermob-proxy")
server.start()
proxy = server.create_proxy()
from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.set_proxy(proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)
proxy.new_har("BETA_HAR") # Creates a new HAR
driver.get("https://www.google.com/") # Gets google.com and loads page in browser.
proxy.har # Returns a HAR JSON blob
server.stop()
This code was taken from http://browsermob-proxy-py.readthedocs.org/en/latest/. When running the above code, FireFox will attempt to get google.com, but will never succeed in loading the page. Eventually it will time out without producing any errors. And BETA_HAR.har can't be found anywhere on my hard drive. I have also noticed that, when trying to use this browser to visit any other site, it will similarly fail to load (I suspect this is due to the proxy not being configured properly).

Try this:
from browsermobproxy import Server
from selenium import webdriver
import json
server = Server("path/to/browsermob-proxy")
server.start()
proxy = server.create_proxy()
profile = webdriver.FirefoxProfile()
profile.set_proxy(self.proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)
proxy.new_har("http://stackoverflow.com", options={'captureHeaders': True})
driver.get("http://stackoverflow.com")
result = json.dumps(proxy.har, ensure_ascii=False)
print result
proxy.stop()
driver.quit()

I use phantomJS, here is an example of how to use it with python:
import browsermobproxy as mob
import json
from selenium import webdriver
BROWSERMOB_PROXY_PATH = '/usr/share/browsermob/bin/browsermob-proxy'
url = 'http://google.com'
s = mob.Server(BROWSERMOB_PROXY_PATH)
s.start()
proxy = s.create_proxy()
proxy_address = "--proxy=127.0.0.1:%s" % proxy.port
service_args = [ proxy_address, '--ignore-ssl-errors=yes', ] #so that i can do https connections
driver = webdriver.PhantomJS(service_args=service_args)
driver.set_window_size(1400, 1050)
proxy.new_har(url)
driver.get(url)
har_data = json.dumps(proxy.har, indent=4)
screenshot = driver.get_screenshot_as_png()
imgname = "google.png"
harname = "google.har"
save_img = open(imgname, 'a')
save_img.write(screenshot)
save_img.close()
save_har = open(harname, 'a')
save_har.write(har_data)
save_har.close()
driver.quit()
s.stop()

What worked for me was to downgrade java version to java11. I used jenv to install and manage multiple java versions.

When you do:
proxy.har
You need to parse that response, proxy.har is a JSON object, so if you need to generate a file, you need to do this:
myFile = open('BETA_HAR.har','w')
myFile.write( str(proxy.har) )
myFile.close()
Then you will find your .har

Finding your HAR file
Inherently, the HAR object generated by the proxy is just that: an object in memory. The reason you can't find it on your hard drive is because it's not being saved there unless you write it there yourself. This is a pretty simple operation, as the HAR is just JSON.
with open("harfile", "w") as harfile:
harfile.write(json.dumps(proxy.har))
Why does ALPHA not work?
When you start dumping your HAR file, you'll find that your HAR file is empty with the ALPHA script. This is because you are not adding the proxy to the settings for Firefox, meaning that it will just connect directly bypassing your proxy.
What about BETA?
This code is written correctly as far as connecting to the proxy, although personally I prefer adding the proxy to the capabilities and passing those through. The code for that is:
cap = webdriver.DesiredCapabilities.FIREFOX.copy()
proxy.add_to_capabilities(cap)
driver = webdriver.Firefox(capabilities=cap)
I would guess that your issue lies with the proxy itself. Check the bmp.log and/or server.log files in the location of the python script and see what it is saying if something is going wrong.
Another alternative is that selenium is reporting back that the webpage has loaded before it actually has finished getting all of the elements, and as such your proxy is shutting down too early. Try making the script wait a bit longer before shutting down the proxy, or running it interactively through the interpreter.

Categories

Resources