I'm working on a selenium test and I need to get a chrome extension from the chrome app store to use in the test. Right now it's a manual process to update to a newer version of the extension.
Current Flow:
1. Manual download extension through a chrome extension downloader.
2. Store the .crx file in a location visible to the selenium test.
3. Execute test with that extension.
I was hoping that google had an API that could be hit in order to download the extension but I've been unable to find anything to that effect. Has anyone ran into a situation like this and been able to solve it?
Basically you just have to capture the redirect url and then request that.
In python:
pluginId = id at end of url on plugin page. option 2 on here explains it well
blah=requests.get(url,params{'prodversion':'57.0','x':"id=pluginId",'response':'redirect'},verify=False,stream=True)
blahFile = requests.get(blah.url)
extension = open("yourExtension.crx", 'wb')
extension.write(blahFile.content)
extension.close()
Related
I have about 8 reports that I need to pull from a system every week which takes quite a bit of time so I am working on automating this process. I am using requests to login to the site and download the files. However, when I download the file using my python script the file comes back blank. When I use the same link to download from the browser its not blank. Below is my code:
payload = {
'txtUsername': 'uid',
'txtPassword': 'pass'
}
domain = 'https://example.com/login.aspx?ReturnUrl=%2fiweb%2f'
path = 'C:\\Users\\workspace\\data-in\\'
with requests.Session() as s:
p = s.post(domain, data=payload)
r = s.get('https://example.com/forms/MSWordFromSql.aspx?ContentType=excel&object=Organization&FormKey=f326228c-3c49-4531-b80d-d59600485557')
with open(path + 'report1.xls', 'wb') as f:
f.write(r.content)
A little about the url. When I was looking for the url I found that it's wrapped in some JS.
Export Raw Data to Excel
However, when I take a look at the path from which the files was downloaded the true location for the report is this:
https://example.com/forms/MSWordFromSql.aspx?ContentType=excel&object=Organization&FormKey=f326228c-3c49-4531-b80d-d59600485557
This is the URL I am using in my code to download a report. After I run the script the file is created, named and saved to the correct directory but its empty. As I mentioned at the top of the thread, if I simply copy the URL about to the browser it downloads the report with no problem.
I was also thinking about using Selenium to get this done but the issue is I cannot rename the files while they are being downloaded. I need each file to have a specific name because all of the downloaded reports are then used in another automation script.
As #Lucas mentioned, your Python code likely sends a different request than your browser does, and thus receives a different response.
I'd use the browser dev tools to inspect the request the browser makes to initiate the download. Use "Copy as curl" and try to reproduce the correct behavior from the command line.
Then reduce the differences between the curl request and the one your python code makes by removing unnecessary parts from the curl invocations and adding the necessary headers to your python code. https://curl.trillworks.com/ can help with the latter.
When I try to use Cookies file (sqlite fomatted) on Linux (Ubuntu) and Windows I fall into troubles with decryption of 'encrypted_value'. Is there any chance to make Cookies file compatible with two systems?
Basically selenium driver use Cookies file for various purpouses, and everything works on Linux. Sometimes my action is needed, so I want to have this Cookies file on my desktop which works on Windows, but when I download it directly and copy-paste it to my profile directory, my chromedriver logs error:
[4708:4884:0604/082853.607:ERROR:os_crypt_win.cc(61)] Failed to decrypt: The parameter is incorrect. (0x57)
I assume that there is some problem with 'encrypted_value' column decryption but I am unable to work-around this issue.
I use selenium for python this is snippet where I create options for my webdriver:
def create_options_for_webdriver(session_directory):
print('Creating options for webdriver!')
options = Options()
options.add_argument("user-data-dir=my_userdir")
options.add_argument("user-agent=my_useragent")
options.add_argument('--disable-background-networking ')
options.add_argument('--disable-client-side-phishing-detection')
options.add_argument('--disable-default-apps')
options.add_argument('--disable-hang-monitor')
options.add_argument('--disable-popup-blocking')
options.add_argument('--disable-prompt-on-repost')
options.add_argument('--disable-sync')
options.add_argument('--disable-web-resources')
options.add_argument('--enable-automation')
options.add_argument('--enable-blink-features=ShadowDOMV0')
options.add_argument('--force-fieldtrials=SiteIsolationExtensions/Control')
options.add_argument('--ignore-certificate-errors')
options.add_argument('--no-first-run')
options.add_argument('--password-store=basic')
options.add_argument('--use-mock-keychain')
return options
before option creating I create minimal directory structure which looks like my_userdir/Default/, and I download Cookies file to Default folder.
... when I download it directly and copy-paste it to my profile directory, my chromedriver logs error:
[... ERROR:os_crypt_win.cc(61)] Failed to decrypt: The parameter is incorrect. (0x57)
It looks like it is not possible. Or maybe not possible they way you are trying to do it. It takes extra effort.
The question Decrypting Chrome's cookies on windows has a link to os_crypt_win.cc. os_crypt_win.cc uses DPAPI, which is the old WinCrypt gear. DPAPI ties encryption to a user's Windows login. DPAPI also places a MAC on the encrypted data. I believe the MAC is the reason for the message you are seeing: "The parameter is incorrect". DPAPI sees the MAC over the encrypted data is wrong, and it gives you the generic error message.
So if you truly want to use the Linux cookie on Windows, you will need to decrypt it using the Linux spec, and then re-encrypt it using the Windows spec.
If you are going to pursue it, then you may want to visit this BlackHat talk: Reversing dpapi and stealing windows secrets offline. It will allow you to encrypt the user's data on Linux for Windows.
Coming back with update!
It appears that, as #jww mentioned process is more complicated, but just a little :)
In order to make Cookies fully compatible with any OS some special treatment must be applied.
In my case I used pickle library to create compatible file. To achieve it, things needs to be done as follow:
from selenium.webdriver import Chrome
import pickle
driver = Chrome()
####here you do some job which generate cookies like FB login or whatever
input("Press any key to close session") #you cant simply close browser, in order to make it work browser have to be closed in console so the rest of script will execute
pickle.dump(driver.get_cookies(), open('cookies.pkl',"wb"))
driver.quit()
print("Session closed!")
This will create cookies.pkl file which can be accessed under any OS like that:
import pickle
from selenium import Chrome
driver = Chrome()
for cookie in pickle.load(open("cookies.pkl"."rb")):
driver.add_cookie(cookie)
### anything you want to execute
As I mentioned this kind of cookies file will work under any OS, sadly it forced me to use selenium, but better this than nothing :)
I want to access Session Buddy using Python.
In my case "access" means to get all currently opened URLs from Chrome.
Session Buddy allows you to save all opened URLs into a .csv file.
To do so you need to "setup" a few things (simplified: press buttons) and then all URLs are downloaded to Chromes /Downloads directory.
I would like to fully automate this process though. This means python needs to access Session Buddy, initiate the download and then save the file to the directory you want it to.
I can't use requests or something though since an extension won't work using an URL. This is what the extension calls: chrome-extension://edacconmaakjimmfgnblocblbcdcpbko/main.html
In general, I don't necessarily want to use Session Buddy to get all the URLs, it just seems to be the easiest way..
So, in summary, I just want to ask: How can I automatically use Python to fetch all currently opened URLs in my Chrome Browser (using Session Buddy)?
I'm thankful for any kind of help.
You can use selenium webdriver and load .crx(extension) file to automate
chrome_options = Options()
chrome_options.add_extension('path_to_extension')
driver = webdriver.Chrome(executable_path=executable_path,chrome_options=chrome_options)
driver.get("http://stackoverflow.com")
driver.quit()
I would like to write a script (in python) which scans the machine.(assumption is system has Linux) and retrieve the list of browser's installed. Looking for suggestions to implement it. I am using selenium to open links
browser = webdriver.Firefox()
Here we have to mention Firefox to open link in Firefox browser. What if user don't have Firefox in machine (chrome installed)?
I had already searched but haven't got any result.
P.S: If system has windows/Mac OS
A better approach: use try/except block
try:
browser = webdriver.Firefox()
browser.get('url')
except (IOException, Exception):
pass
It will help in cases where driver is not able to find the browser or there is some problem while launching.
You can scan all .desktop files in /usr/share/applications looking for WebBrowser in Categories field.
I have to validate that a web application, when executed in the client browser, is fetching some assets (*.js) from a particular remote server.
Say two options exist: whether it gets the script from server A or it gets a copy from server B. I need to assert (based on some preconditions) that the script was downloaded from server A.
The question: Is there a way to inspect source url of loaded javascript using selenium (preferably with python)?
Here it is a possible solution to extract url of javascript libraries from the stackoverflow site.
You should adapt the solution to the site you are working on.
driver = webdriver.Firefox()
driver.get("http://www.http://stackoverflow.com/")
link= driver.find_elements_by_tag_name('script')
for i in link:
print i.get_attribute("src")
Example of output:
http://rules.quantcount.com/rules-p-c1rF4kxgLUzNc.js
http://edge.quantserve.com/quant.js
http://b.scorecardresearch.com/beacon.js
https://www.google-analytics.com/analytics.js
https://ajax.googleapis.com/ajax/libs/jquery/1.12.4/jquery.min.js
https://cdn.sstatic.net/Js/stub.en.js?v=9798373e8e81
There are various strategies to locate elements in a page.
You can use the most appropriate one for your case (http://selenium-python.readthedocs.io/locating-elements.html)