Trying the code in the quick start guide here, I find the code tries to open a google URL using some firefox browser profile
https://developers.google.com/calendar/quickstart/python
If it needs to use a browser, I would like it to use a specific browser profile. How can I make it use that profile?
Alternatively, if I can prevent it from needing to use a browser at all, that would work too (I've already given it a credentials.json, but I guess I need some additional security credential to get access which can only be obtained using a real browser?).
Related
I have been using selenium to scrape data from Google Ads Account, I log-in manually using my profile data first and then save the cookies to be used to later to automatically login and then run the scraping task, some times the script runs on VM, not the machine the cookies was created.
Everything was going fine but sometimes, approximately every 10 days, I get an email that there's suspicious activity in my account and google sign me out automatically and I have to change my google account password recreate the cookies manually.
I'm thinking that the problem comes from the fact that the cookies was created on a machine and being used on other but not sure.
I'm thinking of creating the cookies on the VM and only using it there, but not sure if that would work?!
That might be because google detects, that your using the same cookie in a browser with different useragent & device-metrics.
To bypass that, you could try using Selenium-Profiles and start it, using a profile previously exported from your local browser.
I m a newbie so I will try to explain myself in a way it makes sense.
I produced my first ever python script to scrape data from a web page I use regularly at work. It just prints out couple of values in the console that previously I had to consult manually.
My problem is that every time I execute the script and the browser opens up, it seems the cache is cleared and I have to log in into that work webpage using my personal credentials and do the 2 factor authentication with my phone.
I m wondering wether there is a way to keep the cache for that browser (if I previously already logged into the web page) so I don´t need to go through authentication when I launch my script.
I m using selenium webdriver and chrome, and the option I have configured are these (in screenshot below). Is there perhaps another option I could add to keep cache?
Current options for browser
I tried to find info in the web but so far nothing.Many sites offer a guide on how to perform login by adding lines of code with the username and the password, but I would like to avoid that option as I still would need to use my phone for the 2 factor authentication, and also because this script could be used by some other colleagues in the future.
Thanks a lot for any tip or info :)
After days browsing everywhere, I found this post:
How to save and load cookies using Python + Selenium WebDriver
the second answer is actually the one that saved my life; I just had to add this to my series of options:
chrome_options.add_argument("user-data-dir=selenium")
see the provided link for the complete explanation of the options and imports to use.
Adding that option, I run the script for the first time and I still have to do the login manually and undergo authentication. But when I run it for the second time I don´t need any manual input; the data is scraped from the web, the result is returned and no need any manual action from me.
If anybody is interested in the topic please ping me.
Thanks!
I am searching for a way which allows me to interact with a webrowser (Firefox,Chrome/Chromium,Edge are the most important).
I am currently using pyautogui, to locate login,password fields to put the login data into them. But since you can extract much easier informations when you can use IDs or xPath or other identifiers on webpages, it would make sense to use that.
I tried Firefox with selenium but I run in some problems. Can I attache it to a user created session (do I need the processID or something like that?). (Can I choose between the normal private session of the current profile?
I need a solution which works on Windows and Linux(it would be nice if the major Linux distros would support it. But the most important distros are Fedora/Ubuntu for me.) mac would be optional but since I do not got any mac I am not able to test it anyway.
The way with debugger mode or similar does not work really well for me since the browser needs to get started in a special way.
Would it possible to use something like this:
Can Selenium interact with an existing browser session? ,
When I can retrieve the this information some how form the existing browser?
driver.command_executor._url
driver.session_id
(But when I understand that currently it only works with browsers started with selenium?)
When I use Selenum and start a browserwindow with it can I login to a website and the user is logged in on the webside on his browser window too(if they us the same profile)? (Or does selenium separate cookies?)
If you need additional information or have some hints please post them so I can see them.
Thank you in advance for your help
It seems that it is not possible to connect to a web browser which was opened by the user to my understanding. How ever I found two possible solutions which I am currently trying to evaluate.
Using pyautogui to access the web browser over scanned images and control it with keyboard and mouse. (It is possible to access the console with the right combinations too).
The other solution is maybe more stable. Writing an browser extension which controls the browser.
Are there any alternatives to Selenium that don't require a web driver or browser to operate? I recently moved my code over to a Google Cloud VM instance, and when I run it there are multiple errors. I've been trying to get it to work for hours but just can't (no luck with PhantomJS, Chrome and GeckoDriver - tried re-downloading browsers, editing the sources.list file e.c.t.).
The page I'm web scraping uses JavaScript to load in numbers, which I was I initially chose Selenium. Everything else works perfectly though!
You could simply use the request library.
https://requests.readthedocs.io/en/master/
https://anaconda.org/anaconda/requests
You would then need to send a GET or POST request to the server.
If you do not know how to generate a proper POST request, simply try to "record" it.
If you have chrome, got to the page you want to navigate, press F12, navigate to the "Network" section and write method:POST into the filter.
Further info here:
https://stackoverflow.com/a/39661536/11971785
At first it is a bit more confusing than selenium, but once you understand it its waaaay better in my opinion.
Also the Java values shown on the page can usually be simply read out of the java code which is returned by your request.
No web driver or anything required and a lot more stable and customizable.
I want to fill some field of a webpage and then send a request to it
but this website has a very powerful login page to avoid sending requests for login from a robot
so I can't log in with selenium but after login, I can use selenium and I can send requests, on the other hand, I write this program for an app so I can't open a web driver and then work on it
I need to work on a tab that exists
I want to program work on a session that a human opened
From what I understand from your question is that you want to open an existing browser that is launched by a human.
There are multiple ways to achieve this and as mentioned by Devansh in the comment, You can use the session ID to get the already launched browser and execute your test script on it.
However, there is another way that might be able to solve your issue of executing scripts on already open connection or logged in user.
You can use the profiles of browsers for this scenario, User profiles in a browser are like user accounts on a computer.
You can use the answer to this question to create and use profiles in your script:
Opening an existing tab/logged in user using Chrome Webdriver
Now You can manually log in the required account for the website on the above profile you are struggling to log in and then launch the scripts.