Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
In my current job we have a web based Business intelligence tool where every morning i have to create data extracts from the system and paste them into a PowerPoint presentation. I wish to automate this as its repetitive and time consuming (we have also had several redundancies and i have been allocated the analysts work so i also want to try get home before 10pm :)). The bottle neck for generating these reports is running them on the website and then exporting the results to excel as this manual process can take anything from 10 minutes to an hour of waiting.
I would like to create a script that will open up the web page, make the selections in listboxes containing such information as location product etc as well as a date chooser, press an apply button once the report has generated then export it. This would happen during the period when no one is in the office so the files would be ready when i come in to analysis as opposed to just generating the reports
A second smaller question, Is there a quick way to identify the listboxes using firefox or IE explorer so that they can be referenced in the code?
Is this possible in Python?
Our IT department are also quite strict so for example i can not install new software but can install libraries for Python
Could anyone point me in the direction of sample code particularly referencing listboxes or date objects?
Thank you very much for your time
All this can be done automated using selenium[1] . If you know the class name/id etc for the listboxes, selenium allows you to send click events to the browser for checking/unchecking listboxes. Read up [2] on filling HTML forms using selenium. You can find the relevant code in the documentation links below.
[1] http://selenium-python.readthedocs.org/)
[2] http://selenium-python.readthedocs.org/en/latest/navigating.html#filling-in-forms
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 1 year ago.
Improve this question
So, I have made a python script that posts online ad on a certain website for me automatically (kind of web scraper). But, I need that ad to be posted (AKA that script ran) every day at approximately 12h, so the ad wouldn't get pushed too far so the people could see it constantly.
The only problem is that I don't have access to my PC every day at that time (I'm usually busy), so I need a website that I can post my script to that will execute it every day at a given time. Is there a website that I can use for this purpose?
You can try pythonanywhere. But most of them has very limited services allowed when you are not paying. Also troublesome if you need to expose some of your permissions.
Personally, I run services on my personal box, usually keep it on all-day and a scheduler software to run my programs from an starting script.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 2 years ago.
Improve this question
I'm relatively new to programming but I'm beginning work on a project that will be used in my day job. My end goal is to create a program that automatically downloads PDF's from a website once a month and puts them in specific directories on my PC. This is uncharted territory for me and so I'm trying to think through the steps.
The first problem I need to overcome is the website is password protected.
So I need to access the site then automatically enter a username and password and login. I've read Selenium would probably be the module for this.
Can anybody advise please?
Sounds like Selenium is a good fit.
You can simulate button presses, key strokes, mouse swipes, so you can log in, navigate and download your files.
You can even make it run in the background if you make sure it does what it's supposed to do.
This is a fun example
Be careful though, the website may have a way of detecting it incorporated in it.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 1 year ago.
Improve this question
In my python project, I've list of cited papers and for each paper, I need its Author Name and Abstract and Citation Count from google scholars . I was using scholarly . PyPI like this :
search_pub = scholarly.search_pubs(paperName)
docInfo = next(search_pub)
but now I'm getting this error:
Exception: Cannot fetch the page from Google Scholar.
It seems like they've blocked my IP due to multiple requests. Now I'm unable to find any other programmatic way to extract these info. I can have a list of paper references to extract data for.
Can anyone help me out with any python library or guide me to write some piece of code for this?
You can just wait for this temporary ban to expire and keep going. Make sure to insert a time.sleep(...) or similar in your code to stay under their rate limit. Google Scholar has no official API, so scraping is your only option if this is the data you want.
(I am not recommending that you scrape, and please note that Google Scholar disallows robots through their robots.txt)
Google Scholar blocks your IP if you query too much or too often. Even if you make your program sleep, do not make it sleep periodically since they can detect that too. Google considers this as DoS (Denial of Service) attack. Even if you randomise your sleep time, at one point, if you make too many queries, it will flag you. One workaround for this is using rotating proxy services. Look up online, there are plenty of free ones. They offer you User-Agent strings, which if you put randomly for every query you make, you're good to go afaik.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 3 years ago.
Improve this question
I really am lost.
I've written a python script which takes two files, converts one to the same format as the second and outputs the differences.
I now need to create some kind of GUI?
I'm thinking a relatively simple HTML page which allows users to click on two separate boxes to upload each of the files and then submit them - resulting in the output being offered as file which the user can download to the PC.
Any help of direction would be really appreciated. It's only really a guess that HTML will be my best/simplest choice. Working from the command line isn't really an option I can go with. The PC that this will sit on will not have internet access. I intend to have this installed on the PC as an application.
I accept I'll need to make some changes to my Python code. Posted this here as I'm not sure Python questions would have been the best place but I'll move it if suggested.
Please, any help at all will be useful. First timer
This is not a kind of answer that normally gets a reply on StackOverflow because it is much too broad / subjective.
Given the fact that you are quite new I would like to provide a solution anyway that will allow you to build a GUI quite easily like EasyGUI.
A very easy example:
import easygui
easygui.msgbox('This is a basic message box.', 'Title Goes Here')
Other possibilities (less customizable but arguably easier) are Wooey (automatic web interface generation for your script) and Gooey (automatic GUI generation for your script.
Please, for next time, remember to always provide a Minimal Reproducible Example and read https://stackoverflow.com/help/how-to-ask
I don't suggest you use the Django Framework for such a small application.
You should definitely use Flask for this. Flask is more lightweight
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
Suppose there is a password-protected website that I want to access to scrape some info from it and put it into a spreadsheet. For example, it could be my personal credit card account page and I would be scraping info about the latest transactions.
A variation of this would be if the site allowed to download the transaction info as a CSV file, in which case I would want to download that file.
If I want to write such scraper in Python, what packages should I use for the task? Does it depend on how a specific website is implemented, i.e. I might need one tool to scrape one site and another tool to scrape another.
Thank you
I actually did something very similar to this, but in node. Are you definitely wanting to do this in Python?
If you want to stick to Python, take a look at these modules:
BeautifulSoup
requests
Someone wrote a really awesome module combining the above two modules:
Robobrowser
If you would like to venture down the node route, take a look at this:
nightmarejs