Folks, I have a scraping script that I need to run on specific times for live info, but I can't have my computer on me all day. So I thought about running it on an online interpreter, but repl.it doesn't have webdriver and the other I found didn't neither. Could you help me with that?
Thanks
I'm not sure, but I don't guess if you can do it on a free online interpreter!
You can buy a server and use that, You can SSH to it anytime you want, or even better, You can develop a micro web service using Flask or something else to report the data you need!
Other way I can think of is let your computer be online 24/7 and use smtplib to email yourself the data in an interval!
Related
I have been using selenium to do some web scraping and I'm in need for changing my ip. After having done some reserach into this I have discovered that it is fairly easy to setup and use a proxy. However, I am already paying for a VPN and therefore I would like to use it for this application as well. The free proxy lists that I have found have been way to slow to be useful for me.
I did some googling and found vpnc and other libraries but I couldn't get it to work all the way. I'm fairly new to web scraping and python so therefore I would appreciate if someone could help me on my level of knowledge.
Is it possible to do this or am I trying to achieve something that is way to difficult for an amateur like me? I'm trying to set this up on MacOS as well as Windows 7.
I want to create python web scraper to get and format some data for me and output it in json format so that other web pages can access it. I want to put this service on some of the free python hosts out there.
Because this is my first python project I have some questions.
Should I use any of the python web frameworks for this? As I am not really concern about security (I will have only couple of pages with on input) I thought to leave it just as a script.
I do need some small database. What library can you suggest for this?
Are there cron jobs on python web servers?
Do free servers allow site scrapping every X minutes?
I have python 2.7 as default in my linux. Can/ Should I work with it or should I try to get the new version up and running?
yes, it makes life easier. But you have to check what framework can be used on free server. Sometimes you can't install own modules.
sqlite doesn't need installation. mysql and postgres mostly are preinstalled on servers but you have to check it.
mostly yes but you have to check it.
some servers may not allow scraping any sites but you have to check it.
use version which is installed on server so you have to check it.
Some free servers run page 18 hours a day and freezes page on 6 hours a day - but you have to check it.
I am suppose to test web server's load, performance and stress testing. There will be over 100 client machines that will be connecting to it.
I am using python-selenium webdriver to start a grid. the server is considered as a selenium 'hub' and clients as 'nodes'. This part is working fine till now.
Now the hard part, I need to monitor the server's performance, load and stress from another third party system while the scripts are running.
Is there any possibility if this can work ? I tried using many open sources like funkload, locust, web server stress tool 8. But none of them can monitor the load tests that is swarming dynamically on to the load.
While I was browsing in this site; I came across this project on https://github.com/djangofan/WebDriverTestingTemplate. Will this be helpful to my project?
Selenium is a functional tool so it's not a good idea to use it for performance test.
To achieve same you can go with JMETER as it a good open source tool available
still if you want to use selenium then there is some script present to integrate JMeter with selenium. I never tried it but you can try it.
Refer below link for same:-
https://blazemeter.com/blog/jmeter-webdriver-sampler
http://seleniummaster.com/sitecontent/index.php/performance-test-menu/selenium-load-test-menu/174-use-selenium-webdriver-with-jmeter
Hope it will help :)
It is possible to do with Selenium, but it will take much more resources (especially your time).
I would also recommend to try out the
LoadComplete from SmartBear
It is a very simple and intuitive tool, which lets you run and schedule your tests + send a report with execution results.
You can use Apache JMeter to conduct the load from 3rd-party web server.
With PerfMon Metrics Collector plugin you will be able to get server-side health metrics along with the load test results.
See Getting Started: Scripting with JMeter guide and Learn JMeter in 60 minutes for quick ramp-up on Apache JMeter.
I m trying to automate a Web Application validation performed by my team.I have choosen Python as the language to do this, although my exp. with Python is very limited.I have done similar things in the past using Perl. Now the problem is that after posting the url of the website it directs to a logon page which is made in Javascript. From whatever little Python I know, I believe scrapping/parsing website made in Javascript is not possible. I faced the same issue while doing this with Perl as well and wasn't able to proceed.
Any pointers or help in resolving the above issue would be highly appreciated.
Thanks
Spynner may help http://code.google.com/p/spynner/
Maybe you can take a look a Selenium. It's a firefox plugin that enables automation, but it also has a webdriver system where you can write automation scripts in various languages (including python), and a server execute the code in various browsers. I never tried the webdriver part myself, but that should do what you want.
There is a long list of python http proxies. Does anyone have experience with them and have any recommendations? I need to run a proxy for automation tests running on a windows machine in IE. I want a proxy that has an API for monitoring traffic, so I can give go signals to the automation tests when a page appears to have stopped making requests.
The list seems to be quite old. I would have a look at Twisted. According to this documentation there's something available.