I am suppose to test web server's load, performance and stress testing. There will be over 100 client machines that will be connecting to it.
I am using python-selenium webdriver to start a grid. the server is considered as a selenium 'hub' and clients as 'nodes'. This part is working fine till now.
Now the hard part, I need to monitor the server's performance, load and stress from another third party system while the scripts are running.
Is there any possibility if this can work ? I tried using many open sources like funkload, locust, web server stress tool 8. But none of them can monitor the load tests that is swarming dynamically on to the load.
While I was browsing in this site; I came across this project on https://github.com/djangofan/WebDriverTestingTemplate. Will this be helpful to my project?
Selenium is a functional tool so it's not a good idea to use it for performance test.
To achieve same you can go with JMETER as it a good open source tool available
still if you want to use selenium then there is some script present to integrate JMeter with selenium. I never tried it but you can try it.
Refer below link for same:-
https://blazemeter.com/blog/jmeter-webdriver-sampler
http://seleniummaster.com/sitecontent/index.php/performance-test-menu/selenium-load-test-menu/174-use-selenium-webdriver-with-jmeter
Hope it will help :)
It is possible to do with Selenium, but it will take much more resources (especially your time).
I would also recommend to try out the
LoadComplete from SmartBear
It is a very simple and intuitive tool, which lets you run and schedule your tests + send a report with execution results.
You can use Apache JMeter to conduct the load from 3rd-party web server.
With PerfMon Metrics Collector plugin you will be able to get server-side health metrics along with the load test results.
See Getting Started: Scripting with JMeter guide and Learn JMeter in 60 minutes for quick ramp-up on Apache JMeter.
Related
I have a number of web scrapers built with Selenium in Python that run 24/7 for several months of the year. At the moment, all scrapers run locally on my PC. This comes at quite a cost to the performance of my PC and basically means it's constantly working fairly hard.
Ideally I would like to use remote servers instead but not really sure where to start. I would like to be able to run these things remotely, speed is also quite important - the pages need to load fairly quickly. What would be worth looking into? I'm willing to pay for a quality service.
Thanks
The Issue
I am using my laptop with Apache to act as a server for a local project involving tensorflow and python which uses an API written in Flask to service GET and POST requests coming from an app and maybe another user on the local network.The problem is that the initial page keeps loading when I specifically import tensorflow or the object detection folder within the research folder in the tensorflow github folder, and it never seems to finish doing so, effectively getting it stuck. I suspect the issue has to do with the packages being large in size, but I didn't have any issue with that when running the application on the development server provided with Flask.
Are there any pointers that I should look for when trying to solve this issue? I checked the memory usage, and it doesn't seem to be rising substantially, as well as the CPU usage.
Debugging process
I am able to print basic hello world to the root page quite quickly, but I isolated the issue to the point when the importing takes place where it gets stuck.
The only thing I can think of is to limit the number of threads that are launched, but when I limited the number of threads per child to 5 and number of connections to 5 in the httpd-mpm.conf file, it didn't help.
The error/access logs don't provide much insight to the matter.
A few notes:
Thus far, I used Flask's development server with multi-threading enabled to serve those requests, but I found it to be prone to crashing after 5 minutes of continuous run, so I am now trying to use Apache using the wsgi interface in order to use Python scripts.
I should also note that I am not servicing html files, just basic GET and POST requests. I am just viewing them using the browser.
If it helps, I also don't use virtual environments.
I am using Windows 10, Apache 2.4 and mod_wsgi 4.5.24
The tensorflow module being a C extension module, may not be implemented so it works properly in Python sub interpreters. To combat this, force your application to run in the main Python interpreter context. Details in:
http://modwsgi.readthedocs.io/en/develop/user-guides/application-issues.html#python-simplified-gil-state-api
I've got a python script that I want to build a locally-hosted web gui app for, so I can use the modern styling and tools available to web apps.
The scripts I'm running take a while to process, and I want to update the web app with visual updates, or at least something akin to what the console sees when using print() in python.
My initial hosting efforts have been based on this tutorial, and I tried out the methods in this answer to try and get data to update in a streamed fashion, but the pages only showed once the entire script was finished.
I'm wondering whether web.py could help me?
Any guidance, or even the right terms to google would be appreciated. Thanks.
--
Update: I've been reading up on node.js (something I've failed to do for years..) and, please correct me if I'm wrong, but it seems like it could be the answer. I'm even considering re-writing my original functions into node.js given the existence of this serial comms library
I have a web crawling python script that takes hours to complete, and is infeasible to run in its entirety on my local machine. Is there a convenient way to deploy this to a simple web server? The script basically downloads webpages into text files. How would this be best accomplished?
Thanks!
Since you said that performance is a problem and you are doing web-scraping, first thing to try is a Scrapy framework - it is a very fast and easy to use web-scraping framework. scrapyd tool would allow you to distribute the crawling - you can have multiple scrapyd services running on different servers and split the load between each. See:
Distributed crawls
Running Scrapy on Amazon EC2
There is also a Scrapy Cloud service out there:
Scrapy Cloud bridges the highly efficient Scrapy development
environment with a robust, fully-featured production environment to
deploy and run your crawls. It's like a Heroku for Scrapy, although
other technologies will be supported in the near future. It runs on
top of the Scrapinghub platform, which means your project can scale on
demand, as needed.
As an alternative to the solutions already given, I would suggest Heroku. You can not only deploy easily a website, but also scripts for bots to run.
Basic account is free and is pretty flexible.
This blog entry, this one and this video contain practical examples of how to make it work.
There are multiple places where you can do that. Just google for "python in the cloud", you will come up with a few, for example https://www.pythonanywhere.com/.
In addition, there are also several cloud IDEs that essentially give you a small VM for free where you can develop your code in a web-based IDE and also run it in the VM, one example is http://www.c9.io.
In 2021, Replit.com makes it very easy to write and run Python in the cloud.
If you have a google e-mail account you have an access to google drive and utilities. Choose for colaboratory (or find it in more... options first). This "CoLab" is essentially your python notebook on google drive with full access to your files on your drive, also with access to your GitHub. So, in addition to your local stuff you can edit your GitHub scripts as well.
There is a long list of python http proxies. Does anyone have experience with them and have any recommendations? I need to run a proxy for automation tests running on a windows machine in IE. I want a proxy that has an API for monitoring traffic, so I can give go signals to the automation tests when a page appears to have stopped making requests.
The list seems to be quite old. I would have a look at Twisted. According to this documentation there's something available.