I saw this post on Medium, and wondered how one might go about managing multiple python scripts.
How I Hacked Amazon's Wifi Button
This describes a system where you need to run one or more scripts continuously to catch and react to events in your network.
My question: Let's say I had multiple python scripts that I wanted to do run while I work on other things. What approaches are available to manage these scripts? I have to imagine there is a better way than having a large number of terminal windows running each script individually.
I am coming back to python, and have no formal training in computer programming, so any guidance you can provide will be greatly appreciated.
Let's say I had multiple python scripts that I wanted to do run. What
approaches are available to manage these scripts? I have to imagine
there is a better way than having a large number of terminal windows
running each script individually.
If you have several .py files in a directory that you want to run, without having a specific order, you can do:
import glob
pyFiles = glob.glob('path/*.py')
for pyFile in pyFiles:
execfile(pyFile)
Your system already runs a large number of background processes, with output to the system log or occasionally to a service-specific log file.
A common arrangement for quick and dirty deployments -- where you don't necessarily want to invest in making the scripts robust and well-behaved enough to run as proper services -- is to start the script inside screen or tmux. You can detach when you don't need to be looking at it, and can reattach at any time -- even from a remote login -- to view the output, or to troubleshoot.
Take a look at luigi (I've not used it).
https://github.com/spotify/luigi
These days (five years after the question was asked) a lot of people use docker compose. But that's a little heavy weight depending on what you want to do.
I just saw today the script server of bugy. Maybe it might be a solution for you or somebody else.
(I am just trying to find a tampermonkey script structure for python..)
Related
Before I buy my first setup. I'll launch my deep-learning-pipline on sth like vast.ai.
I never did it before, but How can I protect my script from being "stolen"?
This should be a serious launch and take around 7 days to finish training.
google colab doesn allow enough memory & ram for what i need ( need around 64GB ram)
is there a way to run a python script encrypted? (note: it makes use of libaries)
It is hard to run python encrypted. However, you could try to store the code into encrypted disk space.
There are some ways, from fully obfuscating your script to converting your script into equivalent cython or creating an executable out of it using the likes of Nuitka.
You may also implement some important logic in C/C++ (as extensions) and then call them in your script.
You may also set up a server where you feel is OK and send the bits that needs to be executed, basically create a distributed system.
As you can see there are many ways, and the deeper you go the more complex it gets.
Also you might want to have a look here as well.
I have a requirement to test two applications (via automation using Python).
The requirement is for example we have a system called “www.abc.com”
where we develop and merge code in every 2 weeks and then we create a another system called “www.xyz.com” ( basically it is backup to the first system ), everytime we do a release and add/edit in the main system, we update in our back up system.
Now the question is i need to tests both the system, after every release (every 2 weeks) to see if they both are in sync (identical).
how do i fire a python automation test script (multiple tests) to check if for example databases, servers, UI, front end, check if code base are same in both systems? can i do that if yes any help and advice , please suggest so that i can implement possible solutions .
There are several ways you could approach this:
Assuming you are using some sort of source control you could write a script to make sure that the repo is up to date and then report back the results. See here and here. This probably won't cover the data in your databases, but there are numerous ways to back database backups and it will depend what programs you are using.
Another or additional way you might check is to write a script to gather a list of hashes or checksums of all the files you care about in both systems and then compare the list for differences.
Okay, so basically I am creating a website. The data I need to display on this website is delivered twice daily, where I need to read the delivered data from a file and store this new data in the database (instead of the old data).
I have created the python functions to do this. However, I would like to know, what would be the best way to run this script, while my flask application is running? This may be a very simple answer, but I have seen some answers saying to incorporate the script into the website design (however these answers didn't explain how), and others saying to run it separately. The script needs to run automatically throughout the day with no monitoring or input from me.
TIA
Generally it's a really bad idea to put a webserver to handle such tasks, that is the flask application in your case. There are many reasons for it so just to name a few:
Python's Achilles heel - GIL.
Sharing system resources of the application between users and other operations.
Crashes - it happens, it could be unlikely but it does. And if you are not careful, the web application goes down along with it.
So with that in mind I'd advise you to ditch this idea and use crontabs. Basically write a script that does whatever transformations or operations it needs to do and create a cron job at a desired time.
Here are the facts:
I have a collection of python scripts that work together as part of a program
It takes a user input and outputs .docx and .excel files
Right now, only executable through command-line/editor (i.e. you call one script, it does the rest)
Non-CS people have difficulties downloading the necessary modules, and running the program
I want to make my program easily executable/portable to different OSs. So far, I have the following options:
Create a website that takes input visually, and does what it has to do. How do I get the website to refer to my code? Do I have to put it in some sort of web server?
Create an executable that others can simply run (Not sure if this is possible)
Create a GUI that does everything visually.
What are the pros and cons of each one of these options? Which one would you say is most feasible?
My apologies if I said something silly. I'm still learning most of this stuff.
Thank you!
I have to setup a program which reads in some parameters from a widget/gui, calculates some stuff based on database values and the input, and finally sends some ascii files via ftp to remote servers.
In general, I would suggest a python program to do the tasks. Write a Qt widget as a gui (interactively changing views, putting numbers into tables, setting up check boxes, switching between various layers - never done something as complex in python, but some experience in IDL with event handling etc), set up data classes that have unctions, both to create the ascii files with the given convention, and to send the files via ftp to some remote server.
However, since my company is a bunch of Windows users, each sitting at their personal desktop, installing python and all necessary libraries on each individual machine would be a pain in the ass.
In addition, in a future version the program is supposed to become smart and do some optimization 24/7. Therefore, it makes sense to put it to a server. As I personally rather use Linux, the server is already set up using Ubuntu server.
The idea is now to run my application on the server. But how can the users access and control the program?
The easiest way for everybody to access something like a common control panel would be a browser I guess. I have to make sure only one person at a time is sending signals to the same units at a time, but that should be doable via flags in the database.
After some google-ing, next to QtWebKit, django seems to the first choice for such a task. But...
Can I run a full fledged python program underneath my web application? Is django the right tool to do so?
As mentioned previously, in the (intermediate) future ( ~1 year), we might have to implement some computational expensive tasks. Is it then also possible to utilize C as it is within normal python?
Another question I have is on the development. In order to become productive, we have to advance in small steps. Can I first create regular python classes, which later on can be imported to my web application? (Same question applies for widgets / QT?)
Finally: Is there a better way to go? Any standards, any references?
Django is a good candidate for the website, however:
It is not a good idea to run heavy functionality from a website. it should happen in a separate process.
All functions should be asynchronous, I.E. You should never wait for something to complete.
I would personally recommend writing a separate process with a message queue and the website would only ask that process for statuses and always display a result immediatly to the user
You can use ajax so that the browser will always have the latest result.
ZeroMQ or Celery are useful for implementing the functionality.
You can implement functionality in C pretty easily. I recomment however that you write that functionality as pure c with a SWIG wrapper rather that writing it as an extension module for python. That way the functionality will be portable and not dependent on the python website.