We have around 250 identical linux server which runs a business critical web application for a bank. Basically we do a lot of scripting work but now i want to centralize that only in one location. That means run on one server and and deploy it in many. I know you guys must be thinking that this is an easy task and can be done with a shell script. But again we need to create many different different scripts to do our work
I know python has a big library and this can be possible but i dont know how. To cut in short i need all scripts in one file and based on the argument it will execute it according.
For example in a python program we have a function where we can mix them to perform different result.
So you please let me know how to go about it
This is a very general question, so I'll respond with two different frameworks that are made using Python to facilitate bulk system administration tasks.
func - Func is part of the
Fedora project and so is specialized
to their architecture. If your hosts
are all RedHat/CentOS based, this is
the solution for you.
fabric - Fabric is more generic
but does generally the same thing at
a high level. If your environment
is heterogenous (full of different
types of systems/distributions),
this is probably what you want.
You could also try any of the distributed computing packages. Pyro is one of them that might interest you.
Related
We have a solution written in C#/.NET Framework 4.7. It has a lot of infrastructure code related to environment configurations, database access, logging, exception handling etc.
Our co-workers are eager to contribute to the project with Python code that makes a lot of special calculations. Ideally we want to pass configuration plus (big amount of) input data to their code and get back (big amount of) results without resorting to database integration. Is there a viable way to do so? Main goals are: 1) not to rewrite Python code to C# 2) not to duplicate configuration/database related code in Python to make future maintenance easier
Ideally we want to pass configuration plus (big amount of) input data to their code and get back (big amount of) results [...]
So if all you do is want to pass parameters to a python script and get back the result, there is a pretty easy way to do so. I suggest you take a look at IronPython. Not only can you do that, but you can also easily import things from .NET/C# in your python scripts if there is ever a need for that.
Yes this is exactly what Unix (e.g. Gnu/Linux) dose. The Unix philosophy is about creating many (usually small) programs (that usually do one thing well), and connecting them to create a system that is greater than the parts. To do this we use inter-process communication, usually pipelines / streams.
An alternate approach is to compile the C# into a library, that can be called form the python.
I want to create an GUI application, that would communicate with Asterisk server and provide functions, such as call forwarding, originating calls, etc.
I wanted to use Kivy (Python GUI framework), but there here is so many different tools (AGI, AMI, FastAGI) and libraries (Pyst2, StarPy, etc.) to manage asterisk, that i don't even know where to start.
I have already written some code (using Pyst2 asterisk manager) but I have a feeling, that this is not the best solution, as said application should be able to have multiple instances open simultaneously and AMI would be too messy for that purpose.
Could someone give me some advice or suggestions what tools would be best to use in this case?
You can start from nice book, like ORelly "The future of telephony".
There is no way do something really valuable without know of asterisk dialplan. All that technology used for different things.
I'm working on a framework for Digital Forensic Investigators to use to compare files with each other for my Master's capstone project. However, I ran into a bit of a snag...
I'm trying to implement multiprocessing on the comparisons since using a single core seems to be really slow. The trouble I'm having, however, is when the code goes to enter information into an SQLite database. It will occasionally get a "Database is locked" error when two cores complete at nearly the same time.
So, simple side of my question, is it unsafe to operate database functions within a multiprocessing environment due to the errors I'm encountering? If not, is there a method of going about this that is safe and won't result in random errors?
Thanks!
Your problem is that you are trying to have multiple writers access a toy database -- i.e. sqlite -- which is stored in a single file. Using Lock might help, but it's going to kill your multiprocess throughput because of all the waiting-for-the-lock time. In essence, the lock choke point will serialize your program.
Setting up either MySQL or Postgres on almost any platform is straightforward, and there are several excellent Python modules for accessing them. Using one of those will completely eliminate this problem.
Update for an extended response to comment:
I always ask clients / students, "What problem are you trying to solve?" I'm assuming that you are not trying to create a database system, simply to use one. SQLite3 is fine for a well-defined set of problems, but multiprocess access is not one of them. I could veer off into asking what aspect of your project requires multiprocess access, but I'll assume that you have already determined that this is needed. I don't know either your programming skills or your understanding of how a database works, so forgive me if the following is a bit basic.
Normally you need a database (my preference is Postgres), and a Python module that understands all of the fiddly details of how to talk to that database. Then you need to know what it is you want the DBMS to do for you. The Good News is that you are hardly the first to go down this path.
The Postgres Wiki is full of good stuff. See their page on Python Drivers. Psycopg2 is the category leader and runs on Win/Linux/Mac. Also check out PyPi, the Python Package Index, for many well-written extensions.
If you want to stay more object-oriented, as opposed to writing straight SQL, you might want to look at an ORM like SQLAlchemy. This is another category leader that is well-maintained and widely deployed.
The value of using an ORM is that you can (mostly) keep your head in ObjectLand, where most of your problem lives, and not get tangled up in the cognitive dissonance created by object-oriented programming vs. relational database management, which are two very different views of the world of data.
If you need more help, email me. My address is in my profile.
You can make use of Lock. Take a look at https://docs.python.org/2/library/multiprocessing.html#synchronization-between-processes
I am going to write configuration tool for my Ubuntu based system. Next I would like to write frontends (text, GUI and web). But it is the most complicated project I wanted to write and I am not sure about general architecture I should use.
At the current I have functions and classes for changing system config. But these functions will probably grow & change. #Abki gave me advice how to write interface for frontends. I am going to make base classes for this interface but I don't know how to connect it with backend and next with frontends. Probably I should use design patterns like fasade, wrapper or something else.
It looks like (without interface_to_backend layer):
I don't care about UI and functions to change system config now. But I don't know how to write middle layer so It would be easy to connect it with the rest and extend functionality i the future.
I need general ideas, design patterns, advices how to implement this in Python.
I'm not sure this is entirely appropriate for SO but I'm intrigued and so I'll bite. As a rubyist I can't help much with the Python but here is some opinion on pattens from my experience.
My initial suggestion is you should review a few of the contenders out there. Specifically I'd be looking at cfengine, chef and bcfg2. They each tell a different story but if I'd summarise I'd say:
Chef has a lovely dsl syntax but is let down by a complicated architecture
bcfg2 is written in python but seems to have an annoying tendency to use XML :(
cfengine has the strongest theoretical underpinnings in promise theory (which is v.interesting BTW) but is C based.
Wikipedia also provides a pretty impressive list of configuration management tools that you will find useful.
In regard to designing your own tool I'd suggest there are three principles you want to pursue:
Simplicity, the simpler you make this the better. Simple in terms of scope, configuration and use are all important.
You'll need a single way to store data, you need to be able to trace the choices as they are made and not trample other people's changes (especially in a team environment).
Security, most configuration management tools need root privileges at some point. So you need to make sure that users can trust the code they're running.
You could use Fabric with Python as described in the article Ubuntu Server Setup with Python Fabric
The Wikipedia article at Comparison of open source configuration management software has several other tools that use Python to do this.
I like the approach taken by SALT.
If you write the GUI, text/CLI, and Web interfaces using Python, they can all use the same Python module. That way a change in one interface transparently affects the others. Plus all of those are in Python's area of strength.
Would it be possible to make a python cluster, by writing a telnet server, then telnet-ing the commands and output back-and-forth? Has anyone got a better idea for a python compute cluster?
PS. Preferably for python 3.x, if anyone knows how.
The Python wiki hosts a very comprehensive list of Python cluster computing libraries and tools. You might be especially interested in Parallel Python.
Edit: There is a new library that is IMHO especially good at clustering: execnet. It is small and simple. And it appears to have less bugs than, say, the standard multiprocessing module.
You can see most of the third-party packages available for Python 3 listed here; relevant to cluster computation is mpi4py -- most other distributed computing tools such as pyro are still Python-2 only, but MPI is a leading standard for cluster distributed computation and well looking into (I have no direct experience using mpi4py with Python 3, yet, but by hearsay I believe it's a good implementation).
The main alternative is Python's own built-in multiprocessing, which also scales up pretty well if you have no interest in interfacing existing nodes that respect the MPI standards but may not be coded in Python.
There is no real added value in rolling your own (as Atwood says, don't reinvent the wheel, unless your purpose is just to better understand wheels!-) -- use one of the solid, tested, widespread solutions, already tested, debugged and optimized on your behalf!-)
Look into these
http://www.parallelpython.com/
http://pyro.sourceforge.net/
I have used both and both are exellent for distributed computing
for more detailed list of options see
http://wiki.python.org/moin/ParallelProcessing
and if you want to auto execute something on remote machine , better alternative to telnet is ssh as in http://pydsh.sourceforge.net/
What kind of stuff do you want to do? You might want to check out hadoop. The backend, heavy lifting is done in java, but has a python interface, so you can write python scripts create and send the input, as well as process the results.
If you need to write administrative scripts, take a look at the ClusterShell Python library too, or/and its parallel shell clush. It's useful when dealing with node sets also (man nodeset).
I think IPython.parallel is the way to go. I've been using it extensively for the last year and a half. It allows you to work interactively with as many worker nodes as you want. If you are on AWS, StarCluster is a great way to get IPython.parallel up and running quickly and easily with as many EC2 nodes as you can afford. (It can also automatically install Hadoop, and a variety of other useful tools, if needed.) There are some tricks to using it. (For example, you don't want to send large amounts of data through the IPython.parallel interface itself. Better to distribute a script that will pull down chunks of data on each engine individually.) But overall, I've found it to be a remarkably easy way to do distributed processing (WAY better than Hadoop!)
"Would it be possible to make a python cluster"
Yes.
I love yes/no questions. Anything else you want to know?
(Note that Python 3 has few third-party libraries yet, so you may wanna stay with Python 2 at the moment.)