There is a Scala library I'd like to use, namely BIDMach, however I need to be able to use it from Python rather than in Scala. I've been trying to think of different ways of possible being able communicate between the library and Python code, such as creating an HTTP server in Scala and calling this from Python, using something like JPype to try and use Scala libraries in Python, and different types of interprocess communication. However, none of them seem to work very well, and would seem to require a large amount of reimplementation of what is already in the library. Does anyone know of a good way of going about this?
edit: In terms of exactly what I think I'd like to do, ideally I'd be able to get close to almost all of the libraries functionality usable in Python, however that probably isn't realistic. It would be nice if some of the Scala classes were easily usable in Python, without too much repeated implementation effort. The reason I don't think what I've looked into so far will work well is because it requires a fair bit of reimplementation of what is already in the library (i.e. representing something like a matrix in JSON, as a way of transporting data to/from Python/Scala)
Related
I'm having a problem creating a inter-process communication for my python application. I have two python scripts at hand, let's say A and B. A is used to open a huge file, keep it in memory and do some processing that Mysql can't do, and B is a process used to query A very often.
Since the file A needs to read is really large, I hope to read it once and have it hang there waiting for my Bs' to query.
What I do now is, I use cherrypy to build a http-server. However, I feel it's kind of awkward to do so since what I'm trying to do is absolutely local. So, I'm wondering are there other more organic way to achieve this goal?
I don't know much about TCP/socket etc. If possible, toy examples would be appreciate (please include the part to read file).
Python has good support for ZeroMQ, which is much easier and more robust than using raw sockets.
The ZeroMQ site treats Python as one of its primary languages and offers copious Python examples in its documentation. Indeed, the example in "Learn the Basics" is written in Python.
i am learning python.. i want to do certain kind of scripting in python.. like,
i want to communicate 'wmic' commands through dos promt.. store the result a file..
access some sqlite database, take the data it has and compare with the result i stored..
now, what i dont get is that, how should i proceed? is there any specific frameworks or modules/libraries? like, win32api / com or what else?
pls guide me what things i should follow/learn to accomplish what i intend to do..
thanks
for your project look here
http://tgolden.sc.sabren.com/python/wmi/index.html
http://docs.python.org/library/sqlite3.html
here is list of generic python modules
http://docs.python.org/modindex.html
http://docs.python.org/ - here you can find all information you need with examples.
One of the particularly attractive features of Python is the "batteries included" philosophy: The standard library is huge, and extremely well thought out in 90% of the modules I've ever used. Conversely, this means that a good approach is to learn it first before branching out and installing third-party libraries that may be much less well supported and, ultimately, have no advantage.
In practice, this means I'd recommend keeping https://docs.python.org/release/2.6.5/library/index.html close to your heart, and delve into the documentation for the sqlite3 and probably the subprocess modules. You may or may not want win32api later (I've never worked with wmic, so I'm not sure what you'd need), or come back here when you have concrete questions and already explored the standard library offering.
My machine has two audio inputs: a mic in that I use for gaming, and a line in that I use for guitar. When using one it's important that the other be muted to remove hiss/static, so I was hoping to write a small script that would toggle which one was muted (it's fairly inconvenient to click through the tray icon, switch to my input device, mute and unmute).
I thought perhaps I could do this with pywin32, but everything I could find seemed specific to setting the output volume rather than input, and I'm not familiar enough with win32 to even know where to look for better info.
Could anybody point me in the right direction?
Disclaimer: I'm not a windows programming guru by any means...but here's my best guess
Per the pywin32 FAQ:
How do I use the exposed Win32 functions to do xyz?
In general, the trick is to not
consider it a Python/PyWin32 question
at all, but to search for
documentation or examples of your
problem, regardless of the language.
This will generally give you the
information you need to perform the
same operations using these
extensions. The included
documentation will tell you the
arguments and return types of the
functions so you can easily determine
the correct way to "spell" things in
Python.
Sounds like you're looking to control the "endpoint device" volumes (i.e. your sound card / line-in). Here's the API reference in that direction.
Here's a slightly broader look at controlling audio devices in windows if the previous wasn't what you're looking for.
Here's a blog entry from someone who did what you're trying to do in C# (I know you specified python, but you might be able to extract the correct API calls from the code).
Good luck! And if you do get working code, I'm interested to see it.
I had a similar problem and couldn't figure out how to use Windows API's to do what I wanted. I ended up just automating the GUI with AutoIt. I think that will be the fastest and easiest solution (albeit a "hacky" one). As I answered earlier today, you can use AutoIT from within Python.
You are probably better off using ctypes - pywin32 is good if you are using one of the already included APIs, but I think you'll be out of luck with the sound APIs. Together with the example code from the C# link provided by tgray, use ctypes and winmm.dll, or alternatively, use SWIG to wrap winmm.dll. This may well be quicker as you won't have to build C structure mapping types in ctypes for the types such as MIXERCONTROLDETAILS which are used in the API calls.
tgray seems to have pointed you in the right direction, but once you find out the right Win32 APIs to deal with, you have a couple of options:
1) Try using pywin32...but it may or may not wrap the functionality you need (it probably doesn't). So you probably only want to do this if you need to use COM to get at the functionality you need.
2) Use ctypes. It's generally pretty easy to wrap just about any C functionality with ctypes.
3) If the C# example looks like what you need, you should be able to translate it to IronPython with fairly little effort. Might be easier than using the C API. YMMV, of course.
I want to start writing a http proxy that will modify responses according to some rules/filters I will configure. However, before I start coding it, I want to make sure I'm making the right choice in going with Python. Later, this tool would have to be able to process a lot of requests, so, I would like to know I can count on it later on to be able to perform when "push comes to shove".
As long as the bulk of the processing uses Python's built-in modules it should be fine as far as performance. The biggest strength of Python is its clear syntax and ease of testing/maintainability. If you find that one section of your code is slowing down the process, you can rewrite that section and use it as a C module, while keeping the bulk of your control code in Python.
However if you're looking to make the most optimized Python Code you may want to check out this SO post.
Yes, I think you will find Python to be perfectly adequate for your needs. There's a huge number of web frameworks, WSGI libraries, etc. to choose from, or learn from when building your own.
There's an interesting post on the Python History blog about how Python was supporting high performance websites in 1996.
This will depend on the library you use more than the language itself. The twisted framework is known to scale well.
Here's a proxy server example in python/twisted to get you started.
Bottomline: choose your third party tools wisely and I'm sure you'll be fine.
Python performs pretty well for most tasks, but you'll need to change the way you program if you're used to other languages. See Python is not Java for more info.
If plain old CPython doesn't give the performance you need, you have other options as well.
As has been mentioned, you can extend it in C (using a tool like swig or Pyrex). I also hear good things about PyPy as well, but bear in mind that it uses a restricted subset of Python. Lastly, a lot of people use psyco to speed up performance.
Would it be possible to make a python cluster, by writing a telnet server, then telnet-ing the commands and output back-and-forth? Has anyone got a better idea for a python compute cluster?
PS. Preferably for python 3.x, if anyone knows how.
The Python wiki hosts a very comprehensive list of Python cluster computing libraries and tools. You might be especially interested in Parallel Python.
Edit: There is a new library that is IMHO especially good at clustering: execnet. It is small and simple. And it appears to have less bugs than, say, the standard multiprocessing module.
You can see most of the third-party packages available for Python 3 listed here; relevant to cluster computation is mpi4py -- most other distributed computing tools such as pyro are still Python-2 only, but MPI is a leading standard for cluster distributed computation and well looking into (I have no direct experience using mpi4py with Python 3, yet, but by hearsay I believe it's a good implementation).
The main alternative is Python's own built-in multiprocessing, which also scales up pretty well if you have no interest in interfacing existing nodes that respect the MPI standards but may not be coded in Python.
There is no real added value in rolling your own (as Atwood says, don't reinvent the wheel, unless your purpose is just to better understand wheels!-) -- use one of the solid, tested, widespread solutions, already tested, debugged and optimized on your behalf!-)
Look into these
http://www.parallelpython.com/
http://pyro.sourceforge.net/
I have used both and both are exellent for distributed computing
for more detailed list of options see
http://wiki.python.org/moin/ParallelProcessing
and if you want to auto execute something on remote machine , better alternative to telnet is ssh as in http://pydsh.sourceforge.net/
What kind of stuff do you want to do? You might want to check out hadoop. The backend, heavy lifting is done in java, but has a python interface, so you can write python scripts create and send the input, as well as process the results.
If you need to write administrative scripts, take a look at the ClusterShell Python library too, or/and its parallel shell clush. It's useful when dealing with node sets also (man nodeset).
I think IPython.parallel is the way to go. I've been using it extensively for the last year and a half. It allows you to work interactively with as many worker nodes as you want. If you are on AWS, StarCluster is a great way to get IPython.parallel up and running quickly and easily with as many EC2 nodes as you can afford. (It can also automatically install Hadoop, and a variety of other useful tools, if needed.) There are some tricks to using it. (For example, you don't want to send large amounts of data through the IPython.parallel interface itself. Better to distribute a script that will pull down chunks of data on each engine individually.) But overall, I've found it to be a remarkably easy way to do distributed processing (WAY better than Hadoop!)
"Would it be possible to make a python cluster"
Yes.
I love yes/no questions. Anything else you want to know?
(Note that Python 3 has few third-party libraries yet, so you may wanna stay with Python 2 at the moment.)