I'm working in the VFX industry and we deal with different software packages that ship with their own Python interpreter. We are running on Linux and use modules to handle our environments to make sure that people are using the correct version of all applications depending on the project they are working on.
Since months, we are trying to setup an environment that supports multiple versions of Python. And what is blocking right now are additional Python libraries that we are using in our in-house tools, like sqlalchemy, psycopg2, openpyxl, zmq, etc.
So far, for each project, we have config file that defines the version of each package to be use including the additional Python modules. And to use the correct Python version of each Python module, we look up the main Python interpreter defined in that same modules definition file. This works as long as the major and minor versions of all Python interpreters do line up.
But now, I would like to start an application that ships with a Python 3.7 interpreter and another application with a Python 3.9 interpreter and so on. All applications do you use our in-house tools which need the additional Python modules. Of course, this fails when trying to import any additional module.
For now, the only solution that I see is to install the corresponding Python modules in the 'site-packages' of each application that comes with its own Python interpreter. That should work. But this means that we have to install for each application version all necessary Python modules (ideally, the same version of it to avoid compatibility issues) and when we decide to update one of them, this needs to be done again for all 3rd party applications.
That does not sound super-efficient to me.
Do you have similar experiences and what did you came up with to handle this? I know, that there are more advanced packages like rez to handle complex environment setups, but although I do not know the details of rez, I could imagine that the problems stays the same. I guess that it is not possible to globally populate PYTHONPATH with additional modules so that it works on multiple Python interpreter versions.
Another solution that I could imagine is to make sure that on startup of each application that needs additional Python modules, we do our own sys.path modification depending on the interpreter version. That would imply some coding but we could keep a global version handling without installing them everywhere.
Anyway, if you have any further hints, please let me know.
Greets,
Carlo
I'm looking for a way to call my python project and display the console on my c# app.
My python project is a bit particular, i use some specific librairy, i even have to install a package whl not avalible with pip, avalible only with python3.7(not with 3.8) etc. So to run the project, i need python 3.7 exactly.
The problem is also that i want to deploy my c# app in ClickOnce to be use by clients, without having them to install a local python version.
I've seen on the net two ways to do work with python and c#, that both doesn't seem working for my need.
using python in c# app
1.Call python in a shell
I've imported the python project in my c# app, i've called the python.exe avalible in the venv and i've deployed the app. Everithing seems to work, but i discover that python executable in the venv refer to the local python installation and doesn't seem autonomus. So it was only working for me and not the clients.
Use IronPython in c#
Really far from working, starting by the encoding blowing from everywhere, even in imported librairy like numpy. I've referenced my venv librairy in SetSearchPaths(), and even with that, it's doesn't seem to work.
Any suggestions? The best way i think, would be to have a python.exe independant following the project that can load my venv.
Python.Included and the Numpy.NET package made with it may work for this:
https://github.com/henon/Python.Included
https://github.com/SciSharp/Numpy.NET
They are introduced in this post from 2019 and seem to be in active development today, https://medium.com/scisharp/using-python-libraries-in-net-without-a-python-installation-11124d6190cf
That solution does not use IronPython, but Python.NET: https://github.com/pythonnet/pythonnet
IronPython is an implementation of Python in C#, championed by others and then Microsoft itself back in 2010 era. Later Microsoft dropped it and largely the whole notion of supporting dynamic languages on .NET (they made a DLR system for it back then). It is very cool and works for pure Python code.
But NumPy and many useful and popular Python modules are written in C, using the C API of Python.org 's default C written Python implementation, aka. CPython. This is what Microsoft decided to back too instead, because C written Python modules don't work (easily and well) with IronPython. Also IronPython remains at Python 2.7.
Python.NET just bridges the .NET land with the normal CPython interpreter, so that you can call code cross the language boundary. So Numpy and everything works the same as with Python usually. Which is what you want.
Python.Included is one way to deploy that in C# projects - which may or may not work for you, but at least provides a starting point:
Python.Included is an automatic deployment mechanism for .NET packages
which depend on the embedded Python distribution. This allows
libraries depending on Python and/or Python packages to be deployed
via Nuget without having to worry about any local Python
installations.
It packages embedded Python (python-3.7.3-embed-amd64.zip) in its .NET
assembly and automatically deploys it in the user's home directory
upon first execution. On subsequent runs, it will find Python already
deployed and therefor doesn't install it again.
If you don't want to use that install mechanism, I think you can just bundle the CPython interpreter with your C# application and use the Python.NET mechanism to call that from your app's directory instead.
I am trying to debug an application that is written in Python 2. Problem is, before application can be run, it must be installed (according to the documentation) like this:
python setup.py develop
Application (Cuckoo Sandbox) then can be run like this:
cuckoo -d
With the new Python package developing and testing code now works slightly different than it used to be. As one
will first have to Install Cuckoo before being able to use it in the first place, a simple modify-and-test development
sequence doesn’t work out-of-the-box as it used to do.
Is it possible to somehow debug such application?
Later in the (unfinished) docs there is a mention of debugging web interface which I am not sure would get me to the code I need to debug and it also requires PyCharm Professional. I am using PyCharm Community but I am willing to use any other free IDE or debugger.
It will be difficult to debug if you are working only with compiled .pyo and .pyc files, which is all that you might have from an installed application. It is possible that the install includes some source files. You will have to figure out where it is installed, which is platform specific. Since many projects (including cuckoo) are open source, I would recommend cloning the Github repository and debugging from that.
So I wrote a Python script which does some simple stuff. It was originally going to run on a Unix server but due to crappy network security settings which TPTB refuse to change, we need to run it on a Windows server instead. However, the administrators of said Windows server refuse to do anything helpful like install Python.
What are my options for running a Python script on Windows without Python?
Consideration 1:
Something like Py2Exe - I found this after a quick Google search and it seems promising. From what I can tell, it'll generate a bunch of files but we can just xcopy that directory to our Windows machine and it will be completely isolated and not have any external dependencies. Does anyone have any insight on how well this works? Obviously, it depends on my Python script but fortunately this script is quite simple and only uses built in Python libraries such as urllib2 and urlparse.
Consideration 2:
We can assume the Windows server has at least some version of the .NET Framework installed too, which brings IronPython to mind. I've never used this before, but I've always wanted to. From what I can tell, it will compile Python code into CLS compliant IL code which can be run natively under the .NET runtime. However, does this require additional .NET libraries to be installed on the server? Can I just bundle those DLLs with my program? Or, does it require I rewrite my Python script to call into .NET Framework specific classes instead of using things like urllib2 or urlparse?
Thanks!
PS - The ironic part: I actually barely know Python and I'm a .NET expert, but I wrote the script in Python because I was told it would run on a Unix server. Had I known we'd end up running this on a Windows server, I'd have written the thing in C# to begin with in about 1/10th of the time. Fail.
Will they let you copy executables onto the server at all? If so then you should be able to do a non-admin installation of Python or use Portable Python which can just be copied into a folder without any installation at all.
Nothing wrong with Py2exe, but it does mean you then have to build the script into a fresh executable each time you update it. Also Py2exe has a slightly longer startup time than a Python interpreter because it has to extract the Python dlls into a temporary folder each time it runs; that only matters of course if you run your script a lot.
I'm a .NET developer who knows very little about Python, but want to give it a test drive for a small project I'm working on.
What tools and packages should I install on my machine? I'm looking for a common, somewhat comprehensive, development environment.
I'll likely run Ubuntu 9.10, but I'm flexible. If Windows is a better option, that's fine too.
Edit: To clarify, I'm not looking for the bare minimum to get a Python program to run. I wouldn't expect a newbie .NET dev to use notepad and a compiler. I'd recommend Visual Studio, NUnit, SQL Server, etc.
Your system already has Python on it. Use the text editor or IDE of your choice; I like vim.
I can't tell you what third-party modules you need without knowing what kind of development you will be doing. Use apt as much as you can to get the libraries.
To speak to your edit:
This isn't minimalistic, like handing a .NET newbie notepad and a compiler: a decent text editor and the stdlib are all you really need to start out. You will likely need third-party libraries to develop whatever kind of applications you are writing, but I cannot think of any third-party modules all Python programmers will really need or want.
Unlke the .NET/Windows programming world, there is no one set of dev tools that stands above all others. Different people use different editors a whole lot. In Python, a module namespace is fully within a single file and project organization is based on the filesystem, so people do not lean on their IDEs as hard. Different projects use different version control software, which has been booming with new faces recently. Most of these are better than TFS and all are 1000 times better than SourceSafe.
When I want an interactive session, I use the vanilla Python interpreter. Various more fancy interpreters exist: bpython, ipython, IDLE. bpython is the least fancy of these and is supposed to be good about not doing weird stuff. ipython and IDLE can lead to strange bugs where code that works in them doens't work in normal Python and vice-versa; I've seen this first hand with IDLE.
For some of the tools you asked about and some others
In .NET you would use NUnit. In Python, use the stdlib unittest module. There are various third-party extensions and test runners, but unittest should suit you okay.
If you really want to look into something beyond this, get unittest2, a backport of the 2.7 version of unittest. It has incorporated all the best things from the third-party tools and is really neat.
In .NET you would use SQL Server. In Python, you may use PostgreSQL, MySQL, sqlite, or some other database. Python specifies a unified API for databases and porting from one to another typically goes pretty smoothly. sqlite is in the stdlib.
There are various Object Relational Models to make using databases more abstracted. SQLAlchemy is the most notable of these.
If you are doing network programming, get Twisted.
If you are doing numerical math, get numpy and scipy.
If you are doing web development, choose a framework. There are about 200000: Pylons, zope, Django, CherryPy, werkzeug...I won't bother starting an argument by recommending one. Most of these will happily work with various servers with a quick setting.
If you want to do GUI development, there are quite a few Python bindings. The stdlib ships with Tk bindings I would not bother with. There are wx bindings (wxpython), GTK+ bindings (pygtk), and two sets of Qt bindings. If you want to do native Windows GUI development, get IronPython and do it in .NET. There are win32 bindings, but they'll make you want to pull your hair out trying to use them directly.
In order to reduce the chance of effecting/hosing the system install of python, I typically install virtualenv on the ubuntu python install. I then create a virtualenv in my home directory so that subsequent packages I install via pip or easy_install do not effect the system installation. And I add the bin from that virtualenv to my path via .bashrc
$ sudo apt-get install python-virtualenv
$ virtualenv --no-site-packages ~/local
$ PATH=~/local/bin:$PATH #<----- add this to .bashrc to make it permanent
$ easy_install virtualenv #<--- so that project environments are based off your local environment rather than the system, probably not necessary
Install your favorite editor, I like emacs + rope, but editors are a personal preference and there are plenty of choices.
When I start a new project/idea I create a new virtual environment for that project, so that I don't effect dependencies anywhere else. Since I would hate for some of my projects to break due to an upgrade of a library both that project and the new one depends on.
~/projects $ virtualenv --no-site-packages my_new_project.env
~/projects/my_new_project.env $ source bin/activate
(my_new_project.env)~/projects/my_new_project.env $ easy_install paste ipython #whatever else I think I need
(my_new_project.env)~/projects/my_new_project.env $ emacs ./ & # start hacking
When creating a new package...in order to have something that will be easy_installable/pippable use paster create
(my_new_project.env)~/projects/my_new_project.env$ paster create new_package
(my_new_project.env)~/projects/my_new_project.env/new_package$ python setup.py develop new_package
That's the common stuff as far as I can think of it. Everything else would be editor/version control tool specific
Since I'm accustomed to Eclipse, I find Eclipse + PyDev convenient for Python. For quick computations, Idle is great.
I've used Python on Windows and on Ubuntu, and Linux is much cleaner.
If you launch a terminal and type python you'll get an interpreter, where you can start trying stuff.
Just in case you haven't seen it, check out the book Dive Into Python, is free on-line.
http://www.diveintopython.org/
Follow the examples in the book using the interpreter.
For storing your work you could use any editor; Vim or EMACS could be the most powerful, but also the most difficult to learn at first. If you want a more "traditional" IDE, you could try WingIDE.
http://www.wingware.com/
After you start to get more comfortable with python you should try an enhanced interpreter; try ipython.
http://ipython.scipy.org/moin/
When you start to develop a more serious project you'll need to get additional modules. Here you have two options; 1) Use your distribution tools to install additional modules; or 2) Download the modules you need directly from their sites and install them manually. You'll be responsible to upgrade them of course.
You'll have to decide for yourself which way to go. Personally I prefer to download and install additional modules manually.
Python (duh), setuptools or pip, virtualenv, and an editor. I suggest geany, but that's just me. And of course, any other Python modules you'll need.
Getting to Python from .NET world
Jumping into the Linux world from a .NET / WIndows background can be a bit disconcerting (but I do encourage you to keep trying Linux)
But I would suggest to anyone coming from Windows, to stick with Windows for a little while. goto www.Activestate.com and download their Python package - it includes the full win32com extentions by Mark Hammond and it also includes a complete, fast IDE "pythonwin"
I have done real professional development with just this setup alone on a windows box - one 14MB .msi and off you go !
Now to use Python on the DLR (Dynamic common language runtime) you need to download IronPython. THis is a seperate interpreter, that was also originally written by Mark Hammond at Microsoft and is at ironpython.org.
With this you can run code like (from wikipedia) ::
import clr
clr.AddReference("System.Windows.Forms")
from System.Windows.Forms import MessageBox
MessageBox.Show("Hello World")
Now you can access any .NET code from python.
If you're just starting out with Python, I'd actually argue against bringing in the complexity of virtualenv (which I think can be pretty overwhelming), at least until you've got a firm grasp of Python basics (especially regarding library/dependency management).
If you're using Ubuntu and the Gnome desktop environment, gedit is the default (gui) text editor, and has great support for Python built in. So my recommendation is to start with the pre-installed Python and gedit (which is pretty extensible on its own).
You don't need much. Python comes with "Batteries Included."
Visual Studio == IDLE. You already have it. If you want more IDE-like environment, install Komodo Edit.
NUnit == unittest. You already have it in the standard library.
SQL Server == sqlite. You already have it in the standard library.
Stop wasting time getting everything ready. It's already there in the basic Python installation.
Get to work.
Linux, BTW, is primarily a development environment. It was designed and built by developers for developers. Windows is an end-user environment which has to be supplemented for development.
Linux was originally focused on developers. All the tools you need are either already there or are part of simple yum or RPM installs.
You would probably like to give NetBeans Python IDE a shot. You can choose to use either Windows/Linux.
Database: sqlite (inbuilt). You might want SQLAlchemy though.
GUI: tcl is inbuilt, but wxPython or pyQt are recommended.
IDE: I use idle (inbuilt) on windows, TextMate on Mac, but you might like PyDev. I've also heard good things about ulipad.
Numerics: numpy.
Fast inline code: lots of options. I like boost weave (part of scipy), but you could look into ctypes (to use dlls), Cython, etc.
Web server: too many options. Django (plus Apache) is the biggest.
Unit testing: inbuilt.
Pyparsing, just because.
BeautifulSoup (or another good HTML parser).
hg, git, or some other nice VC.
Trac, or another bug system.
Oh, and StackOverflow if you have any questions.
Pycharm Community is worth to try.