I'm working on a Python application. Sometimes the interpreter crashes when in a third party C++ DLL.
I'm thinking about writing a Python extension that installs a handler for unhandled structured exceptions (Windows) in order to write a minidump to the disk and log the stack trace of every Python thread.
Two questions:
Does a Python extension with a similar purpose already exist? According to my own Google search, nothing seems to be publicly available, but maybe I didn't search enough.
Is it feasible to implement something like this? (I'm experienced in C++ and Windows programming, but have never implemented a Python extension...)
Check out FaultHandler on PyPI.
I recently wanted to do the same thing and created minidumper to do so, and did a small write-up of it here.
Related
I'm working on an (Windows and Mac) application that uses Python as an embedded scripting language.
The application includes an internal text editor, implemented using Scintilla, and I'm using Jedi for autocompletion, which generally works great.
However, when attempting autocompletion on a computer that does not have a separate installation of Python, Jedi raises an error:
jedi.api.environment.InvalidPythonEnvironment:
Could not get version information for 'python':
FileNotFoundError(2, 'The system cannot find the file specified', None, 2, None)
Digging into the code, I can see that the underlying code that is throwing the FileNotFoundError is when Jedi attempts to run python using subprocess.Popen. Python is not installed on the computer, so this fails.
I can also reproduce the same issue on a computer that does have Python installed by editing my Path environment variable not to include the location of python.exe.
Ideally, we don't want users of our application to have to install Python just to get autocompletion working.
My questions:
Is it possible to get Jedi not to spawn subprocesses, and instead run its code inside the same instance of Python within which it itself is running? I couldn't find anything about this in the documentation or the source code that deals with Environments, and extrapolating from the discussion here I suspect the answer might be no.
Is it possible somehow to get Jedi to use the same python37.dll that our application is using for its functionality, instead of looking for a .exe file that does not exist?
Is there any way we could make some kind of minimal Python installation within our existing app installation that uses the same DLLs/Python Lib etc? How could I go about doing this?
Is there any other way to get Jedi autocompletion working in our app without requiring the user to install Python, or including a full Python installer as part of our build process?
Is it possible to get Jedi not to spawn subprocesses, and instead run its code inside the same instance of Python within which it itself is running? I couldn't find anything about this in the documentation or the source code that deals with Environments, and extrapolating from the discussion here I suspect the answer might be no.
This is definitely possible. All the tools are there. There are discussions ongoing here: https://github.com/davidhalter/jedi-vim/issues/870.
IMO a patch to Jedi is needed that uses an jedi.api.environment.InterpreterEnvironment in some cases like yours. It's definitely possible, it's just buggy at the moment.
I was reading up on people who could create scripts to inject code into a Windows process, to make the running process malfunction or function in the way the injector desires. I have also seen that this is called "DLL injection", if I'm not mistaken.
I have been searching around how this is done on Python, and couldn't find a clean answer.
How is this implemented on Python? You would need Windows compatibility such as pywin32 modules, but then what from there? I have seen this done on C, but never on Python. Python was created in C so it should be possible to do this.
I have followed the documentation provided for the Python comtypes module (http://pythonhosted.org/comtypes/server.html) and built a simple COM server.
It seems to be working correctly. I can see that the typelib and interface are registered (using Oleviewer) and from a Python script I can call the (single) method that adds two integers.
However, I cannot make it work with Excel (Office 2007, running on Win7). The following simple function does not work (execution stops when MyMethod is called)
Function test(a, b)
Dim x As New MyTypeLib.MyObject
ab = x.MyMethod(a, b)
Debug.Print "d" & CStr(ab)
test = ab
End Function
Is there any way to debug this?
More comments
There is indeed light at the end of the tunnel. The suggestion to use VBScript has been helpful. The following short script does work (providing independent confirmation that the COM server works).
Dim x,ab
Set x = CreateObject("MyTypeLib.MyObject")
ab = x.MyMethod(1, 2)
MsgBox CStr(ab)
However, VBScript is calling the server using late binding and I was trying to get the early binding to work.
So, I changed my Excel VBA function to use late binding and expected things to work, but they do not.
Here is the modified VBA:
Function test(a, b)
Dim x As Object
Set x = CreateObject("MyTypeLib.MyObject")
gadd = x.MyMethod(1, 5)
End Function
So my server can be used with late binding from Python and VBScript, but not Excel!
Using Process Monitor I can see that in both late and early binding cases Excel tries unsuccessfully to load msvcr90.dll, even though this DLL is installed on the system.
Here is a screenshot from Process Monitor, showing where Office Excel 2007 on 64-bit Windows 7 Home Edition starts to try to load msvcr90.dll
When I posted this question, I needed to know a bit more about how to debug COM servers (well a lot more actually!). It turns out that there were several problems that stopped my server from working. I think finding out about the tools is probably as much of interest as the actual problems, so this answer will try to cover both.
Initially, it was helpful to use VBScript to provide an independent verification that the server was registered and working (the simple code is shown above). VBScript uses late binding and I wanted early binding, but it was nevertheless helpful because I then realised that Excel did not work with the late binding VBA code either (on machine A)!
Something odd happened too with VBScript. When I first wrote scripts they ran by default (clicking on the file in Windows Explorer), but then for no apparent reason they stopped working. I found that I needed to specify the full path to the 32-bit version (C:\WINDOWS\SysWOW64\cscript.exe) for them to work.
Also worth noting is that by using cscript.exe, instead of wscript.exe, debugging messages embedded in my Python modules appear on the Windows CMD window.
The next breakthrough came by using the Dependency Walker program depends.exe. This is often mentioned in posts like mine where people are trying to figure out why DLLs are not working.
Dependency Walker is available from it's own website, but that version does not work properly with Windows 7 (see Profiling x86 executable with Dependency Walker hangs on Windows 7 x64) so I found that I needed to install WDK 8.1 (from here: http://msdn.microsoft.com/en-US/windows/hardware/gg454513).
DW is very powerful and a bit overwhelming at first. However, by reading its logging output it became clear that Excel 2007 uses msvcr80.dll and that the server's attempt to load msvcr90.dll was failing (even though the DLL was properly installed on the machine). This was not a problem on machine B because Excel 2013 uses msvcr90.dll.
Now, in these tests I was creating the server using Python comtypes interactively (there is no server DLL involved). But there is also the possibility of creating a DLL by using the py2exe tool. I was trying to do this too, the DLL failed to register using regsvr32.
DW helped debug this problem too. Happily, in the logging output I noticed a stack trace of a Python exception. It was the GetModule() in the example. It turns out that this call is needed only once, to create some Python classes that are then cached on the machine. Removing this command fixed the problem.
Also, although it may not have caused a problem, the official current version of py2exe is 0.6.9 but this caused some files that are no longer available to be bundled (specifically zlib.pyd). The unofficial version 0.6.10 (available here: http://www.lfd.uci.edu/~gohlke/pythonlibs/#py2exe) solved that issue.
So I now have the possibility of compiling a COM server DLL, in which I can include the required msvcr90.dll, and, yes, it works on both machines :-)
In summary then. Windows 7, with its mixture of 32-bit and 64-bit libraries is a nightmare. VBScript helps, by providing an independent environment for testing a server. Dependency Walker really is amazing, but takes ages to learn to use (other useful tools in the WDK include oleviewer, process-monitor and process-explorer).
Although I found many answers and discussions about this question, I am unable to find a solution particular to my situation. Here it is:
I have a main program written in FORTRAN.
I have been given a set of python scripts that are very useful.
My goal is to access these python scripts from my main FORTRAN program. Currently, I simply call the scripts from FORTRAN as such:
CALL SYSTEM ('python pyexample.py')
Data is read from .dat files and written to .dat files. This is how the python scripts and the main FORTRAN program communicate to each other.
I am currently running my code on my local machine. I have python installed with numpy, scipy, etc.
My problem:
The code needs to run on a remote server. For strictly FORTRAN code, I compile the code locally and send the executable to the server where it waits in a queue. However, the server does not have python installed. The server is being used as a number crunching station between universities and industry. Installing python along with the necessary modules on the server is not an option. This means that my “CALL SYSTEM ('python pyexample.py')” strategy no longer works.
Solution?:
I found some information on a couple of things in thread Is it feasible to compile Python to machine code?
Shedskin, Psyco, Cython, Pypy, Cpython API
These “modules”(? Not sure if that's what to call them) seem to compile python script to C code or C++. Apparently not all python features can be translated to C. As well, some of these appear to be experimental. Is it possible to compile my python scripts with my FORTRAN code? There exists f2py which converts FORTRAN code to python, but it doesn't work the other way around.
Any help would be greatly appreciated. Thank you for your time.
Vincent
PS: I'm using python 2.6 on Ubuntu
One way or another, you'll need to get the Python runtime on your server, otherwise it won't be possible to execute Python bytecode. Ignacio is on the right track with suggesting invoking libpython directly, but due to Fortran's parameter-passing conventions, it will be a lot easier for you to write a C wrapper to handle the interface between Fortran and the CPython embedding API.
Unfortunately, you're doing this the hard way -- it's a lot easier to write a Python program that can call Fortran subroutines than the other way around.
You don't want any of those. What you should do is use FORTRAN's FFI to talk with libpython and frob its API.
This question already has an answer here:
Closed 11 years ago.
Possible Duplicate:
Calling Python from Objective-C
I'm a long-time Python programmer and short-time Cocoa programmer. I'm just getting started with PyObjC and it's really amazing how easy it it is to get stuff done. That said, I wanted to try using pure ObjC for my controller with PyObjC models. I might be enjoy letting Python be Python and Objective-C be Objective-C. I figured it was worth a try, anyways.
Except I can't figure out or find anything about how to call Python from Objective-C, only the other way around. Can someone point me to any resources on this? (Maybe it's on the PyObjC site but I just don't know what I'm looking for?)
Edit: I'm most interested, at the basic level, in being able to call a Python module and get some native ObjC data types back.
There are several possible approaches. The most tempting is to use py2app to compile a loadable bundle from your python code from which you can access the principal class using NSBundle. Unfortunately, this use case hasn't gotten much love from the py2app developers, and I've found several bugs in 10.5 and 10.6, including a rather nasty memory leak when passing data from python back in to Objective-C. I wouldn't recommend using py2app at thist point.
The second approach is invert the embedding. Write a Python cocoa app and load your Objective-C code from a bundle at startup (even in main()). If you already have a large Objective-C app, this may take a bit of work. The only downside, that I'm ware of, is that you won't be able to use GC in your Objective-C code, but this is really a universal limitation in working with PyObjC.
Finally, you can instantiate a python interpreter in your Objective-C code to load your python code. This is obviously more involved, but may the best option if you already have a large Objective-C codebase into which you want to inject your python code. The main.m file from the Python-Cococa application template in Xcode is a good place to start to see this in action.
Whoops, guess I should've searched a bit more:
Calling Python from Objective-C