I have a python script that calls a USB-based data-acquisition C# dotnet executable. The main python script does many other things, e.g. it controls a stepper motor. We would like to check the relative timing of various operations, for that purpose the dotnet exe generates a log with timestamps from C# Stopwatch.GetTimestamp(), which as far as I know yields the same number as calls to win32 API QueryPerformanceCounter().
Now I would like to get similar numbers from the python script. time.clock() returns such values, unfortunately it subtracts the value obtained at the time of 1st call to time.clock(). How can I get around this? Is it easy to call QueryPerformanceCounter() from some existing python module or do I have to write my own python extension in C?
I forgot to mention, the python WMI module by Tim Golden does this:
wmi.WMI().Win32_PerfRawData_PerfOS_System()[0].Timestamp_PerfTime
, but it is too slow, some 48ms overhead. I need something with <=1ms overhead. time.clock() seems to be fast enough, as is c# Stopwatch.GetTimestamp().
TIA,
Radim
Have you tried using ctypes?
from ctypes import *
val = c_int64()
windll.Kernel32.QueryPerformanceCounter(byref(val))
print val.value
You could just call the C# StopWatch class directly from Python couldn't you? Maybe a small wrapper is needed (don't know Python/C# interop details - sorry) - if you are already using C# for data acquisition, doing the same for timings via Stopwatch should be simpler than anything else you can do.
Related
How can I run c/c++ code within python in the form:
def run_c_code(code):
#Do something to run the code
code = """
Arbitrary code
"""
run_c_code(code)
It would be great if someone could provide an easy solution which does not involve installing packages. I know that C is not a scripting language but it would be great if it could do a 'mini'-compile that is able to run the code into the console. The code should run as it would compiled normally but this needs to be able to work on the fly as the rest of the code runs it and if possible, run as fast as normal and be able to create and edit variables so that python can use it. If necessary, the code can be pre-compiled into the code = """something""".
Sorry for all the requirements but if you can make the c code run in python then that would be great. Thanks in advance for all the answers..
As somebody else already pointed out, to run C/C++ code from "within" Python, you'd have to write said C/C++ code into an own file, compile it correctly, and then execute that program from your Python code.
You can't just type one command, compile it, and execute it. You always have to have the whole "framework" set up. You can't compile a program when you haven't yet written the } that ends the class/function/statement 20 lines later on. At this point you'd already have to write the whole C/C++ program for it to work. It's simply not meant to be interpreted on the run, line by line. You can do that with python, bash/dash/batch, and a few others. But C/C++ definitely isn't one of them.
With those come several issues. Firstly, the C/C++ part probably needs data from the Python part. I don't know of any way of doing it in RAM alone (maybe there is one, but I don't know), so the Python part would have to write it into a file, the C/C++ part would read and process it, then put the processed data into another file, and then the Python part would have to read that and continue.
Which brings another point up. Here we're already getting into multi-threading territory, because the moment you execute that C/C++ program you're dealing with a second thread. So, somehow, you'd have to coordinate those programs so that the Python part only continues once the C/C++ part is done. Shouldn't be a huge problem to get running, but it can be a nightmare to performance and RAM if done wrongly.
Without knowing to what extent you use that program, I also like to add that C/C++ isn't platform-independent like Python. You'll have to compile that program for every single different OS that you run it on. That may come with minor changes to the code and in general just a lot of work because you have to debug and test it for every single system.
To sum up, I think it may be better to find another solution. I don't know why you'd want to run this specific part in C/C++, but I'd recommend trying to get it done in one language. If there's absolutely no way you can get it done in Python (which I doubt, there's libraries for almost everything), you should get your Python to C/C++ instead.
If you want to run C/C++ code - you'll need either a C/C++ compiler, or a C/C++ interpreter.
The former is quite easy to arrange (though probably not suitable for an end user product) and you can just compile the code and run as required.
The latter requires that you attempt to process the code yourself and generate python code that you can then import. I'm not sure this one is worth the effort at all given that even websites that offer compilation tools wrap gcc/g++ rather than implement it in javascript.
I suspect that this is an XY problem; you may wish to take a couple of steps back and try to explain why you want to run c++ code from within a python script.
I'm using github's linguist to identify unknown source code files. Running this from the command line after a gem install github-linguist is insanely slow. I'm using python's subprocess module to make a command-line call on a stock Ubuntu 14 installation.
Running against an empty file: linguist __init__.py takes about 2 seconds (similar results for other files). I assume this is completely from the startup time of Ruby. As #MartinKonecny points out, it seems that it is the linguist program itself.
Is there some way to speed this process up -- or a way to bundle the calls together?
One possibility is to just adapt the linguist program (https://github.com/github/linguist/blob/master/bin/linguist) to take multiple paths on the command-line. It requires mucking with a bit of Ruby, sure, but it would make it possible to pass multiple files without the startup overhead of Linguist each time.
A script this simple could suffice:
require 'linguist/file_blob'
ARGV.each do |path|
blob = Linguist::FileBlob.new(path, Dir.pwd)
# print out blob.name, blob.language, blob.sloc, etc.
end
I have some Python source code that manipulates lists of lists of numbers (say about 10,000 floating point numbers) and does various calculations on these numbers, including a lot of numpy.linalg.norm for example.
Run time had not been an issue until we recently started using this code from a C# UI (running this Python code from C# via IronPython). I extracted a set of function calls (doing things as described in the 1st paragraph) and found that this code takes about 4x longer to run in IronPython compared to Python 2.7 (and this is after excluding the startup/setup time in C#/IronPython). I'm using a C# stopwatch around the repeated IronPython calls from C# and using the timeit module around an execfile in Python 2.7 (so the Python time results include more operation like loading the file, creating the objects, ... whereas the C# doesn't). The former requires about 4.0 seconds while the latter takes about 0.9 seconds.
Would you expect this kind of difference? Any ideas how I might resolve this issue? Other comments?
Edit:
Here is a simple example of code that runs about 10x slower on IronPython on my machine (4 seconds in Python 2.7 and 40 seconds in IronPython):
n = 700
for i in range(n-1):
for j in range(i, n):
dist = np.linalg.norm(np.array([i, i, i]) - np.array([j, j, j]))
You're using NUMPY?! You're lucky it works in IronPython at all! The support is being added literally as we speak!
To be exact, there's a CPython-extension-to-IronPython interface project and there's a native CLR port or numpy. I dunno which one you're using but both ways are orders of magnitude slower that working with the C version in CPython.
UPDATE:
The Scipy for IronPython port by Enthought that you're apparently using looks abandoned: last commits in the repos linked are a few years old and it's missing from http://www.scipy.org/install.html, too. Judging by the article, it was a partial port with interface in .NET and core in C linked with a custom interface. The previous paragraph stands for it, too.
Using the information from Faster alternatives to numpy.argmax/argmin which is slow , you may get some speedup if you limit data passing back and forth between the CLR and the C core.
I've just about finished coding a decently sized disease transmission model in C#. However, I'm fairly new to .NET and am unsure how to proceed. Currently I just double-click on the .exe file and the model imports config setting from text files, does its thing, and outputs the results into a text file.
What I would like to do next is write a Python script to do the following:
Run the simulation N times (N > 1000)
After each run rename the output file and store (i.e. ./output.txt -> ./acc/outputN.txt)
Aggregate, parse, and analyze the outputs
Output the result in some clean format (possibly excel)
The majority of my programming experience to date has been in C/C++ on linux. I'm fairly confident about the last two items; however, I have no idea how to proceed for the first two. Here are some specific questions I'd like advice on:
What is the easiest/best way to run my C# .exe from a python script?
Does anyone have advice on the best way to do filesystem operations in Python on a Windows system?
Thanks!
As of Python 2.6+ you should be using the subprocess module: (Docs)
import subprocess
for v in range(1000):
cmdLine = r"c:\path\to\my\app.exe"
subprocess.Popen(subprocess)
subprocess.Popen(r"move output.txt ./acc/output-%d.txt" % (v))
The answer to your problems can be found in 'os' in the python standard library. Documentation for doing various operations, such as handling files and starting processes, can be found here.
Process management (Running your C# program) can be found here and file operations are here.
EDIT: Actually, instead of the above process link, you should use the subprocess module.
I'm developing my own Python code interpreter using the Python C API, as described in the Python documentation. I've taken a look on the Python source code and I tried to follow the same steps that are carried out in the standard interpreter when executing a py file. These steps (sequence of C API function calls) are basically:
PyRun_AnyFileExFlags()
PyRun_SimpleFileExFlags()
PyRun_FileExFlags()
PyArena_New()
PyParser_ASTFromFile()
run_mod()
PyAST_Compile()
PyEval_EvalCode()
PyEval_EvalCodeEx()
PyThreadState_GET()
PyFrame_New()
PyEval_EvalFrameEx()
The only difference in my code is that I do manually the AST compilation, frame creation, etc. and then I call PyEval_EvalFrame.
With this, I am able to execute an arbitrary .py file with my program, as if it were the normal Python interpreter. My problem comes when the code that my program is executing makes use of the time module: all time module operations get blocked in the GIL! For example, if the Python code calls time.sleep(1), this call is blocked and never gets executed.
Obviously I am doing something wrong that blocks the GIL (and therefore blocks the time module) but I dont know how to correct it. The last statement in my code where I have control is in PyEval_EvalFrameEx, and from that point on, everything runs "as in regular Python interpreter", I think.
Anybody had a similar problem? What am I doing wrong, so that I block the time module?
Hope somebody can help me...
Thanks for your time. Best regards,
R.
You need to provide more detail.
How does your interpreter's behavior differ from the standard interpreter?
If you just want to run arbitrary source files, why are you not calling one of the higher level interfaces, like PyRun_SimpleFile? Did your code call Py_Initialize?