Develop Reference

python subprocess console output - eclipse calls batch calls python calls exe

I have the situation that I want the job done from eclipse so I use eclipse's .launch configuration. I tried to make it run python directly but got error: error 193 (%1 is not a valid Win32 app). where %1 is probably my python script.
I decided to create a simple batch script that calls this big wild python animal.
I did a lot of combinations and found this the best (batch outputs some strings, runs python, waits for it, the outputs some strings again):
start /b /wait "Python_script.py" "%1" "%2" "%3" "%4" "%5"
It worked until python itself started to run exe file.
Once again I tried a lot of combinations:
os.system([exe, arg1, arg2, ...]) and
subprocess.call(..) and subprocess.check_output(..)
-> I either didn't see the output in eclipse console, or the output was delayed or there was only python / or only exe's output in console.
finally I used subprocess.Popen(...) and it's nearly allright - the only defect is that the output from python script don't wait for exe's process to finish, and when I use subprocess.Popen(...).wait() exe passes output to console but the WHOLE output from python script is delayed until the 'exe' terminates. I want to delay only the part of pythons script output that is written after the exe is called.
how to achieve this 'partly console output delay' is the main topic
advices on python and eclipse .launch configuration will be appreciated
general advices on how does the communication between this processes(?) work will be appreciated
Thanks!

It sounds to me like you have three different processes you're trying to get to work together, you've tried a whole bunch of stuff to get it working, and the code is complex enough that you can't easily post it here. That makes it pretty hard to get a good answer (Stack Overflow works much better with focused questions), but here's the general approach I'd take:
Does your script run if you try to run Python_script.py directly from the command prompt?
If it doesn't, then look into registering the .py file type in Windows.
If it does, then maybe Eclipse launch configurations don't support or don't properly support Windows registered file types. There should be no need to mess with batch files and start; just replace Python_script.py in your launch configuration with c:\Python27\python.exe Python_script.py (or similar).
Get your script working from a command prompt - able to run, with proper Python and subprocess output, and waiting for everything to terminate.
If things work from the command prompt and still don't work from Eclipse, then post a new question with a small snippet of code showing what you're trying and a description of what's wrong. subprocess.call, subprocess.check_output, and Popen all have different uses, so it's hard to give general advice besides just referring to the documentation.

How to properly debug a python3.3 curses application using pdb?

When using pdb to debug a curses application, the interactive debugger is useless, since curses messes up the terminal screen. Debugging post mortem works though, but that is a bit limited.
So what we probably need is having the debugger work in a terminal separately from the debuggee (the application that is being debugged).
Some alternatives which apply remote debugging (such as xpdb) appear either not to work with python 3.3 or give weird errors for other reasons.
So how can I use pdb in a different terminal, or in another proper way?

Use some debugger's functionalities for attach to a running process. For instance you can try:
gdb python <pid>
See how here Python Wiki DebuggingWithGdb.
being the pid of the process you want to debug. Also there is WinPdb that allows you to connect to a remote or local process. WinPdb is well documented and I think is your best option.

I've found that this bit of advice from the Python documentation helps:
A common problem when debugging a curses application is to get your
terminal messed up when the application dies without restoring the
terminal to its previous state. In Python this commonly happens when
your code is buggy and raises an uncaught exception. Keys are no
longer echoed to the screen when you type them, for example, which
makes using the shell difficult. In Python you can avoid these
complications and make debugging much easier by importing the module
curses.wrapper. It supplies a wrapper() function that takes a
callable. It does the initializations described above, and also
initializes colors if color support is present. It then runs your
provided callable and finally deinitializes appropriately. The
callable is called inside a try-catch clause which catches exceptions,
performs curses deinitialization, and then passes the exception
upwards. Thus, your terminal won’t be left in a funny state on
exception.
Please see here for info.

Can't daemonize Python script

I have a problem with deamonizing of my Python application.
As basic daemon on Python, I have used this example. It works, but only for some simple actions. But when i've tried to deamonize my app - nothing was happen.
This is my crawler code, (it's too big for posting it on SO) and Recursion()._recurse(url, 1) - is the instance what starts crawler (it works independently).
How can I deamonize my application? Thanks to all replies in advance!

Note: Is not actually daemonizing, but is an approach I've had used and had made me happy when I've needed it
If you're on Unix you can use screen command. I've used that for a while to "daemonize" Django's Server with a lot of success.
Here are some usage examples:
In Unix, what is screen, and how do I use it?
Screen Command Examples: Get Control of Linux / Unix Terminal

If you just want to run the script in the backround you can use nohup, just add an '&' to the end of the script. For example: python myscript.py &. However this is different from daemonizing
Another option would be to create a cron job

subprocess running python getting import error

I'm trying to run a python script from a python program by kicking it off from subprocess (The reason is that the main program has to have exited when the script runs, with a combination of wx.CallAfter and Close). However when the script runs I get an error on line 1 with ImportError: No module named os which makes me think it's something to do with the PythonPath, but I can run the script just fine from a terminal.
Why can't the script see any core modules when run this way?
Edit:
The line in question is:
wx.CallAfter(subprocess.Popen,'python %s "%s" %s %s'%(os.path.join(BASE_DIR,"updatecopy.py"),BASE_DIR,pos[0],pos[1]),shell=True)
BASE_DIR is just the directory that the script lives in.

subprocess is there because os.exec* has been deprecated so I wouldn't suggest using that in place of Popen as someone suggested.
I've seen this issue crop up when running from a frozen process. If that is the case then you're most likely inheriting a weird environment for the new python process.
Most frozen scripts will be trying to run from a zip file, in which case it's no wonder that Python can't find anything, it's all trapped in a zip file :)
If this is the situation then try running using the python executable that you are using to run the frozen script. It should be able to deal with the special environment.

Maybe you could use os.execv instead of Popen.
From os/python docs:
These functions all execute a new program, replacing the current process; they do not return. On Unix, the new executable is loaded into the current process, and will have the same process id as the caller. Errors will be reported as OSError exceptions.
(emphasis mine)

How to properly use PyDev with two different Python versions with scripts that are recalling other python scripts?

The story began with a very strange error while I was running my script from PyDev. Running the same script from outside will not encounter the same problem.
Fatal Python error: Py_Initialize: can't initialize sys standard streams
File "C:\Python26\lib\encodings\__init__.py", line 123
raise CodecRegistryError,\
^
SyntaxError: invalid syntax
This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.
I was able to find why this is happening: In PyDev I use two different Python versions: 3.1 that is the default installation and 2.6 as the alternative one.
My Windows Environment does not contains PYTHONHOME, CLASSPATH, PYTHONPATH but PyDev does add them.
Now the problem is at one stage my python script does execute another python script using os.system(python second.py) and the second script will fail with the above error.
Now I'm looking to find a way to prevent this issue, issue that is happening because it will run the execute the default python using the settings for the non-default one (added by PyDev).
I do not want to change the standard call (python file.py) but I want to be able to run my script from pydev without problem and being able to use default or alternative python environment.
Any ideas?

I found a solution that seams acceptable specially because it will not interfere with running the scripts on other systems, just to run python -E second.py - this will force Python to ignore PYTHON* environment variables.

I may not be understanding this quite right, but I think you're invoking a script from pydev that works okay, but this script executes another script which requires a different version.
While this would unfortunately be installation-specific, you could use os.system("c:\absolute\path\to\proper\version\of\python.exe second.py").
If PyDev is setting up conflicting environmental variables, you may want to look into subprocess over os.system.
http://docs.python.org/library/subprocess.html#using-the-subprocess-module
This will allow you to invoke a process with a handle, so you can optionally wait for it to terminate. It will also allow you to pass environment variables upon execution.

I believe your call should be:
import sys
os.system(sys.executable+ ' second.py')
So that you guarantee you're using the same interpreter you're currently running and not launching the other one (or did you really mean to use the other interpreter?)