I am new to Python and Multiprocessing.
Summarized issue: I am running a .py executable which contains a Pool.map(function,list_args) the function it calls contain the line os.system() which it executes a .csh file in the same directory as the initial .py executable. This .csh executable creates a new folder and moves the working directory into it. I believe this where my code fails, I believe that the os.system() does not wait for the .csh executable to finish.
(Ubuntu is the OS on my computer).
Here is part of the code:
import sys
import os
import multiprocessing as mp
import time
def phenix(FILES):
os.system("mkdir temp_"+str(FILES))
time.sleep(1)
os.system("cp 5k10_only_atoms.pdb.mtz "+str(FILES)+" temp_"+str(FILES)+"/")
retval = os.getcwd()
time.sleep(1)
os.chdir(str(retval)+"/temp_"+str(FILES))
time.sleep(1)
os.system("./phenix.get_cc_mtz_pdb 5k10_only_atoms.pdb.mtz "+str(FILES)+" scale=True fix_xyz=True >> grabado")
os.chdir(retval)
#THE USER INPUTS VALUES FOR THE VARIABLES cores AND list_files
if __name__ == '__main__':
p = mp.Pool(cores)
p.map(phenix,list_files)
The command line
./phenix.get_cc_mtz_pdb 5k10_only_atoms.pdb.mtz .....and so on
Runs some calculations comparing values insides the files 5k10_only_atoms.pdb.mtz and FILES (another file e.g. "file-01"). It is this command line phenix.get_cc_mtz_pdb that creates another folder (called temp_dir) and moves the working directory into it to do some calculations and write out files.
Again I am pretty sure that this is where something goes wrong because the phenix command never does anything besides creating a folder named temp_dir. It is only when phenix (a .csh executable) tries to change the working directory that os.system() does not wait for any return and my .py executable continues on to the next command.
Is their some way to let my phenix executable finish its job with out os.system() not waiting for it to finish?
NOTE: I am not able to modify this phenix (.csh) executable. Also the phenix command line does work when I run it directly on the terminal.
Related
So basically I am running a script in Python that executes other 5 .py scripts as well, just like this:
exec(open('statcprs.py').read())
exec(open('dragndownf.py').read())
exec(open('lowprbubble.py').read())
exec(open('wshearstr.py').read())
exec(open('slices.py').read())
These .py files uses Paraview (another software) to run some stuff, so If I only run "statcprs.py", it will open Paraview's terminal and run the script. The problem is, from the first one "statcprs.py" to the second one "dragndownf.py" it doesn't interrupt the software, and it continue running it, interefering with scripts from both .py files.
I would like to execute the first one, stop and then start the second one from scratch without connection between them. is this somehow possible?
I think the problem is this line (line 1) which opens the terminal:
#!/usr/bin/env pvpython
The following will execute a list of python scripts in the same folder as the driver script:
import os
from pathlib import Path
import subprocess
import sys
scripts = [
'statcprs.py',
'dragndownf.py',
'lowprbubble.py',
'wshearstr.py',
'slices.py',
]
parent = Path(__file__).resolve().parent
for script in scripts:
script_path = parent / script
subprocess.call([sys.executable, script_path])
I try to do the following:
1 VBA script calls shell using
RetVal = Shell(fullpythonexepath & fullscriptpath)
2 Shell get follwing command
fullpythonexepath & fullscriptpath
3 Python Script
import numpy as np
thefilepath = "input.txt" # is in the same folder as the script
theoutputpath = "output.txt" # is in the same folder as the script
ssc=np.loadtxt(thefilepath, delimiter='\t', skiprows=0, usecols=range(2)) #error line
#some other code to fit input data
np.savetxt(theoutputpath, sscnew, header=top, fmt='%1.2f\t%1.2f')
When I run this the output file doesn't get printed, which means the script doesn't run properly.
Now to the funny thing: When I run the python script from IDLE it runs fine. When I run it from the shell using the command:
fullpythonexepath & fullscriptpath
it says :
I tried inputting the fullpath to the input.txt and output.txt. When I do this and run it in the IDLE it does not find the file anymore. When called from the shell it doesn't work either.
Python obviousely does not find the path to the input.txt
the issue is, to my understanding, not related to VBA. The error occures also when using shell commands
Solution was to put :
import os
os.chdir(r"fullpath")
into my script this changes the current working directory to my path and then input.txt gets found.
When you start the script from shell the working directory for the script will be the directory from which it is called upon.
Most IDLE define their own working directory.
To check I suggest doing:
os.getcwd()
in both cases and look what directory is used in both cases
I am using Vizard to create an .exe file off a python script. I need this script to create a folder which resides next to the .exe file
if getattr(sys, 'frozen', False):
logging.warning('Application is exe')
loggingPath = os.path.dirname(sys.executable)
logging.warning(os.getcwd())
elif __file__:
loggingPath = os.path.dirname(__file__)
logging.warning('Application is script')
logging.warning(os.getcwd())
if not os.path.exists(loggingFolder):
logging.warning('Directory not existing... creating..')
os.makedirs(loggingFolder)
works fine when I execute from the IDE, but in an exe file it throws data in the Appdata Folder in Windows/Users/Temp/randomfoldername.
Also, I always Application is script, even when its packed into exe.
Can someone point me in the right direction here?
Thanks in advance
The sys module does not have any attribute frozen, which results in the first if statement always returning False.
sys.executable will give the path to the python interpreter binary, ie. for Windows the path of your python.exe file, which I cannot see why you would need for this.
If what you want is to ensure that the file running is a .exe file, then make a folder next to it, it may be simpler to just check if the filename ends with .exe?
if __file__.endswith('.exe'):
loggingFolder = os.path.join(os.path.dirname(__file__), 'foldername')
if not os.path.exists(loggingFolder):
os.makedirs(loggingFolder)
if you just want to create a folder at runtime, then another (possibly easier) method is to run your vizard program from within a batch file and create the folder first in the batch file
e.g. create run_viz_prog.bat with content like this :-
mkdir new_folder
my_viz_prog.exe
I'm writing a python (ver 2.7) script to automate the set of commands in this Getting Started example for INOTOOL.
Problem: When I run this entire script, I repeatedly encounter these errors:
Current Directory is not empty
No project is found in this directory
No project is found in this directory
But, when I run a first script only up till the code line marked, and manually type in the next three lines, or when I run these last three lines (starting from the "ino init -t blink" line) after manually accessing the beep folder, then I am able to successfully execute the same code.
Is there a limitation with os.system() that I'm encountering?
My code:
import os,sys
def upload()
os.system("cd /home/pi/Downloads")
os.system("mkdir beep")
os.system("cd beep") #will refer to this code junction in question description
os.system("ino init -t blink")
os.system("ino build")
os.system("ino upload")
sys.exit(0)
Yes, when os.system() commands are run for cd , it does not actually change the current directory for the python process' context. From documentation -
os.system(command)
Execute the command (a string) in a subshell. This is implemented by calling the Standard C function system(), and has the same limitations. Changes to sys.stdin, etc. are not reflected in the environment of the executed command.
So even though you are changing directory in os.system() call, the next os.system call still occurs in same directory. Which could be causing your issue.
You shoud try using os.chdir() to change the directory instead of os.system() calls.
The Best would be to use subprocess module as #PadraicCunningham explains in his answer.
You can use the subprocess module and os.mkdir to make the directory, you can pass the current working directory cwd to check_callso you actually execute the command in the directory:
from subprocess import check_call
import os
def upload():
d = "/home/pi/Downloads/beep"
os.mkdir(d)
check_call(["ino", "init", "-t", "blink"],cwd=d)
check_call(["ino", "build"],cwd=d)
check_call(["ino", "upload"],cwd=d)
A non-zero exit status will raise CalledProcessError which you may want to catch but once successful you know the commands all returned a 0 exit status.
I'm trying to use py2exe to compile a python script into an executable.
I've set up the setup.py file just like it's described in documentation:
from distutils.core import setup
import py2exe
setup(console=['agent.py', 'test.py'])
The agent.py file simply uses subprocess.Popen to open another script:
import sys
import subprocess
print 'is this working?'
child = subprocess.Popen([sys.executable, 'test.py'])
The test.py file is
while 0 == 0:
print 'test'
When running this as a python script, it works fine. When running as a py2exe-compiled executable, it does not run.
When I try to change the file reference in agent.py from 'test.py' to 'test.exe', running the compiled agent.exe simply prints 'is this working?' on an infinite loop. What have I done wrong?
sys.executable points to agent.exe instead of python.exe when run as compiled executable. You need to change your Popen to:
child = subprocess.Popen(['test.exe'])
when running compiled executable. You can use hasattr(sys, "frozen") to determine whether you're in frozen (py2exe) or not (Python script) mode.
That didn't quite work, but all I had to do was replace your answer with the full path name. Thanks! This worked:
app_path = os.path.realpath(os.path.join(
os.path.dirname(sys.executable), 'test.exe'))
child = subprocess.Popen(app_path)