I am trying to merge 2 python scripts(before.py and after.py) to one as I need to set a cron job for the python script to run both scripts in one execution.
The 2 scripts are connected wherein the file that is processed by after.py (tmp1.txt) is being generated by before.py.
Original order of script is
python3 before.py someCsv.csv > tmp1.txt
python3 after.py tmp1.txt >tmp2.txt
in before.py, I added this
subprocess.call(' python3 after.py tmp1.txt > tmp2.txt', shell=True)
Which I believe will allow to execute a run a shell command inside python.
If I run the script directly (after.py), it returns 44,623 records.
But if i run after.py within before.py via subprocess.call(), it returns only 44,619 records.
I am using the same file in both scripts ,that is tmp1.txt.
I tried subprocess.call(), subprocess.run(), and os.system(). It seems that I am doing something wrong here.
instead of outputting the result to a new file via ">" in python shell, I just created a new file within before.py which guarantees that before after.py will run, tmp1.txt is 100% completed. Which allows me to get the full 44,623 records.
This is the end of before.py. Instead of directly outputting to a file within the python shell, I just created the file within the script itself.
Related
The following block executes and extracts files until it gets to one particular file and then it just hangs.
#Unzip
unzip_cmd = f"unzip temp/temp.zip -d temp/unzip_output/"
os.system(unzip_cmd)
When executing in terminal, the command runs the entire way through. I've tried using subprocess.run() with the same result.
I'm attempting to run a Linux script through Python's subprocess module. Below is the subprocess command:
result = subprocess.run(['/dir/scripts/0_texts.sh'], shell=True)
print(result)
Here is the 0_texts.sh script file:
cd /dir/TXTs
pylanguagetool text_0.txt > comments_0.txt
The subprocess command executes the script file, writing a new comments_0.txt in the correct directory. However, there's an error in the execution. The comments_0.txt contains an error of "input file is required", and the subprocess result returns returncode=2. When I run the pylanguagetool text_0.txt > comments_0.txt directly in the terminal the command executes properly, with the comments_0.txt written with the proper input file of text_0.txt.
Any suggestions on what I'm missing?
There is some ambiguity here in that it's not obvious which shell is run each time 0_texts.sh is invoked, and whether it has the values you expect of environment variables like PATH, which could result in a different copy of pylanguagetool running from when you call it at the command line.
First I'd suggest removing the shell=True option in subprocess.run, which is only involving another, potentially different shell here. Next I would change subprocess.run(['/dir/scripts/0_texts.sh']) to subprocess.run(['bash', '/dir/scripts/0_texts.sh']) (or whichever shell you wanted to run, probably bash or dash) to remove that source of ambiguity. Finally, you can try using type pylanguagetool in the script, invoking pylanguagetool with its full path, or calling bash /dir/scripts/0_texts.sh from your terminal to debug the situation further.
A bigger-picture issue is, pyLanguageTool is a Python library, so you're almost certainly going to be better off calling its functions from your original Python script directly instead of using a shell script as an intermediary.
I'm trying to automate my workflow. So I'm generating a test list for my system in MATLAB. In the same script, I'm then waiting for a new log file which only gets generated when I run a python script in a certain directory through windows powershell.
That python script would take the test list generated and produce a new log file. My code looks for the date of modification of that file, if it has changed, it means a new log file has been produced. The code then extracts certain strings of characters and then plot the result.
My question is how can I invoke powershell from MATLAB to invoke that python script execution command in a specific directory?
Regards
If you really want to commit to this Powershell-in-the-middle approach, you can call it with system
system( 'powershell -command "Some Powershell command"' );
You can collect outputs from this if there are any.
Seems like it would be easier to call the Python script from Matlab though and cut out the call to Powershell, this could also be done with the system command (i.e. without also calling Powershell in between). Or you could invoke Python directly.
Why running a python file doesn't require the x permission when running it like this:
python script.py
But it does when it's being run as:
./script.py
Because what you are running with python script.py is the python program; then, it loads and runs the script that you specified in parameters, that is script.py (basically a text file). The script file doesn't need to be an executable because what is executed here is the python interpreter (the python binary itself, that should obviously have the x permission).
With .\script.py, you try to run directly your script (still the same text file) as a program. When you do that, you want it to be parsed with the interpreter that you specified in the first line of your script code, the "shebang", e.g. #!/usr/bin/env python. If it's not set with the x permission, the OS doesn't try to "execute" your file (though it might try to open it with the default program, where applicable), so, it will not care about the shebang.
The file itself it interpreted (read) rather than actually executed in your first example. The python application is what needs execute rights.
In the second example the file itself is being executed, so needs those rights in order to proceed.
When we run a script as python script.py we actually invoke the python interpreter which is generally located at /usr/bin/python (The output of which python will tell you where exactly).
The interpreter in turn reads the scripts and executes its code.
It is the interpreter that has the execute permission.
When a script is executed as ./script.py then the script is executed directly and hence the script requires execute permission. The interpreter used is specified by shebang line.
When the kernel finds that the first two bytes are #! then it uses the rest of the line as interpreter and passes the file as argument. Note that to do this the file needs to have execute permission. In the former case we are indirectly doing what the kernel would do had we executed the script as ./script.py
In short for executing by method1 the interpreter needs only read permission but for later it needs to execute it directly
I have a Python3 script that can take optional command line parameters which the script reads via sys.argv. My script works fine if I run it from a terminal window, however any time I add it to the PA schedule tab with parameters, I get a bash 126 exit code in the log. According to the bash appendix this exit code means Permission problem or command is not an executable. So, it would appear that the scripts running from a terminal window don't run in the same context as scripts run from schedule, because my script runs fine from a bash terminal.
What I need is to figure out a way to get scripts to run with optional command line parameters via the schedule tab.
I figure I could write another script that calls the first script with the optional parameters, however it seems to me there should be another way to get around this where you can just put the parameters you wish to run the script with directly on the line you give to the schedule tab to run. Otherwise I'll have to write a script for each separate set of parameters I want to call with versus just an entry to schedule the script to run.
Thanks to #Lattyware for this solution. If you have this problem you can solve it by going to a terminal and adding the execute bit on your script. You can accomplish adding the execute permission using chmod such as:
chmod 700 foo.py
You may also want to verify your file permissions prior to any changes to ensure you don't add or remove something you didn't intend to.
You can read up on chmod here if you need to.