Develop Reference

Linux bash executables behave differently depending on whether I am activating from the command line or with os.system('')

I've been attempting to execute a certain CLI from within python and store the output for later use within the same script. I suspect this question has a simple answer, but if one wishes to go through the entire pipeline, here is the tool in question.
wget http://rna.urmc.rochester.edu/Releases/current/RNAstructureForLinux.tgz
tar xvf as usual, go inside the resulting directory and execute 'make all', the executables I use in the bash script are within the 'exe' directory.
I attempted to execute the commands with os.system(), but with little luck. The CLI I am using; however, seems to be running. The function which I have set to execute the os.system() commands contains the following block.
txt = open('home/spectre/tools/RNAstructure/exe/RNAStructure_nucleic_acid.txt',"w")
txt.write('AAGGCTGTCCAGGCGCAATGTGGTGGCTGCTTCTCTGGGGAGTCCTCCAGGCTTGCCCAACCCGGGGCTCCGTCCTCTTGGCCCAAGAGCTACCCCAGCAGCTGACATCCCCCGGGTACCCAGAGCCGTATGGCAAAGGCCAAGAGAGCAGCACGGACATCAAGGCTCCAGAGGGCTTTGCTGTGAGGCTCGTCTTCCAGGACTTCGACCTGGAGCCGTCCCAGGACTGTGCAGGGGACTCTGTCACAGTGAGCTGGGGATGGGGGGGGTCCCGCCAGGACTGTGGCCAGGGAGATTCCCGGGGTTGTGGGAAGTGGCGGTGCCCTGAATCCCCCATCTGGAGGAGGGATGAAT')
os.system(' cd ~/tools/RNAstructure/exe ; ./python_RNA_structure.sh')
nucleotides, structure, MFE =
RNAStructure_from_file('home/spectre/tools/RNAstructure/exe/RNAStructure_bracket_output.txt')
The executable *.sh file contains this.
#!/bin/bash
cd ~/tools/RNAstructure/exe
./Fold RNAStructure_nucleic_acid.txt RNAStructure_nucleic_acid_output.txt
./ct2dot RNAStructure_nucleic_acid_output.txt -1 RNAStructure_bracket_output.txt
If I execute the bash script from the command line the output should look a little like this
Initializing nucleic acids...
Using auto-detected DATAPATH: "../data_tables" (set DATAPATH to avoid this warning).
done.
98% \[==================================================\] \\ done.
Writing output ct file...done.
Single strand folding complete.
Converting CT file...
Using auto-detected DATAPATH: "../data_tables" (set DATAPATH to avoid this warning).
CT file conversion complete.
If I execute the bash script form the python file.
Initializing nucleic acids...
Using auto-detected DATAPATH: "../data_tables" (set DATAPATH to avoid this warning).
Error reading sequence. The file did not contain any nucleotides.
Single strand folding complete with errors.
Converting CT file...
Using auto-detected DATAPATH: "../data_tables" (set DATAPATH to avoid this warning).
CT file conversion complete.
It looks an awful lot like my CLI can find the files it needs inside the terminal, but not outside of it. I haven't experimented with any parameters like trying absolute paths, but I understood by using os.system() I could execute a bash script, but it is not clear to me why this is changing how that script behaves.
What I've done to resolve the problem:
reopening the file seems to resolve the problem, but I am still working out why.

The problem seems to resolve when I reopen the file within the python script like so:
txt = open('home/spectre/tools/RNAstructure/exe/RNAStructure_nucleic_acid.txt',"w")
txt.write('AAGGCTGTCCAGGCGCAATGTGGTGGCTGCTTCTCTGGGGAGTCCTCCAGGCTTGCCCAACCCGGGGCTCCGTCCTCTTGGCCCAAGAGCTACCCCAGCAGCTGACATCCCCCGGGTACCCAGAGCCGTATGGCAAAGGCCAAGAGAGCAGCACGGACATCAAGGCTCCAGAGGGCTTTGCTGTGAGGCTCGTCTTCCAGGACTTCGACCTGGAGCCGTCCCAGGACTGTGCAGGGGACTCTGTCACAGTGAGCTGGGGATGGGGGGGGTCCCGCCAGGACTGTGGCCAGGGAGATTCCCGGGGTTGTGGGAAGTGGCGGTGCCCTGAATCCCCCATCTGGAGGAGGGATGAAT')
txt = open('home/spectre/tools/RNAstructure/exe/RNAStructure_nucleic_acid.txt')
os.system(' cd ~/tools/RNAstructure/exe ; ./python_RNA_structure.sh')
nucleotides, structure, MFE =
RNAStructure_from_file('home/spectre/tools/RNAstructure/exe/RNAStructure_bracket_output.txt')
I am not sure why this resolves the problem, I found this solution serendipitously. I'll update the answer when I figure out why, unless someone wants to beat me to it. It's magic to me for now.
It seems that after opening the file, RNAStructure_nucleic_acid.txt, and assigning it to the txt variable for writing, I need to reopen it after writing is complete. Otherwise the file is blank when I try printing it's output within the program, but after the program finishes executing, the file contains the correct text.

Running and piping text into a .exe file from Python

I am trying to a run a .exe file from python and pipe a string into it. The .exe itself opens a command box and requires a series of string inputs that can be entered in one go on a series of lines (as below)
In bash the solution would be:
printf "test.dat\nMoreinput\nMoreinput" | ~/Desktop/Median_filt_exes/ascxyz.exe
To recreate this in python I have tried:
from subprocess import Popen, PIPE
p = Popen(r"./ascxyz.exe", stdin=PIPE,text=True)
p.communicate("test.dat\nMoreinput/nMoreinput")
There's no error however it doesn't seem to be working (the .exe should create a new file when run successfully). Any help into what I could do to figure out why the exe isnt running properly would be very appreciated!

The immediate problem is probably that you are not terminating the input with a newline. But you really also don't want to do the Popen plumbing yourself.
from subprocess import run
run(['./ascxyz.exe'], text=True,
input="test.dat\nMoreInput\nMoreInput\n")
Notice also how we pass in a list as the first argument, to avoid the complications of shell=True.

Python does not allow additional comment with env python

I have a commenting line problem in python. I created a ages.py with vim and here is my script
#!/usr/bin/env python
ages={"dad":42, "mom":35, "lisa":7}
for item in ages:
print item
When I add a comment above the !/usr/bin, like
# this is a python script
#!/usr/bin/env python
ages={"dad":42, "mom":35, "lisa":7}
for item in ages:
print item
and after I go back to directory, which includes the same script, and run the script with writing to terminal ages.py but I get this error
$ ./ages.py
./ages.py: line 3: mom:35,: command not found
./ages.py: line 5: syntax error near unexpected token `print'
./ages.py: line 5: 'print item'
I know that when I remove the #!/usr/bin/env python line # works perfectly for commenting. However, I would like to add this line to run the script only writing its name to prompt. Otherwise, I have to write python ages.py every single time to run it, which I see as a burden.
1st Q: How can I comment to .py script with still having the '#!/usr/bin/env python' line
P.S: I've already tried """ ''' before and after comment line, it does not work either.
2nd Q: Is there a way to run a .py script in python environment? For instance, I can run .m file in octave simply writing its name to command line. But I could not figure out if there is a same way in python?
P.S: I know the questions are so easy and deserve -1, however, for new python users, the all answers will provide great information, thanks for all of them. It is so subtle solution but without it, I've constantly got the errors

That first line is a very important one. It's called the Hashbang and sometimes known as the shebang. It tells the operating system what interpreter to use to execute the script. When the shebang is used, it has to be the first line. Other variations include
#!/bin/sh
#!/usr/bin/perl/
#!/usr/bin/python
These are for system default sh, perl and python. Any other comments in your code has to be after this line.

the line containing #!/usr/bin/env python must be first, you can add your comments below;
yes, you can do it like this: exec(open('yourscript.py').read()) - however, results might sometimes differ from your expectations.

Capture dynamic command prompt output in text file [duplicate]

This question already has answers here:
how to direct output into a txt file in python in windows
(6 answers)
Closed 6 years ago.
I am running a python script which checks for the modifications of files in a folder. I want that output to be printed in a file. The problem is that the output is DYNAMIC , the cmd is always open and when a file is modified, I will have an information right-ahead about that in the cmd window. All the solutions which I found were matching the situations were I just run a command and I finish with that.
I tryed with:
python script.py > d:\output.txt but the output.txt file is empty
An example of the command prompt windows, after I run the command python script.py and I touch the 2 files, the command prompt will look like this. I want to capture that output.
Solution: In the python script which I use, add to the logging.basicConfig function, one more argument : filename='d:\test.log'

The issue is output buffering. If you wait long enough, you'll eventually see data show up in the file in "blocks". There are a few ways around it, for example:
Run python with the -u (unbuffered) flag
Add a sys.stdout.flush() after all print statements (which can be simplified by replacing stdout with a custom class to do it for you; see the linked question for more)
Add flush=True option to print statements if your version of Python supports it
If appropriate, use the logging module instead of print statements.

python test.py>test.txt
It's working for me in windows cmd prompt

As I see it the simplest would be to add the file handling (the writing to output.txt ) inside your script. Thus, when it is time to print the information you need to have (as your example shows when you touch two files you print two lines), you can open the file, write the specific line and close it after it is done (then you can see the updated output.txt).
Get the file path for the output.txt as a command line argument like
python script.py --o 'd:\output.txt'
for example.

Dropping a file onto a python script in windows - avoid Windows use of backslashes in the argument

I know all about how Windows uses backslashes for filenames, etc., and Unix uses forward. However, I never use backslashes with strings I create in my code. However:
When windows explorer "drops" a file onto a python script, the string it passes contains backslashes. These translate into escape sequences in the strings in the sys.argv list and then I have no way to change them after that (open to suggestions there)
Is there any way I can somehow make windows pass a literal string or ... any other way I can solve this problem?
I'd love my script to be droppable, but the only thing preventing me is windows backslashes.
EDIT:
Sorry everyone, the error was actually not the passing of the string - as someone has pointed out below, but this could still help someone else:
Make sure you use absolute path names because when the Windows shell will NOT run the script in the current directory as you would from a command line. This causes permission denied errors when attempting to write to single-part path-names that aren't absolute.

Cannot reproduce. This:
import os, sys
print sys.argv
print map(os.path.exists, sys.argv)
raw_input()
gives me this:
['D:\\workspaces\\generic\\SO_Python\\9266551.py', 'D:\\workspaces\\generic\\SO_Python\\9254991.py']
[True, True]
after dropping the second file onto the first one. Python 2.7.2 (on Windows). Can you try this code out?