Develop Reference

Syntax error in a correct python script when redirected to stdin

Using the bash shell on Ubuntu Linux and Python 2.7.6, if I run the python script below with the command,
python test.py
I get this output:
outer
inner
2
If I, instead, use redirection of stdin like this,
python -i < test.py
The output is,
>>> outer
>>> >>> ... ... ... File "<stdin>", line 4
print i
^
SyntaxError: invalid syntax
>>>
So, clearly, I shouldn't use the -i option with shell redirection, ie with the '<' character in the command. This combination apparently leads to some manipulation of white space which, by the time python parses the input, is a python syntax error.
Unfortunately, I run into the same problem when running python scripts in the Django python shell and the command to run the Django python shell has no way to disable interactive python mode and no way to pass the name of a python script file.
Furthermore, I believe this worked on a different Linux box or earlier on the same Linux box so I wonder if there is some environment setting that would fix it.
Is there a way to change the behavior of python's interactive mode or is there a way to disable interactive mode in the Django shell so I can run general python scripts without false syntax errors? Or am I simply suffering from a bug in Django or python??
Incidentally, this issue sounds very much like the problem described in https://stackoverflow.com/a/13703413/2950621 but the accepted answer says to alter empty lines in the script. Yet my script has no empty lines.
The python script follows. (I wrote it to flag when the indented lines are not executed):
print 'outer'
i = 1
if True:
print 'inner'
i = 2
print i
I suppose some readers might suspect a hidden character is to blame. Here is the octal dump so you can see there is nothing but what you see above (not even a tab):
$ od -a test.py
0000000 p r i n t sp ' o u t e r ' nl i sp
0000020 = sp 1 nl i f sp T r u e : nl sp p r
0000040 i n t sp ' i n n e r ' nl sp i sp =
0000060 sp 2 nl p r i n t sp i nl

You're having the opposite of the problem in the linked question. The -i makes python read the script in interactive mode (the ">>> " and "... " are prompts), and in interactive mode, you to press return (i.e. enter a blank line) to exit an indented section. The linked question involved blank lines in the middle of an indented section, which confused the interpreter. You have the opposite: you end an indented section without a blank line. That's perfectly ok in non-intereactive mode, but in interactive mode it's a no-no.
Solution: either put blank lines at the end of indented sections (and never in the middle), don't tell the interpreter to read it in interactive mode. I'm not sure what you're actually trying to accomplish with the -i flag, but note that python -i test.py (without the redirect) will run the script in non-interactive mode and then enter interactive mode and read from stdin.

How to use pipe in command using python sub process call [duplicate]

This question already has answers here:
How to use `subprocess` command with pipes
(7 answers)
Closed 7 years ago.
I am trying to filter out first 3 line of /proc/meminfo using pipe and head command.
so basically i need to run this in Python:
cat /proc/meminfo | head -3
I am using below line in my code :
subprocess.call(["cat", "/proc/meminfo", "|", "head", "-3"])
While just using subprocess.call(["cat", "/proc/meminfo"]) I am getting whole list but I am just interested in first 3 line.
Using above command is giving me below error:
cat: invalid option -- '3'
Try `cat --help' for more information.
Any suggestions?

/proc/meminfo is just a file. You don't need a subprocess to read it. Simply open and read it as a file. Here is all you need:
fh = open('/proc/meminfo', 'r')
lines = fh.readlines()
fh.close()
first_lines = lines[:3]
The first_lines list will contain the first three lines (including trailing newline characters).

To use pip you have to enable shell as shell=True, however it's not advisable specifically because of security reason . You can do this alternative,
import subprocess
ps = subprocess.Popen(('cat', '/proc/meminfo'),stdout=subprocess.PIPE)
output = subprocess.check_output(('head', '-3'), stdin=ps.stdout)
print output

The pipe is a shell syntax element. You need to run the code in a shell to use a pipe:
subprocess.call(["cat /proc/meminfo | head -3"], shell=True)
From the manual:
If shell is True, the specified command will be executed through the shell. This can be useful if you are using Python primarily for the enhanced control flow it offers over most system shells and still want convenient access to other shell features such as shell pipes, filename wildcards, environment variable expansion, and expansion of ~ to a user’s home directory.

Well head actually accepts an argument, so the pipe is not actually necessary. The following should give the expected result.
subprocess.call(["head", "-3", "/proc/meminfo"])

following this document
In default, subprocess.call with shell=False will disables all shell based features including pipe. When using shell=True, pipes.quote() can be used to properly escape whitespace and shell metacharacters in strings that are going to be used to construct shell commands.
you can use this code
subprocess.call("cat /proc/meminfo | head -3", shell=True)

Python equivalent to perl -pe?

I need to pick some numbers out of some text files. I can pick out the lines I need with grep, but didn't know how to extract the numbers from the lines. A colleague showed me how to do this from bash with perl:
cat results.txt | perl -pe 's/.+(\d\.\d+)\.\n/\1 /'
However, I usually code in Python, not Perl. So my question is, could I have used Python in the same way? I.e., could I have piped something from bash to Python and then gotten the result straight to stdout? ... if that makes sense. Or is Perl just more convenient in this case?

Yes, you can use Python from the command line. python -c <stuff> will run <stuff> as Python code. Example:
python -c "import sys; print sys.path"
There isn't a direct equivalent to the -p option for Perl (the automatic input/output line-by-line processing), but that's mostly because Python doesn't use the same concept of $_ and whatnot that Perl does - in Python, all input and output is done manually (via raw_input()/input(), and print/print()).
For your particular example:
cat results.txt | python -c "import re, sys; print ''.join(re.sub(r'.+(\d\.\d+)\.\n', r'\1 ', line) for line in sys.stdin)"
(Obviously somewhat more unwieldy. It's probably better to just write the script to do it in actual Python.)

You can use:
$ python -c '<your code here>'

You can in theory, but Python doesn't have anywhere near as much regex magic that Perl does, so the resulting command will be much more unwieldy, especially as you can't use regular expressions without importing re (and you'll probably need sys for sys.stdin too).
The Python equivalent of your colleague's Perl one-liner is approximately:
import sys, re
for line in sys.stdin:
print re.sub(r'.+(\d\.\d+)\.\n', r'\1 ', line)

You have a problem which can be solved several ways.
I think you should consider using regular expression (what perl is doing in your example) directly from Python. Regular expressions are in the re module. An example would be:
import re
filecontent = open('somefile.txt').read()
print re.findall('.+(\d\.\d+)\.$', filecontent)
(I would prefer using $ instead of '\n' for line endings, because line endings are different between operational systems and file encodings)
If you want to call bash commands from inside Python, you could use:
import os
os.system(mycommand)
Where command is the bash command. I use it all the time, because some operations are better to perform in bash than in Python.
Finally, if you want to extract the numbers with grep, use the -o option, which prints only the matched part.

Perl (or sed) is more convenient. However it is possible, if ugly:
python -c 'import sys, re; print "\n".join(re.sub(".+(\d\.\d+)\.\n","\1 ", l) for l in sys.stdin)'

Quoting from https://stackoverflow.com/a/12259852/411282:
for ln in __import__("fileinput").input(): print ln.rstrip()
See the explanation linked above, but this does much more of what perl -p does, including support for multiple file names and stdin when no filename is given.
https://docs.python.org/3/library/fileinput.html#fileinput.input

You can use python to execute code directly from your bash command line, by using python -c, or you can process input piped to stdin using sys.stdin, see here.

Learn Python the Hard Way, Exercise 15

I'm trying to solve exercise 15's extra credit questions of Zed Shaw's Learn Python the Hard Way but I've ran into a problem. The code is as follows:
from sys import argv
script, filename = argv
txt = open(filename)
print "Here's your file %r:" % filename
print txt.read()
print "I'll also ask you to type it again:"
file_again = raw_input("> ")
txt_again = open(file_again)
print txt_again.read()
print txt_again.read()
I understand all the code that has been used, but extra credit question 7 asks:
Startup python again and use open from the prompt. Notice how you can open files and run read on them right there?
I've tried inputting everything I could think of in terminal (on a mac) after first starting up python with the 'python' command, but I can't get the code to run. What should I be doing to get this piece of code to run from the prompt?

Zed doesn't say to run this particular piece of code from within Python. Obviously, that code is getting the filename value from the parameters you used to invoke the script, and if you're just starting up the Python shell, you haven't used any parameters.
If you did:
filename = 'myfilename.txt'
txt = open(filename)
then it would work.

I just started with open(xyz.txt)
Well, yes, of course that isn't going to work, because you don't have a variable xyz, and even if you did, it wouldn't have an attribute txt. Since it's a file name, you want a string "xyz.txt", which you create by putting it in quotes: 'xyz.txt'. Notice that Python treats single and double quotes more or less the same; unlike in languages like C++ and Java, there is not a separate data type for individual characters - they're just length-1 strings.

Basically, just like in this transcript (I've added blank lines to aid readability):
pax:~$ python
Python 2.7.1+ (r271:86832, Apr 11 2011, 18:05:24)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> xyz = open ("minimal_main.c")
>>> print xyz.read()
int main (void) {
return 0;
}
>>> xyz.close()
>>> <CTRL-D>
pax:~$ _
All it's showing you is that you don't need a script in order to run Python commands, the command line interface can be used in much the same way.

print open('ex15_sample.txt').read()
After running python in terminal, we'll use open('filename.txt') to open the file and using the dot operator we can apply the read() function directly on it.

After running Python in terminal,
abc = open ("ex15_sample.txt")
print abc.read()
That should do.

Is Python's seek() on OS X broken?

I'm trying to implement a simple method to read new lines from a log file each time the method is called.
I've looked at the various suggestions both on stackoverflow (e.g. here) and elsewhere for simulating "tail" functionality; most involve using readline() to read in new lines as they're appended to the file. It should be simple enough, but can't get it to work properly on OS X 10.6.4 with the included Python 2.6.1.
To get to the heart of the problem, I tried the following:
Open two terminal windows.
In one, create a text file "test.log" with three lines:
one
two
three
In the other, start python and execute the following code:
Python 2.6.1 (r261:67515, Feb 11 2010, 00:51:29)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.stat('test.log')
posix.stat_result(st_mode=33188, st_ino=23465217, st_dev=234881025L, st_nlink=1, st_uid=666, st_gid=20, st_size=14, st_atime=1281782739, st_mtime=1281782738, st_ctime=1281782738)
>>> log = open('test.log')
>>> log.tell()
0
>>> log.seek(0,2)
>>> log.tell()
14
>>>
So we see with the tell() that seek(0,2) brought us to the end of the file as reported by os.stat(), byte 14.
In the first shell, add another two lines to "test.log" so that it looks like this:
one
two
three
four
five
Go back to the second shell, and execute the following code:
>>> os.stat('test.log')
posix.stat_result(st_mode=33188, st_ino=23465260, st_dev=234881025L, st_nlink=1, st_uid=666, st_gid=20, st_size=24, st_atime=1281783089, st_mtime=1281783088, st_ctime=1281783088)
>>> log.seek(0,2)
>>> log.tell()
14
>>>
Here we see from os.stat() that the file's size is now 24 bytes, but seeking to the end of the file somehow still points to byte 14?? I've tried the same on Ubuntu with Python 2.5 and it works as I expect. I tried with 2.5 on my Mac, but got the same results as with 2.6.
I must be missing something fundamental here. Any ideas?

How are you adding two more lines to the file?
Most text editors will go through operations a lot like this:
fd = open(filename, read)
file_data = read(fd)
close(fd)
/* you edit your file, and save it */
unlink(filename)
fd = open(filename, write, create)
write(fd, file_data)
The file is different. (Check it with ls -li; the inode number will change for almost every text editor.)
If you append to the log file using your shell's >> redirection, it'll work exactly as it should:
$ echo one >> test.log
$ echo two >> test.log
$ echo three >> test.log
$ ls -li test.log
671147 -rw-r--r-- 1 sarnold sarnold 14 2010-08-14 04:15 test.log
$ echo four >> test.log
$ ls -li test.log
671147 -rw-r--r-- 1 sarnold sarnold 19 2010-08-14 04:15 test.log
>>> log=open('test.log')
>>> log.tell()
0
>>> log.seek(0,2)
>>> log.tell()
19
$ echo five >> test.log
$ echo six >> test.log
>>> log.seek(0,2)
>>> log.tell()
28
Note that the tail(1) command has an -F command line option to handle the case where the file is changed, but a file by the same name exists. (Great for watching log files that might be periodically rotated.)

Short answer: no, your assumptions are.
Your text editor is creating a new file with the same name, not modifying the old file in place. You can see in your stat result that the st_ino is different. If you were to do os.fstat(log.fileno()), you'd get the old size and old st_ino.
If you want to check for this in your implementation of tail, periodically compare the st_ino of the stat and fstat results. If they differ, there's a new file with the same name.