I am creating a simple python file in unix, just to open and write some test in it, but getting error while execution. Using Python 2.4.3
file = open(“testfile.txt”, “w”)
file.write(“This is a test”)
file.write(“To add more lines.”)
file.close()
Error:
./test.py: line 1: syntax error near unexpected token `('
./test.py: line 1: `file = open(“testfile.txt”, “w”)'
I believe you are using curly quotes “” (e.g. from Microsoft Word, etc..) rather than actual single and double quote chars '' "".
Make sure you are using a regular text editor, not a rich text editor. That's the problem.
The problem is that “ is not a valid quote in Python. Try copying and pasting this code into your file/terminal and you should then realise the difference.
file = open("testfile.txt", "w")
file.write("This is a test")
file.write("To add more lines.")
file.close()
In addition to the "smart" quotes that need to be plain ASCII " characters you need a "shebang" line as the first line of the script. Otherwise it is likely to be treated as a shell script and handed to /bin/sh for execution. You should insert this as the first line of the file:
#!/usr/bin/env python
Or run it via python ./x.py.
I think quotes are the problem.
can you try a content manager
with open('testfile.txt', 'w') as output_file:
output_file.write("Your Text Here")
use of context managers is to properly manage resources. In fact, that's the reason we use a context manager . The act of opening a file consumes a resource (called a file descriptor), and this resource is limited by your OS. Similarly writing.
That is to say, there are a maximum number of files a process can have open at one time.
Related
Having an odd problem.
I have a Django app that opens a file (represented as a Django FieldFile) and reads each row using readline() as below:
with file.open(mode='r') as f:
row = f.readline()
# do something with row...
The file is text, utf-8 encoded and lines are terminated with \r\n.
The problem is each row is being read as the hex representation of the string, so instead of "Hello" I get "48656c6c6f".
A few stranger things:
It previously worked properly, but at some point an update has broken it (I've tried rolling back to previous commits and it is still wonky, so possibly a dependency has updated and not something from my requirements.txt). Missed it in my testing because it is in a very rarely used part of the app.
If I read the same file using readlines() instead of readline() I see the correct string representation of the file wrapped in [b'...']
The file reads normally if I do it using straight Python open() and readline() from an interpreter
Forcing text mode with mode='rt' doesn't change the behaviour, neither does mode='rb'
The file is stored in a Minio bucket, so the defaut storage is storages.backends.s3boto3.S3Boto3Storage from django-storages and not the default Django storage class. This means that boto3, botocore and s3fs are also in the mix, making it more confusing for me to debug.
Scratching my head at why this worked before and what I'm doing wrong.
Environment is Python 3.8, Django 2.2.8 and 3.0 (same result) running in Docker containers.
EDIT
Let me point out that the fix for this is simply using
row = f.readline().decode()
but I would still like to figure out what's happening.
EDIT 2
Further to this, FieldFile.open() is reading the file as a binary file, whereas plain Python open() is reading the file as a text file.
This seems very weird.
I think you will see the solution immediately after trying following (I will then update my answer or delete it if it really doesn't help to find it, but I'm quite confident)
A assume, that there is some code, that is monkeypatching file.open or the django view function.
What I suggest is:
Start your code with manage.py runserver
Ad following code to manage.py (as the very first lines)
import file
print("ID of file.open at manage startup is", id(file.open)
Then add code to your view directly one line above the file.open
print("ID of file.open before opening is", id(file.open)
If both ids are different, then something monkeypatched your open function.
If both are the same, then the problem must be somewhere else.
If you don not see the output of these two prints, something might have monkeypatched your view.
If this doesn't work, then try to use
open() instead of file.open()
Is there any particular reason you use file.open()
Addendum 1:
So what you sai is, that file is an object instance of a class is it a FileField?
In any case can you obtain the name of the file and open it with a normal open() to see whether it is only file.open() that does funny things or whether it is also open() reading it this stange way.
Did you just open the file from command line with cat filename (or if under windows with type filename?
If that doesn't work we could add traces to follow each line of the source code that is being executed.
Addendum 2:
Well if you can't try this in a manage.py runserver, what happens if you try to read the file with a manage.py shell?
Just open the shell and type something like:
from <your_application>.models import <YourModel>
entry = <YourModel>.objects.get(id=<idofentry>)
line1 = entry.<filefieldname>.open("r").read().split("\n")[0]
print("line1 = %r" % line1)
If this is still not conclusive, (but only if you can reproduce the issue with the management shell, then create a small file containing the lines.
from <your_application>.models import <YourModel>
entry = <YourModel>.objects.get(id=<idofentry>)
import pdb; pdb.set_trace()
line1 = entry.<filefieldname>.open("r").read().split("\n")[0]
print("line1 = %r" % line1)
And import it from the management shell.
The code should enter the debugger and now you can single step through the open function and see whether you end up on sime weird function in some monkeypatch.
So I have a python file with a ton of lines in it that I want to read into python then echo into another file over a socket.
Assuming I have file foo.py
import os
os.popen('some command blah')
print("some other commands, doesn't matter")
Then I try and open the file, read all the lines, and echo each line into a new file.
Something along the lines of
scriptCode = open(os.path.realpath(__file__)).readlines()
for line in scriptCode:
connection.send("echo " + line + " >> newfile.py")
print("file transfered!")
However, when I do this, the command is executed in the remote shell.
So my question:
How do I safely echo text into a file without executing any keywords in it?
What have I tried?
Adding single quotes around line
Adding single quotes around line and then a backslash to single quotes in line
Things I've considered but haven't tried yet:
Base64 encoding the line until on the remote machine then decoding it (I don't want to do this because there's no guarentee it'll have this command)
I know this is odd. Why am I doing this?
I'm building a pentesting reverse shell handler.
shlex.quote will:
Return a shell-escaped version of the string s. The returned value is a string that can safely be used as one token in a shell command line, for cases where you cannot use a list.
Much safer than trying to quote a string by yourself.
I have a Python script that runs properly on my laptop, but when running on my raspberry pi, the following code does not seem to be working properly. Specifically, "TextFile.txt" is not being updated and/or saved.
openfile = open('/PATH/TextFile.txt','w')
for line in lines:
if line.startswith(start):
openfile.write(keep+'\n')
print ("test 1")
else:
openfile.write(line)
print ("test 2")
openfile.close()
I am seeing "test 1" and "test 2" in my output, so I know that the code is being reached, paths are correct, etc
It may be due to a permissions problem. I am running the script from the terminal by using:
usr/bin/python PATH/script.py
Python is owned by "root" and script.py is owned by "Michael".
My first guess:
Does the file exist? If it does not exist then you cannot write to it. Try this to create the file if it does not exist: file = open('myfile.dat', 'w+')
Additionally manually opening and closing file handles is bad practice in python. The with statement handles the opening and closing of the resource automatically for you:
with open("myfile.dat", "w+") as f:
#doyourcalculations with the file object here
for line in f:
print line
All, thank you for your input. I was able to figure out that it was writing to the new file, but it was overwriting with the same text. The reason was because ".startswith" was returning false when I expected true. The misconception was due to the difference between how Windows and Unix treat new line characters (/n /r).
Since your code is running, there should be a file somewhere.
You call "PATH/script.py", but there is "/PATH/TextFile.txt" in your program. Is the slash before PATH a mistake? Have you checked the path in your program is really where you are looking for the output file?
I am beginning to learn Python and want to use it to automate a process.
The process consists in
modifying a few lines of a file
use the file as the input for an executable
save, move, etc
repeat
The problem is that the file I'm trying to modify was written in a language that utilizes the SUB character to run. Therefore, when I try
with open(myFile,'r') as file:
data = list(file)
data does not contain any information beyond the SUB character.
Therefore, I need to be able to do two things:
Read the whole file in python (without exiting prematurely at the SUB character locations) so that I can modify it.
Be able to run it on the executable (that is, the SUB characters need to be back at their respective places).
Any suggestions on how to go about solving this problem?
Thanks
Use the binary mode to open file.
with open(myFile,'rb') as file:
for line in file:
print line
Are you on Windows? Quoted from your link to the SUB character:
In CP/M, 86-DOS, MS-DOS, PC DOS, DR-DOS and their various derivatives, character 26 was also used to indicate the end of a character stream, and thereby used to terminate user input in an interactive command line window (and as such, often used to finish console input redirection, e.g. as instigated by COPY CON: TYPEDTXT.TXT).
While no longer technically required to indicate the end of a file many text editors and program languages up to the present still support this convention...
Python 2.7 in text mode will stop at a CTRL-Z character (hex 1A), so open the file in binary mode:
Example:
# Create a file with embedded character 1Ah
with open('sub.txt','wb') as f:
f.write(b'abc\x1adef')
# Open in default (text) mode and read as much as possible
with open('sub.txt','r') as f:
print repr(f.read())
# Open in binary mode
with open('sub.txt','rb') as f:
print repr(f.read())
Output:
'abc'
'abc\x1adef'
I'm trying to automate downloading of some text files from a z/os PDS, using Python and ftplib.
Since the host files are EBCDIC, I can't simply use FTP.retrbinary().
FTP.retrlines(), when used with open(file,w).writelines as its callback, doesn't, of course, provide EOLs.
So, for starters, I've come up with this piece of code which "looks OK to me", but as I'm a relative Python noob, can anyone suggest a better approach? Obviously, to keep this question simple, this isn't the final, bells-and-whistles thing.
Many thanks.
#!python.exe
from ftplib import FTP
class xfile (file):
def writelineswitheol(self, sequence):
for s in sequence:
self.write(s+"\r\n")
sess = FTP("zos.server.to.be", "myid", "mypassword")
sess.sendcmd("site sbd=(IBM-1047,ISO8859-1)")
sess.cwd("'FOO.BAR.PDS'")
a = sess.nlst("RTB*")
for i in a:
sess.retrlines("RETR "+i, xfile(i, 'w').writelineswitheol)
sess.quit()
Update: Python 3.0, platform is MingW under Windows XP.
z/os PDSs have a fixed record structure, rather than relying on line endings as record separators. However, the z/os FTP server, when transmitting in text mode, provides the record endings, which retrlines() strips off.
Closing update:
Here's my revised solution, which will be the basis for ongoing development (removing built-in passwords, for example):
import ftplib
import os
from sys import exc_info
sess = ftplib.FTP("undisclosed.server.com", "userid", "password")
sess.sendcmd("site sbd=(IBM-1047,ISO8859-1)")
for dir in ["ASM", "ASML", "ASMM", "C", "CPP", "DLLA", "DLLC", "DLMC", "GEN", "HDR", "MAC"]:
sess.cwd("'ZLTALM.PREP.%s'" % dir)
try:
filelist = sess.nlst()
except ftplib.error_perm as x:
if (x.args[0][:3] != '550'):
raise
else:
try:
os.mkdir(dir)
except:
continue
for hostfile in filelist:
lines = []
sess.retrlines("RETR "+hostfile, lines.append)
pcfile = open("%s/%s"% (dir,hostfile), 'w')
for line in lines:
pcfile.write(line+"\n")
pcfile.close()
print ("Done: " + dir)
sess.quit()
My thanks to both John and Vinay
Just came across this question as I was trying to figure out how to recursively download datasets from z/OS. I've been using a simple python script for years now to download ebcdic files from the mainframe. It effectively just does this:
def writeline(line):
file.write(line + "\n")
file = open(filename, "w")
ftp.retrlines("retr " + filename, writeline)
You should be able to download the file as a binary (using retrbinary) and use the codecs module to convert from EBCDIC to whatever output encoding you want. You should know the specific EBCDIC code page being used on the z/OS system (e.g. cp500). If the files are small, you could even do something like (for a conversion to UTF-8):
file = open(ebcdic_filename, "rb")
data = file.read()
converted = data.decode("cp500").encode("utf8")
file = open(utf8_filename, "wb")
file.write(converted)
file.close()
Update: If you need to use retrlines to get the lines and your lines are coming back in the correct encoding, your approach will not work, because the callback is called once for each line. So in the callback, sequence will be the line, and your for loop will write individual characters in the line to the output, each on its own line. So you probably want to do self.write(sequence + "\r\n") rather than the for loop. It still doesn' feel especially right to subclass file just to add this utility method, though - it probably needs to be in a different class in your bells-and-whistles version.
Your writelineswitheol method appends '\r\n' instead of '\n' and then writes the result to a file opened in text mode. The effect, no matter what platform you are running on, will be an unwanted '\r'. Just append '\n' and you will get the appropriate line ending.
Proper error handling should not be relegated to a "bells and whistles" version. You should set up your callback so that your file open() is in a try/except and retains a reference to the output file handle, your write call is in a try/except, and you have a callback_obj.close() method which you use when retrlines() returns to explicitly file_handle.close() (in a try/except) -- that way you get explict error handling e.g. messages "can't (open|write to|close) file X because Y" AND you save having to think about when your files are going to be implicitly closed and whether you risk running out of file handles.
Python 3.x ftplib.FTP.retrlines() should give you str objects which are in effect Unicode strings, and you will need to encode them before you write them -- unless the default encoding is latin1 which would be rather unusual for a Windows box. You should have test files with (1) all possible 256 bytes (2) all bytes that are valid in the expected EBCDIC codepage.
[a few "sanitation" remarks]
You should consider upgrading your Python from 3.0 (a "proof of concept" release) to 3.1.
To facilitate better understanding of your code, use "i" as an identifier only as a sequence index and only if you irredeemably acquired the habit from FORTRAN 3 or more decades ago :-)
Two of the problems discovered so far (appending line terminator to each character, wrong line terminator) would have shown up the first time you tested it.
Use retrlines of ftplib to download file from z/os, each line has no '\n'.
It's different from windows ftp command 'get xxx'.
We can rewrite the function 'retrlines' to 'retrlines_zos' in ftplib.py.
Just copy the whole code of retrlines, and chane the 'callback' line to:
...
callback(line + "\n")
...
I tested and it worked.
you want a lambda function and a callback. Like so:
def writeLineCallback(line, file):
file.write(line + "\n")
ftpcommand = "RETR {}{}{}".format("'",zOsFile,"'")
filename = "newfilename"
with open( filename, 'w' ) as file :
callback_lambda = lambda x: writeLineCallback(x,file)
ftp.retrlines(ftpcommand, callback_lambda)
This will download file 'zOsFile' and write it to 'newfilename'