Develop Reference

Confusing problem >> FileNotFoundError: [Errno 2] No such file or directory:

This problem puzzled me.
Maybe the problem is in the code, I hope you take a look
with open(training_images_labels_path,'r') as file:
lines = file.readlines()
He says that the file does not exist
FileNotFoundError: [Errno 2] No such file or directory: '\\Desktop\\project\\data\\generated\\training_images_labels.txt'
Though the file exists
I need solutions

If it says that the file does not exist though the file exists, it means the path has been not given properly. Try giving the path correctly.
Method 1:
Giving correct path 'C:\\Users\\Public\\Desktop\\project\\data\\generated\\training_images_labels.txt' or
'C:\\Users\\<insert your username>\\Desktop\\project\\data\\generated\\training_images_labels.txt' is your path if I guess correctly
Method 2:
Using os module ( Recommended )
mydir = 'C:/Users/Public/Desktop/project/data/generated'
myfile = 'training_images_labels.txt'
training_images_labels_path = os.path.join(mydir, myfile)
with open(training_images_labels_path,'r') as file:
lines = file.readlines()
Method 3:
You can also try changing the working directory to the location where your data is present. ie Desktop>project>data>generated here and open the file with file name. ie
with open('training_images_labels.txt','r') as file:
lines = file.readlines()

I had the same problem with importing an excel file, which of course exists in the same directory with my .py file. The chosen solution above did not help me, and actually I didn't understand those three methods, as I am working on mac OS.
This method worked for me: in Spyder, I usually run the file by pressing the keys "Shift + Enter", which in this case made the problem. So, my solution was, to press on the "Run file" button (or the fn+F5 key) instead.
Maybe someone wants to explain the difference.

Looks like its a windows path you are working and i believe path really thrown in the error is wrong when compared to the actual where the txt file resides.. just cross check once, if that's the case try to pass the correct path in to the variable "training_images_labels_path"

Can you tell how you created this path.Some advise.
use path separator library to generate path to avoid this error.
training_images_labels_path
further try to navigate parent directory using python and print pth.may be some new line or linux/windwos convereted path or other special character in path. navigating parent directory and listing will solve
if still not solve try to navigatep parent-parent dir and print path
try hard

Watch your path if its correct or not. I had the same problem and it turned out i didnt had the good path set

What is the best way to read a JSON file and obtain the values without the invisible characters in Python?

I have a simple JSON file that I was supposed to use as a configuration file, it contains the default directories for whoever is running the script using their MacBooks:
{
"main_sheet_path": "/Users/jammer/Documents/Studios⁩/⁨CAT/⁨000-WeeklyReports⁩/2020/",
"reference_sheet_path": "/Users/jammer/Documents/DownloadedFiles/"
}
I read the JSON file and obtain the values using this code:
with open('reportconfig.json','r') as j:
config_data = json.load(j)
main_sheet_path = str(config_data.get('main_sheet_path'))
reference_sheet_path = str(config_data.get('reference_sheet_path'))
I use the path to check for a source file's existence before doing anything with it:
source_file = 'source.xlsx'
source_file = main_sheet_path + filename
if not os.path.isfile(source_file) :
print ('ERROR: Source file \'' + source_file + '\' NOT FOUND!')
return
Note that the filename is inputted as a parameter when the script is run (there are multiple files, the script has to know which one to target).
The file is there for sure but the script never seems to "see" it so I get that "ERROR" that I printed in the above code. Why do I think there are invisible characters? Because when I copy and paste from what was printed in the "error" notice above into the terminal, the last few characters of the file name always gets substituted by some invisible characters and hitting backspace erases characters where the cursor isn't supposed to be.
How do I know for sure that the file is there and that my problem is with reading the JSON file and not in the Directory names or anywhere else in the code? Because I finally gave up on using a JSON config file and went with a configuration file like this instead:
#!/usr/local/bin/python3.7
# -*- coding: utf-8 -*-
file_paths = { "main_sheet_path": "/Users/jammer/Documents/Studios⁩/⁨CAT/⁨000-WeeklyReports⁩/2020/",
"reference_sheet_path": "/Users/jammer/Documents/DownloadedFiles/"
}
I then just import the file and obtain the values like this:
import reportconfig as cfg
main_sheet_path = cfg.file_paths['main_sheet_path']
reference_sheet_path = cfg.file_paths['reference_sheet_path']
...
This workaround works perfectly — I don't get the "error" that the file isn't there when it is and the rest of the script is executed as expected. When the file isn't there, I get the proper "error" I expect and copying-and-pasting the full path and filename from the "error message" gives me the complete file name and hitting the backspace erases the right characters (no funny behavior, no invisible characters).
But could anyone please tell me how read the JSON file properly without getting those pesky invisible characters? I've spent hours trying to figure it out including searching seemingly related questions in stackoverflow but couldn't find the answer. TIA!

I think there is just a typo error in this code:
source_file = 'source.xlsx'
source_file = main_sheet_path + filename
Maybe filename is set to some other file which is not present hence it is giving you error.
Try to set filename='source.xlsx'
Maybe it will help

Python filename if statement not working

I have a python script that I need to run from the windows command line. The line
for filename in os.listdir(os.getcwd() + "\\sampdirectory1\\sampdirectory2"):
if filename.startswith("sample.csv"):
os.remove("sample.csv")
keeps giving me the error
The system cannot find the file specified 'sample.csv'
Well the file doesn't exist yet, it's created in the script for the first time then edited by the script every time after that. What I don't understand is why it's trying the do os.remove on sample.csv, when the if statement should fail, meaning the remove shouldn't be reached.

You can't delete it while holding on to it because "On Windows, attempting to remove a file that is in use causes an exception to be raised"
https://docs.python.org/2/library/os.html#os.remove

There are 2 things to notice here:
first: the folder/destination are different. you should be using
os.remove(os.getcwd() + "\sampdirectory1\sampdirectory2" + "sample.csv")
second: a more elegant solution would be
try:
os.remove(os.getcwd() + "\\sampdirectory1\\sampdirectory2" + "sample.csv")
except:
print ('no such file/directory')
pass

There might be a file .\sampdirectory1\sampdirectory2\sample.csv, so the condition is valid. But you're trying to delete a file .\sample.csv (sample.csv in current directory) that doesn't exist and you're getting the error.
Moreover, there might be a file .\sampdirectory1\sampdirectory2\sample.csvSOMETHING, so the condition is still valid and you're getting the error.

You need to do os.remove(filename) instead of os.remove("sample.csv")
Because at first sample.csv is not the file that you are checking if it exists before removing. And even the filename is sample.csv you need to precise to full path of the file.
And as you are iterating over listing the directory files you don't need to check if the file exists.
So if you want to remove files whose names start with sample.csv, the code should be as below:
for filename in os.listdir(os.getcwd() + "\\sampdirectory1\\sampdirectory2"):
if filename.startswith("sample.csv"):
os.remove(filename)
But if you want to remove only sample.csv, then you don't need any loop. Just do
filename = os.path.join(os.getcwd(), "sampdirectory1\\sampdirectory2\\sample.csv")
if os.path.exists(filename):
os.remove(filename)

Python script not writing to txt file

I have a Python script that runs properly on my laptop, but when running on my raspberry pi, the following code does not seem to be working properly. Specifically, "TextFile.txt" is not being updated and/or saved.
openfile = open('/PATH/TextFile.txt','w')
for line in lines:
if line.startswith(start):
openfile.write(keep+'\n')
print ("test 1")
else:
openfile.write(line)
print ("test 2")
openfile.close()
I am seeing "test 1" and "test 2" in my output, so I know that the code is being reached, paths are correct, etc
It may be due to a permissions problem. I am running the script from the terminal by using:
usr/bin/python PATH/script.py
Python is owned by "root" and script.py is owned by "Michael".

My first guess:
Does the file exist? If it does not exist then you cannot write to it. Try this to create the file if it does not exist: file = open('myfile.dat', 'w+')
Additionally manually opening and closing file handles is bad practice in python. The with statement handles the opening and closing of the resource automatically for you:
with open("myfile.dat", "w+") as f:
#doyourcalculations with the file object here
for line in f:
print line

All, thank you for your input. I was able to figure out that it was writing to the new file, but it was overwriting with the same text. The reason was because ".startswith" was returning false when I expected true. The misconception was due to the difference between how Windows and Unix treat new line characters (/n /r).

Since your code is running, there should be a file somewhere.
You call "PATH/script.py", but there is "/PATH/TextFile.txt" in your program. Is the slash before PATH a mistake? Have you checked the path in your program is really where you are looking for the output file?

Downloading text files with Python and ftplib.FTP from z/os

I'm trying to automate downloading of some text files from a z/os PDS, using Python and ftplib.
Since the host files are EBCDIC, I can't simply use FTP.retrbinary().
FTP.retrlines(), when used with open(file,w).writelines as its callback, doesn't, of course, provide EOLs.
So, for starters, I've come up with this piece of code which "looks OK to me", but as I'm a relative Python noob, can anyone suggest a better approach? Obviously, to keep this question simple, this isn't the final, bells-and-whistles thing.
Many thanks.
#!python.exe
from ftplib import FTP
class xfile (file):
def writelineswitheol(self, sequence):
for s in sequence:
self.write(s+"\r\n")
sess = FTP("zos.server.to.be", "myid", "mypassword")
sess.sendcmd("site sbd=(IBM-1047,ISO8859-1)")
sess.cwd("'FOO.BAR.PDS'")
a = sess.nlst("RTB*")
for i in a:
sess.retrlines("RETR "+i, xfile(i, 'w').writelineswitheol)
sess.quit()
Update: Python 3.0, platform is MingW under Windows XP.
z/os PDSs have a fixed record structure, rather than relying on line endings as record separators. However, the z/os FTP server, when transmitting in text mode, provides the record endings, which retrlines() strips off.
Closing update:
Here's my revised solution, which will be the basis for ongoing development (removing built-in passwords, for example):
import ftplib
import os
from sys import exc_info
sess = ftplib.FTP("undisclosed.server.com", "userid", "password")
sess.sendcmd("site sbd=(IBM-1047,ISO8859-1)")
for dir in ["ASM", "ASML", "ASMM", "C", "CPP", "DLLA", "DLLC", "DLMC", "GEN", "HDR", "MAC"]:
sess.cwd("'ZLTALM.PREP.%s'" % dir)
try:
filelist = sess.nlst()
except ftplib.error_perm as x:
if (x.args[0][:3] != '550'):
raise
else:
try:
os.mkdir(dir)
except:
continue
for hostfile in filelist:
lines = []
sess.retrlines("RETR "+hostfile, lines.append)
pcfile = open("%s/%s"% (dir,hostfile), 'w')
for line in lines:
pcfile.write(line+"\n")
pcfile.close()
print ("Done: " + dir)
sess.quit()
My thanks to both John and Vinay

Just came across this question as I was trying to figure out how to recursively download datasets from z/OS. I've been using a simple python script for years now to download ebcdic files from the mainframe. It effectively just does this:
def writeline(line):
file.write(line + "\n")
file = open(filename, "w")
ftp.retrlines("retr " + filename, writeline)

You should be able to download the file as a binary (using retrbinary) and use the codecs module to convert from EBCDIC to whatever output encoding you want. You should know the specific EBCDIC code page being used on the z/OS system (e.g. cp500). If the files are small, you could even do something like (for a conversion to UTF-8):
file = open(ebcdic_filename, "rb")
data = file.read()
converted = data.decode("cp500").encode("utf8")
file = open(utf8_filename, "wb")
file.write(converted)
file.close()
Update: If you need to use retrlines to get the lines and your lines are coming back in the correct encoding, your approach will not work, because the callback is called once for each line. So in the callback, sequence will be the line, and your for loop will write individual characters in the line to the output, each on its own line. So you probably want to do self.write(sequence + "\r\n") rather than the for loop. It still doesn' feel especially right to subclass file just to add this utility method, though - it probably needs to be in a different class in your bells-and-whistles version.

Your writelineswitheol method appends '\r\n' instead of '\n' and then writes the result to a file opened in text mode. The effect, no matter what platform you are running on, will be an unwanted '\r'. Just append '\n' and you will get the appropriate line ending.
Proper error handling should not be relegated to a "bells and whistles" version. You should set up your callback so that your file open() is in a try/except and retains a reference to the output file handle, your write call is in a try/except, and you have a callback_obj.close() method which you use when retrlines() returns to explicitly file_handle.close() (in a try/except) -- that way you get explict error handling e.g. messages "can't (open|write to|close) file X because Y" AND you save having to think about when your files are going to be implicitly closed and whether you risk running out of file handles.
Python 3.x ftplib.FTP.retrlines() should give you str objects which are in effect Unicode strings, and you will need to encode them before you write them -- unless the default encoding is latin1 which would be rather unusual for a Windows box. You should have test files with (1) all possible 256 bytes (2) all bytes that are valid in the expected EBCDIC codepage.
[a few "sanitation" remarks]
You should consider upgrading your Python from 3.0 (a "proof of concept" release) to 3.1.
To facilitate better understanding of your code, use "i" as an identifier only as a sequence index and only if you irredeemably acquired the habit from FORTRAN 3 or more decades ago :-)
Two of the problems discovered so far (appending line terminator to each character, wrong line terminator) would have shown up the first time you tested it.

Use retrlines of ftplib to download file from z/os, each line has no '\n'.
It's different from windows ftp command 'get xxx'.
We can rewrite the function 'retrlines' to 'retrlines_zos' in ftplib.py.
Just copy the whole code of retrlines, and chane the 'callback' line to:
...
callback(line + "\n")
...
I tested and it worked.

you want a lambda function and a callback. Like so:
def writeLineCallback(line, file):
file.write(line + "\n")
ftpcommand = "RETR {}{}{}".format("'",zOsFile,"'")
filename = "newfilename"
with open( filename, 'w' ) as file :
callback_lambda = lambda x: writeLineCallback(x,file)
ftp.retrlines(ftpcommand, callback_lambda)
This will download file 'zOsFile' and write it to 'newfilename'