Earlier this week I put together my first ever working python script. I am trying to create another one in a different directory, but when I try to run any script at all (using Notepad++) I get an error that shows Python seems to be trying to access the old directory and not finding it, even though I haven't told it to look in the old directory. Now the original script doesn't work either. This is what the error message looks like, no matter what I try to run:
python "C:\Users\me\Documents\oldDirectory\oldScript.py"
Process started (PID=12884) >>>
Traceback (most recent call last):
File "C:\Users\me\Documents\oldDirectory\oldScript.py", line 13, in <module>
month = pd.read_csv(sheet)
File "C:\Users\me\AppData\Local\Programs\Python\Python39\lib\site-packages\pandas\io\parsers.py", line 610, in read_csv
return _read(filepath_or_buffer, kwds)
File "C:\Users\me\AppData\Local\Programs\Python\Python39\lib\site-packages\pandas\io\parsers.py", line 462, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
File "C:\Users\me\AppData\Local\Programs\Python\Python39\lib\site-packages\pandas\io\parsers.py", line 819, in __init__
self._engine = self._make_engine(self.engine)
File "C:\Users\me\AppData\Local\Programs\Python\Python39\lib\site-packages\pandas\io\parsers.py", line 1050, in _make_engine
return mapping[engine](self.f, **self.options) # type: ignore[call-arg]
File "C:\Users\me\AppData\Local\Programs\Python\Python39\lib\site-packages\pandas\io\parsers.py", line 1867, in __init__
self._open_handles(src, kwds)
File "C:\Users\me\AppData\Local\Programs\Python\Python39\lib\site-packages\pandas\io\parsers.py", line 1362, in _open_handles
self.handles = get_handle(
File "C:\Users\me\AppData\Local\Programs\Python\Python39\lib\site-packages\pandas\io\common.py", line 642, in get_handle
handle = open(
FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\me\Documents\oldDirectory\Table.csv'
<<< Process finished (PID=12884). (Exit code 1)
As you might guess from the error message, my original script used pd.read_csv() and accessed Table.csv. No matter what I try to run, I get this error. What's going on?
You can check again the path of you file to read. If it's not wrong, try without the '.csv' at the end of the file path.
The problem was with how I was using Notepad++ and not the code I was running. To run a script in that environment you press F6 and a dialog pops up asking what you want to execute. This does NOT default to whichever .py you have open at the moment, so when it hit ctrl+F6 to skip the dialog, it kept trying to run my old script.
Related
Premise
I'm trying to convert some PDF to images via pdf2image and poppler, to then run some computervision tasks on.
The conversion itsself works fine.
However, the conversion creates some artifacts for each page in the pdf as it is being converted, which I would like to be deleted at the end of the function. To facilitate this, I am using tempfile.TemporaryDirectory(). The function looks as follow:
with tempfile.TemporaryDirectory() as path:
images_from_path: [Image] = convert_from_path(
os.path.join(path_superfolder, "calibration_target.pdf"),
size=(2480, 3508),
output_folder=path, poppler_path=r'E:\poppler-22.04.0\Library\bin')
if len(images_from_path) >= page:
images_from_path[page - 1].save(os.path.join(path_superfolder, "result.jpg"))
Problem
The trouble is, that the program always crashes with the following errors, after transforming the PDF and writing the required image to a file.
Traceback (most recent call last):
File "C:\Python310\lib\shutil.py", line 617, in _rmtree_unsafe
os.unlink(fullname)
PermissionError: [WinError 32] The process cannot access the file, because it is being used by another process: 'C:\\Users\\tobia\\AppData\\Local\\Temp\\tmp24c4bmzv\\bd76d834-672e-49fc-ac30-7751b7b660d0-01.ppm'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python310\lib\tempfile.py", line 843, in onerror
_os.unlink(path)
PermissionError: [WinError 32] The process cannot access the file, because it is being used by another process: 'C:\\Users\\tobia\\AppData\\Local\\Temp\\tmp24c4bmzv\\bd76d834-672e-49fc-ac30-7751b7b660d0-01.ppm'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python310\lib\code.py", line 90, in runcode
exec(code, self.locals)
File "<input>", line 1, in <module>
File "E:\PyCharm 2022.2.3\plugins\python\helpers\pydev\_pydev_bundle\pydev_umd.py", line 198, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
File "E:\PyCharm 2022.2.3\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "D:\Dokumente\Uni\Informatik\BA_Thesis\tumexam-scheduling-codebase\generate_data.py", line 393, in <module>
extract_calibration_page_as_image_from_pdf()
File "D:\Dokumente\Uni\Informatik\BA_Thesis\tumexam-scheduling-codebase\generate_data.py", line 190, in extract_calibration_page_as_image_from_pdf
tmp_dir.cleanup()
File "C:\Python310\lib\tempfile.py", line 873, in cleanup
self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
File "C:\Python310\lib\tempfile.py", line 855, in _rmtree
_shutil.rmtree(name, onerror=onerror)
File "C:\Python310\lib\shutil.py", line 749, in rmtree
return _rmtree_unsafe(path, onerror)
File "C:\Python310\lib\shutil.py", line 619, in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
File "C:\Python310\lib\tempfile.py", line 846, in onerror
cls._rmtree(path, ignore_errors=ignore_errors)
File "C:\Python310\lib\tempfile.py", line 855, in _rmtree
_shutil.rmtree(name, onerror=onerror)
File "C:\Python310\lib\shutil.py", line 749, in rmtree
return _rmtree_unsafe(path, onerror)
File "C:\Python310\lib\shutil.py", line 600, in _rmtree_unsafe
onerror(os.scandir, path, sys.exc_info())
File "C:\Python310\lib\shutil.py", line 597, in _rmtree_unsafe
with os.scandir(path) as scandir_it:
NotADirectoryError: [WinError 267] Directory name invalid: 'C:\\Users\\tobia\\AppData\\Local\\Temp\\tmp24c4bmzv\\bd76d834-672e-49fc-ac30-7751b7b660d0-01.ppm'
When stepping through the cleanup routine, everything seems fine, the path is correct and it starts deleting files, until at some point the internal path variable gets jumbled up and the routine crashes, because obviously a file is not a directory. To me it seems like a race condition is causing problems here.
What I have already tried
Rewriting the function to not use with and instead explicitly call the routine with tmp_dir.cleanup()
Just creating the directory without populating it with the conversion artifacts. The cleanup works in this case.
The documentation for tempfile mentions Permission errors occuring when files are open. The files are however only used in this function and if this is what is causing the error, I am unsure where the files are still opened or which function is causing this. My suspicion of course would be the conversion function.
While experimenting some more and writing this question, I found a working solution:
with tempfile.TemporaryDirectory() as path:
images_from_path: [Image] = convert_from_path(
os.path.join(path_superfolder, f"calibration_target_{exam_type}.pdf"),
size=(2480, 3508),
output_folder=path, poppler_path=r'E:\poppler-22.04.0\Library\bin')
if len(images_from_path) >= page:
images_from_path[page - 1].save(os.path.join(path_superfolder, "result.jpg"))
images_from_path = []
It seems that somehow, the routine had trouble cleaning up, because the converted images, are actually the artifacts created by pdf2image and were still being held by my data structure. Resetting the data structure, before implicitly initiating the cleanup fixed the issue.
If there is a better way of tackling this issue, please do not hesitate to inform me.
I have a python script that is being run by cron. The script imports the pandas module and uses read_csv to load a csv to a data frame and then later saves it to another csv. 'apath' is the absolute path to the file:
statedata_raw=pd.read_csv(apath+'statedata.csv')
statedata_raw.to_csv(apath+'state_data.csv',index=False)
The permissions on the csv file are set correctly -rwxr-xr-x
when I run it in the command line, everything works fine. When I run it via cron I get the following error:
Traceback (most recent call last):
File "/users/maderman/wdtest.py", line 21, in <module>
statedata_raw=pd.read_csv(apath+'statedata.csv')
File "/opt/miniconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 676, in parser_f
return _read(filepath_or_buffer, kwds)
File "/opt/miniconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 448, in _read
parser = TextFileReader(fp_or_buf, **kwds)
File "/opt/miniconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 880, in __init__
self._make_engine(self.engine)
File "/opt/miniconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 1114, in _make_engine
self._engine = CParserWrapper(self.f, **self.options)
File "/opt/miniconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 1891, in __init__
self._reader = parsers.TextReader(src, **kwds)
File "pandas/_libs/parsers.pyx", line 374, in pandas._libs.parsers.TextReader.__cinit__
File "pandas/_libs/parsers.pyx", line 678, in pandas._libs.parsers.TextReader._setup_parser_source
OSError: Initializing from file failed
I verified that pandas itself is loading and that the to_csv is working by replacing the read_csv. When I replaced the read_csv with the following code to manually create a dataframe, everything worked fine, running in command line and running in cron:
cat=['a','a','a','a','a','b','b','b','b','b']
val=[1,2,3,4,5,6,7,8,9,10]
columns=['cat','val']
data=[cat,val]
dict={key:value for key,value in zip(columns,data)}
statedata_raw=pd.DataFrame(data=dict)
I found another post that suggested passing the argument engine='python' to the read_csv, but that didn't do anything.
So I know that:
cron is running python fine
it can import pandas and run a couple of different pandas functions.
the file permissions are fine
The issue seems to be specifically related to the read_csv commmand.
Any suggestions would be appreciated.
The framing on this question was wrong and it boiled down to a permissions issue. A better question was posted and answered here: stackoverflow.com/questions/62353610
Beginner here. trying to load this table via python so i can figure out how i can manipulate it and gain some insight with the eventual intention of calculating the WOE and/or running a regression.
The command ran fine on a test db of two rows i created so it must be something to do with the format of the csv im trying to use. Its a file with 8000 customers and 50 associated variables including some dates and then counts, sums and averages for 30, 60 and 90 day windows of a number of different factors. Could any of this be the reason i get the error message at the bottom?
(* are just redaction's)
data = pd.read_csv("C:\Users\******\Desktop\*******.csv")
>>> data = pd.read_csv(r"C:\Users\******\Desktop\**************")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\******\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\io\parsers.py", line 702, in parser_f
return _read(filepath_or_buffer, kwds)
File "C:\Users\******\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\io\parsers.py", line 429, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
File "C:\Users\******\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\io\parsers.py", line 895, in __init__
self._make_engine(self.engine)
File "C:\Users\******\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\io\parsers.py", line 1122, in _make_engine
self._engine = CParserWrapper(self.f, **self.options)
File "C:\Users\******\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\io\parsers.py", line 1853, in __init__
self._reader = parsers.TextReader(src, **kwds)
File "pandas/_libs/parsers.pyx", line 387, in pandas._libs.parsers.TextReader.__cinit__
File "pandas/_libs/parsers.pyx", line 705, in pandas._libs.parsers.TextReader._setup_parser_source
FileNotFoundError: [Errno 2] File b'C:\\Users\\******\\Desktop\\**************' does not exist: b'C:\\Users\\******\\Desktop\\**************'
....
add r(raw string) before ":
data = pd.read_csv(r"C:\Users******\Desktop*******.csv")
You should replace single backslash with double backslash, like so
data = pd.read_csv("C:\\Users******\\Desktop*******.csv")
or prefix path with r
data = pd.read_csv(r"C:\Users******\Desktop*******.csv")
See here for full description on which characters need escaping in python strings.
Its better to create a separate folder where keep both data and your csv file...
Then just read by only file name... Try to press tab when you are in parenthesis
because it will give you also suggestion where you will see if the file is available or not.
df = pd.read_csv('filename.csv)
I'm running the next piece of code to check the type of my file:
import subprocess as sub
output = sub.check_output(["file", "test.py"]).decode('ascii')
#output=sub.check_output(["file","C:/Users/Roger.That/PycharmProjects/test/test.py"]).decode('ascii')
and I keep getting the following error:
Traceback (most recent call last):
File "C:/Users/Roger That/PycharmProjects/test/test.py", line 2, in <module>
output = sub.check_output(["file", "test.py"]).decode('ascii')
File "C:\Users\Roger That\AppData\Local\Programs\Python\Python37-32\lib\subprocess.py", line 395, in check_output
**kwargs).stdout
File "C:\Users\Roger That\AppData\Local\Programs\Python\Python37-32\lib\subprocess.py", line 472, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Users\Roger That\AppData\Local\Programs\Python\Python37-32\lib\subprocess.py", line 775, in __init__
restore_signals, start_new_session)
File "C:\Users\Roger That\AppData\Local\Programs\Python\Python37-32\lib\subprocess.py", line 1178, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
The full path of my test.py : "C:/Users/Roger.That/PycharmProjects/test/test.py"
output from gitbash :
$ file /c/Users/Roger\ That/PycharmProjects/test/test.py
/c/Users/Roger That/PycharmProjects/test/test.py: ASCII text, with CRLF line terminators
Any chance that it is related to the fact that I have a dot between "Roger" and "That"? Although it doesn't work also when I use the relative path (only the file name)
Update
I changed my user's dir name from Roger that to Roger.that, but it still didn't help:
python /c/Users/Roger.That/PycharmProjects/test/test.py
same error..
checked also :
import subprocess as sub
output = sub.check_output(["ls" "-l"])
got same error
I created from scratch the virtual environment and it solved my problem.
I'm attempting to try a t-test in python using the pandas module. However, the same error keeps occuring in which my target file cannot be found. In this case, the target file is brain_size.csv, where the separators are semi-colons. The values which are left blank are represented by a period.
Here's what I have keyed in:
import pandas as pd
data = pd.read_csv('This PC\Desktop\brain_size.csv', sep=';', na_values='.')
Here's the error message. It's a long string
File "C:\Users\Tina Gnali\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 562, in parser_f
return _read(filepath_or_buffer, kwds)
File "C:\Users\Tina Gnali\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 315, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
File "C:\Users\Tina Gnali\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 645, in __init__
self._make_engine(self.engine)
File "C:\Users\Tina Gnali\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 799, in _make_engine
self._engine = CParserWrapper(self.f, **self.options)
File "C:\Users\Tina Gnali\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 1213, in __init__
self._reader = _parser.TextReader(src, **kwds)
File "pandas\parser.pyx", line 358, in pandas.parser.TextReader.__cinit__ (pandas\parser.c:3427)
File "pandas\parser.pyx", line 628, in pandas.parser.TextReader._setup_parser_source (pandas\parser.c:6861)
OSError: File b'This PC\\Desktop\x08rain_size.csv' does not exist
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
I want to ask:
What am I doing wrong? Why I can't retrieve the target file?
Why my error elicits such a long error message?
What does the "parser" module do?
The problem is with using backslashes "\". You must avoid that. Backslash is reserved for something called escape characters, like new line being denoted with "\n" and stuff. Either use double backslashes "\\" or just forwardslashes "/" or raw literals in your read_csv():
"C:\\Users\\blabla\\"
or
"C:/Users/blabla/"
or
r"C:\Users\blabla\"
Regarding how to identify the error, look for the "error" string in the error message. It is here:
OSError: File b'This PC\\Desktop\x08rain_size.csv' does not exist
This tells you that Python is looking for a file called 'x08rain_size.csv', and obviously you don't have such a file. But what is x08rain? Could it be that b is replaced with x08 when you place a backslash in front of it? Let's ask this to Python:
In [247]: '\b'
Out[247]: '\x08'
There we go!
Sometimes you may not be able to use
"C:\\Users\\blabla\\" or "C:/Users/blabla/"
Solution 1. The other option could be this:
Open Anaconda prompt or cmd, then change the path. Lets assume you are in drive "c" and your folder is in drive e. So after opening the cmd write "e:" and hit enter. Then the command will show you "E:\>". Now you should write "cd E:\Users\blabla\desired_folder". After running that, you should write "jupyter notebook" and run it. It will generate and open a new notebook in the same folder that has your file.
Solution 2. The other simple solution is, after opening the jupyther notebook> file> use the folder icon and choose the right folder.
May be the sep is different it may be "," try it and if still it doesnt works try removing sep and na values and try to keep the file in same directory where the program is present or give actual path