When I run the predict.py file alone, it finds and reads the data.csv file. but it fails by running the predict.py file from the asd.py file in another file path;
My Files
-sbin
-master
+asd.py
-scripts
-bash
-dataset
+data.csv
+predict.py
asd.py
import os
import sys
runPath = os.path.dirname(os.path.realpath(__file__))
sys.path.append(os.path.join(runPath, "../../scripts/bash"))
from predict import pred
pred().main()
predict.py
import pandas as pd
class pred:
def main(self):
data = pd.read_csv('dataset/data.csv', sep=';')
Could the reason for this error be caused by changing the path of operation? Or I didn't get it because of another mistake.
Error:
FileNotFoundError: File b'dataset/data.csv' does not exist
Longer answer on the comment above:
This happens because although you appended the scripts folder to your sys path in order to import stuff from predict.py, whenever you call that code inside asd.py, it will run from the calling script's (asd.py) current working directory.
What that means for you is that the relative path dataset/dataset.csv does not exist in the current working directory of asd.py (sbin/master) and consequently the code will raise a FileNotFound exception.
The way to fix this and be able to run your code from anywhere, would be to give predict.py the absolute path of your dataset file.
To do that in a way that is not hardcoded, I would do what you did to get your runPath, namely get the absolute path of your predict.py file inside a varible and use os.path.join to join that to the dataset file. That way you can always be sure the dataset file will be found by whatever calling script uses the code in predict.py.
Example below:
predict.py
import pandas as pd
import os
current_dir = os.path.dirname(os.path.realpath(__file__))
class pred:
def main(self):
data_fullpath = os.path.join(current_dir, 'dataset/dataset.csv')
data = pd.read_csv(data_fullpath, sep=';')
I think you should use absolute path, instead of relative path
sys.path documentations says:
A list of strings that specifies the search path for modules
So I would not expect that relative paths starts from the path you set in asd.py
The script is trying to open dataset/data.csv starting from current folder where the script was started.
In your case I would try to pass that path somehow to the second script.
Related
This question already has answers here:
How do you properly determine the current script directory?
(16 answers)
How to know/change current directory in Python shell?
(7 answers)
Closed 5 years ago.
How do I determine:
the current directory (where I was in the shell when I ran the Python script), and
where the Python file I am executing is?
To get the full path to the directory a Python file is contained in, write this in that file:
import os
dir_path = os.path.dirname(os.path.realpath(__file__))
(Note that the incantation above won't work if you've already used os.chdir() to change your current working directory, since the value of the __file__ constant is relative to the current working directory and is not changed by an os.chdir() call.)
To get the current working directory use
import os
cwd = os.getcwd()
Documentation references for the modules, constants and functions used above:
The os and os.path modules.
The __file__ constant
os.path.realpath(path) (returns "the canonical path of the specified filename, eliminating any symbolic links encountered in the path")
os.path.dirname(path) (returns "the directory name of pathname path")
os.getcwd() (returns "a string representing the current working directory")
os.chdir(path) ("change the current working directory to path")
Current working directory: os.getcwd()
And the __file__ attribute can help you find out where the file you are executing is located. This Stack Overflow post explains everything: How do I get the path of the current executed file in Python?
You may find this useful as a reference:
import os
print("Path at terminal when executing this file")
print(os.getcwd() + "\n")
print("This file path, relative to os.getcwd()")
print(__file__ + "\n")
print("This file full path (following symlinks)")
full_path = os.path.realpath(__file__)
print(full_path + "\n")
print("This file directory and name")
path, filename = os.path.split(full_path)
print(path + ' --> ' + filename + "\n")
print("This file directory only")
print(os.path.dirname(full_path))
The pathlib module, introduced in Python 3.4 (PEP 428 — The pathlib module — object-oriented filesystem paths), makes the path-related experience much much better.
pwd
/home/skovorodkin/stack
tree
.
└── scripts
├── 1.py
└── 2.py
In order to get the current working directory, use Path.cwd():
from pathlib import Path
print(Path.cwd()) # /home/skovorodkin/stack
To get an absolute path to your script file, use the Path.resolve() method:
print(Path(__file__).resolve()) # /home/skovorodkin/stack/scripts/1.py
And to get the path of a directory where your script is located, access .parent (it is recommended to call .resolve() before .parent):
print(Path(__file__).resolve().parent) # /home/skovorodkin/stack/scripts
Remember that __file__ is not reliable in some situations: How do I get the path of the current executed file in Python?.
Please note, that Path.cwd(), Path.resolve() and other Path methods return path objects (PosixPath in my case), not strings. In Python 3.4 and 3.5 that caused some pain, because open built-in function could only work with string or bytes objects, and did not support Path objects, so you had to convert Path objects to strings or use the Path.open() method, but the latter option required you to change old code:
File scripts/2.py
from pathlib import Path
p = Path(__file__).resolve()
with p.open() as f: pass
with open(str(p)) as f: pass
with open(p) as f: pass
print('OK')
Output
python3.5 scripts/2.py
Traceback (most recent call last):
File "scripts/2.py", line 11, in <module>
with open(p) as f:
TypeError: invalid file: PosixPath('/home/skovorodkin/stack/scripts/2.py')
As you can see, open(p) does not work with Python 3.5.
PEP 519 — Adding a file system path protocol, implemented in Python 3.6, adds support of PathLike objects to the open function, so now you can pass Path objects to the open function directly:
python3.6 scripts/2.py
OK
To get the current directory full path
>>import os
>>print os.getcwd()
Output: "C :\Users\admin\myfolder"
To get the current directory folder name alone
>>import os
>>str1=os.getcwd()
>>str2=str1.split('\\')
>>n=len(str2)
>>print str2[n-1]
Output: "myfolder"
Pathlib can be used this way to get the directory containing the current script:
import pathlib
filepath = pathlib.Path(__file__).resolve().parent
If you are trying to find the current directory of the file you are currently in:
OS agnostic way:
dirname, filename = os.path.split(os.path.abspath(__file__))
If you're using Python 3.4, there is the brand new higher-level pathlib module which allows you to conveniently call pathlib.Path.cwd() to get a Path object representing your current working directory, along with many other new features.
More info on this new API can be found here.
To get the current directory full path:
os.path.realpath('.')
Answer to #1:
If you want the current directory, do this:
import os
os.getcwd()
If you want just any folder name and you have the path to that folder, do this:
def get_folder_name(folder):
'''
Returns the folder name, given a full folder path
'''
return folder.split(os.sep)[-1]
Answer to #2:
import os
print os.path.abspath(__file__)
I think the most succinct way to find just the name of your current execution context would be:
current_folder_path, current_folder_name = os.path.split(os.getcwd())
If you're searching for the location of the currently executed script, you can use sys.argv[0] to get the full path.
For question 1, use os.getcwd() # Get working directory and os.chdir(r'D:\Steam\steamapps\common') # Set working directory
I recommend using sys.argv[0] for question 2 because sys.argv is immutable and therefore always returns the current file (module object path) and not affected by os.chdir(). Also you can do like this:
import os
this_py_file = os.path.realpath(__file__)
# vvv Below comes your code vvv #
But that snippet and sys.argv[0] will not work or will work weird when compiled by PyInstaller, because magic properties are not set in __main__ level and sys.argv[0] is the way your executable was called (it means that it becomes affected by the working directory).
Having this directory structure. fileOpener.py opens the testfile.txt. And fileCallingFileOpener.py, fileCallingFileOpener2.py and fileCallingFileOpener3.py call a method from fileOpener.py that opens testfile.txt. What would be the correct path written in fileOpener.py so that it always work? Even in other computers.
\parentDirectory
\subfldr1
-fileOpener.py
-testfile.txt
\subfldr2
-fileCallingFileOpener.py
\subfldr2
-fileCallingFileOpener2.py
-fileCallingFileOpener3.py
I am asking something like:
os.path.join("..", "subfldr1", "testfile.txt")
But make it generic so it doesn't depends from where you call it.
You could try to get the location of the fileOpener module using __file__ and then make the path relative to that:
# parentDirectory/subfldr1/fileOpener.py
from pathlib import Path
def read_file():
file_path = Path(__file__).parent / "testfile.txt"
with file_path.open("r", encoding="utf-8") as file:
...
An alternative is to just use an absolute path to a file created in a temp directory, if necessary, or configurable by the end user with, for example, an environment variable.
I have a file in a package I am making that uses os.getcwd() to return the directory of the file.
For example:
# myfile1py
import os
def getfiledir():
return os.getcwd()
The above code returns C:\Users\Someone\Python36-32\Lib\site-packages\, which is the correct directory of the file. When I import it, though, an issue occurs.
# myfile2.py
import myfile
print(myfile.getfiledir())
The above code is the code in myfile2.py. After importing myfile1.py, I ran the function getfiledir() and it returns the directory of myfile2.py (C:\Users\Someone\Documents) instead of the directory of the imported file (myfile1.py).
How do I get the code in myfile2.py return the directory of myfile1.py?
Keep in mind that I want the code that returns the directory to be in the imported file (myfile1.py), not the importer file.
Any help would be greatly appreciated.
With the os module. It's on the standard library.
import os
os.path.dirname(myfile.__file__)
Further details: How to retrieve a module's path?
The following code using Path() is losing relative information:
src_file=inspect.getfile(CompileTypeData)
logger.debug(f'SRC_FILE: {src_file}')
src_path = Path(src_file).resolve()
logger.debug(f'SRC_PATH: {src_path}')
logger.debug(f'SRC_DIRNAME: {src_path.parent}')
Produces this:
DEBUG:from_project_types_file:SRC_FILE: ../../build_compile_mod/compile_type.py
DEBUG:from_project_types_file:SRC_PATH: /build_compile_mod/compile_type.py
DEBUG:from_project_types_file:SRC_DIRNAME: /build_compile_mod
What happened to my relative paths? It's my understanding that resolve() should make this an absolute path rather than losing data.
my_path.resolve() accepts an additional parameter, strict that defaults to False. If set to True, it raises a FileNotFoundError if the path cannot be resolved to an absolute path. That could be used to assess whether pathlib has enough data to create an absolute path from the string you obtained with inspect.getfile().
inspect.getfile() does not always return an absolute path, but you can always convert that path using os.path.realpath(inspect.getfile(obj)), or using Path.relative_to(...) method in combination with os.path.realpath(__file__).
Example of Relative Paths and sys.path.insert
If you put relative paths in to sys.path using
sys.path.insert and change directories while
running the script, then you'll get wrong answers
trying to use __file__ or anything derived from
it.
Say we have this:
/tmp/test1
/tmp/test2
/tmp/test2/mymod.py
/tmp/rundir/rundir2
So we CD to `/tmp/test2/ and launch the script.
% cd /tmp/test2
Now we create this script in Test2 that defines
show_path() which finds the path to mymod.py
and prints it.
Notice that sys.path.insert uses a relative path
this is the problem.
import sys
sys.path.insert(0,'../test1')
from mymod import *
import inspect
from pathlib import Path
def show_path():
myClassFile=inspect.getfile(MyClass)
print(Path(myClassFile).resolve())
So now we run our script. It starts in /tmp/test2/
it then changes directory and runs show_path().
Notice that show_path() points to nowhere because
of the relative path.
import os
os.chdir('/tmp/rundir/rundir2')
show_path()
This gives the following incorrect output.
/private/tmp/rundir/test1/mymod.py
Is there a universal approach in Python, to find out the path to the file that is currently executing?
Failing approaches
path = os.path.abspath(os.path.dirname(sys.argv[0]))
This does not work if you are running from another Python script in another directory, for example by using execfile in 2.x.
path = os.path.abspath(os.path.dirname(__file__))
I found that this doesn't work in the following cases:
py2exe doesn't have a __file__ attribute, although there is a workaround
When the code is run from IDLE using execute(), in which case there is no __file__ attribute
On Mac OS X v10.6 (Snow Leopard), I get NameError: global name '__file__' is not defined
Test case
Directory tree
C:.
| a.py
\---subdir
b.py
Content of a.py
#! /usr/bin/env python
import os, sys
print "a.py: sys.argv[0]=", sys.argv[0]
print "a.py: __file__=", __file__
print "a.py: os.getcwd()=", os.getcwd()
print
execfile("subdir/b.py")
Content of subdir/b.py
#! /usr/bin/env python
import os, sys
print "b.py: sys.argv[0]=", sys.argv[0]
print "b.py: __file__=", __file__
print "b.py: os.getcwd()=", os.getcwd()
print
Output of python a.py (on Windows)
a.py: __file__= a.py
a.py: os.getcwd()= C:\zzz
b.py: sys.argv[0]= a.py
b.py: __file__= a.py
b.py: os.getcwd()= C:\zzz
Related (but these answers are incomplete)
Find path to currently running file
Path to current file depends on how I execute the program
How can I know the path of the running script in Python?
Change directory to the directory of a Python script
First, you need to import from inspect and os
from inspect import getsourcefile
from os.path import abspath
Next, wherever you want to find the source file from you just use
abspath(getsourcefile(lambda:0))
You can't directly determine the location of the main script being executed. After all, sometimes the script didn't come from a file at all. For example, it could come from the interactive interpreter or dynamically generated code stored only in memory.
However, you can reliably determine the location of a module, since modules are always loaded from a file. If you create a module with the following code and put it in the same directory as your main script, then the main script can import the module and use that to locate itself.
some_path/module_locator.py:
def we_are_frozen():
# All of the modules are built-in to the interpreter, e.g., by py2exe
return hasattr(sys, "frozen")
def module_path():
encoding = sys.getfilesystemencoding()
if we_are_frozen():
return os.path.dirname(unicode(sys.executable, encoding))
return os.path.dirname(unicode(__file__, encoding))
some_path/main.py:
import module_locator
my_path = module_locator.module_path()
If you have several main scripts in different directories, you may need more than one copy of module_locator.
Of course, if your main script is loaded by some other tool that doesn't let you import modules that are co-located with your script, then you're out of luck. In cases like that, the information you're after simply doesn't exist anywhere in your program. Your best bet would be to file a bug with the authors of the tool.
This solution is robust even in executables:
import inspect, os.path
filename = inspect.getframeinfo(inspect.currentframe()).filename
path = os.path.dirname(os.path.abspath(filename))
I was running into a similar problem, and I think this might solve the problem:
def module_path(local_function):
''' returns the module path without the use of __file__. Requires a function defined
locally in the module.
from http://stackoverflow.com/questions/729583/getting-file-path-of-imported-module'''
return os.path.abspath(inspect.getsourcefile(local_function))
It works for regular scripts and in IDLE. All I can say is try it out for others!
My typical usage:
from toolbox import module_path
def main():
pass # Do stuff
global __modpath__
__modpath__ = module_path(main)
Now I use _modpath_ instead of _file_.
You have simply called:
path = os.path.abspath(os.path.dirname(sys.argv[0]))
instead of:
path = os.path.dirname(os.path.abspath(sys.argv[0]))
abspath() gives you the absolute path of sys.argv[0] (the filename your code is in) and dirname() returns the directory path without the filename.
The short answer is that there is no guaranteed way to get the information you want, however there are heuristics that work almost always in practice. You might look at How do I find the location of the executable in C?. It discusses the problem from a C point of view, but the proposed solutions are easily transcribed into Python.
See my answer to the question Importing modules from parent folder for related information, including why my answer doesn't use the unreliable __file__ variable. This simple solution should be cross-compatible with different operating systems as the modules os and inspect come as part of Python.
First, you need to import parts of the inspect and os modules.
from inspect import getsourcefile
from os.path import abspath
Next, use the following line anywhere else it's needed in your Python code:
abspath(getsourcefile(lambda:0))
How it works:
From the built-in module os (description below), the abspath tool is imported.
OS routines for Mac, NT, or Posix depending on what system we're on.
Then getsourcefile (description below) is imported from the built-in module inspect.
Get useful information from live Python objects.
abspath(path) returns the absolute/full version of a file path
getsourcefile(lambda:0) somehow gets the internal source file of the lambda function object, so returns '<pyshell#nn>' in the Python shell or returns the file path of the Python code currently being executed.
Using abspath on the result of getsourcefile(lambda:0) should make sure that the file path generated is the full file path of the Python file.
This explained solution was originally based on code from the answer at How do I get the path of the current executed file in Python?.
This should do the trick in a cross-platform way (so long as you're not using the interpreter or something):
import os, sys
non_symbolic=os.path.realpath(sys.argv[0])
program_filepath=os.path.join(sys.path[0], os.path.basename(non_symbolic))
sys.path[0] is the directory that your calling script is in (the first place it looks for modules to be used by that script). We can take the name of the file itself off the end of sys.argv[0] (which is what I did with os.path.basename). os.path.join just sticks them together in a cross-platform way. os.path.realpath just makes sure if we get any symbolic links with different names than the script itself that we still get the real name of the script.
I don't have a Mac; so, I haven't tested this on one. Please let me know if it works, as it seems it should. I tested this in Linux (Xubuntu) with Python 3.4. Note that many solutions for this problem don't work on Macs (since I've heard that __file__ is not present on Macs).
Note that if your script is a symbolic link, it will give you the path of the file it links to (and not the path of the symbolic link).
You can use Path from the pathlib module:
from pathlib import Path
# ...
Path(__file__)
You can use call to parent to go further in the path:
Path(__file__).parent
Simply add the following:
from sys import *
path_to_current_file = sys.argv[0]
print(path_to_current_file)
Or:
from sys import *
print(sys.argv[0])
If the code is coming from a file, you can get its full name
sys._getframe().f_code.co_filename
You can also retrieve the function name as f_code.co_name
The main idea is, somebody will run your python code, but you need to get the folder nearest the python file.
My solution is:
import os
print(os.path.dirname(os.path.abspath(__file__)))
With
os.path.dirname(os.path.abspath(__file__))
You can use it with to save photos, output files, ...etc
import os
current_file_path=os.path.dirname(os.path.realpath('__file__'))