I have a function that loads data using the current path like this: open('./filename', 'rb'). When I call it from a module located in the same package, it works, but when I import its package from a module in a different package and call it, I get an error telling me that the path './filename' does not exist. The error is raised by the call to open. What is causing this, and how do I fix this?
I'm not aware of best practices, but modules have a __file__ attribute set to a string representation of the name of the file they were loaded from. Thus, you can do this:
import os.path
# Get the directory this module is being loaded from
module_directory = os.path.dirname(__file__)
# Get the path to the file we want to open
file_path = os.path.join(module_directory, 'filename')
with open(file_path, 'rb') as f:
# do what you want with the file
Related
Having this directory structure. fileOpener.py opens the testfile.txt. And fileCallingFileOpener.py, fileCallingFileOpener2.py and fileCallingFileOpener3.py call a method from fileOpener.py that opens testfile.txt. What would be the correct path written in fileOpener.py so that it always work? Even in other computers.
\parentDirectory
\subfldr1
-fileOpener.py
-testfile.txt
\subfldr2
-fileCallingFileOpener.py
\subfldr2
-fileCallingFileOpener2.py
-fileCallingFileOpener3.py
I am asking something like:
os.path.join("..", "subfldr1", "testfile.txt")
But make it generic so it doesn't depends from where you call it.
You could try to get the location of the fileOpener module using __file__ and then make the path relative to that:
# parentDirectory/subfldr1/fileOpener.py
from pathlib import Path
def read_file():
file_path = Path(__file__).parent / "testfile.txt"
with file_path.open("r", encoding="utf-8") as file:
...
An alternative is to just use an absolute path to a file created in a temp directory, if necessary, or configurable by the end user with, for example, an environment variable.
I have a file in a package I am making that uses os.getcwd() to return the directory of the file.
For example:
# myfile1py
import os
def getfiledir():
return os.getcwd()
The above code returns C:\Users\Someone\Python36-32\Lib\site-packages\, which is the correct directory of the file. When I import it, though, an issue occurs.
# myfile2.py
import myfile
print(myfile.getfiledir())
The above code is the code in myfile2.py. After importing myfile1.py, I ran the function getfiledir() and it returns the directory of myfile2.py (C:\Users\Someone\Documents) instead of the directory of the imported file (myfile1.py).
How do I get the code in myfile2.py return the directory of myfile1.py?
Keep in mind that I want the code that returns the directory to be in the imported file (myfile1.py), not the importer file.
Any help would be greatly appreciated.
With the os module. It's on the standard library.
import os
os.path.dirname(myfile.__file__)
Further details: How to retrieve a module's path?
When I run the predict.py file alone, it finds and reads the data.csv file. but it fails by running the predict.py file from the asd.py file in another file path;
My Files
-sbin
-master
+asd.py
-scripts
-bash
-dataset
+data.csv
+predict.py
asd.py
import os
import sys
runPath = os.path.dirname(os.path.realpath(__file__))
sys.path.append(os.path.join(runPath, "../../scripts/bash"))
from predict import pred
pred().main()
predict.py
import pandas as pd
class pred:
def main(self):
data = pd.read_csv('dataset/data.csv', sep=';')
Could the reason for this error be caused by changing the path of operation? Or I didn't get it because of another mistake.
Error:
FileNotFoundError: File b'dataset/data.csv' does not exist
Longer answer on the comment above:
This happens because although you appended the scripts folder to your sys path in order to import stuff from predict.py, whenever you call that code inside asd.py, it will run from the calling script's (asd.py) current working directory.
What that means for you is that the relative path dataset/dataset.csv does not exist in the current working directory of asd.py (sbin/master) and consequently the code will raise a FileNotFound exception.
The way to fix this and be able to run your code from anywhere, would be to give predict.py the absolute path of your dataset file.
To do that in a way that is not hardcoded, I would do what you did to get your runPath, namely get the absolute path of your predict.py file inside a varible and use os.path.join to join that to the dataset file. That way you can always be sure the dataset file will be found by whatever calling script uses the code in predict.py.
Example below:
predict.py
import pandas as pd
import os
current_dir = os.path.dirname(os.path.realpath(__file__))
class pred:
def main(self):
data_fullpath = os.path.join(current_dir, 'dataset/dataset.csv')
data = pd.read_csv(data_fullpath, sep=';')
I think you should use absolute path, instead of relative path
sys.path documentations says:
A list of strings that specifies the search path for modules
So I would not expect that relative paths starts from the path you set in asd.py
The script is trying to open dataset/data.csv starting from current folder where the script was started.
In your case I would try to pass that path somehow to the second script.
Let us have a look at the Directory structure of my Python Package
packman
weights.py
functions:
weigh()
force()
relatives.py
functions:
roll()
torque()
__init__.py
data
work.txt
rastor.txt
Now I have two questions.
Firstly suppose I want to access work.txt, from the function weigh() inside weights.py how would I address it?
I initially tried with this
f = open("data/work.txt")
While this method does succesfully work when the code is run inside main. However it fails to find the file when it is used as a package and it raises the issue
FileNotFoundError: [Errno 2] No such file or directory: 'data/work.txt'
How should I write the address of work.txt to make it more universal?
My other question is when I want to call the function weigh() of weights.py from the function roll() inside relatives.py, how would I do it?
I usually have a main.py or similar single entry point for my applications. Then I can do something like this to get the path for my application:
import os
app_location = os.path.dirname(os.path.abspath(__file__))
Now you can pass this location to your other modules or perhaps even use the same idea in them to get their location. Since you now have this location, you can easily do something like this:
data_location = os.path.join(app_location, 'data', 'work.txt')
with open(data_location) as f:
# do something with the file object
for line in f:
print(line)
As for your second question, just import weights into your relatives.py script and call weights.weigh()
I am writing my first python package which I want to upload on PyPI. I structured my code based on this blog post.
I want to store user setting in a config.ini file. Read it once(every time the package is run) in separate python module in same package and save user setting in global variables of that module. Later import those in other modules.
To recreate the error I just edited few lines of code, in the template described in the blog post. (Please refer to it since it would take too much typing to recreate entire thing here in question.)
The only difference is that my stuff.py reads from config file like this:
from ConfigParser import SafeConfigParser
config = SafeConfigParser()
config.read('config.ini')
TEST_KEY = config.get('main', 'test_key')
Here are the contents of config.ini (placed in same dir as stuff.py):
[main]
test_key=value
And my bootstrap.py just imports and print the TEST_KEY
from .stuff import TEST_KEY
def main():
print(TEST_KEY)
But on executing the package, the import fails give this error
Traceback (most recent call last):
File "D:\Coding\bootstrap\bootstrap-runner.py", line 8, in <module>
from bootstrap.bootstrap import main
File "D:\Coding\bootstrap\bootstrap\bootstrap.py", line 11, in <module>
from .stuff import TEST_KEY
File "D:\Coding\bootstrap\bootstrap\stuff.py", line 14, in <module>
TEST_KEY = config.get('main', 'test_key')
File "C:\Python27\Lib\ConfigParser.py", line 607, in get
raise NoSectionError(section)
ConfigParser.NoSectionError: No section: 'main'
Import keeps giving ConfigParser.NoSectionError, but if you build/run only stuff.py(I use sublime3), the module gives no errors and printing TEST_KEY gives value as output.
Also, this method of import does work when I just use 3 files(config, stuff, main) in a dir and just execute the main as a script. But there I had to import it like this
from stuff import TEST_KEY
I'm just using the explicit relative imports as described in that post but don't have enough understanding of those. I guess the error is due to project structure and import, since running stuff.py as standalone script raises no ConfigParser.NoSectionError.
Other method to read the config file once and then use data in other modules will be really helpful as well.
There are two aspects to this question. First is the weird behavior of ConfigParser. When ConfigParser is unable to locate the .ini file; it never gives, for some annoying reason, an IOError or an error which indicates that it is unable to read the file.
In my case it keeps giving ConfigParser.NoSectionError when the section is clearly present. When I caught the ConfigParser.NoSectionError error it gave an ImportError! But it never tells you that it is simply unable to read the file.
Second is how to safely read the data files that are included in your package. The only way I found to do this was to use the __file__ parameter. This is how you would safely read the config.ini in the above question, for Python27 and Python3:
import os
try:
# >3.2
from configparser import ConfigParser
except ImportError:
# python27
# Refer to the older SafeConfigParser as ConfigParser
from ConfigParser import SafeConfigParser as ConfigParser
config = ConfigParser()
# get the path to config.ini
config_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'config.ini')
# check if the path is to a valid file
if not os.path.isfile(config_path):
raise BadConfigError # not a standard python exception
config.read(config_path)
TEST_KEY = config.get('main', 'test_key') # value
This relies on the fact that config.ini is located inside our package bootstrap and is expected to be shipped with it.
The important bit is how you get the config_path:
config_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'config.ini')
__file__ refers to the location of current script that is being executed. In my question that means the location of stuff.py, that is inside the bootstrap folder, and so is config.ini.
The above line of code then means; get the absolute path to stuff.py; from that get path to the directory containing it; and join that with config.ini (since it is in same directory) to give absolute path to the config.ini. Then you can proceed to read it and raise an exception just in case.
This will work even when you release your package on pip and a user installs it from there.
As a bonus, and digressing from the question a bit, if you are releasing your package on pip with data files inside your package, you must tell setuptools to include them inside you package when you build sdist and bdists. So to include the config.ini in above package add the following lines to setup class call in setup.py:
include_package_data = True,
package_data = {
# If any package contains *.ini files, include them
'': ['*.ini'],
},
But it still may not work in some cases eg. building wheels etc. So you also do the same in your MANIFEST.IN file:
include LICENSE
include bootstrap/*.ini
abhimanyuPathania : The issue is with path of config.ini in stuff.py. Change config.read('config.ini') to config.read('./bootstrap/config.ini') in stuff.py. I tried the solution. It works for me.
Enjoying Pythoning...