Develop Reference

python yaml path after deployment

So this is a question about how to handle settings files and relative paths in python (probably also something about best practice).
So I have coded a smaller project that i want to deploy to a docker image and everything is set up now except when I try to run the python task (Through cron) I get the error: settings/settings.yml not found.
tree .
├───settings
│ └───settings/settings.yml
└───main.py
And am referencing the yml file as
open('settings/settings.yml', 'r') as f:
config = yaml.load(f, Loader=yaml.FullLoader)
I can see this is what is causing the problem but am unsure about how to fix it. I wish to reference the main file basically by using the entry_points from setuptools in the future so my quick fix with cd'ing before python main.py will not be a lasting solution.

Instead of hardcoding a path as a string, you can find the directories and build the file path with os.path. For example:
import os
import yaml
current_dir = os.path.dirname(os.path.abspath(__file__))
settings_dir = os.path.join(current_dir, "settings")
filename = "settings.yml"
settings_path = os.path.join(settings_dir, filename)
with open(settings_path, "r") as infile:
settings_data = yaml.load(infile)
This way it can be run in any file system and the python file can be called from any directory.

issues while reading a file in python

I'm struggling with reading a file in python, the py file and CSV file are in the same folder but the VSCode makes an error and can't find the file:
import csv
with open('file.csv','r') as f:
reader = reader(f)
...
how can I fix this??
and the error is:
Exception has occurred: FileNotFoundError
[Errno 2] No such file or directory: 'file.csv'

If you run:
import os
os.getcwd()
You'll find out your current working directory which I assume is not the one you were expecting. If you're running the python script through VS code it could be using it could be the directory which you have open on the left hand side.
So either run the python using the correct working directory or use an absolute path like this:
import csv
with open('pathname/file.csv','r') as f:
reader = reader(f)

There might be an issue with your relative path settings.
Try this:
import os
import csv
dir = os.path.dirname(__file__)
filename = os.path.join(dir, 'file.csv')
with open(filename,'r') as f:
reader = reader(f)

Are you using spyder?
If so, please check if the current working path is the path your py file locates.

import csv
with open('file.csv','r') as f:
reader = csv.reader(f)
in this case your file.csv should be in folder where is your python script (current working folder)
or, instead of 'file.csv' you can put absolute path

Creating a submodule that relies on a properties file

I have the following structure
main.py
module/
properties.yaml
file.py
file.py relevant code:
def read_properties():
with open('properties.yaml') as file:
properties = yaml.load(file)
main.py relevant code:
from module import file
file.read_properties()
When read_properties() is called within main.py, I get the following error: FileNotFoundError: [Errno 2] No such file or directory: 'properties.yaml'
What is the recommended way of allowing my module to access the properties file even when imported?

Provide the absolute path to properties.yaml:
with open('/Users/You/Some/Path/properties.yaml') as file:

As JacobIRR said in his answer, it is best to use the absolute path to the file. I use the os module to construct the absolute path based on the current working directory. So for your code it would be something like:
import os
working_directory = os.path.dirname(__file__)
properties_file = os.path.join(working_directory, 'module', 'properties.yaml')

Based on answers from #JacobIRR and #BigGerman
I ended up using pathlib instead of os, but the logic is the same.
Here is the syntax with pathlib for those interested:
in file.py:
from pathlib import Path
properties_file = Path(__file__).resolve().parent/"properties.yaml"
with open(properties_file) as file:
properties = yaml.load(file)

python; reading file path error

i have a directory structure;
DIR1:
----outerPyFile.py
----DIR2:
--------innerPyFile.py
--------DIR3:
------------fileToRead.csv
I'm reading fileToRead.csv in innerPyFile: pd.read_csv('DIR3/fileToRead.csv')
works fine if i run innerPyFile.py individually
Now when import innerPyFile module inside outerPyFile.py as
import innerPyFile
-- FileNotFoundError: DIR3\\fileToRead.csv. does not exist
i tried replacing path with absolute path in innerPyFile as pd.read_csv(os.path.abspath('DIR3/fileToRead.csv'))
still, when i run outerPyFile i get,
FileNotFoundError C:\\\DIR1\\\DIR3\\\fileToRead.csv does not exist,
here the code omitted DIR2 so i changed code as pd.read_csv(os.path.abspath('DIR2/DIR3/fileToRead.csv'))
Now the code structure works file when i run outerPyFile.py which is acceptable.
but here the problem will arise when i run innerPyFile individually because it will search for DIR2 which is not there in CWD of innerPyFile.
anyone can suspect this behavior,
please revert me what is going on?
FYI, I've also tried pathLib module which didn't solve the issue.

Try this:
innerPyFile.py
import os
script_path = os.path.abspath(__file__) # i.e. /path/to/dir/foobar.py
script_dir = os.path.split(script_path)[0] #i.e. /path/to/dir/
rel_path = "DIR3/fileToRead.csv"
abs_file_path = os.path.join(script_dir, rel_path)
pd.read_csv(abs_file_path)
outerPyFile.py
import DIR2.innerPyFile
#......do something.....

Relative file paths in python [duplicate]

Say I have a Python project that is structured as follows:
project
/data
test.csv
/package
__init__.py
module.py
main.py
__init__.py:
from .module import test
module.py:
import csv
with open("..data/test.csv") as f:
test = [line for line in csv.reader(f)]
main.py:
import package
print(package.test)
When I run main.py I get the following error:
C:\Users\Patrick\Desktop\project>python main.py
Traceback (most recent call last):
File "main.py", line 1, in <module>
import package
File "C:\Users\Patrick\Desktop\project\package\__init__.py", line 1, in <module>
from .module import test
File "C:\Users\Patrick\Desktop\project\package\module.py", line 3, in <module>
with open("../data/test.csv") as f:
FileNotFoundError: [Errno 2] No such file or directory: '../data/test.csv'
However, if I run module.py from the package directory, I don’t get any errors. So it seems that the relative path used in open(...) is only relative to where the originating file is being run from (i.e __name__ == "__main__")? I don't want to use absolute paths. What are some ways to deal with this?

Relative paths are relative to current working directory.
If you do not want your path to be relative, it must be absolute.
But there is an often used trick to build an absolute path from current script: use its __file__ special attribute:
from pathlib import Path
path = Path(__file__).parent / "../data/test.csv"
with path.open() as f:
test = list(csv.reader(f))
This requires python 3.4+ (for the pathlib module).
If you still need to support older versions, you can get the same result with:
import csv
import os.path
my_path = os.path.abspath(os.path.dirname(__file__))
path = os.path.join(my_path, "../data/test.csv")
with open(path) as f:
test = list(csv.reader(f))
[2020 edit: python3.4+ should now be the norm, so I moved the pathlib version inspired by jpyams' comment first]

For Python 3.4+:
import csv
from pathlib import Path
base_path = Path(__file__).parent
file_path = (base_path / "../data/test.csv").resolve()
with open(file_path) as f:
test = [line for line in csv.reader(f)]

This worked for me.
with open('data/test.csv') as f:

My Python version is Python 3.5.2 and the solution proposed in the accepted answer didn't work for me. I've still were given an error
FileNotFoundError: [Errno 2] No such file or directory
when I was running my_script.py from the terminal. Although it worked fine when I run it through Run/Debug Configurations from the PyCharm IDE (PyCharm 2018.3.2 (Community Edition)).
Solution:
instead of using:
my_path = os.path.abspath(os.path.dirname(__file__)) + some_rel_dir_path
as suggested in the accepted answer, I used:
my_path = os.path.abspath(os.path.dirname(os.path.abspath(__file__))) + some_rel_dir_path
Explanation:
Changing os.path.dirname(__file__) to os.path.dirname(os.path.abspath(__file__))
solves the following problem:
When we run our script like that: python3 my_script.py
the __file__ variable has a just a string value of "my_script.py" without path leading to that particular script. That is why method dirname(__file__) returns an empty string "". That is also the reason why my_path = os.path.abspath(os.path.dirname(__file__)) + some_rel_dir_path is actually the same thing as my_path = some_rel_dir_path. Consequently FileNotFoundError: [Errno 2] No such file or directory is given when trying to use open method because there is no directory like "some_rel_dir_path".
Running script from PyCharm IDE Running/Debug Configurations worked because it runs a command python3 /full/path/to/my_script.py (where "/full/path/to" is specified by us in "Working directory" variable in Run/Debug Configurations) instead of justpython3 my_script.py like it is done when we run it from the terminal.

Try
with open(f"{os.path.dirname(sys.argv[0])}/data/test.csv", newline='') as f:

I was surprised when the following code worked.
import os
for file in os.listdir("../FutureBookList"):
if file.endswith(".adoc"):
filename, file_extension = os.path.splitext(file)
print(filename)
print(file_extension)
continue
else:
continue
So, I checked the documentation and it says:
Changed in version 3.6: Accepts a path-like object.
path-like object:
An object representing a file system path. A path-like object is
either a str or...
I did a little more digging and the following also works:
with open("../FutureBookList/file.txt") as file:
data = file.read()