Develop Reference

python not able to find the certificate file path

My program structure is of the form:
-worker
-worker
--lib
----file.py
----cert.pem
-app.py
Inside file.py I have following code:
import os
BASE_DIR = os.path.dirname(__file__)
filepath = os.path.join(BASE_DIR, 'cert.pem')
When I run the service which is through app.py, it fails at file.py line L2 stating:
OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /usr/local/lib/python3.7/dist-packages/worker/lib/../lib/cert.pem
Could someone have any idea why python is taking "lib" directory twice and putting ".." in between?

pathlib is usually best to handle paths.
import pathlib
BASE_DIR = pathlib.Path(__file__).parent.absolute()
filepath = BASE_DIR / "cert.pem"
print(filepath)
# >>> /Users/felipe/Desktop/example_project/cert.pem
Note the example.py (above) is also inside example_project folder.

AS posted by #Selcuk: This worked for me.:
BASE_DIR = os.path.realpath(os.path.dirname(__file__)) – Selcuk

Use / for path on Windows, and have stdlib functions use it as well (glob, os.path, etc.)

How to force Python to use / as path separator (for glob, os.path, etc.), even on Windows?
I tried os.sep = '/' but it does not work:
import os, glob
os.sep = '/'
os.path.sep = '/'
for f in glob.glob('D:/Temp/**/*', recursive=True):
print(f)
# D:/Temp\New folder
# D:/Temp\New Text Document.txt
print(os.path.join('D:/', 'Temp', 'hello'))
# D:/Temp\hello
I'd like to avoid "hacks" like having to add .replace('\\', '/') for each path, so linked questions like Why not os.path.join use os.path.sep or os.sep? don't answer it.
I also tried:
import posixpath as path
print(path.normpath(path.join('D:\\Temp', 'hello')))
# D:\Temp/hello # path.join has correctly use forward slashes
but here \isn't replaced by /.

How to move down to a parent directory in Python?

The following code in Python gives me the current path.
import os
DIR = os.path.dirname(os.path.dirname(__file__))
How can I now use the variable DIR to go down one more directory? I don't want to change the value of DIR as it is used elsewhere.
I have tried this:
DIR + "../path/"
But it does not seems to work.

Call one more dirname:
os.path.join(os.path.dirname(DIR), 'path')

Try:
import os.path
print(os.path.abspath(os.path.join(DIR, os.pardir)))

When you join a path via '+' you have to add a 'r':
path = r'C:/home/' + r'user/dekstop'
or write double backslashes:
path = 'C://home//' + 'user//dekstop'
Anyway you should never use that!
That's the best way:
import os
path = os.path.join('C:/home/', 'user/dekstop')

Platform independent path concatenation using "/" , "\"?

In python I have variables base_dir and filename. I would like to concatenate them to obtain fullpath. But under windows I should use \ and for POSIX / .
fullpath = "%s/%s" % ( base_dir, filename ) # for Linux
How can I make this platform independent?

You want to use os.path.join() for this.
The strength of using this rather than string concatenation etc is that it is aware of the various OS specific issues, such as path separators. Examples:
import os
Under Windows 7:
base_dir = r'c:\bla\bing'
filename = r'data.txt'
os.path.join(base_dir, filename)
'c:\\bla\\bing\\data.txt'
Under Linux:
base_dir = '/bla/bing'
filename = 'data.txt'
os.path.join(base_dir, filename)
'/bla/bing/data.txt'
The os module contains many useful methods for directory, path manipulation and finding out OS specific information, such as the separator used in paths via os.sep

Use os.path.join():
import os
fullpath = os.path.join(base_dir, filename)
The os.path module contains all of the methods you should need for platform independent path manipulation, but in case you need to know what the path separator is on the current platform you can use os.sep.

Digging up an old question here, but on Python 3.4+ you can use pathlib operators:
from pathlib import Path
# evaluates to ./src/cool-code/coolest-code.py on Mac
concatenated_path = Path("./src") / "cool-code\\coolest-code.py"
It's potentially more readable than os.path.join() if you are fortunate enough to be running a recent version of Python. But, you also tradeoff compatibility with older versions of Python if you have to run your code in, say, a rigid or legacy environment.

import os
path = os.path.join("foo", "bar")
path = os.path.join("foo", "bar", "alice", "bob") # More than 2 params allowed.

I've made a helper class for this:
import os
class u(str):
"""
Class to deal with urls concat.
"""
def __init__(self, url):
self.url = str(url)
def __add__(self, other):
if isinstance(other, u):
return u(os.path.join(self.url, other.url))
else:
return u(os.path.join(self.url, other))
def __unicode__(self):
return self.url
def __repr__(self):
return self.url
The usage is:
a = u("http://some/path")
b = a + "and/some/another/path" # http://some/path/and/some/another/path

Thanks for this. For anyone else who sees this using fbs or pyinstaller and frozen apps.
I can use the below which works perfect now.
target_db = os.path.join(os.path.abspath(os.path.dirname(sys.argv[0])), "sqlite_example.db")
I was doing this fugliness before which was obviously not ideal.
if platform == 'Windows':
target_db = (os.path.abspath(os.path.dirname(sys.argv[0])) + "\\" + "sqlite_example.db")
if platform == 'Linux' or 'MAC':
target_db = (os.path.abspath(os.path.dirname(sys.argv[0])) + "/" + "sqlite_example.db")
target_db_path = target_db
print(target_db_path)

Build the full path filename in Python

I need to pass a file path name to a module. How do I build the file path from a directory name, base filename, and a file format string?
The directory may or may not exist at the time of call.
For example:
dir_name='/home/me/dev/my_reports'
base_filename='daily_report'
format = 'pdf'
I need to create a string '/home/me/dev/my_reports/daily_report.pdf'
Concatenating the pieces manually doesn't seem to be a good way. I tried os.path.join:
join(dir_name,base_filename,format)
but it gives
/home/me/dev/my_reports/daily_report/pdf

This works fine:
os.path.join(dir_name, base_filename + '.' + filename_suffix)
Keep in mind that os.path.join() exists only because different operating systems use different path separator characters. It smooths over that difference so cross-platform code doesn't have to be cluttered with special cases for each OS. There is no need to do this for file name "extensions" (see footnote) because they are always preceded by a dot character, on every OS.
If using a function anyway makes you feel better (and you like needlessly complicating your code), you can do this:
os.path.join(dir_name, '.'.join((base_filename, filename_suffix)))
If you prefer to keep your code clean, simply include the dot in the suffix:
suffix = '.pdf'
os.path.join(dir_name, base_filename + suffix)
That approach also happens to be compatible with the suffix conventions in pathlib, which was introduced in python 3.4 a few years after this question was asked. New code that doesn't require backward compatibility can do this:
suffix = '.pdf'
pathlib.PurePath(dir_name, base_filename + suffix)
You might be tempted to use the shorter Path() instead of PurePath() if you're only handling paths for the local OS. I would question that choice, given the cross-platform issues behind the original question.
Warning: Do not use pathlib's with_suffix() for this purpose. That method will corrupt base_filename if it ever contains a dot.
Footnote: Outside of Microsoft operating systems, there is no such thing as a file name "extension". Its presence on Windows comes from MS-DOS and FAT, which borrowed it from CP/M, which has been dead for decades. That dot-plus-three-letters that many of us are accustomed to seeing is just part of the file name on every other modern OS, where it has no built-in meaning.

If you are fortunate enough to be running Python 3.4+, you can use pathlib:
>>> from pathlib import Path
>>> dirname = '/home/reports'
>>> filename = 'daily'
>>> suffix = '.pdf'
>>> Path(dirname, filename).with_suffix(suffix)
PosixPath('/home/reports/daily.pdf')

Um, why not just:
>>> import os
>>> os.path.join(dir_name, base_filename + "." + format)
'/home/me/dev/my_reports/daily_report.pdf'

Is not it better to add the format in the base filename?
dir_name='/home/me/dev/my_reports/'
base_filename='daily_report.pdf'
os.path.join(dir_name, base_filename)

Just use os.path.join to join your path with the filename and extension. Use sys.argv to access arguments passed to the script when executing it:
#!/usr/bin/env python3
# coding: utf-8
# import netCDF4 as nc
import numpy as np
import numpy.ma as ma
import csv as csv
import os.path
import sys
basedir = '/data/reu_data/soil_moisture/'
suffix = 'nc'
def read_fid(filename):
fid = nc.MFDataset(filename,'r')
fid.close()
return fid
def read_var(file, varname):
fid = nc.Dataset(file, 'r')
out = fid.variables[varname][:]
fid.close()
return out
if __name__ == '__main__':
if len(sys.argv) < 2:
print('Please specify a year')
else:
filename = os.path.join(basedir, '.'.join((sys.argv[1], suffix)))
time = read_var(ncf, 'time')
lat = read_var(ncf, 'lat')
lon = read_var(ncf, 'lon')
soil = read_var(ncf, 'soilw')
Simply run the script like:
# on windows-based systems
python script.py year
# on unix-based systems
./script.py year

from pathlib import Path
# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent
TEMPLATE_PATH = Path.joinpath(BASE_DIR,"templates")
print(TEMPLATE_PATH)

Adding code below for better understanding:
import os
def createfile(name, location, extension):
print(name, extension, location)
#starting creating a file with some dummy contents
path = os.path.join(location, name + '.' + extension)
f = open(path, "a")
f.write("Your contents!! or whatever you want to put inside this file.")
f.close()
print("File creation is successful!!")
def readfile(name, location, extension):
#open and read the file after the appending:
path = os.path.join(location, name + '.' + extension)
f = open(path, "r")
print(f.read())
#pass the parameters here
createfile('test','./','txt')
readfile('test','./','txt')