I have created a function where it will dynamically create a json file.
Currently i am using pathlib module which can be found in python 3 . However the server that i am deploying my script will only have python 2.7 and below.
The current approach im doing now work perfectly fine with pathlib module. When this was triggered in a higher environment, error im receiving is pathlib module not found. Hence i would go without using pathlib module when locating the base path to inject json content to publish_workbook.json
import json
import os
from os import listdir
from os.path import isfile, join
from pathlib import Path
def createJson():
print("Creating Json File ....... ")
..
..
event_dict = json.loads(_json)
_jsonFile = json.dumps(event_dict, indent = 4, sort_keys=True)
#Convert path to python 2.7 library orusing os.path
base = Path('/home/reporting/job/')
_saveFile = base / ('publish_workbook' + '.json')
print("Saving json file on : " + str(_saveFile))
_saveFile.write_text(_jsonFile)
createJson()
What is the best approach i can go with this ?
Install pathlib2 on the server, which is a backported version of pathlib. It's no longer supported but then neither is Python 2.7.
You'll then only have to change the import line to
from pathlib2 import Path
Just do it as strings. It's actually fewer characters to type. Pathlib has some cool concepts, but I'm not convinced it's a net win. And, of course, for Python 2 you don't have the print function.
base = '/home/reporting/job/'
_saveFile = base + 'publish_workbook.json'
print "Saving json file on : " + _saveFile
_saveFile.write_text(_jsonFile)
Related
I am trying to make it so when this runs it runs the newest version in the list.
Here is the directory I want:
ClockOS:
bootLoader.py (the file I am starting at)
Versions:
Version 0.1.1:
main.py (Ignore because tis an older version)
Version 0.1.2:
main.py (The one I want to run/import)
I have tried loading through os.dir, and using sys.path, and adding an empty __init__.py
Anyone know how to get this working?
My code:
import os
import re
import sys
versionList = []
for filename in os.listdir('Versions'):
if filename.startswith('Version'):
versionList.append(filename)
print(f"Loaded {filename}")
def major_minor_micro(version):
major, minor, micro = re.search('(\d+)\.(\d+)\.(\d+)', version).groups()
return int(major), int(minor), int(micro)
latest = str(max(versionList, key=major_minor_micro))
print("Newest Version: ", latest)
os.path.dirname('Versions/'+str(latest))
sys.path.insert(1, latest)
print(sys.path)
import main
Lastly, I need this to work on both windows and linix, if that's possible.
You need to use importlib to do what you want. Here is an example on how to import my_function from a file:
import importlib
spec = importlib.util.spec_from_file_location('some_module_name', filename)
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)
mod.my_function(1)
Probably the question is a duplicate of Import arbitrary python source file. (Python 3.3+)
I define parameters for my programs in a python module for simplicity. These parameters are then loaded using import. Therefore, I have to make sure to always load from the working directory, and nowhere else (independent form the location of the executed script or available modules in python path).
I found two solutions. First modifying the path:
import sys
from os import getcwd
import_path = sys.path
sys.path = [str(getcwd(), ]
import xxx
sys.path = import_path
or using importlib
from pathlib import Path
from importlib.util import module_from_spec, spec_from_file_location
spec = spec_from_file_location('xxx', str(Path('.').expanduser()/'xxx.py'))
xxx = module_from_spec(spec)
spec.loader.exec_module(xxx)
Of course this can be wrapped into a context manager or a function respectively.
What would the pythonic way be to do this? Do those two approaches have advantages and disadvantages?
I checked How can I import a Python library located in the current working directory?
as well as Import python package from local directory into interpreter, but they lack the focus on robustness.
Local imports can be achieved by modifying the path. A context manager is a suitable solution:
import sys
from pathlib import Path
from contextlib import contextmanager
#contextmanager
def local_import(dir_=None):
"""Only import modules within `dir_` (default: cwd)."""
if dir_ is None:
dir_ = Path.cwd()
else:
dir_ = Path(dir_).absolute().resolve(strict=True)
import_path0 = sys.path[0]
sys.path[0] = str(dir_)
try:
yield
finally:
sys.path[0] = import_path0
Then the local import can be done using the standard import syntax
with local_import():
import xxx
This solution relys on the order, in which the paths are scanned, thus we temporarily replace sys.path[0]. We replace it, instead of prepending to avoid import conflicts with the script directory.
Note
You have to be careful to avoid name conflicts, as the import statement is used, modules with identical names will be imported only once. Thus if different modules with the same name exist in the working directory and in the original sys.path[0], only one of them will be imported. Thus, local_import should only be used for scripts, that only use the standard library or installed third party libraries, but not for scripts that import other scripts from the directory. For the unlikely case that you want to import different files with the same name, the following function can be used:
import uuid
from importlib.util import module_from_spec, spec_from_file_location
def import_file(file, content=None):
"""Try importing `file` as module avoiding name clashes.
If `content` is given `content = import_file('file.py', 'content')`
roughly corresponds to `from file import content`
else `file = import_file('file.py')`
roughly corresponds to `import file`.
Parameters
----------
file : str or Path
The Python file corresponding to the module.
content : str, optional
What to import from the module (optional).
"""
file = Path(file).expanduser().resolve(strict=True)
print(file)
spec = spec_from_file_location(file.stem + str(uuid.uuid4()), str(file))
module = module_from_spec(spec)
spec.loader.exec_module(module)
if content:
print(module)
return getattr(module, content)
else:
return module
import tempfile
tmp = tempfile.NamedTemporaryFile(delete=True)
try:
# do stuff with temp
tmp.write(b'def fun():\n\tprint("hello world!")\n')
if __name__ == '__main__':
func = __import__(tmp.name)
func.fun()
finally:
tmp.close() # deletes the file
So I want to create a temporary file, add some source code to it and then import the module and call the function, but I always run into this error:
ModuleNotFoundError: No module named '/var/folders/3w/yyp887lx4018h9s5sr0bwhkw0000gn/T/tmp84bk0bic'
It doesn't seem to find the module of the temporary file. How do I solve this?
There are a few problems with your code:
Your filename does not end with .py, but Python modules are expected to. You can fix this by setting suffix='.py' in NamedTemporaryFile().
__import__() is not the right way to load a module from a full path. See here: How to import a module given the full path?
You do not flush after writing and before importing, so even if Python does find the file, it may well be empty. Add tmp.flush() after writing to fix this.
Importing can only be done from certain directories which are part of the PYTHON_PATH. You can extend that. Then you will have to use __import__() with a module name (not a path in the file system). You will have to deal with the suffix for the temp file.
I implemented a simple version using the local directory for the temp module file and a version using a proper tempfile:
#!/usr/bin/env python3
import sys
import os
import tempfile
SCRIPT = '''\
def fun():
print("hello world!")
'''
# simple version using the local directory:
with open('bla.py', 'w') as tmp_module_file:
tmp_module_file.write(SCRIPT)
import bla
bla.fun()
# version using the tempfile module:
tmpfile = tempfile.NamedTemporaryFile(suffix='.py', delete=True)
try:
tmpfile.write(SCRIPT.encode('utf8'))
tmpfile.flush()
tmpmodule_path, tmpmodule_file_name = os.path.split(tmpfile.name)
tmpmodule_name = tmpmodule_file_name[:-3] # strip off the '.py'
sys.path.append(tmpmodule_path)
tmpmodule = __import__(tmpmodule_name)
finally:
tmpfile.close()
tmpmodule.fun()
I was using python 2.6 for a script, but for requirement limitations I have to downgrade my script to python 2.5, how can I get the relative path using python 2.5?
I was previously using:
os.path.relpath(path[, start])
But since this is new from 2.6 I can't use it anymore.
Thanks and regards!
Try Relative paths in Python it should have valuable information to you.
>>> import sys
>>> import os.path
>>> sys.path[0]
'C:\\Python25\\Lib\\idlelib'
>>> os.path.relpath(sys.path[0], "path_to_libs") # if you have python 2.6
>>> os.path.join(sys.path[0], "path_to_libs")
'C:\\Python25\\Lib\\idlelib\\path_to_libs'
EDIT: Found something more http://www.saltycrane.com/blog/2010/03/ospathrelpath-source-code-python-25/
It is a re-implementation by James Gardner:
from posixpath import curdir, sep, pardir, join
def relpath(path, start=curdir):
"""Return a relative version of a path"""
if not path:
raise ValueError("no path specified")
start_list = posixpath.abspath(start).split(sep)
path_list = posixpath.abspath(path).split(sep)
# Work out how much of the filepath is shared by start and path.
i = len(posixpath.commonprefix([start_list, path_list]))
rel_list = [pardir] * (len(start_list)-i) + path_list[i:]
if not rel_list:
return curdir
return join(*rel_list)
How do I get the current file's directory path?
I tried:
>>> os.path.abspath(__file__)
'C:\\python27\\test.py'
But I want:
'C:\\python27\\'
The special variable __file__ contains the path to the current file. From that we can get the directory using either pathlib or the os.path module.
Python 3
For the directory of the script being run:
import pathlib
pathlib.Path(__file__).parent.resolve()
For the current working directory:
import pathlib
pathlib.Path().resolve()
Python 2 and 3
For the directory of the script being run:
import os
os.path.dirname(os.path.abspath(__file__))
If you mean the current working directory:
import os
os.path.abspath(os.getcwd())
Note that before and after file is two underscores, not just one.
Also note that if you are running interactively or have loaded code from something other than a file (eg: a database or online resource), __file__ may not be set since there is no notion of "current file". The above answer assumes the most common scenario of running a python script that is in a file.
References
pathlib in the python documentation.
os.path - Python 2.7, os.path - Python 3
os.getcwd - Python 2.7, os.getcwd - Python 3
what does the __file__ variable mean/do?
Using Path from pathlib is the recommended way since Python 3:
from pathlib import Path
print("File Path:", Path(__file__).absolute())
print("Directory Path:", Path().absolute()) # Directory of current working directory, not __file__
Note: If using Jupyter Notebook, __file__ doesn't return expected value, so Path().absolute() has to be used.
In Python 3.x I do:
from pathlib import Path
path = Path(__file__).parent.absolute()
Explanation:
Path(__file__) is the path to the current file.
.parent gives you the directory the file is in.
.absolute() gives you the full absolute path to it.
Using pathlib is the modern way to work with paths. If you need it as a string later for some reason, just do str(path).
Try this:
import os
dir_path = os.path.dirname(os.path.realpath(__file__))
import os
print(os.path.dirname(__file__))
I found the following commands return the full path of the parent directory of a Python 3 script.
Python 3 Script:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
from pathlib import Path
#Get the absolute path of a Python3.6 and above script.
dir1 = Path().resolve() #Make the path absolute, resolving any symlinks.
dir2 = Path().absolute() #See #RonKalian answer
dir3 = Path(__file__).parent.absolute() #See #Arminius answer
dir4 = Path(__file__).parent
print(f'dir1={dir1}\ndir2={dir2}\ndir3={dir3}\ndir4={dir4}')
REMARKS !!!!
dir1 and dir2 works only when running a script located in the current working directory, but will break in any other case.
Given that Path(__file__).is_absolute() is True, the use of the .absolute() method in dir3 appears redundant.
The shortest command that works is dir4.
Explanation links: .resolve(), .absolute(), Path(file).parent().absolute()
USEFUL PATH PROPERTIES IN PYTHON:
from pathlib import Path
#Returns the path of the current directory
mypath = Path().absolute()
print('Absolute path : {}'.format(mypath))
#if you want to go to any other file inside the subdirectories of the directory path got from above method
filePath = mypath/'data'/'fuel_econ.csv'
print('File path : {}'.format(filePath))
#To check if file present in that directory or Not
isfileExist = filePath.exists()
print('isfileExist : {}'.format(isfileExist))
#To check if the path is a directory or a File
isadirectory = filePath.is_dir()
print('isadirectory : {}'.format(isadirectory))
#To get the extension of the file
fileExtension = mypath/'data'/'fuel_econ.csv'
print('File extension : {}'.format(filePath.suffix))
OUTPUT:
ABSOLUTE PATH IS THE PATH WHERE YOUR PYTHON FILE IS PLACED
Absolute path : D:\Study\Machine Learning\Jupitor Notebook\JupytorNotebookTest2\Udacity_Scripts\Matplotlib and seaborn Part2
File path : D:\Study\Machine Learning\Jupitor Notebook\JupytorNotebookTest2\Udacity_Scripts\Matplotlib and seaborn Part2\data\fuel_econ.csv
isfileExist : True
isadirectory : False
File extension : .csv
works also if __file__ is not available (jupyter notebooks)
import sys
from pathlib import Path
path_file = Path(sys.path[0])
print(path_file)
Also uses pathlib, which is the object oriented way of handling paths in python 3.
IPython has a magic command %pwd to get the present working directory. It can be used in following way:
from IPython.terminal.embed import InteractiveShellEmbed
ip_shell = InteractiveShellEmbed()
present_working_directory = ip_shell.magic("%pwd")
On IPython Jupyter Notebook %pwd can be used directly as following:
present_working_directory = %pwd
I have made a function to use when running python under IIS in CGI in order to get the current folder:
import os
def getLocalFolder():
path=str(os.path.dirname(os.path.abspath(__file__))).split(os.sep)
return path[len(path)-1]
Python 2 and 3
You can simply also do:
from os import sep
print(__file__.rsplit(sep, 1)[0] + sep)
Which outputs something like:
C:\my_folder\sub_folder\
This can be done without a module.
def get_path():
return (__file__.replace(f"<your script name>.py", ""))
print(get_path())