I'm planning to have project with following structure:
./runme.py
./my_modules/__init__.py
./my_modules/global_imports.py
./my_modules/user_defined_functions.py
Idea is to store important variables in global_imports.py from where they will be imported into runme.py using from my_modules.global_imports import * (I know it is a bad practice import modules this way, but I promise there will be just few variables with not colliding names)
Four questions:
Two of the variables contained inside global_imports.py should be SCRIPT_PATH and SCRIPT_DIR. I've tried SCRIPT_PATH = os.path.realpath(__file__) and SCRIPT_DIR = os.path.dirname(SCRIPT_PATH) but it returns path (directory) for global_imports.py not for runme.py. How can I get path (directory) of runme.py?
Inside global_imports.py I will probably import modules such as os and sys. I also need to import those modules inside runme.py. Is this considered as problem, when modules are imported first from another module and later from main script or vice versa?
Is it possible to import variables from global_imports.py into user_defined_functions.py? I consider this as bad practice I'm just curious.
Is there better approach to separate project into modules?
Addressing your questions in order:
In the first variable SCRIPT_DIR you are getting the full path of the file global_imports.py, which would be something like this:
SCRIPT_PATH = '/home/../../my_project/my_modules/global_imports.py'
now in order to get the directory of runme.py, we should consider another variable:
SCRIPT_PATH_DIR = os.path.dirname(SCRIPT_PATH)
this will give us the path
SCRIPT_PATH_DIR = '/home/../../my_project/my_modules/'
now to get to its parent directory which contains runme.py we can get like this:
SCRIPT_DIR = os.path.abspath(os.path.join(SCRIPT_PATH_DIR, os.pardir))
Now, SCRIPT_DIR gives the path of runme.py i.e.:
SCRIPT_DIR = '/home/../../my_project/'
As per our project structure, runme.py should only conrtain an import to the main module and then command to run the app. So, it shouldn't contain any other imports. Still if you need to use a module then explicitly import it in the runme.py as one of the Zen of Python says 'Explicit is better than implicit'
Yes, it is possible, you can do it like this:
from .global_imports import variable_name
but in general you should have a separate config.py or settings.py file in '/../my_project/' directory which should contain all the settings and variables which you may need to use anywhere in the project.
This approach is good enough as far as I've seen. Your main project directory contains runme.py and other modules which are used inside the project. 'my_modules' is one of the modules I think. You can have more of such modules inside the project directory. A better approach is to have settings and configurations inside one of the modules(such as my_modules) only and to have other modules for the functionality.
Hope this helps, please comment if something is unclear.
Related
I'm currently trying to import another .py file that's within a different directory, but am having some problems when it comes to doing so. I've tried various methods, and am currently attempting the method of adding the directory to the sys.path list. If I print this list once adding the directory, either absolute or relative, it appears in this list. However, my IDE (VSCode) still flags up saying the file cannot be found.
I'm aware that if I use VSCode's feature to add this to their 'extraPaths' list, it then works. However, I'm trying to avoid a dependency on my VS settings, thus avoid using this feature where possible.
Here is my file:
import sys
sys.path.append(r'c:\Users\Kiana\Documents\MyStuff\Home\Python\Pocket-Pet\game_scripts')
import pet
for p in sys.path:
print( p )
The output of the print statements (when 'import pet' is commented out):
My project structure:
I have tried both 'insert' and 'append' methods, as well as absolute and relative paths. Any ideas? Cheers
have you tried to use os.path.join('C:/', 'code', 'my-library') ? I'm not a windows user so I can't test it by myself
I use to import classes of programs relative to my script path like this
home_dir = os.path.dirname(os.path.realpath(__file__))
sys.path.append(os.path.join(home_dir, "/../"))
from folder.program import MyClass
I'm creating a package with the following structure
/package
__init__.py
/sub_package_1
__init__.py
other_stuff.py
/sub_package_2
__init__.py
calc_stuff.py
/results_dir
I want to ensure that calc_stuff.py will save results to /results_dir, unless otherwise specified (yes, I'm not entirely certain having a results directory in my package is the best idea, but it should work well for now). However, since I don't know from where, or on which machine calc_stuff will be run, I need the package, or at least my_calc.py, to know where it is saved.
So far the two approaches I have tried:
from os import path
saved_dir = path.join(path.dirname(__file__), 'results_dir')
and
from pkg_resources import resource_filename
filepath = resource_filename(__name__, 'results_dir')
have only given me paths relative to the root of the package.
What do I need to do to ensure a statement along the lines of:
pickle.dump(my_data,open(os.path.join(full_path,
'results_dir',
'results.pkl'), 'wb')
Will result in a pickle file being saved into results_dir ?
I'm not entirely certain having a results directory in my package is the best idea, me either :)
But, if you were to put a function like the following inside a module in subpackage2, it should return a path consisting of (module path minus filename, 'results_dir', the filename you passed the function as an argument):
def get_save_path(filename):
import os
return os.path.join(os.path.dirname(__file__), "results_dir", filename)
C:\Users\me\workspaces\workspace-oxygen\test36\TestPackage\results_dir\foo.ext
I am trying to import one python script from another. I have a few common functions defined in one script and then lots of other scripts that want to import those functions. No classes, just functions.
The importing script needs to import from a relative path e.g. ../../SharedScripts/python/common.py
I then a have a few functions def f1(...) defined which I will call.
I found the imp module which seemed to be the right thing to use but I was unable to figure out the exact syntax that would work for my example.
Can someone suggest the correct code to use or the simplest approach if imp is not the right module?
SOLUTION from the answers below I was able to get this working...
projectKey = 'THOR'
# load the shared script relative to this script
sys.path.append(os.path.dirname(__file__) + '/../../SharedScripts/python')
import jira
jira.CheckJiraCommitMessage(sys.argv[1], sys.argv[2], projectKey)
Where I had an empty __init__.py and a jira.py in the SharedScripts/python directory with plain function definitions.
Why not adding ../../SharedScripts/python/ to the python path? Then you could use common.py like any other module:
import common
common.f1()
You can alternate the Python path through the system variable PYTHONPATH or by manipulating it directly from python: sys.path.append("../../SharedScripts/python/")
Please notice that it is probably wiser to work with absolute pathes... (The current directory of the app could change)
To get the absolute path could can call use the function os.path.abspath: os.path.abspath('../../SharedScripts/python/')
A possible way is to add the directory to the Python path before doing the import.
#!/usr/bin/env python
import sys
sys.path.append('../../SharedScripts/python')
import common
I have a Python project with the following directory structure:
project/
project/src/
project/src/somecode.py
project/src/mypackage/mymodule.py
project/src/resources/
project/src/resources/datafile1.txt
In mymodule.py, I have a class (lets call it "MyClass") which needs to load datafile1.txt. This sort of works when I do:
open ("../resources/datafile1.txt")
Assuming the code that creates the MyClass instance created is run from somecode.py.
The gotcha however is that I have unit tests for mymodule.py which are defined in that file, and if I leave the relative pathname as described above, the unittest code blows up as now the code is being run from project/src/mypackage instead of project/src and the relative filepath doesn't resolve correctly.
Any suggestions for a best practice type approach to resolve this problem? If I move my testcases into project/src that clutters the main source folder with testcases.
I usually use this to get a relative path from my module. Never tried in a unittest tho.
import os
print(os.path.join(os.path.dirname(__file__),
'..',
'resources'
'datafile1.txt'))
Note: The .. tricks works pretty well, but if you change your directory structure you would need to update that part.
On top of the above answers, I'd like to add some Python 3 tricks to make your tests cleaner.
With the help of the pathlib library, you can explicit your ressources import in your tests. It even handles the separators difference between Unix (/) and Windows ().
Let's say we have a folder structure like this :
`-- tests
|-- test_1.py <-- You are here !
|-- test_2.py
`-- images
|-- fernando1.jpg <-- You want to import this image !
`-- fernando2.jpg
You are in the test_1.py file, and you want to import fernando1.jpg. With the help to the pathlib library, you can read your test resource with an object oriented logic as follows :
from pathlib import Path
current_path = Path(os.path.dirname(os.path.realpath(__file__)))
image_path = current_path / "images" / "fernando1.jpg"
with image_path.open(mode='rb') as image :
# do what you want with your image object
But there's actually convenience methods to make your code more explicit than mode='rb', as :
image_path.read_bytes() # Which reads bytes of an object
text_file_path.read_text() # Which returns you text file content as a string
And there you go !
in each directory that contains Python scripts, put a Python module that knows the path to the root of the hierarchy. It can define a single global variable with the relative path. Import this module in each script. Python searches the current directory first so it will always use the version of the module in the current directory, which will have the relative path to the root of the current directory. Then use this to find your other files. For example:
# rootpath.py
rootpath = "../../../"
# in your scripts
from rootpath import rootpath
datapath = os.path.join(rootpath, "src/resources/datafile1.txt")
If you don't want to put additional modules in each directory, you could use this approach:
Put a sentinel file in the top level of the directory structure, e.g. thisisthetop.txt. Have your Python script move up the directory hierarchy until it finds this file. Write all your pathnames relative to that directory.
Possibly some file you already have in the project directory can be used for this purpose (e.g. keep moving up until you find a src directory), or you can name the project directory in such a way to make it apparent.
You can access files in a package using importlib.resources (mind Python version compatibility of the individual functions, there are backports available as importlib_resources), as described here. Thus, if you put your resources folder into your mypackage, like
project/src/mypackage/__init__.py
project/src/mypackage/mymodule.py
project/src/mypackage/resources/
project/src/mypackage/resources/datafile1.txt
you can access your resource file in code without having to rely on inferring file locations of your scripts:
import importlib.resources
file_path = importlib.resources.files('mypackage').joinpath('resources/datafile1.txt')
with open(file_path) as f:
do_something_with(f)
Note, if you distribute your package, don't forget to include the resources/ folder when creating the package.
The filepath will be relative to the script that you initially invoked. I would suggest that you pass the relative path in as an argument to MyClass. This way, you can have different paths depending on which script is invoking MyClass.
I want to dynamically import a list of modules. I'm having a problem doing this. Python always yells out an ImportError and tells me my module doesn't exist.
First I get the list of module filenames and chop off the ".py" suffixes, like so:
viable_plugins = filter(is_plugin, os.listdir(plugin_dir))
viable_plugins = map(lambda name: name[:-3], viable_plugins)
Then I os.chdir to the plugins directory and map __import__ the entire thing, like so:
active_plugins = map(__import__, viable_plugins)
However, when I turn active_plugins into a list and try to access the modules within, Python will throw out an error, saying it cannot import the modules since they don't appear to be there.
What am I doing wrong?
Edit: By simply using the interactive interpreter, doing os.chdir and __import__(modulefilename) produces exactly what I need. Why isn't the above approach working, then? Am I doing something wrong with Python's more functional parts?
It says it can't do it, because even though you're changing your directory to where the modules are, that directory isn't on your import path.
What you need to do, instead of changing to the directory where the modules are located, is to insert that directory into sys.path.
import sys
sys.path.insert(0, directory_of_modules)
# do imports here.