I am attempting to grade some python submissions that are in separate folders for each student. To do this, there is a function, say f() which I want to run. I understand that if my current path is the same as the one where the file is located, I can simply do
import filename
filename.f()
However, are there better ways? For instance, let's say the directory structure is as follows:
main.py
student/run_this.py
I know that if there is a "__init__.py" file in the student folder, I can just type
import student.run_this
However, without that file, it doesn't work.
Some similar questions I found were
Import module from subfolder
How to do relative imports in Python?
http://www.daniweb.com/software-development/python/threads/192000/import-from-a-subdirectory-of-a-directory-on-pythonpath
but none of these gave particularly satisfying answers.
create an __init__.py module inside the folder student which should contain
from . import *
You can then call any modules from student folder to its parent folder modules as
import student.module.py
If you post any other errors you are facing, we can help further.
Related
The Challenge
I am working on a Python project that will act as a translation layer for the SCPI command line interface on scientific instruments.
It is similar to the concept described in Multi layer package in python however my situation is slightly more complex and I can't seem to figure out how to make it work.
The following is a representation of my project structure (I am happy to change it if it is required):
Some things to keep in mind
The names of the files in translator_lib.instruments.supplierX.moduleX are only class1.py, class2.py and class3.py, the rest of the filename is for reference to describe where they are derived from.
Many of the modules and classes inside of supplier1 and supplier2 have the same names (as in the example).
Every directory contains a __init__.py file
The __init__.py files (based on the example layout) look as follows (Note I only have one module for testing)
translator_lib\__init__.py
from .instruments import supplier1
__all__ = ['supplier1']
translator_lib\instruments\__init__.py
from .supplier1 import module1
__all__ = ['module1']
What I'm trying to do
Compile three libraries called my_translator_lib, supplier1_translator_lib and supplier2_translator_lib.
Reason
The development team would import my_translator_lib to do what they need to, but if we want to send sample code to supplier1, we want to send them the supplier1_translator_lib and they should only be able to import supplier1_translator_lib
Example 1 : Developer
from translator_lib.instruments import supplier1
from translator_lib.instruments import supplier2
class DoStuff:
__init__(self):
self.sup1_class1 = supplier1.module1.class1.class1()
self.sup2_class1 = supplier2.module1.class1.class1()
Example 2 : Supplier 1
from supplier1_translator_lib import module1
from supplier1_translator_lib import module2
class DoStuff:
__init__(self):
self.class1 = module1.class1.class1()
self.class2 = module1.class2.class2()
I've tried multiple combinations and sections from How to create a Python library and Deep dive: Create and publish your first Python library. I manage to create a library and install it, but ultimately I can only import my_translator_lib and nothing else is visible or available.
Any help in this regard would be truly appreciated.
have you created the __init__.py files?
you need to have then in every subfolder to your module.
Files named __init__.py are used to mark directories on disk as Python package directories. Try to follow this:
mydir/spam/__init__.py
mydir/spam/module.py
If this not solve your problem, as a last resource you can try to sys.path.append('path_to_other_modules')
Im having trouble importing a class on a python module.
Here are my directory structure:
_wikiSpider
+scrapy.cfg
_wikiSpider
+__init__.py
+items.py
+items.pyc
+settings.py
+settings.pyc
+pipelines.py
_spiders
+__init__.py
+__init__.pyc
+articleSpider.py
+articleSpider.pyc
+items.py
Code breaks at this line:
from wikiSpider.items import Article
Im not sure why, since class Article is defined at items.py (deepest folder)
Can someone give me an explanation?
Like others, I didn't have a circular reference problem. I'd like to generalize the solution here just a bit more though.
Any file name conflict can cause this. You could have multiple sub files with the same name (as above).
Or it could be the file you're working in.
Eg: trello.py as a pet project.
from trello import TrelloApi
Import reference will import itself before importing the pip installed package. Attempts to import trello and reference objects directly will fail with "NameError: name '' is not defined"
You have an items.py in both your root and _spiders folder. To reference a file in a subfolder you need the folder name and the file.
from _spiders.items import Article
assuming the file that imports this code is in your root directory. Python uses a you are here, to current file location, for it's directory hierarchy.
Best solution :
Rename class name with temporary name
Put the same temporary name in import statement in __init__.py
Now that works, put your old name again
from main wikiSpider directive try:
from _wikiSpider._spiders.items import Article
orelse from terminal open your _spiders directive and try:
from items import Article
Here we want to open the items.py file where we created Article class, so when you give some wrong directive or file , it cant find the items.py file that you created, so it shows 'Cannot import error'
What worked for me is removing the __pycache__ folder. I was moving class files around and changing the directory hierarchy up and the cache must have had old names/locations. Deleting it and running my program again created a new cache and the error was no longer present.
I'm planning to have project with following structure:
./runme.py
./my_modules/__init__.py
./my_modules/global_imports.py
./my_modules/user_defined_functions.py
Idea is to store important variables in global_imports.py from where they will be imported into runme.py using from my_modules.global_imports import * (I know it is a bad practice import modules this way, but I promise there will be just few variables with not colliding names)
Four questions:
Two of the variables contained inside global_imports.py should be SCRIPT_PATH and SCRIPT_DIR. I've tried SCRIPT_PATH = os.path.realpath(__file__) and SCRIPT_DIR = os.path.dirname(SCRIPT_PATH) but it returns path (directory) for global_imports.py not for runme.py. How can I get path (directory) of runme.py?
Inside global_imports.py I will probably import modules such as os and sys. I also need to import those modules inside runme.py. Is this considered as problem, when modules are imported first from another module and later from main script or vice versa?
Is it possible to import variables from global_imports.py into user_defined_functions.py? I consider this as bad practice I'm just curious.
Is there better approach to separate project into modules?
Addressing your questions in order:
In the first variable SCRIPT_DIR you are getting the full path of the file global_imports.py, which would be something like this:
SCRIPT_PATH = '/home/../../my_project/my_modules/global_imports.py'
now in order to get the directory of runme.py, we should consider another variable:
SCRIPT_PATH_DIR = os.path.dirname(SCRIPT_PATH)
this will give us the path
SCRIPT_PATH_DIR = '/home/../../my_project/my_modules/'
now to get to its parent directory which contains runme.py we can get like this:
SCRIPT_DIR = os.path.abspath(os.path.join(SCRIPT_PATH_DIR, os.pardir))
Now, SCRIPT_DIR gives the path of runme.py i.e.:
SCRIPT_DIR = '/home/../../my_project/'
As per our project structure, runme.py should only conrtain an import to the main module and then command to run the app. So, it shouldn't contain any other imports. Still if you need to use a module then explicitly import it in the runme.py as one of the Zen of Python says 'Explicit is better than implicit'
Yes, it is possible, you can do it like this:
from .global_imports import variable_name
but in general you should have a separate config.py or settings.py file in '/../my_project/' directory which should contain all the settings and variables which you may need to use anywhere in the project.
This approach is good enough as far as I've seen. Your main project directory contains runme.py and other modules which are used inside the project. 'my_modules' is one of the modules I think. You can have more of such modules inside the project directory. A better approach is to have settings and configurations inside one of the modules(such as my_modules) only and to have other modules for the functionality.
Hope this helps, please comment if something is unclear.
This sounds ridiculous as there appears to be an unlimited number of responses to this question on this site - but I can't find a straightforward solution without temporarily changing my system path for each reload (or init, which doesn't work for my setup). I'm looking for a secure, non-hacky way of getting this done.
Simply put - I have a directory structure as follows:
**root**
>main.py
>**modules**
>>rivescript.py
>>js.py
>**plugins**
>>weather.py
>>synd.py
To make it simple, I would like to import every available module in the presented subdirectories (modules, plugins) natively in main.py
Pseudo:
#main.py
import "./modules/*.py" as modules_*
import "./plugins/*.py" as plugins_*
And be able to call functions as something like:
plugins_weather.get("3088")
modules_rivescript.RiveScript.reply("localuser", language_input)
Any suggestions? Speed and resource consumption are a big thing for this project.
First, you should put __init__.py files (which could be empty) in modules/ and plugins/ directories, to mark them as packages.
Now, you are able to import your modules in main.py:
import modules.js as js
import modules.rivescript as rivescript
import plugins.weather as weather
import plugins.synd as synd
weather.get("3088") # Usage example
I have a Python project with the following directory structure:
project/
project/src/
project/src/somecode.py
project/src/mypackage/mymodule.py
project/src/resources/
project/src/resources/datafile1.txt
In mymodule.py, I have a class (lets call it "MyClass") which needs to load datafile1.txt. This sort of works when I do:
open ("../resources/datafile1.txt")
Assuming the code that creates the MyClass instance created is run from somecode.py.
The gotcha however is that I have unit tests for mymodule.py which are defined in that file, and if I leave the relative pathname as described above, the unittest code blows up as now the code is being run from project/src/mypackage instead of project/src and the relative filepath doesn't resolve correctly.
Any suggestions for a best practice type approach to resolve this problem? If I move my testcases into project/src that clutters the main source folder with testcases.
I usually use this to get a relative path from my module. Never tried in a unittest tho.
import os
print(os.path.join(os.path.dirname(__file__),
'..',
'resources'
'datafile1.txt'))
Note: The .. tricks works pretty well, but if you change your directory structure you would need to update that part.
On top of the above answers, I'd like to add some Python 3 tricks to make your tests cleaner.
With the help of the pathlib library, you can explicit your ressources import in your tests. It even handles the separators difference between Unix (/) and Windows ().
Let's say we have a folder structure like this :
`-- tests
|-- test_1.py <-- You are here !
|-- test_2.py
`-- images
|-- fernando1.jpg <-- You want to import this image !
`-- fernando2.jpg
You are in the test_1.py file, and you want to import fernando1.jpg. With the help to the pathlib library, you can read your test resource with an object oriented logic as follows :
from pathlib import Path
current_path = Path(os.path.dirname(os.path.realpath(__file__)))
image_path = current_path / "images" / "fernando1.jpg"
with image_path.open(mode='rb') as image :
# do what you want with your image object
But there's actually convenience methods to make your code more explicit than mode='rb', as :
image_path.read_bytes() # Which reads bytes of an object
text_file_path.read_text() # Which returns you text file content as a string
And there you go !
in each directory that contains Python scripts, put a Python module that knows the path to the root of the hierarchy. It can define a single global variable with the relative path. Import this module in each script. Python searches the current directory first so it will always use the version of the module in the current directory, which will have the relative path to the root of the current directory. Then use this to find your other files. For example:
# rootpath.py
rootpath = "../../../"
# in your scripts
from rootpath import rootpath
datapath = os.path.join(rootpath, "src/resources/datafile1.txt")
If you don't want to put additional modules in each directory, you could use this approach:
Put a sentinel file in the top level of the directory structure, e.g. thisisthetop.txt. Have your Python script move up the directory hierarchy until it finds this file. Write all your pathnames relative to that directory.
Possibly some file you already have in the project directory can be used for this purpose (e.g. keep moving up until you find a src directory), or you can name the project directory in such a way to make it apparent.
You can access files in a package using importlib.resources (mind Python version compatibility of the individual functions, there are backports available as importlib_resources), as described here. Thus, if you put your resources folder into your mypackage, like
project/src/mypackage/__init__.py
project/src/mypackage/mymodule.py
project/src/mypackage/resources/
project/src/mypackage/resources/datafile1.txt
you can access your resource file in code without having to rely on inferring file locations of your scripts:
import importlib.resources
file_path = importlib.resources.files('mypackage').joinpath('resources/datafile1.txt')
with open(file_path) as f:
do_something_with(f)
Note, if you distribute your package, don't forget to include the resources/ folder when creating the package.
The filepath will be relative to the script that you initially invoked. I would suggest that you pass the relative path in as an argument to MyClass. This way, you can have different paths depending on which script is invoking MyClass.