Pretty new to python but trying to write a reusable package for some data processing I've been doing a lot recently.
My project I'm wanting to structure like
src/
mypackage/*.py
test/
test-type1/*.py
test-type2/*.py
...
Each 'test-type' will have multiple scripts underneath it. My issue is when I try to import mypackage like in the root directory it explodes. I'm curious what a good way to handle this is? I can't just use relative paths since I get an error
attempted relative import with no known parent package
Any help would be appreciated.
Edit:
If a test file was in the root directory I could just do. With the change in directories I can't do that without getting the relative import error.
import mypackage.{file}
Edit2:
Example of code of one of my tests. Folder structure
rl/
__init__.py
callbacks/
__init__.py
checkpoint.py
earlystop.py
progress.py
training/
__init__.py
train.py
tests/
cartpole-v1/
train.py
''' Code
import gym
import torch
import pandas as pd
import matplotlib.pyplot as plt
from pole_actor import PoleActor
from ...rl.training import train
from ...rl.callbacks import Checkpoint, EarlyStop, ProgressBarCallback
def a2c(env: gym.Env):
actor = PoleActor(batch_size=50,max_memory=500,optimizer=torch.optim.Adam,lr=3e-4)
rewards = train(env,actor,3000,callbacks=[
Checkpoint(save_path='models/pole-callback.ach5',patience=100),
EarlyStop(patience=500,delay=1000),
#Need a better way of selecting the 'best' model, ideally we should always get a good model out of here
ProgressBarCallback()
])
smoothed = pd.Series.rolling(pd.Series(rewards),10).mean()
plt.plot()
plt.xlabel('Episodes')
plt.ylabel('Reward')
plt.plot(rewards)
plt.plot(smoothed)
plt.show(block=True)
if __name__ == "__main__":
env = gym.make("CartPole-v1")
a2c(env)
You could use these lines of code before importing the library:
import os,sys,inspect
current_dir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
parent_dir = os.path.dirname(current_dir)
sys.path.insert(0, parent_dir)
this code will import the packages from the parent directory instead of the current directory.
Related
I know there are many questions related to import errors and file/folder structure, but none seem to have the same structure as my file tree I have been struggling with ordering my files and importing for several hours and would appreciate some guidance.
My file tree is as follows
-MainFolder
-program.py
-HelperFolder
-helper1.py
-helper2.py
-HelperTestFolder
-helpertest.py
The helper functions need to import each other and the program.py has to import helper1 and helper2. The helpertest.py function has to import helper1.py and helper2.py
I've tried using an __init__.py file for the HelperFolder, various combinations of import .file, from file import *, etc. No matter what I do, I seem to get thrown some error (NameErrors, ImportErrors, etc) by either the helper1 or helper2.py file, the program.py file, or the helpertest.py file.
I'm sorry for the vague question, but is there a "right" way to do this that I'm missing?
In program.py:
from HelperFolder import helper1, helper2
In helper1.py:
import helper2
It can be tricky to manage circular dependencies. You're sure you need helper1 and helper2 to import each other?
In helpertest.py:
import os, sys
sys.path.append( os.path.realpath(os.path.dirname(__file__) + '/..' ))
import helper1
import helper2
I am working with the following directory in Python3.8:
package/
__init__.py
/folder1
__init__.py
file1.py
/folder2
__init__.py
file2.py
/folder3
__init__.py
file3.py
I would like to import a function from file3 into file2. What is the easiest way to do it? I would also like to avoid manually appending to PYTHONPATH, to make this as smooth as possible for the next person pulling the project.
So far tried relative imports, which didn't work, possibly because I did not specify something in the init, which is currently empty. The error I am getting using:
from ..package.folder3.file3 import function_name
is
ImportError: attempted relative import with no known parent package
Thanks for all help!
The answer to your question is pretty simple, you need to add your package path to the system path.
Here is a complete example:
In file3.py, let's create a simple function
def func():
print("Hello from file3")
In file2.py, we can import func function like so:
import os
import sys
sys.path.append(os.path.abspath('../../..'))
# import func now from file3
from package.folder3.file3 import func
func() #should return "Hello from file3"
Hopefully, this answers your question!
I have a following directory structure:
main_work/
data_processing/
data_extraction.py
data_selector.py
data_selector.py:
from data_extraction import connect_to_db
If I start working in main_work and want to do
from data_processing import data_selector as ds
it throws me an error:
ModuleNotFoundError: No module named 'data_extraction'
So, the import in data_selector.py takes into account only current folder (main_work) and doesnt go one level deeper into data_processing.
I need to be able to do from main_work following:
from data_processing import data_selector as ds
and from data_processing following:
import data_selector
How do I achieve those two functionalities?
You may want to add your folder to the PYTHONPATH, this way python will know where to look for the modules you want to import.
You need the following structure
main_work/
data_processing/
__init__.py
data_extraction.py
data_selector.py
and in data_selector.py
from .data_extraction import connect_to_db
the . specifies that it should look in the same directory as the data_selector module
I have a directory structure like
Documents/
Project_1/
module_1.py
Project_2/
module_2.py
If I want to import module_1.py into module_2.py, what is the syntax for it?
I have tried variations on
import ../Project_1/module_1
And
from .Project_1 import module_1
But I can't get it to work and have only done this once before in another project.
You have two alternatives;
from Documents.project_1 import module1.py
or
import Documents.project_1.module1.py
It is a crude solution, but I ended up with something like this:
#This thing modifies the path
from sys import path
#This gets our user directory
from os import environ
#add module_1's location to path
path.append(environ['USERPROFILE'] + "\\Documents\\Project_1")
#import it now
import module_1
This is not the most elegant solution, but it can work on almost any Windows machine, assuming the folders are placed in their Documents.
The code about Environ could be reasonably replaced to match another directory, though.
Currently have the following file hierarchy:
\package
__init__.py
run_everything.py
\subpackage
__init__.py
work.py
work1.py
work2.py
\test
__init__.py
test_work.py
test_work1.py
My first question is regarding relative imports. Suppose in \subpackage\work.py I have a function called custom_function(), and I would like to test that function in test_work.py. For some reason I can not figure out how to make this import from one module to another. Trying from .. subpackage.work1 import custom_function() does not seem to work, and yields the error Attempted relative import in non-package Is there any way to resolve this?
2)
I would like to run all test files from run_everything.py with one function, would adding a suite() function in each test_work*.py file, which adds each unit_testing class to suite.addTest(unittest.makeSuite(TestClass)), and finally importing them into the top-level run_everything.py be the most conventional way in Python2.7?
Here is a hack*
Insert the path's to "subpackage" and "test" to your python path in run_everything using:
import sys
sys.path.insert(0, '/path/to/package/subpackage')
sys.path.insert(0, '/path/to/package/test')
And then, you can import all your files using vanilla imports in run_everything:
import work, work1, work2
import test_work, test_work1
*This won't permanently affect your PYTHONPATH.