I am using the following block of code in a python file names app.py to read JSON data:
import json
with open('./mapping/canada_provinces.geojson', 'r') as f:
countryGeoJSONData = json.load(f)
What this block of code seems to do is that the variable countryGeoJsonData variable cannot be imported by any other files within the same directory, such as by using the following import code in another file:
from app import countryGeoJSONData
Attempting the above, I get an error message that the name countryGeoJSONData could not be imported from app.py.
Why is this happening? Is the with block creating some sort of local context? The variable can be used outside of the with block in the same file. Why can it not be imported into another file?
To help reproduce the situation, here's the contents of app.py:
import pandas as pd
import json
# Read in Report Data
indicators = pd.read_excel(".\data\In Depth_All Data Export Report.xlsx",3)
contextual = pd.read_excel(".\data\In Depth_All Data Export Report.xlsx",4)
#open the GeoJSON file to show canadian provinces on the map
with open('./mapping/canada_provinces.geojson', 'r') as f:
countryGeoJSONData = json.load(f)
#unique available indicators
availableIndicators = indicators["Indicator"].unique()
#unqiue provinces
provinces = indicators[indicators["Reporting level"]=="Province"]["Province/territory"].unique()
And then in layout.py, I have the following import code:
from app import indicators, contextual, availableIndicators, provinces, countryGeoJSONData
which leads to the following error:
ImportError: cannot import name 'countryGeoJSONData' from 'app'
However, if I insert the following code after the with block:
importableJSON= countryGeoJSONData
then the new variable could be imported into layout.py with no problem. This is why I thought being inside a with block was causing a problem.
Project folder structure:
project
-data (includes data files)
-mapping (includes geojson file)
app.py
layout.py
Got it.
The with statement have nothign to do with it - but your relative filename probably do.
If you tested the module, importing it in the same folder the data folder ("mapping/") is in, it will work.
It will fail if you try importing this module from any other folder.
The solution for this is to use the module's __file__ special variable to find the absolute path for your your data file. pathlib.Path allows that with minimal fuzz:
import json
from pathlib import Path
with (Path(__file__).parent / 'mapping/canada_provinces.geojson').open() as f:
countryGeoJSONData = json.load(f)
(The Path object overrides the "/" separator so Paths can be compound with strings - that is not a typo)
Related
I have encountered issue while working on my Django project.
I have multiple classes inside views.py.
It had 1200 lines so I decided to move these views to seperated files inside new folder.
Now one files, for example Customer.py has 2 classes for different operations.
This is my project structure before splitting views.py:
MyProject
core
- urls.py
api
- views.py
manage.py
Project structure after splitting views.py
MyProject
core
- urls.py
api
- view
- *all the files with multiple classes in each file*
manage.py
After splitting views.py I needed to import all classes from all files inside of the view folder inside core/urls.py.
I have been trying to find out few hours now and can figure it out...
My current solution is that in urls.py im doing
from api.view import *
while having init.py inside view folder which is doing
from .oneOfManyFiles import *
for all classes...
I highly dont like this solution, I would love to find out some good looking simple elegant solution. Is here anyone who can help me ?
Big thanks
Avoiding explicit imports
You can't avoid that, at least not without breaking code completion on IDEs.
import importlib
import os
filedir = os.path.dirname(__file__)
modules = [f[:-3] for f in os.listdir(filedir) if os.path.isfile(os.path.join(filedir, f)) and f.endswith('.py') and f != '__init__.py']
for module_name in modules:
module = importlib.import_module(f".{module_name}", package=__name__)
names = [x for x in module.__dict__ if not x.startswith("_")]
globals().update({name: getattr(module, name) for name in names})
I wouldn't consider that good looking, simple or elegant.
Self-replacing generated imports
If what's bothering you is having to do it by hand, then probably the best thing you could do is write a script to generate the import statements and replace itself when run.
import os
filedir = os.path.dirname(__file__)
modules = [f[:-3] for f in os.listdir(filedir) if os.path.isfile(os.path.join(filedir, f)) and f.endswith('.py') and f != '__init__.py']
imports = [f" from .{module} import *\n" for module in sorted(modules)]
with open(__file__, 'r') as f:
this_script = list(f)[:17]
with open(__file__, 'w') as f:
f.write(f"""{''.join(this_script)[:-1]}
try:
{''.join(imports)[:-1] if imports else ' pass'}
except ImportError:
pass
""")
try:
from .oneOfManyFiles import *
except ImportError:
pass
For code completion, either run the project or python -m api.view.__init__.
I'm writing a python script to import a setting file from another project.
This is the structure of the project:
- root
- ...
- folder_1
- setting_folder
- __init__.py
- setting_1.py
- setting_2.py
- setting_3.py
Here the file contents:
init.py
from .setting_1 import *
from .setting_2 import *
setting_1.py
foo = "foo1"
setting_2.py
foo = "foo2"
try:
from .setting_3 import *
except ImportError:
pass
setting_3.py
foo = "foo3"
What I need to do is, from a script outside the project, load setting_2.py and get the value of foo variable (foo3 due to relative import).
Suppose that I run my script from directory "C:\Users\bar\Desktop".
My idea to achieve this goal is to copy setting_2.py in another directory outside the project (let's say a), create an empty file init.py in a, append to PYTHONPATH "C:\Users\bar\Desktop" and then import the module.
Here the code:
import os
import importlib.util
with open(setting_2_path, "r") as f:
test_file_content = f.read()
setting_tmp_path = r"C:\Users\bar\Desktop\a\setting_2.py"
with open(setting_tmp_path, "w") as f:
f.write(test_file_content)
init_tmp_path = r"C:\Users\bar\Desktop\a\__init__.py"
with open(init_tmp_path, "w") as f:
f.write("")
current_env = os.environ.copy()
current_env.update({'PYTHONPATH': r"C:\Users\bar\Desktop"})
spec_module = importlib.import_module('a.setting_2')
print(getattr(spec_module, "foo"))
This is working well, but in production this script will be in another project and I cannot create a folder at the same level of the script.
I can create the folder but in another directory.
To simulate this scenario, suppose that I run the script from C:\Users\bar\Desktop\bar2 and the folder is C:\Users\bar\Desktop\a.
In this case I have the following error:
ImportError: No module named 'a'
How can I fix the problem?
import os
import sys
cwd = os.getcwd() # This is current working directory, as in your assumption it is: C:\Users\bar\Desktop\bar2
path_to_settings_files = os.path.join(os.path.dirname(cwd), 'a') # This is where settings are, as in your assumption it is: C:\Users\bar\Desktop\a
sys.path.append(path_to_settings_files) # This line guides interpreter to find settings.py in loading modules
import setting_1
import setting_2
import setting_3
print(setting_1.foo)
print(setting_2.foo)
print(setting_3.foo)
Here is my directory on raspberry pi:
Home
|__pi
|__test1
|__test2
|__ abc.py
I am running a python code (abc.py) that edit and then dump a YAML file. But the file is saved in the same directory (i.e in test2). Is it possible to dump the YAML file in another directory (i.e in test1)?
If yes, then please let me know the code.
Here is my python code:
import sys
from ruamel.yaml import YAML
inp = """\
# example
name:
# details
family: Smith # very common
given: Alice # one of the siblings
"""
yaml = YAML()
code = yaml.load(inp)
code['name']['given'] = 'Bob'
yaml.dump(code, sys.stdout)
The second parameter to the .dump() method can be a stream like sys.stdout or a stream that is an opened file:
with open('../code1/output.yaml', 'w') as fp:
yaml.dump(code, fp)
As you are on Python3, you also have pathlib as part of the standard library and you can pass a pathlib.Path as second parameter to .dump():
from pathlib import Path
output = Path('../code1/output.yaml')
yaml.dump(code, output)
The Path will be properly opened for writing and closed.
(You can also load from a Path by doing, yaml.load(Path('some/path/to/a/file')))
import tempfile
tmp = tempfile.NamedTemporaryFile(delete=True)
try:
# do stuff with temp
tmp.write(b'def fun():\n\tprint("hello world!")\n')
if __name__ == '__main__':
func = __import__(tmp.name)
func.fun()
finally:
tmp.close() # deletes the file
So I want to create a temporary file, add some source code to it and then import the module and call the function, but I always run into this error:
ModuleNotFoundError: No module named '/var/folders/3w/yyp887lx4018h9s5sr0bwhkw0000gn/T/tmp84bk0bic'
It doesn't seem to find the module of the temporary file. How do I solve this?
There are a few problems with your code:
Your filename does not end with .py, but Python modules are expected to. You can fix this by setting suffix='.py' in NamedTemporaryFile().
__import__() is not the right way to load a module from a full path. See here: How to import a module given the full path?
You do not flush after writing and before importing, so even if Python does find the file, it may well be empty. Add tmp.flush() after writing to fix this.
Importing can only be done from certain directories which are part of the PYTHON_PATH. You can extend that. Then you will have to use __import__() with a module name (not a path in the file system). You will have to deal with the suffix for the temp file.
I implemented a simple version using the local directory for the temp module file and a version using a proper tempfile:
#!/usr/bin/env python3
import sys
import os
import tempfile
SCRIPT = '''\
def fun():
print("hello world!")
'''
# simple version using the local directory:
with open('bla.py', 'w') as tmp_module_file:
tmp_module_file.write(SCRIPT)
import bla
bla.fun()
# version using the tempfile module:
tmpfile = tempfile.NamedTemporaryFile(suffix='.py', delete=True)
try:
tmpfile.write(SCRIPT.encode('utf8'))
tmpfile.flush()
tmpmodule_path, tmpmodule_file_name = os.path.split(tmpfile.name)
tmpmodule_name = tmpmodule_file_name[:-3] # strip off the '.py'
sys.path.append(tmpmodule_path)
tmpmodule = __import__(tmpmodule_name)
finally:
tmpfile.close()
tmpmodule.fun()
I have a project named AsyncDownloaderTest with main.py and AsyncDownloader.py in same directory.I have just started learning python but it seems issue is with the import.
main.py
from .AsyncDownloader import AsyncDownloader
ad = AsyncDownloader()
ad.setSourceCSV("https://people.sc.fsu.edu/~jburkardt/data/csv/grades.csv","First name")
print(ad.printURLs)
AsyncDownloader.py
import pandas as pd
class AsyncDownloader:
"""Download files asynchronously"""
__urls = None
def setSourceCSV(self, source_path, column_name):
self.source_path = source_path
self.column_name = column_name
# TODO Check if path is a valid csv
# TODO Store the urls found in column in a list
my_csv = pd.read_csv(source_path, usecols=[column_name], chunksize=10)
for chunk in my_csv:
AsyncDownloader.urls += chunk.column_name
def printURLs(self):
print(AsyncDownloader.urls)
I am getting the following error
ModuleNotFoundError: No module named '__main__.AsyncDownloader'; '__main__' is not a package
Do you have __init__.py in the same directory as AsyncDownloader.py? That should do it.
__init__.py is an empty file that signals that the directory contains packages and makes functions and classes importable from .py files in that directory.
You can probably lose the leading . in from .AsyncDownloader as well. If you like, you can make the import absolute by changing it to:
from enclosing_folder.AsyncDownloader import AsyncDownloader