First software job and l inherited a codebase with heavy JSON usage. I'm trying to use Python to load a text file from accessing a JSON element. My relative path understanding is limited but l'm able to load python modules in the same sub directory (not using JSON though). This data file is in the same folder.
I parse and store data within the JSON element here:
with open(cfg) as fp:
edict = json.load(fp)
if edict["dtype"] == "custom" :
data = edict["dtype"]["custom"]["datapath"]
Relevant section of JSON file:
{
"dtype" : {
"custom" : {
"datapath" : "DataPipeLine/example_data.txt",
"type" : "T",
"X" : "encoder",
"Y" : "encoder"
}
}
}
I get an error from passing the data variable into a function later in the program:
UnboundLocalError: local variable referenced before assignment error is raised when you try to assign a value to a local variable before it has been declared.
There are too many errors in your code
I assumed your project directory like this
.
├── DataPipeLine
│ └── example_data.txt
├── cfg.json
└── main.py
There is the right code example
import json
from pathlib import Path
# Get current directory
CUR_DIR = Path(__file__).parent.absolute()
def load_data(path: Path):
print(path.read_text())
# cfg file path
cfg = CUR_DIR / 'cfg.json'
with cfg.open() as fp:
edict = json.load(fp)
# Check there is custom node or not
if edict["dtype"]["custom"]:
# Generate your datapath
data_path = CUR_DIR / edict["dtype"]["custom"]["datapath"]
load_data(data_path)
If my answer solved your problem please accept it as the right answer.
Related
I have this code in my_program.py:
from my_module import do_stuff_with_file
do_stuff_with_file("hi.txt")
And this is my_module.py:
def do_stuff_with_file(fileName):
print(fileName)
# do stuff with the file
A file not found error arises when my_module.py is not in the same directory as my_program.py. This problem was solved using this code (my_module.py).
def do_stuff_with_file(fileName):
fileName = os.path.join(os.path.dirname(sys.modules['__main__'].__file__), fileName)
print(fileName)
# do stuff with file
The issue arises when my_program.py is imported by a file in a different directory.
How can I fix this?
Given the following file hierarchy :
stack_overflow/
├─ q67993523/
├─ my_module/
│ ├─ __init__.py
├─ 67993523.py
├─ hi.txt
With the following file content :
# 67993523.py
from my_module import do_stuff_with_file
do_stuff_with_file("hi.txt")
# my_module/__init__.py
def do_stuff_with_file(filename):
print(f"{filename!s} content is :")
with open(filename, "rt") as file:
print(file.read())
and the file hi.txt :
Hello !
When I run C:\path\to\python.exe C:/stack_overflow/q67993523/67993523.py (with the path to q67993523 included in my PYTHONPATH), with my current directory being q67993523/, I get :
hi.txt content is :
Hello !
But if I change my current dir to q67993523/my_module/ and execute the exact same command, I get :
hi.txt content is :
Traceback:
[...]
FileNotFoundError: [Errno 2] No such file or directory: 'hi.txt'
because relative to the current working directory q67993523/my_module/ there is no file hi.txt, the file would be ../hi.txt.
I think what you are doing is an instance of the XY problem.
What you are trying to achieve is to find a file given its name but not the location. It is very difficult to do, prone to error, and would include lots of hacks to work.
I don't think it is actually what you want to do. What you want to do, I presume, is not to search for files but just to use them. So you should not lose the precious information of their location.
For example, in your main script (mine is 67993523.py), you know that the file is right there, in the same directory. But if you just send hi.txt, because the function don't know the file location of the code that called her, it does not know where to search for the file.
Instead, give the complete file location, namely the absolute path.
If I change my main script to :
# 67993523.py
from pathlib import Path
from my_module import do_stuff_with_file
the_directory_of_this_pyfile = Path(__file__).parent
do_stuff_with_file((the_directory_of_this_pyfile / "hi.txt").absolute())
And run it with my current directory being q67993523/, I get :
C:\stack_overflow\q67993523\hi.txt content is :
Hello !
And when I change my current directory to q67993523/my_module/, I get the same thing :
C:\stack_overflow\q67993523\hi.txt content is :
Hello !
The difference is that in your script, the hi.txt filename assumes that your current working directory is q67993523/. If you have a different current working directory (because Pytest, because running the script for anywhere you want, ... see the comment from #tdelaney) then there is no ./hi.txt file, so it will fail.
I encourage you to learn on the topic of current working directory
and how to express the current Python file directory.
I have a following python3 script, that uses the config function to load data from a txt file that sits in the same directory where the script is located.
I've implemented the configparser module to extract the data, and pathlib module to set the absolute path to that file
from pathlib import Path
try:
import ConfigParser as configparser
except:
import configparser
def config():
parser = configparser.ConfigParser()
config_file = Path('config.txt').resolve()
parser.read(config_file)
return parser
then i pass it as an argument to the next method, which then gets the needed variables from the
config file:
def search_for_country(config):
country_name = config.get("var", "country_name")
the config.txt file is structured like this:
[var]
country_name = Brazil
The problem is: everything works fine if I run the script via Terminal from the same directory, but eventually it is intended to be run as a cron job, and if i try to execute it from one directory above, it returns the following error:
File "test/script.py", line 28, in search_for_country
country_name = config.get("var", "country_name")
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/configparser.py", line 781, in get
d = self._unify_values(section, vars)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/configparser.py", line 1149, in _unify_values
raise NoSectionError(section) from None
configparser.NoSectionError: No section: 'var'
Which seems to be telling that it cannot find the txt file.
So far I've tried out different options, for example using parser.read_file() instead of parser.read(), also tried this: https://stackoverflow.com/a/35017127/13363008
But none seem to be working. Maybe anyone could think of a cause to this problem?
so for Your problem :
import pathlib
path_own_dir = pathlib.Path(__file__).parent.resolve()
path_conf_file = path_own_dir / 'config.txt'
assert path_conf_file.is_file()
but why to store such config as text in the first place ?
"config is code" - so why limit Yourself with text config files ?
my usual way is :
# conf.py
class Conf(object):
def __init__(self):
country_name: str = 'Brazil'
conf=Conf()
# main.py
from .conf import conf
print(conf.county_name)
# override settings in the object
conf.county_name = 'Argentina'
print(conf.county_name)
it has so many advantages like having the correct data type,
having the IDE hints, avoiding the parsers, dynamically change settings, etc ...
I am using the following block of code in a python file names app.py to read JSON data:
import json
with open('./mapping/canada_provinces.geojson', 'r') as f:
countryGeoJSONData = json.load(f)
What this block of code seems to do is that the variable countryGeoJsonData variable cannot be imported by any other files within the same directory, such as by using the following import code in another file:
from app import countryGeoJSONData
Attempting the above, I get an error message that the name countryGeoJSONData could not be imported from app.py.
Why is this happening? Is the with block creating some sort of local context? The variable can be used outside of the with block in the same file. Why can it not be imported into another file?
To help reproduce the situation, here's the contents of app.py:
import pandas as pd
import json
# Read in Report Data
indicators = pd.read_excel(".\data\In Depth_All Data Export Report.xlsx",3)
contextual = pd.read_excel(".\data\In Depth_All Data Export Report.xlsx",4)
#open the GeoJSON file to show canadian provinces on the map
with open('./mapping/canada_provinces.geojson', 'r') as f:
countryGeoJSONData = json.load(f)
#unique available indicators
availableIndicators = indicators["Indicator"].unique()
#unqiue provinces
provinces = indicators[indicators["Reporting level"]=="Province"]["Province/territory"].unique()
And then in layout.py, I have the following import code:
from app import indicators, contextual, availableIndicators, provinces, countryGeoJSONData
which leads to the following error:
ImportError: cannot import name 'countryGeoJSONData' from 'app'
However, if I insert the following code after the with block:
importableJSON= countryGeoJSONData
then the new variable could be imported into layout.py with no problem. This is why I thought being inside a with block was causing a problem.
Project folder structure:
project
-data (includes data files)
-mapping (includes geojson file)
app.py
layout.py
Got it.
The with statement have nothign to do with it - but your relative filename probably do.
If you tested the module, importing it in the same folder the data folder ("mapping/") is in, it will work.
It will fail if you try importing this module from any other folder.
The solution for this is to use the module's __file__ special variable to find the absolute path for your your data file. pathlib.Path allows that with minimal fuzz:
import json
from pathlib import Path
with (Path(__file__).parent / 'mapping/canada_provinces.geojson').open() as f:
countryGeoJSONData = json.load(f)
(The Path object overrides the "/" separator so Paths can be compound with strings - that is not a typo)
I have a project named AsyncDownloaderTest with main.py and AsyncDownloader.py in same directory.I have just started learning python but it seems issue is with the import.
main.py
from .AsyncDownloader import AsyncDownloader
ad = AsyncDownloader()
ad.setSourceCSV("https://people.sc.fsu.edu/~jburkardt/data/csv/grades.csv","First name")
print(ad.printURLs)
AsyncDownloader.py
import pandas as pd
class AsyncDownloader:
"""Download files asynchronously"""
__urls = None
def setSourceCSV(self, source_path, column_name):
self.source_path = source_path
self.column_name = column_name
# TODO Check if path is a valid csv
# TODO Store the urls found in column in a list
my_csv = pd.read_csv(source_path, usecols=[column_name], chunksize=10)
for chunk in my_csv:
AsyncDownloader.urls += chunk.column_name
def printURLs(self):
print(AsyncDownloader.urls)
I am getting the following error
ModuleNotFoundError: No module named '__main__.AsyncDownloader'; '__main__' is not a package
Do you have __init__.py in the same directory as AsyncDownloader.py? That should do it.
__init__.py is an empty file that signals that the directory contains packages and makes functions and classes importable from .py files in that directory.
You can probably lose the leading . in from .AsyncDownloader as well. If you like, you can make the import absolute by changing it to:
from enclosing_folder.AsyncDownloader import AsyncDownloader
I have a config file abc.txt which looks somewhat like:
path1 = "D:\test1\first"
path2 = "D:\test2\second"
path3 = "D:\test2\third"
I want to read these paths from the abc.txt to use it in my program to avoid hard coding.
In order to use my example, your file "abc.txt" needs to look like this.
[your-config]
path1 = "D:\test1\first"
path2 = "D:\test2\second"
path3 = "D:\test2\third"
Then in your code you can use the config parser.
import ConfigParser
configParser = ConfigParser.RawConfigParser()
configFilePath = r'c:\abc.txt'
configParser.read(configFilePath)
As human.js noted in his comment, in Python 3, ConfigParser has been renamed configparser. See Python 3 ImportError: No module named 'ConfigParser' for more details.
You need a section in your file:
[My Section]
path1 = D:\test1\first
path2 = D:\test2\second
path3 = D:\test2\third
Then, read the properties:
import ConfigParser
config = ConfigParser.ConfigParser()
config.readfp(open(r'abc.txt'))
path1 = config.get('My Section', 'path1')
path2 = config.get('My Section', 'path2')
path3 = config.get('My Section', 'path3')
If you need to read all values from a section in properties file in a simple manner:
Your config.cfg file layout :
[SECTION_NAME]
key1 = value1
key2 = value2
You code:
import configparser
config = configparser.RawConfigParser()
config.read('path_to_config.cfg file')
details_dict = dict(config.items('SECTION_NAME'))
This will give you a dictionary where keys are same as in config file and their corresponding values.
details_dict is :
{'key1':'value1', 'key2':'value2'}
Now to get key1's value :
details_dict['key1']
Putting it all in a method which reads sections from config file only once(the first time the method is called during a program run).
def get_config_dict():
if not hasattr(get_config_dict, 'config_dict'):
get_config_dict.config_dict = dict(config.items('SECTION_NAME'))
return get_config_dict.config_dict
Now call the above function and get the required key's value :
config_details = get_config_dict()
key_1_value = config_details['key1']
Generic Multi Section approach:
[SECTION_NAME_1]
key1 = value1
key2 = value2
[SECTION_NAME_2]
key1 = value1
key2 = value2
Extending the approach mentioned above, reading section by section automatically and then accessing by section name followed by key name.
def get_config_section():
if not hasattr(get_config_section, 'section_dict'):
get_config_section.section_dict = collections.defaultdict()
for section in config.sections():
get_config_section.section_dict[section] = dict(config.items(section))
return get_config_section.section_dict
To access:
config_dict = get_config_section()
port = config_dict['DB']['port']
(here 'DB' is a section name in config file
and 'port' is a key under section 'DB'.)
A convenient solution in your case would be to include the configs in a yaml file named
**your_config_name.yml** which would look like this:
path1: "D:\test1\first"
path2: "D:\test2\second"
path3: "D:\test2\third"
In your python code you can then load the config params into a dictionary by doing this:
import yaml
with open('your_config_name.yml') as stream:
config = yaml.safe_load(stream)
You then access e.g. path1 like this from your dictionary config:
config['path1']
To import yaml you first have to install the package as such: pip install pyyaml into your chosen virtual environment.
This looks like valid Python code, so if the file is on your project's classpath (and not in some other directory or in arbitrary places) one way would be just to rename the file to "abc.py" and import it as a module, using import abc. You can even update the values using the reload function later. Then access the values as abc.path1 etc.
Of course, this can be dangerous in case the file contains other code that will be executed. I would not use it in any real, professional project, but for a small script or in interactive mode this seems to be the simplest solution.
Just put the abc.py into the same directory as your script, or the directory where you open the interactive shell, and do import abc or from abc import *.
Since your config file is a normal text file, just read it using the open function:
file = open("abc.txt", 'r')
content = file.read()
paths = content.split("\n") #split it into lines
for path in paths:
print path.split(" = ")[1]
This will print your paths. You can also store them using dictionaries or lists.
path_list = []
path_dict = {}
for path in paths:
p = path.split(" = ")
path_list.append(p)[1]
path_dict[p[0]] = p[1]
More on reading/writing file here.
Hope this helps!
For Pyhon 3.X:
Notice the lowercase import configparser makes a big difference.
Step 1:
Create a file called "config.txt" and paste the below two lines:
[global]
mykey = prod/v1/install/
Step 2:
Go to the same directory and create a testit.py and paste the code below into a file and run it. (FYI: you can put the config file anywhere you like you, you just have to change the read path)
#!/usr/bin/env python
import configparser
config = configparser.ConfigParser()
config.read(r'config.txt')
print(config.get('global', 'mykey') )