I have the following file structure:
app/
data/
data.csv
module/
__init__.py
script.py
__init__.py
config.py
My config class contains a set of Pandas dataframes read in from csv files. These dataframes are used across my code base:
import pandas as pd
class Config:
data = pd.read_csv('data/data.csv')
data2 = pd.read_csv('data/data2.csv')
My script contains:
from config import Config
Which returns:
FileNotFoundError: File b'data/data.csv' does not exist
if the run.py script is stored anywhere but the top level.
Is there any way to have class that has some stored file paths without causing a file not found error if importing the class in different scripts across the code base?
Try to specify the file in path resolved by the current file. For example, if your app/config.py is this:
import pandas as pd
class Config:
data = pd.read_csv('data/data.csv')
data2 = pd.read_csv('data/data2.csv')
You want to write the path like this:
mydir = os.path.dirname(os.path.abspath(__file__))
data = pd.read_csv(os.path.join(mydir, 'data/data.csv'))
data2 = pd.read_csv(os.path.join(mydir, 'data/data2.csv'))
which __file__ is the magic variable in Python to tell you which file you're in.
Basically, whenever your script try to open a file, it is relative to your current directory. Therefore, you need the above trick to find the file relative to the script's directory instead of current dir.
Related
After using the example from the docs, for dcc.Upload, I am able to save the csv file in my project directory by using to_csv(location2), location2 is a file string to my project directory ('E:/dash_app/components/new.csv')
it important to note that the goal is to have a multipage project where I can toggle the displayed screen from a dropdown in a navbar. so the project should run correctly until the csv file is uploaded and there after it will display all the graphs and charts and information that is returned from the csv.file .
My project structure looks as follows:
Dash_app
__pycache__
.vscode
Components
Import_csv.py
csv_clean.py
app.py
Index.py
requirements
Under components directory there are many more components that get imported into index.py.
I import import_csv into index.py, the dcc.Upload component is in csv_upload.py, therefore by running index.py it runs import_csv.py where the csv file can be uploaded into my directory.
The problem is that I use functions from csv_clean.py to take the uploaded csv file as an input in order to return values so that I can use those values in other scrips in the 'components' directory (for graph and card modules).
The input to all the functions in csv_clean.py will be location2.
I receive the error: in " csv_clean.py data = pd.read(location)" no such file or directory exist : location2
The csv_clean.py script looks like as follows:
`` Import pandas as pd from components.import_csv import import location2
def daily_roi_income(file):
csv_df = pd.read_csv(file)
roi = csv_df.loc[csv_df.Remark=='Roi Income', ['Date', 'Credit']]
return roi
Roi = daily_roi_income(location2)
``
So this function cannot work unless the csv file exists. And I cannot run Index.py to upload the csv unless this works.
What is an approach to take if code is dependent on other code to run first.
Thanks in advance.
I stetted up a Spyder project where I have codes located in a specific folder and different datas in different folders. Basically, I would like to read those files using the relative path or a simple approach. Let's take the project tree below as example:
Project Tree
I am trying to read "dummy_csv.csv" using "dummy_code.py".
What I am currently doing is this:
import pandas as pd
filepath= "../../../../../dummy_folder02/untitled folder/untitled folder/untitled
folder/dummy_data/dummy_csv.csv"
pd.read_csv(filepath)
I wonder if there is a more elegant/cleaner way of doing this...
You can include the root dir of your data in the system path variable, and then use just the relavtive path:
import sys
sys.path.append(<absolute path to root data dir>)
filepath = "<relative path to csv file, in relation to the absolute path added to sys.path>"
for example:
sys.path.append("C:/my_datasets/dummy_folder02")
filepath = "untitled folder/untitled folder/untitled folder/dummy_data/dummy_csv.csv"
I have a folder named test, and I want read its test.xlsx in Python under different computer environments .
#For Mac:
df = pd.read_excel('/Users/Steven/Desktop/test/test.xlsx')
#For Linux:
df = pd.read_excel('/Users/Users/Desktop/test/test.xlsx')
#For Windows:
df = pd.read_excel('C:/Users/Desktop/test/test.xlsx')
I'm wondering is there any way to read this file in a general way without changing its file path. Please share any ideas that may help, thanks.
df = pd.read_excel('../test/test.xlsx')
As far as I understood, your idea is to read a file using the concept of relative path. To do so, we can use os. Let's suppose you have a parent folder, containing /test/test.xlsx. In this same parent folder, we can insert our python program (let's call it my_pretty_program.py):
import os
import pandas as pd
#gets your current directory
dirname = os.path.dirname(__file__)
#concatenates your current directory with your desired subdirectory
results = os.path.join(dirname, r'test\test.xlsx')
#reads the excel file in a dataframe
df = pd.read_excel(results, engine='python')
print(df)
And just like this you can do the trick. Remember to have the structure as:
Parent/
|-- my_pretty_program.py
|-- test/
| |-- test.xlsx
This is a problem that has been previously solved (cannot write file with full path in Python) however I followed the advice in the previous answer and it didn't work and that's why I'm posting this.
I'm trying to access a csv file to load into the pandas dataframe.
import os
output_path = os.path.join('Desktop/My_project_folder', 'train.csv')
This is returning:
IOError: File Desktop/My_project_folder/train.csv does not exist
edit: I don't understand because the train.csv file exists in my project folder.
The os.path.join() function is platform agnostic meaning it can run across multiple OS (PC, Mac, Linux) without having the need to specify directories or subdirectories with forward or back slashes. Hence, simply separate paths and file names by commas:
myDir = '/path/to/Desktop/My_project_folder'
output_path = os.path.join(myDir, 'train.csv')
However, if Python script resides in same directory as data, have script detect its own path and then import data frame into pandas and avoiding hard-coding whole path names:
import os
import pandas as pd
# SET CURRENT DIRECTORY
cd = os.path.dirname(os.path.abspath(__file__))
traindf = read_csv(os.path.join(cd, 'train.csv'))
i am trying to import specified modules from test_file hierarchy
something like :
test_case1.py
test_subsuite_2
test_sub_2.1.1.py
test_suite2
is it possible to do a run import on this hierarchy
/project/main.py
/project/test_files
test_files folder hierarchy is like this :
test_files
test_suite1
test_case1.py
test_subsuite_1
test_sub1_1.py
test_sub1_2.py
test_subsuite_2
test_subsuite_2_1
test_sub_2.1.1.py
test_sub2_1.py
test_sub3_2.py
test_suite2
test_case2.py
test_subsuite2_1
test_sub21_1.py
test_sub21_2.py
test_subsuite2_2
test_sub22_1.py
test_sub23_2.py
Create an empty file called __init__.py in all you folders. Then you can import using . as a folder separator. Documentation here.
The key is to create a blank file __init__.py on all subfolders containing the files you want to import.
In your case, you will have to create the __init__.py files on all of the following folders-
test_files
test_files\test_suite1
test_files\test_suite1\test_subsuite_2
test_files\test_suite1\test_subsuite_2\test_subsuite_2_1
Also, when you import files be careful about specifying the import path correctly, with the entire path from the topmost level specified, and with different folder levels separated by a .
For example, you must import test_case1 by specifying :
from test_files.test_suite1 import test_case1
Similarly, test_subsuite_2 can be imported by specifying :
from test_files.test_suite1 import test_subsuite_2