Develop Reference

How To Have Python Import A Module From Another Directory

So I have a Python3 Script that I am trying to implement. The overall project has a repository file tree that looks like this:
.
├── Dockerfile
├── LICENSE
├── Makefile
├── README.md
├── functions
│ ├── README.md
│ ├── __init__.py
│ ├── aws
│ │ ├── README.md
│ │ ├── __init__.py
│ │ ├── __pycache__
│ │ │ ├── boto3_helper.cpython-310.pyc
│ │ │ └── boto3_helper.cpython-39.pyc
│ │ ├── boto3_helper.py
│ │ ├── boto3_session_create.py
│ │ ├── boto3_session_mock.py
│ │ ├── describe_direct_connect.py
│ │ ├── describe_vpc.py
│ │ ├── get_network_payload.py
│ │ ├── get_routing_tables.py
│ │ ├── get_vpc.py
│ │ ├── make_diagram_network_block.py
│ │ └── transform.py
│ ├── helpers
│ │ ├── __init__.py
│ │ ├── get_payload.py
│ │ └── helpers.py
│ └── peering
│ ├── __init__.py
│ └── peering.py
├── images
│ └── diagram.png
├── main.py
├── modules
│ ├── README.md
│ ├── audit
│ │ └── audit.py
│ ├── awsmaster
│ │ └── awsmaster.py
│ ├── cad
│ │ └── cad.py
│ ├── canary
│ │ └── canary.py
│ ├── cas-nonprod
│ │ └── casnonprod.py
│ ├── cas-prod
│ │ └── casprod.py
│ ├── css-dev
│ ├── css-nonprod
│ ├── css-prod
│ ├── css-staging
│ ├── didev
│ ├── dr
│ ├── eng
│ ├── factory-nonprod
│ ├── factory-sandbox
│ ├── factory-staging
│ ├── hsm-nonprod
│ ├── hsm-prod
│ ├── iis-nonprod
│ ├── iis-prod
│ ├── it
│ ├── log-archive
│ ├── net
│ │ ├── images
│ │ │ └── net.png
│ │ ├── net.py
│ │ └── net_payload.json
│ ├── octal
│ ├── octane
│ ├── pa-nonprod
│ ├── pa-prod
│ ├── platform-systems
│ ├── pte
│ ├── rf-regression
│ ├── sec
│ ├── shared-svcs
│ ├── taf
│ └── yukon
└── requirements.txt
I have done research that basically says use sys or use from path.to.folder import file. However, when I run the python function it doesn't actually import the module.
# How to use this file to generate a diagram:
# $> python3 diagram.py
# https://diagrams.mingrammer.com/docs/getting-started/installation
import json
import traceback
from diagrams import Cluster, Diagram
from diagrams.aws.network import PrivateSubnet
from diagrams.aws.management import OrganizationsAccount
from functions.aws import get_vpc
graph_attr = {
"bgcolor": "transparent",
"margin": "-1, -2",
"size": "50,50!"
}
def get_network_payload():
json_data = get_vpc(json)
with open("net_payload.json", "w") as outfile:
outfile.write(json_data)
def impinjnetcluster():
with Diagram(filename="images/net", show=False, direction="LR", graph_attr=graph_attr):
with Cluster("impinjnet", graph_attr={"margin": "40", "fontsize": "18"}):
impinjnet = OrganizationsAccount("Impinj-Net")
with Cluster("vpc-###1", graph_attr={"margin": "40", "fontsize": "18"}):
net_subnet1 = PrivateSubnet("###")
with Cluster("vpc-###2", graph_attr={"margin": "40", "fontsize": "18"}):
net_subnet2 = PrivateSubnet("###")
impinjnet
try:
get_network_payload()
impinjnetcluster()
except:
traceback.print_exc()
Basically I am trying to do the following:
Get the network payload
load the network payload into a dynamic block
generate image
However, I can't even get the network payload to run it just does this:
Traceback (most recent call last):
File "/Users/rbarrett/Git/Impinj-di/aws_network_design/modules/net/net.py", line 10, in <module>
from functions.aws import functions, get_vpc
ModuleNotFoundError: No module named 'functions'
I put the __init__.py as an empty file in every folder that I would need to import as seen from the tree and here is my python paths:
rbarrett#RBARRETT-MBP1  ~/Git/Impinj-di/aws_network_design/modules/net   DI-4894-CreateWorkflows-4 ● ?  python3 -c "import sys; print('\n'.join(sys.path))"
/usr/local/Cellar/python#3.10/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python310.zip
/usr/local/Cellar/python#3.10/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10
/usr/local/Cellar/python#3.10/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload
/usr/local/lib/python3.10/site-packages
I have tried looking at several other questions similar to mine, but none of them have seemed to help me pull in the function I have defined in another file.

So it turns out I had to use sys and then had to use the following in my import statements:
import sys
sys.path.append('../../')
from functions.aws.boto3_helper import <function_name>
Where <function_name> was a function defined within boto3_helper.py, the biggest problem I had was to set sys.path.append('../../') which is where the files I needed to access as a module are located at.

Coverage unittest fails in Python 3.8

I am struggling with the implementation of unittest for subdirectories. I have the following project
project
│ README.md
│ __init__.py
│
└───common
│ __init__.py
│ └───my_func
│ │ __init__.py
│ │ func1.py
│ │ func2.py
│
└───scripts
│ __init__.py
│ └───folder_1
│ │ __init__.py
│ │ code1.py
│ │ code2.py
│
│ └───folder_2
│ │ __init__.py
│ │ code3.py
│ │ code4.py
│ │
│ └───tests
│ │ └───test1
│ │ │ __init__.py
│ │ │ test_code3.py
│ │ │ test_code4.py
│ │ │
│ │ └───test2
│ │ │ __init__.py
│ │ │ test_code3.py
│ │ │ test_code4.py
I set the working directory to be ./project. In my code1.py file I import common.my_func.func1 and it runs normally.
Now, I am trying to implement some unittest functions in ./project/scripts/tests. To do so, I am using the coverage package and running the command: coverage run --source=scripts -m unittest discover scripts/tests. However, when doing so, I get the following error:
ModeluNotFoundError: No module named common.my_func
Weirdly speaking, the scripts works perfectly when I try to run it for one test folder only, and not the whole folder coverage run --source=scripts -m unittest discover scripts/tests/test1.
I tried multiple combinations of removing the source, getting more specific with the folder and on. Have any of you faced similar problems with python 3.8?
Thank you very much in advance,

Is there a way to remove the side navigation bar containing file names in the latest version of streamlit?

I recently upgraded streamlit to version 1.10.0. After upgradation I ran streamlit only to find that there is a side navigation bar displaying python file names present in my directory. I have attached images for the same.
import streamlit as st
import pandas as pd
from services.multiapp import MultiApp
from functools import partial
from pages import show, analysis, help, custom, download, three_d
def refresh():
pass
file = st.file_uploader('Upload CSV file',
type='csv', help="Format")
if file is not None:
# verification
# Prediction
st.button("Refresh", on_click=refresh)
data = pd.read_csv(file)
app = MultiApp()
app.add_app('Show data', partial(show.app, data))
app.add_app('Analysis', analysis.app)
app.add_app('Custom Plots', custom.app)
app.add_app('3-D Plots', partial(three_d.app, data))
app.add_app('Download', partial(download.app, data, file.name))
app.add_app('Help', help.app)
app.run()
Please help me to remove the side navigation bar.
My current working directory has the following structure
├── app.py
├── pages
│ ├── analysis.py
│ ├── custom.py
│ ├── download.py
│ ├── help.py
│ ├── __pycache__
│ │ ├── analysis.cpython-38.pyc
│ │ ├── analysis.cpython-39.pyc
│ │ ├── custom.cpython-38.pyc
│ │ ├── custom.cpython-39.pyc
│ │ ├── download.cpython-38.pyc
│ │ ├── download.cpython-39.pyc
│ │ ├── get_df.cpython-38.pyc
│ │ ├── help.cpython-38.pyc
│ │ ├── help.cpython-39.pyc
│ │ ├── iplots.cpython-38.pyc
│ │ ├── plot.cpython-38.pyc
│ │ ├── show.cpython-38.pyc
│ │ ├── show.cpython-39.pyc
│ │ ├── show_d.cpython-38.pyc
│ │ ├── three_d.cpython-38.pyc
│ │ └── three_d.cpython-39.pyc
│ ├── show.py
│ └── three_d.py
├── __pycache__
│ ├── multiapp.cpython-38.pyc
│ └── plots.cpython-38.pyc
├── README.md
├── services
│ ├── multiapp.py
│ ├── __pycache__
│ │ ├── multiapp.cpython-38.pyc
│ │ └── multiapp.cpython-39.pyc
│ └── services.py
├── Templates
│ ├── links.txt
│ ├── multi-page-app-main
│ │ ├── app.py
│ │ ├── apps
│ │ │ ├── data.py
│ │ │ ├── home.py
│ │ │ ├── model.py
│ │ │ └── __pycache__
│ │ │ ├── data.cpython-38.pyc
│ │ │ ├── home.cpython-38.pyc
│ │ │ └── model.cpython-38.pyc
│ │ ├── multiapp.py
│ │ ├── __pycache__
│ │ │ └── multiapp.cpython-38.pyc
│ │ ├── README.md
│ │ └── requirements.txt
│ ├── streamlit-dashboard-template-main
│ │ ├── app.py
│ │ ├── dashboard.py
│ │ ├── README.md
│ │ ├── stdashdark.png
│ │ └── stdashlight.png
│ ├── streamlit-geospatial-master
│ │ ├── app.py
│ │ ├── apps
│ │ │ ├── basemaps.py
│ │ │ ├── census.py
│ │ │ ├── cesium.py
│ │ │ ├── deck.py
│ │ │ ├── device_loc.py
│ │ │ ├── gee_datasets.py
│ │ │ ├── gee.py
│ │ │ ├── heatmap.py
│ │ │ ├── home.py
│ │ │ ├── housing.py
│ │ │ ├── plotly_maps.py
│ │ │ ├── raster.py
│ │ │ ├── rois.py
│ │ │ ├── timelapse.py
│ │ │ ├── vector.py
│ │ │ ├── wms.py
│ │ │ └── xy.py
│ │ ├── data
│ │ │ ├── cog_files.txt
│ │ │ ├── html
│ │ │ │ └── sfo_buildings.html
│ │ │ ├── realtor_data_dict.csv
│ │ │ ├── us_counties.geojson
│ │ │ ├── us_metro_areas.geojson
│ │ │ ├── us_nation.geojson
│ │ │ └── us_states.geojson
│ │ ├── environment-bk.yml
│ │ ├── index.html
│ │ ├── LICENSE
│ │ ├── multiapp.py
│ │ ├── packages.txt
│ │ ├── Procfile
│ │ ├── README.md
│ │ ├── requirements.txt
│ │ └── setup.sh
│ ├── Streamlit-master
│ │ ├── app.py
│ │ ├── data
│ │ │ ├── demo.wav
│ │ │ ├── Salary_Data.csv
│ │ │ ├── sal.jpg
│ │ │ ├── snippets
│ │ │ └── virtual.mp4
│ │ ├── data.py
│ │ ├── demo.py
│ │ ├── layout.py
│ │ ├── plots.py
│ │ ├── Procfile
│ │ ├── README.md
│ │ ├── requirements.txt
│ │ ├── sidebar.py
│ │ └── widget.py
│ └── streamlit template
│ ├── JC-202103-citibike-tripdata.csv
│ ├── logo.png
│ ├── requirements.txt
│ └── streamlit_template.py
├── test.csv
└── test.py

Streamlit sidebar picks everything up that is located in the "pages" folder (see here: Blog Streamlit)
You could either rename your pages folder to something else or if you like the side bar, you can just collapse it initially:
import streamlit as st
st.set_page_config(initial_sidebar_state="collapsed")
That way the sidebar will be closed on start but can still be opened if needed.

A little late to this question, but I added the following snippet to the pages that I wanted to hide the lists on.
no_sidebar_style = """
<style>
div[data-testid="stSidebarNav"] {display: none;}
</style>
"""
st.markdown(no_sidebar_style, unsafe_allow_html=True)
This got rid of the Streamlit generated page components on the sidebar and let me keep all my other components.

Add this to all the pages you want to hide the page list from, but there should not be any ‘ul’ in your page except for the page list.
st.markdown("<style> ul {display: none;} </style>", unsafe_allow_html=True)

Python Setuptools including data with a specific folder structure

I've done a bit of searching and can't find any examples that match my scenario.
I'm trying to create a x-lang module to return some data that we use internally.
My git repo would contain both the powershell and python version of the module, and each module would reference the static data.
I'm struggling to package it in a way that doesn't have a wierd folder structure and works from a python packaging point of view:
the folder structure looks like this (anonymised)
<Root of Repo>
│ .gitignore
│ MANIFEST.in
│ pyproject.toml
│ README.md
│ requirements.txt
│ setup.py
│
├───data
│ 1.json
│ 2.json
│ 3.json
│ 4.json
│
├───src
│ ├───powershell
│ │ │ readme.md
│ │ │ test.ps1
│ │ │ xlangTest.psd1
│ │ │
│ │ ├───classes
│ │ ├───private
│ │ │ get-something1.ps1
│ │ │
│ │ └───public
│ │ Get-DataExample.ps1
│ │
│ └───python
│ │ 1.py
│ │ 2.py
│ │ 3.py
│ │ 4.py
│ │ 5.py
│ │ 6.py
│ │ 7.py
│ │ 8.py
│ │ 9.py
│ │ __init__.py
│ │
│ └───ff.ff.ff.egg-info
│ dependency_links.txt
│ PKG-INFO
│ SOURCES.txt
│ top_level.txt
│
├───tests
│ ├───powershell
│ │ Get-DataExample.tests.ps1
│ │ get-something1.tests.ps1
│ │
│ └───python
and my setup.py looks like this:
import setuptools
with open("README.md", "r", encoding="utf-8") as fh:
long_description = fh.read()
setuptools.setup(
name="ff.ff.ff",
version="0.0.1",
author="ffff",
author_email="david.wallis#fff.fff",
description="An xlang module test",
long_description=long_description,
long_description_content_type="text/markdown",
url="https://ffff/ffff",
project_urls={
"Bug Tracker": "https://ffff/ffff",
},
classifiers=[
"Programming Language :: Python :: 3",
"Operating System :: OS Independent",
],
package_dir={"": "src"},
# data_files={
# "data":['data/*.json'],
# },
# include_package_data = True,
# package_data = {
# "data" : ["../data/*.json"]
# },
packages=setuptools.find_packages(where="src"),
# install_requires=[
# 'readContent',
# 'json',
# ],
python_requires=">=3.6",
)
manifest.IN contains:
graft data
when I package it I end up with the following in the tar file within the tar.gz
C:.
│ ff.ff.ff-0.0.1.tar
│
└───ff.ff.ff-0.0.1
│ #PaxHeader
│
└───ff.ff.ff-0.0.1
│ MANIFEST.in
│ PKG-INFO
│ pyproject.toml
│ README.md
│ setup.cfg
│ setup.py
│
├───data
│ 1.json
│ 2.json
│ 3.json
│ 4.json
│
└───src
├───python
│ 1.py
│ 2.py
│ 3.py
│ 4.py
│ 5.py
│ 6.py
│ 7.py
│ 8.py
│ 9.py
│ __init__.py
│
└───ff.ff.ff.egg-info
dependency_links.txt
PKG-INFO
SOURCES.txt
top_level.txt
I guess what I'm looking for is a sanity check, is this ok, or can I some how flatten the src/python in the distribution or even rename to the module name?
and I suppose the key point here is that I want the data to be common across the two modules. and ultimatley I want to do the same for test cases next.

Can you help me to resolve the error I am getting after running Stringtie prepDE.py?

When I run prepDE.py job exit with the following error. I am using StringTie/2.1.4 with Python/2.7.18.
Following is the command I was running (inside ADA cluster)
#Loading New modules
module load StringTie/2.1.4-GCC-9.3.0
module load Python/2.7.18-GCCcore-9.3.0
#converting data to DESEQ2 compatible format
prepDE.py
Traceback (most recent call last): File
"/sw/eb/software/StringTie/2.1.4-GCC-9.3.0/bin/prepDE.py", line 58, in
samples = [(i,glob.iglob(os.path.join(opts.input,i,"*.gtf")).next())
for i in next(os.walk(opts.input))1 if re.search(opts.pattern,i)]
StopIteration
I have a directory structure as follows Directory structure
├── ballgown
│ ├── PVX_2d_1
│ │ ├── e2t.ctab
│ │ ├── e_data.ctab
│ │ ├── i2t.ctab
│ │ ├── i_data.ctab
│ │ ├── PVX_2d_1.gtf
│ │ └── t_data.ctab
│ ├── PVX_2d_2
│ │ ├── e2t.ctab
│ │ ├── e_data.ctab
│ │ ├── i2t.ctab
│ │ ├── i_data.ctab
│ │ ├── PVX_2d_2.gtf
│ │ └── t_data.ctab
│ ├── PVX_2d_3
│ │ ├── e2t.ctab
│ │ ├── e_data.ctab
│ │ ├── i2t.ctab
│ │ ├── i_data.ctab
│ │ ├── PVX_2d_3.gtf
│ │ └── t_data.ctab
Can you help me in this regard?
Thanks in advance.

This is an educated guess based on the error and the documentation linked:
-i INPUT, --input=INPUT, --in=INPUT
– a folder containing all sample sub-directories, or a text file with sample ID and path to its GTF file on each line [default: . ]
Try invoking the script with
prepDE.py -i ballgown
since the data subfolders are in a ballgown folder.