Develop Reference

Why does Colab not find the parent directory for relative imports?

I have a python file that I tested locally using VS-Code and the shell. The file contains relative imports and worked as intended on my local machine.
After uploading the associated files to Colab I did the following:
py_file_location = '/content/gdrive/content/etc'
os.chdir(py_file_location)
# to verify whether I got the correct path
!python3
>>> import os
>>> os.getcwd()
output: '/content/gdrive/MyDrive/content/etc'
However, when I run the file I get the following error:
ImportError: attempted relative import with no known parent package
Why is that? Using the same file system and similar shell commands the file worked locally.

A solution that worked for me was to turn my relative paths into static ones, so instead of using:
directory = pathlib.Path(__file__).parent
sys.path.append(str(directory.parent))
sys.path.append(str(directory.parent.parent.parent))
__package__ = directory.name
I needed to make the path static by using resolve():
directory = pathlib.Path(__file__).resolve().parent
sys.path.append(str(directory.parent))
sys.path.append(str(directory.parent.parent.parent))
__package__ = directory.name
It is however still not fully clear to me why this is required when running on Colab but not when running locally.

Trying to search for a file in Python 3.7, but can't get out of current working directory

Playing around with Python and trying to use os.walk() to search for files around my drives:
import os
def find_files(filename, search_path):
result = []
for root, dir, files in os.walk(search_path, topdown=False):
print('Root: ' + root)
print(dir)
if filename in files:
result.append((os.path.join(root, filename)))
return result
When I run the function, it doesn't matter what I put in the search_path, it just goes to the current working directory instead, like os.walk is ignoring everything except the Python directory. Am I doing something wrong?
Using PyCharm 2019 2.5 Community for an IDE and then I tested it in the Python 3.7 terminal and got the same thing.

Your code is correct - it works fine here on Windows and Python 3.8.2. I suspect you're not passing the path correctly - you say in your comment you tried "C:". That is not the path to the C drive, you want "C:\\".

Code works in Visuall Studio Coe when i use debug but not when i run in terminal python

When i run in terminal i get the [Errno 2] No such file or directory error, but when i use debug mode it works.
the error occurs here,
list = open(filename, "r")
the code also works with IDLE

Your debugger is most likely running in a different location than your terminal. cd your terminal to where the file is located, or try using the absolute path instead of a relative one.

If you add these few lines in your code and run it using your IDE or Terminal, you'll notice the difference:
import os
curdir = os.getcwd()
print('My current directory is {}'.format(curdir))
fullpath_to_filename = os.path.join(curdir, filename)
print('When I run `open(filename)`, python sees: {}'.format(fullpath_to_filename))
print('This filepath is {}valid'.format('' if os.path.exists(fullpath_to_filename) else 'not '))
You'll likely experience something similar to below:
# on IDE:
My current directory is c:\Users\me\Projects\
When I run `open(filename)`, python sees: c:\Users\me\Projects\test.txt
This filepath is valid
# on Terminal:
My current directory is c:\Programs\Python38\
When I run `open(filename)`, python sees: c:\Programs\Python38\test.txt
This filepath is not valid
Solution: use absolute path for your filename, or os.chdir(...) to the correct parent path before location filename. I would recommend the former to avoid managing directories.

How to add path to os.environ["PATH"] for python in Windows 10?

I have a script which has an error because 'neato.exe' can't be found in paths. When I look at os.environ["PATH"], indeed C:\\Program Files (x86)\\Graphviz2.38\\bin, the path to neato.exe, is not there. I can do a hack for the time being by adding this line, but this doesn't seem satisfactory.
if not 'C:\\Program Files (x86)\\Graphviz2.38\\bin' in os.environ["PATH"]:
os.environ["PATH"] += os.pathsep + 'C:\\Program Files (x86)\\Graphviz2.38\\bin'
Nonetheless, it shows that the error ValueError("Program %s not found in path." neato.exe) is an accurate error. The script works when I add the path to Neato. I added C:\Program Files (x86)\Graphviz2.38\bin to my Environmental Variables in windows, but to no avail. And I also noticed that there are only a few paths in my Path Env. Vars., not the many that python lists. I am using python 3.7 and also running it using the anaconda navigator. I would like to make a more permanent change so I don't have to edit every script that looks for neato.exe with the silly if statement above. Does anyone know how to change what's in os.environ["PATH"] in anaconda?
I am using networkx, networkx.drawing.nx_agraph.to_agraph. The script agraph.py has this function (_which()), which need to make a path match or it will throw an error.
def _which(self, name):
"""Searches for name in exec path and returns full path"""
import os
import glob
paths = os.environ["PATH"]
if os.name == "nt":
exe = ".exe"
else:
exe = ""
for path in paths.split(os.pathsep):
match = glob.glob(os.path.join(path, name + exe))
if match:
return match[0]
raise ValueError("No prog %s in path." % name)

A few things to note about Windows Path:
Windows Path is a combination of user defined and system defined - the former is appended to the latter
Windows Path has a pretty short length limit
Unfortunately because of the above, you'll probably want to temporarily change the Path for your program to ensure it can find the binary you're looking for.
You could set the PATH environment variable with the Graphviz bin directory at the beginning before executing your script(s).

easy way of installing python apps without using PYTHON path or muli symlink in site-package

I didn't want to install python modules using easy install, symlinks in site-packages or PYTHONPATH.
So, I am trying something that I do wants system wide, then any application installation is done locally. Note, the root password is required only once here.
First create a symblink of.../pythonX.Y/site-packages/mymodules -> /home/me/lib/python_related
So, I create a directory called
/home/me/lib/python_related/
In there:
/home/me/lib/python_related
/home/me/lib/python_related/__init__.py
/home/me/lib/python_related/django_related/
/home/me/lib/python_related/django_related/core
/home/me/lib/python_related/django_related/core/Django1.0
/home/me/lib/python_related/django_related/core/Django1.1
/home/me/lib/python_related/django_related/core/mycurrent_django -> Django1.1/django
/home/me/lib/python_related/django_related/apps
/home/me/lib/python_related/django_related/apps/tagging
/home/me/lib/python_related/django_related/apps/tagging/django-tagging-0.2
/home/me/lib/python_related/django_related/apps/tagging/django-tagging-0.3
/home/me/lib/python_related/django_related/apps/tagging/mycurrent_tagging -> django-tagging-0.3
Now, here is the content of:
/home/me/lib/python_related/__init__.py
==========================================
import sys, os
# tell us where you keep all your modules and this didn't work as it gave me
# the location of the site-packages
#PYTHON_MODULE_PATH = os.path.dirname(__file__)
PYTHON_MODULE_PATH = "/home/me/libs/python_bucket"
def run_cmd(cmd):
"""
Given a command name, this function will run the command and returns the output
in a list.
"""
output = []
phdl = os.popen(cmd)
while 1:
line = phdl.readline()
if line == "":
break
output.append(line.replace("\n", ""))
return output
def install():
"""
A cheesy way of installing and managing your python apps locally without
a need to install them in the site-package. All you'd need is to install
the directory containing this file in the site-package and that's it.
Anytime you have a python package you want to install, just put it in a
proper sub-directory and make a symlink to that directory called mycurrent_xyz
and you are done. (e.g. mycurrent_django, mycurrent_tagging .. etc)
"""
cmd = "find %s -name mycurrent_*" % PYTHON_MODULE_PATH
modules_to_be_installed = run_cmd(cmd)
sys.path += modules_to_be_installed
install()
=======================================================
Now in any new python project, just import your mymodules and that pulls in any apps that you have in the above directory with the proper symbolic link. This way you can have multiple copies of apps and just use the mycurrent_xyz to the one you want to use.
Now here is question. Is this a good way of doing it?

Have a look at virtualenv.
It may do what you are after.