Develop Reference

Python using os and chdir to read and write to iCloud

I am using the following python code to access a folder in iCloud. I am getting an error of:
FileNotFoundError: [Errno 2] No such file or directory:
import os
os.chdir('/Users/me/Library/Mobile\ Documents/com~apple~CloudDocs/jupyter/')
Is there something I am missing?

There is no backslash in the directory name; you aren't using a shell, so the space does not need to be escaped.
os.chdir('/Users/me/Library/Mobile Documents/com~apple~CloudDocs/jupyter/')

Clarify how to specify file paths inside Docker coming from Python

I have created a Docker image with for a Flask app. Inside my Flask app, this is how I specify the file paths.
dPath = os.getcwd() + "\\data\\distanceMatrixv2.csv"
This should ideally resolve in a filepath similar to app/data/distanceMatrixv2.csv. This works when I do a py main.py run in the CMD. After making sure I am in the same directory, creating the image and running the Docker image via docker run -it -d -p 5000:5000 flaskapp, it throws me the error below when I try to do any processing.
FileNotFoundError: /backend\data\distanceMatrixv2.csv not found.
I believe this is due to how Docker resolves file paths but I am a bit lost on how to "fix" this. I am using Windows while my Docker image is built with FROM node:lts-alpine.

node:lts-alpine is based on Alpine which is a Linux distro. So your python program runs under Linux and should use Linux file paths, i.e. forward slashes like
dPath = os.getcwd() + "/data/distanceMatrixv2.csv"
It looks like os.getcwd() returns /backend. So the full path becomes /backend/data/distanceMatrixv2.csv.

The os.path library can do this portably across operating systems. If you write this as
import os
import os.path
dPath = os.path.join(os.getcwd(), "data", "distanceMatrixv2.csv")
it will work whether your (Windows) system uses DOS-style backslash-separated paths or your (Linux) system uses Unix-style forward slashes.
Python 3.4 also added a pathlib library, if you prefer a more object-oriented approach. This redefines the Python / division operator to concatenate paths, using the native path separator. So you could also write
from pathlib import Path
dPath = Path.cwd() / "data" / "distanceMatrixv2.csv"

Why does Colab not find the parent directory for relative imports?

I have a python file that I tested locally using VS-Code and the shell. The file contains relative imports and worked as intended on my local machine.
After uploading the associated files to Colab I did the following:
py_file_location = '/content/gdrive/content/etc'
os.chdir(py_file_location)
# to verify whether I got the correct path
!python3
>>> import os
>>> os.getcwd()
output: '/content/gdrive/MyDrive/content/etc'
However, when I run the file I get the following error:
ImportError: attempted relative import with no known parent package
Why is that? Using the same file system and similar shell commands the file worked locally.

A solution that worked for me was to turn my relative paths into static ones, so instead of using:
directory = pathlib.Path(__file__).parent
sys.path.append(str(directory.parent))
sys.path.append(str(directory.parent.parent.parent))
__package__ = directory.name
I needed to make the path static by using resolve():
directory = pathlib.Path(__file__).resolve().parent
sys.path.append(str(directory.parent))
sys.path.append(str(directory.parent.parent.parent))
__package__ = directory.name
It is however still not fully clear to me why this is required when running on Colab but not when running locally.

os.makedirs() / os.mkdir() on Windows. No error, but no (visible) folder created

I just try to create some new folders with Python's (3.7.3) os.makedirs() and os mkdir().
Apparently, it works fine because no error occurs (Win 10 Home). But as I try to find the created folder in the windows explorer it isn't there.
Trying to create it again with python I get an error:
'[WinError 183] Cannot create a file when that file already exists:'
Strange thing is, all this worked fine on my computer at work (Wind 10 too), and also on my android tablet.
I've already tried to use relative & absolute paths in makedirs / mkdir.
Ok, it's all about these few lines:
import os
# print(os.getcwd()) shows: C:\Users\Andrej\Desktop
# tried relative path..
os.makedirs('Level1/Level2/Level3')
# tried out some absolute paths like...
os.makedirs('C:/Users/Andrej/Desktop/Level1/Level2/Level3')
os.makedirs('C:\\Users\\Andrej\\Desktop\\Level1\\Level2\\Level3')
UPDATE: It works perfectly when I write makedirs command directly in the Windows powershell. The issue above only occurs when I write this code in Visual Code Studio and then start the file from the powershell by typing in: python makedirs.py...

I just had the same thing happen to me, except I was creating a folder in the AppData/Roaming directory and was using the version of Python from the Microsoft Store.
Apparently, files and directories created in AppData using that version of Python, will instead be created in:
%LocalAppData%\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache

I found what causes this problem finally!
All of the created test folders have been moved to a folder named VTROOT. This folder is created by Comodo Internet Security I use...Its "Auto-Containment" moved all folders I set up by running the code from .py-file (only then) in the PowerShell.
Disabling the auto-containment of Comodo solves the problem..Oh my..
Thank you anyways!

the "if" part will check if the file exists already and consequentially solve the ERROR: '[WinError 183] Cannot create a file when that file already exists:'
the "import sys" part will fix the issue with windows where the folder is created but not visible by the user (even with show hidden files turned on)
import sys
import os
path = 'your path'
def make_dir():
if not os.path.exists(Path):
os.makedirs(Path)
or
import sys
import os
path = 'your path'
def make_dir():
mode = 0o666 #read and write for all users and groups
# form more permission codes search for: *chmod number codes*
if not os.path.exists(Path):
os.mkdir(Path, mode)

Try adding the below import statement:
import sys
This should work.

How to load jar file and support files placed at different location

I am creating a python program that runs a jar file. The jar file and some support files are placed in a different location than the python program's directory. I tried adding jar file path to sys.path but it's unable to access the file from there, however the path is added to sys.path correctly. How can I get this working?
jar file location: E:\data
python file location: C:\Users\user\Desktop
I am using subprocess to call the jar file, the code looks like:
import os
import sys
import subprocess as sp
class abc():
def __init__(self):
sys.path.append(r'E:\data')
def run(self):
print sys.path
env = dict(os.environ)
env['JAVA_OPTS'] = '-Xms256m -Xmx256m -Xss1024k'
sp.call(['java', '-jar', 'file.jar'], env=env)
if __name__ == '__main__':
o = abc()
o.run()
After running above code, I get an error saying:
Error: Unable to access jarfile file.jar

What if you just change your working directory:
import os
cwd = os.getcwd() #current directory
os.chdir('path/to/jar')
... # run file
...
os.chdir(cwd)

sys.path and PYTHONPATH are used when importing python modules
When executing commands, the operating system lookup the command in its system path (%PATH% on Windows).
There is no lookup path for data / filenames passed as argument.
When using sp.call() the system path lookup uses whatever directory the script has been launched from. So you need to either change dir to E:\DATA or use the absolute path:
sp.call(['java', '-jar', 'E:\DATA\file.jar'], env=env)
There are plenty of env variable on Windows: https://en.wikipedia.org/wiki/Environment_variable#Default_values