Develop Reference

`importlib` not utilising/recognising path

I am trying to import modules while running my main python script, using a smaller setup.py script. However the importlib command: importlib.util.spec_from_file_location(name, location) doesn't appear to be detecting my small python script. Presumably I'm not filling in the name or location fields correctly.
Example Script A (setup.py):
import os
import pandas as pd
print("success!") # So I can see it has run.
Example Script B (my_script.py):
import importlib
setup_path = ("/home/solebay/My Project Name/")
start_up_script = importlib.util.spec_from_file_location("setup.py", setup_path)
module = importlib.util.module_from_spec(start_up_script)
Running the above snippet returns:
AttributeError: 'NoneType' object has no attribute 'loader'
I subsequently investigated by running type(start_up_script) the result it gives is typeNone.
The paths are correct. I verified this by running the following:
"/home/solebay/My Project Name/"
sudo python3 "/home/solebay/My Project Name/setup.py"
These printed the messages is a directory and success! respectively.
Note: Maurice Meyer succeeded in answering my main question, and so I have marked it as correct. However, I have not achieved my main goal; namely importing modules via another script. So if that is your aim, this question might not be appropriate for you.

The location argument passed to spec_from_file_location has to be the full path to the python script:
import importlib.util
spec = importlib.util.spec_from_file_location(
name='something__else', # name is not related to the file, it's the module name!
location='/tmp/solebay/My Project Name/setup.py' # full path to the script
)
my_mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(my_mod)
print(my_mod)
Out:
success!
<module 'something__else' from '/tmp/solebay/My Project Name/setup.py'>

Cannot set pythonpath environment variable from python

I'm trying to debug a project that has a lot of additional libraries added to PYTHONPATH at runtime before launching the python file.
I was not able to add those commands with tasks.json file prior to debugging python file in Visual Studio code (see post Visual Studio Code unable to set env variable paths prior to debugging python file), so I'm just adding them via an os.system("..") command
I'm only showing 1 of the libraries added below:
# Standard library imports
import os
import sys
os.system("SET PYTHONPATH=D:\\project\\calibration\\pylibrary\\camera")
# Pylibrary imports
from camera import capture
When I debug, it fails on line from camera import capture with:
Exception has occurred: ModuleNotFoundError
No module named 'camera'
File "D:\project\main.py", line 12, in <module>
from camera.capture import capture
I also tried
os.environ['PYTHONPATH']="D:\\project\\pylibrary\\camera" and I still get the same error
Why is it not remembering the pythonpath while running the script?
How else can I define the pythonpath while running Visual Studio Code and debugging the project file?
I know I can add the pythonpath to env variables in windows, but it loads too many libraries and I want it to only remember the path while the python script is executed.
Thanks

Using os.system() won't work because it starts a new cmd.exe shell and sets the env var in that shell. That won't affect the env vars of the python process. Assigning to os.environ['PYTHONPATH'] won't work because at that point your python process has already cached the value, if any, of that env var in the sys.path variable. The solution is to
import sys
sys.path.append(r"D:\project\calibration\pylibrary\camera")

Run all Python scripts in a folder - from Python

How can I write a Python program that runs all Python scripts in the current folder? The program should run in Linux, Windows and any other OS in which python is installed.
Here is what I tried:
import glob, importlib
for file in glob.iglob("*.py"):
importlib.import_module(file)
This returns an error: ModuleNotFoundError: No module named 'agents.py'; 'agents' is not a package
(here agents.py is one of the files in the folder; it is indeed not a package and not intended to be a package - it is just a script).
If I change the last line to:
importlib.import_module(file.replace(".py",""))
then I get no error, but also the scripts do not run.
Another attempt:
import glob, os
for file in glob.iglob("*.py"):
os.system(file)
This does not work on Windows - it tries to open each file in Notepad.

You need to specify that you are running the script through the command line. To do this you need to add python3 plus the name of the file that you are running. The following code should work
import os
import glob
for file in glob.iglob("*.py"):
os.system("python3 " + file)
If you are using a version other than python3, just change the argument from python3 to python

Maybe you can make use of the subprocess module; this question shows a few options.
Your code could look like this:
import os
import subprocess
base_path = os.getcwd()
print('base_path', base_path)
# TODO: this might need to be 'python3' in some cases
python_executable = 'python'
print('python_executable', python_executable)
py_file_list = []
for dir_path, _, file_name_list in os.walk(base_path):
for file_name in file_name_list:
if file_name.endswith('.csv'):
# add full path, not just file_name
py_file_list.append(
os.path.join(dir_path, file_name))
print('PY files that were found:')
for i, file_path in enumerate(py_file_list):
print(' {:3d} {}'.format(i, file_path))
# call script
subprocess.run([python_executable, file_path])
Does that work for you?
Note that the docs for os.system() even suggest using subprocess instead:
The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function.

If you have control over the content of the scripts, perhaps you might consider using a plugin technique, this would bring the problem more into the Python domain and thus makes it less platform dependent. Take a look at pyPlugin as an example.
This way you could run each "plugin" from within the original process, or using the Python::multiprocessing library you could still seamlessly use sub-processes.

os.makedirs() / os.mkdir() on Windows. No error, but no (visible) folder created

I just try to create some new folders with Python's (3.7.3) os.makedirs() and os mkdir().
Apparently, it works fine because no error occurs (Win 10 Home). But as I try to find the created folder in the windows explorer it isn't there.
Trying to create it again with python I get an error:
'[WinError 183] Cannot create a file when that file already exists:'
Strange thing is, all this worked fine on my computer at work (Wind 10 too), and also on my android tablet.
I've already tried to use relative & absolute paths in makedirs / mkdir.
Ok, it's all about these few lines:
import os
# print(os.getcwd()) shows: C:\Users\Andrej\Desktop
# tried relative path..
os.makedirs('Level1/Level2/Level3')
# tried out some absolute paths like...
os.makedirs('C:/Users/Andrej/Desktop/Level1/Level2/Level3')
os.makedirs('C:\\Users\\Andrej\\Desktop\\Level1\\Level2\\Level3')
UPDATE: It works perfectly when I write makedirs command directly in the Windows powershell. The issue above only occurs when I write this code in Visual Code Studio and then start the file from the powershell by typing in: python makedirs.py...

I just had the same thing happen to me, except I was creating a folder in the AppData/Roaming directory and was using the version of Python from the Microsoft Store.
Apparently, files and directories created in AppData using that version of Python, will instead be created in:
%LocalAppData%\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache

I found what causes this problem finally!
All of the created test folders have been moved to a folder named VTROOT. This folder is created by Comodo Internet Security I use...Its "Auto-Containment" moved all folders I set up by running the code from .py-file (only then) in the PowerShell.
Disabling the auto-containment of Comodo solves the problem..Oh my..
Thank you anyways!

the "if" part will check if the file exists already and consequentially solve the ERROR: '[WinError 183] Cannot create a file when that file already exists:'
the "import sys" part will fix the issue with windows where the folder is created but not visible by the user (even with show hidden files turned on)
import sys
import os
path = 'your path'
def make_dir():
if not os.path.exists(Path):
os.makedirs(Path)
or
import sys
import os
path = 'your path'
def make_dir():
mode = 0o666 #read and write for all users and groups
# form more permission codes search for: *chmod number codes*
if not os.path.exists(Path):
os.mkdir(Path, mode)

Try adding the below import statement:
import sys
This should work.

Pydevd with virtual code or (source provider)

we´re having python source code stored in a sql database, the code is build together to a virtual python module and can be executed.
We want to debug this modules but then of course the Eclipse debugger host doesnt know where to find the source code for these modules.
Is there a way to provide pydevd with the location of the source code, even if that means to write down the files to disk?

Write it to the disk and when doing the compile pass the filename for the code (and, when you're not in debug mode, just don't write it and pass '<string>' as the filename).
See the example below:
from tempfile import mktemp
my_code = '''
a = 10
print a
'''
tmp_filename = mktemp('.py', 'temp_file_')
with open(tmp_filename, 'w') as f:
f.write(my_code)
obj = compile(my_code, tmp_filename, 'exec')
exec obj #Place breakpoint here: when stepping in it should get to the code.

You need to add module to PYTHONPATH in Eclipse project settings and import it using the standard Python import. Then PyDev debugger should find it without any problems.