the main module sits within the runner package and executes stuff in the other packages. The main module can also Update the other packages and when that happens I want to reload them in order to get the new functions/modules that were added to those packages.
Project Structure
|--runner
|----main.py
|--core
|----module_1.py
|--configurations
|--utils
But that doesn't work.
I tried the following commands:
importlib.reload - only reloads a single module, using it recursively with sys.modules didn't add the new modules to the import tree. example: if after the update, "core" received a new module "module_new.py" and its imported in "module_1.py" it's not recognized after the reload.
I tried using IPython.lib.deepreload - it didn't work as well.
I've been stuck with this issue for some time, and haven't found any working solution yet.
Suggestions? Thanks
I fixed the issue by restarting the entire program using a while loop from an outer execution script.
Exit code 2: update required
Do
{
$process = Start-Process python -ArgumentList $CommandLine -verb RunAs -PassThru -WindowStyle Minimized -Wait
} WHILE ($process.ExitCode -eq 2)
Modules will be reloaded by import command if they are not in sys.modules dict
# import some standard (non-updatable) modules
import numpy as np
# save set of non-reloadable modules on first run,
# and delete reloadable modules on other runs
if 'init_modules' not in globals():
init_modules = set(sys.modules.keys())
else:
modules = list(sys.modules.keys())
for m in modules:
if m not in init_modules:
del(sys.modules[m])
# import reloadable packages and modules
import MyPackage
Related
I am trying to import modules while running my main python script, using a smaller setup.py script. However the importlib command: importlib.util.spec_from_file_location(name, location) doesn't appear to be detecting my small python script. Presumably I'm not filling in the name or location fields correctly.
Example Script A (setup.py):
import os
import pandas as pd
print("success!") # So I can see it has run.
Example Script B (my_script.py):
import importlib
setup_path = ("/home/solebay/My Project Name/")
start_up_script = importlib.util.spec_from_file_location("setup.py", setup_path)
module = importlib.util.module_from_spec(start_up_script)
Running the above snippet returns:
AttributeError: 'NoneType' object has no attribute 'loader'
I subsequently investigated by running type(start_up_script) the result it gives is typeNone.
The paths are correct. I verified this by running the following:
"/home/solebay/My Project Name/"
sudo python3 "/home/solebay/My Project Name/setup.py"
These printed the messages is a directory and success! respectively.
Note: Maurice Meyer succeeded in answering my main question, and so I have marked it as correct. However, I have not achieved my main goal; namely importing modules via another script. So if that is your aim, this question might not be appropriate for you.
The location argument passed to spec_from_file_location has to be the full path to the python script:
import importlib.util
spec = importlib.util.spec_from_file_location(
name='something__else', # name is not related to the file, it's the module name!
location='/tmp/solebay/My Project Name/setup.py' # full path to the script
)
my_mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(my_mod)
print(my_mod)
Out:
success!
<module 'something__else' from '/tmp/solebay/My Project Name/setup.py'>
I'm trying to debug a project that has a lot of additional libraries added to PYTHONPATH at runtime before launching the python file.
I was not able to add those commands with tasks.json file prior to debugging python file in Visual Studio code (see post Visual Studio Code unable to set env variable paths prior to debugging python file), so I'm just adding them via an os.system("..") command
I'm only showing 1 of the libraries added below:
# Standard library imports
import os
import sys
os.system("SET PYTHONPATH=D:\\project\\calibration\\pylibrary\\camera")
# Pylibrary imports
from camera import capture
When I debug, it fails on line from camera import capture with:
Exception has occurred: ModuleNotFoundError
No module named 'camera'
File "D:\project\main.py", line 12, in <module>
from camera.capture import capture
I also tried
os.environ['PYTHONPATH']="D:\\project\\pylibrary\\camera" and I still get the same error
Why is it not remembering the pythonpath while running the script?
How else can I define the pythonpath while running Visual Studio Code and debugging the project file?
I know I can add the pythonpath to env variables in windows, but it loads too many libraries and I want it to only remember the path while the python script is executed.
Thanks
Using os.system() won't work because it starts a new cmd.exe shell and sets the env var in that shell. That won't affect the env vars of the python process. Assigning to os.environ['PYTHONPATH'] won't work because at that point your python process has already cached the value, if any, of that env var in the sys.path variable. The solution is to
import sys
sys.path.append(r"D:\project\calibration\pylibrary\camera")
I would like to debug some of the basic packages that come with the Python install and/or are built-in packages, including pip and venv.
The desire comes from an error message of file permissions (unable to access a file with an "unprintable file name") some of my team is getting running these commands - see this question for details.
Question
How do you debug the Python source code when trying to catch issues in the main python executable, or when directly running a base python module (see following examples for pip and venv)?
$ python -m pip install --upgrade
$ python -m venv .venv
If it matters, my environment is VSCode, where I am happily able to engage the debugger on any custom script I have written, using the built-in debugger that interacts (I assume) with the main Microsoft Python extension.
You will need to set "justMyCode": false in your launch.json for the debugger to trace into third-party code.
Start by looking at the source code for those modules; the -m switch looks for a package or module to import first. If it's a package, then Python imports the __main__ module in that package and runs it as the main script. If it is a module, the module itself is imported and run as __main__.
Usually the code is structured such that a function is called you can import directly too. You can then just write a bit of code that imports the same function and calls it the same way the __main__ module would. From there on out it is trivial to run this under a debugger.
E.g. pip is a package, so python -m pip will import pip.__main__ and run that as a script. This then triggers:
from pip._internal.cli.main import main as _main # isort:skip # noqa
if __name__ == '__main__':
sys.exit(_main())
to be run. You can do the same in VSCode; import pip._internal.cli.main.main and call it.
You can find the source code for these modules by just importing them and printing out the resulting object:
python -c "import pip; print(pip)"
The representation of a module, if loaded from disk, will include it's filename. If the filename ends in /__init__.py it's a package, so you can also double-check that the __main__.py file exists:
python -c "import pip.__main_; print(pip.__main__)"
You can do the same for the venv module. This one is part of the Python standard library, so the documentation actually links directly to the source code, and the venv.__main__ module just imports venv.main() and calls it.
I'm facing of a strange issue, and after a couple of hour of research I'm looking for help / explanation about the issue.
It's quite simple, I wrote a cgi server in python and I'm working with some libs including pynetlinux for instance.
When I'm starting the script from terminal with any user, it works fine, no bug, no dependency issue. But when I'm trying to start it using a script in rc.local, the following code produce an error.
import sys, cgi, pynetlinux, logging
it produce the following error :
Traceback (most recent call last):
File "/var/simkiosk/cgi-bin/load_config.py", line 3, in
import cgi, sys, json, pynetlinux, loggin
ImportError: No module named pynetlinux
Other dependencies produce similar issue.I suspect some few things like user who executing the script in rc.local (root normaly) and trying some stuff found on the web without success.
Somebody can help me ?
Thanx in advance.
Regards.
Ollie314
First of all, you need to make sure if the module you want to import is installed properly. You can check if the name of the module exists in pip list
Then, in a python shell, check what the paths are where Python is looking for modules:
import sys
sys.path
In my case, the output is:
['', '/usr/lib/python3.4', '/usr/lib/python3.4/plat-x86_64-linux-gnu', '/usr/lib/python3.4/lib-dynload', '/usr/local/lib/python3.4/dist-packages', '/usr/lib/python3/dist-packages']
Finally, append those paths to $PATH variable in /etc/rc.local. Here is an example of my rc.local:
#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing
export PATH="$PATH:/usr/lib/python3.4:/usr/lib/python3.4/plat-x86_64-linux-gnu:/usr/lib/python3.4/lib-dynload:/usr/local/lib/python3.4/dist-packages:/usr/lib/python3/dist-packages"
# Do stuff
exit 0
The path where your modules are install is probably normally sourced by .bashrc or something similar. .bashrc doesn't get sourced when it's not an interactive shell. /etc/profile is one place that you can put system wide path changes. Depending on what Linux version/distro it may use /etc/profile.d/ in which case /etc/profile runs all the scripts in /etc/profile.d, add a new shell script there with execute permissions and a .sh extention.
This concerns the importing of my own python modules in a HTCondor job.
Suppose 'mymodule.py' is the module I want to import, and is saved in directory called a XDIR.
In another directory called YDIR, I have written a file called xImport.py:
#!/usr/bin/env python
import os
import sys
print sys.path
import numpy
import mymodule
and a condor submit file:
executable = xImport.py
getenv = True
universe = Vanilla
output = xImport.out
error = xImport.error
log = xImport.log
queue 1
The result of submitting this is that, in xImport.out, the sys.path is printed out, showing XDIR. But in xImport.error, there is an ImporError saying 'No module named mymodule'. So it seems that the path to mymodule is in sys.path, but python does not find it. I'd also like to mention that error message says that the ImportError originates from the file
/mnt/novowhatsit/YDIR/xImport.py
and not YDIR/xImport.py.
How can I edit the above files to import mymodule.py?
When condor runs your process, it creates a directory on that machine (usually on a local hard drive). It sets that as the working directory. That's probably the issue you are seeing. If XDIR is local to the machine where you are running condor_submit, then it's contents don't exist on the remote machine where the xImport.py is running.
Try using the .submit feature transfer_input_files mechanism (see http://research.cs.wisc.edu/htcondor/manual/v7.6/2_5Submitting_Job.html) to copy the mymodule.py to the remote machines.