I have been working on a python project and I am new to it. I have made a small library for my project in which I have several different modules doing different tasks.
For example: I have 5 modules namely add, subtract, multiply, divide and root. I call all these .pyc files into my main.py file and my code runs properly if all of them are in the same folder.
Now, I want to store my main.py at: D:\project\main.py and these 5 .pyc files at : D:\project\Lib\ (In the Lib folder)
I found a solution as to mention the path of the folder Lib into the code but I can not do so as I need to submit the code somewhere and if they try to run this on their PC, it might not import these files.
What would be the possible solution to this?
Inside the main.py file, add the location of your modules to the path variable as below.
os.environ["PATH"] += os.pathsep + your_path
your_path which is relative to your project should be like './Lib'.
Try to import modules after the above step. I guess it should work fine. Let me know.
Note:
Those .pyc files are cache files from executing the .py files. You need to worry about the .py files and not .pyc files.
Related
I have a code project with various python scripts and modules. The folder structure of the github project is something like this:
/data_collection
/analysis
/modules
/helpers
Most of the scripts in data_collection and analysis will import stuff from modules or helpers. The code for doing this, in an example script /data_collection/pull_data.py, would be something like this:
import sys
sys.path.insert(0, '..')
from modules import my_module
from helpers import my_helper
now, if i simply run this code from the shell (from the dir that the script is in) - easy, it works just fine.
BUT: I want to run this from the crontab. It doesn't work, because crontab's PWD is always the cron user's home dir.
Now, I realise that I could add PWD=/path/to/project at the top of cron. But, what if I also have other project's scripts firing from cron?
I also realise that I could reorganise the whole folder structure of the project, perhaps putting all these folders into a folder called app and adding __init__.py to each folder -- but I'm not really in a position to do that at this moment.
So - I wonder, is there a possibility to achieve the following:
retain the relative paths in sys.path.insert within scripts (or perhaps get some solution that avoids the sys.path business altogether (so that it can run without modification on other systems)
be able to run these scripts from the crontab while also running scripts that live in other project directories from the crontab
Many thanks in advance!
I've created an experimental import library: ultraimport
It allows to do file system based imports with relative (or absolute) paths that will always work, no matter how you run the code or which user is running the code (given the user has read access to the files you're trying to import).
In your example script /data_collection/pull_data.py you would then write:
import ultraimport
my_module = ultraimport('__dir__/../modules/my_module.py')
my_helper = ultraimport('__dir__/../helpers/my_helper.py')
No need for any __init__.py and no need to change your directory structure and no need to change sys.path. Also your current working directory (which you can get running pwd) does not play any role for finding the import files.
I have already tried appending the path to sys.path, but that doesn't solve the problem either. The import works fine with my Python36 interpreter.
My directory structure is:
project (dir)
:main.py
:=> Algorithms (sub-dir)
:==> AlgoOne.py
My code in the entry file which is one level above the Algorithms dir is:
#main.py
from Algorithms.AlgoOne.py import ClassOne
Any ideas why this could happen, or how I instruct jython to read the directory as a module like Python36?
Thanks
Up to this point I have organized my projects in such a way that everything I'm working on is in the same folder, so to play around/debug I have just launched Python from that folder like this:
C:\Users\Username\Dropbox\Projects\MyShinyProject>Python
>>>
However, I want to start organizing things better. I have created some "Utilities" classes that I expect I'll use over and over again. So they should be in their own folder.
So now, say I have a Projects folder (in Windows) with lots of subfolders of things I have been working on:
Projects
Sandbox
Sandbox1
Sandbox2
MyUtilities
__init__.py
Utility1.py
MyShinyProject
__init__.py
ImportantClass.py
I would like to be able to go into the command prompt and use classes/functions from both the MyUtilities folder and from the MyShinyProject folder. However, if I launch Python from inside MyShinyProject, I don't have access to MyUtilities (or vice versa). I've tried doing a relative import like this:
>>>import ..MyUtilities.Utility1
But that doesn't work:
import ..MyUtilities.Utility1
^
SyntaxError
If it matters: I don't use an IDE. I just use Notepad++ and the command prompt. Also, I added the __init__.py files to the folders because I read somewhere you're supposed to do that when you make modules, but I don't understand how to get all of this working correctly, or if I'm even close to doing it right.
I also tried adding my Projects folder to the PATH variable in the Windows environment table, but that doesn't seem to work. Even after adding it importing doesn't work, and when I do this:
import sys
for x in sys.path:
print(x)
...the folder I added to PATH does not appear (I tried adding it to the beginning and the end).
How can I use several of my user created modules together using the command prompt to import them?
Assuming you have __init__.py in your Projects folder, in the console you can do this:
import sys
sys.path.append("C:\Users\Username\Dropbox\Projects")
import Projects.MyUtilities.Utility1
Or if you want to add your desired directory permanently to the python path, you can append your directory to the value of the environment variable called PYTHONPATH.
After trying to deal with relative imports and reading the many StackOverflow posts about it, I'm starting to realize this is way more complicated than it needs to be.
No one else is using this code but me. I'm not creating a tool or an api, so I see no need to create a "package" or a "module". I just want to organize my code into folders, as opposed to having a 50 scripts all in the same directory. It's just ridiculous.
Basically, just want to have three folders and be able to import scripts from wherever I want. One folder that contains some scripts with utility functions. Another folder has the majority of the code, and another folder that contains some experiments (I'm doing machine learning research).
/project/code/
/project/utils/
/project/experiments/
All I need to do is import python files between folders.
One solution I have found is putting __init__.py files in each directory including the root directory where the folders live. But I then need to run my experiments in the parent of that directory.
/experiment1.py
/experiment2.py
/project/__init__.py
/project/code/__init__.py
/project/utils/__init__.py
So the above works. But then I have two problems. My experiments don't live in the same folder as the code. This is just annoying, but I guess I can live with it. But the bigger problem is that I can't put my experiments different folders:
/experiments/experiment1/something.py
/experiments/experiment2/something_else.py
/project/__init__.py
/project/code/__init__.py
/project/utils/__init__.py
I suppose I could symlink the project directory into each experiment folder, but thats ridiculous.
The other way of doing it is treating everything like a module:
/project/__init__.py
/project/code/__init__.py
/project/utils/__init__.py
/project/experiments/__init__.py
/project/experiments/experiment1/something.py
/project/experiments/experiment2/something_else.py
But then I have to run my experiments with python -m project.experiments.experiment1.something which just seems odd to me.
A solution I have found thus far is:
import imp
import os
currentDir = os.path.dirname(__file__)
filename = os.path.join(currentDir, '../../utils/helpful.py')
helpful = imp.load_source('helpful',filename)
This works, but it is tedious and ugly though. I tried creating a script to handle this for me, but then the os.path.dirname(__file__) is wrong.
Surely someone has been in my position trying to organize their python scripts in folders rather than having them all flat in a directory.
Is there a good, simple solution to this problem, or will I have to resort to one of the above?
Running python files as a module seems also odd to me. You can put your experiments in a folder still by putting a main.py file on the root directory. So the folder tree would be the following;
/project/experiments
/project/code
/project/utils
/project/main.py
Call your experiments on your main.py file and make your imports on main.py file. The __init__.pys should also be inside each folder.
By this way, you won't need to run the py files as a python module. Also, the project will have a single entry point which is very useful in many cases.
So I found this tutorial very useful.
The solution I have come up with is as follows:
project/
.gitignore
setup.py
README.rst
MANIFEST.in
code/
__init__.py
something.py
tests/
__init__.py
tests.py
utils/
__init__.py
utils.py
experiments/
experiment1/
data.json
experiment1.py
experiment2/
data.json
experiment2.py
Then I run python setup.py develop in order symlink my code so I can import it anywhere else (you can unlink with python setup.py develop --uninstall).
I still haven't decided whether I like my experiments to live inside the project folder or outside. I don't think it really matters since this code is only for my personal use. But I suppose it would be proper for it to live outside...
I'm trying to migrate to wsgi but I'm getting ImportError exceptions all the time.
I have my modules installed only with .pyc files, I don't have the .py files in the server (I can't change that, sorry). When I add some .py files the ImportError moves away to some other import line.
Is there a way to make WSGI use the .pyc files and work without the .py?
The script with the application entry point is a .py file but it imports some modules which are .pyc
Thanks for your help
You might need to put all the python files on that machine, then convert them to .pyc or .pyo then delete the .py's
I'm suspecting maybe the .pys were generated on a different version of python maybe.
Also paste the import errors .. that might help in diagnoses.