Develop Reference

strategy for making sure imports of custom modules within project work from crontab?

I have a code project with various python scripts and modules. The folder structure of the github project is something like this:
/data_collection
/analysis
/modules
/helpers
Most of the scripts in data_collection and analysis will import stuff from modules or helpers. The code for doing this, in an example script /data_collection/pull_data.py, would be something like this:
import sys
sys.path.insert(0, '..')
from modules import my_module
from helpers import my_helper
now, if i simply run this code from the shell (from the dir that the script is in) - easy, it works just fine.
BUT: I want to run this from the crontab. It doesn't work, because crontab's PWD is always the cron user's home dir.
Now, I realise that I could add PWD=/path/to/project at the top of cron. But, what if I also have other project's scripts firing from cron?
I also realise that I could reorganise the whole folder structure of the project, perhaps putting all these folders into a folder called app and adding __init__.py to each folder -- but I'm not really in a position to do that at this moment.
So - I wonder, is there a possibility to achieve the following:
retain the relative paths in sys.path.insert within scripts (or perhaps get some solution that avoids the sys.path business altogether (so that it can run without modification on other systems)
be able to run these scripts from the crontab while also running scripts that live in other project directories from the crontab
Many thanks in advance!

I've created an experimental import library: ultraimport
It allows to do file system based imports with relative (or absolute) paths that will always work, no matter how you run the code or which user is running the code (given the user has read access to the files you're trying to import).
In your example script /data_collection/pull_data.py you would then write:
import ultraimport
my_module = ultraimport('__dir__/../modules/my_module.py')
my_helper = ultraimport('__dir__/../helpers/my_helper.py')
No need for any __init__.py and no need to change your directory structure and no need to change sys.path. Also your current working directory (which you can get running pwd) does not play any role for finding the import files.

What does "Mark directory as sources root" really do?

In Pycharm, if you right click in a folder inside your project, you can mark it as sources root, so then you can import modules from this folder and subfolders.
However doing it this way will only make your program runnable inside Pycharm, if I try to execute from outside Pycharm (eg from the console) it will complain that certain modules are not found, which is the problem that I'm facing.
If I mark a certain folder as sources root my program runs fine, but I need to understand what does it do so I can configure the program to find this modules even if not using Pycharm.
I want to know what does this option exactly do, and how can I get the same behaviour without using it.
It is just adding a __init__.py file in the root folder?
Is it doing something like:
import sys
sys.path.insert(0, my_folder)

First __init__.py marks a directory as a regular package directory(this is pre 3.3, in 3.3+ its not required anymore). With this, python will look for submodules inside that directory for imports.
"Mark directory as sources root" sets the PATH(or PYTHONPATH) for the environment. In shell, it will be like,
export PYTHONPATH="${PYTHONPATH}:/your/source/root"
PYTHONPATH holds the default search path for module files.
This will enable the interpreter to search for the modules in the appended path. The default value is installation dependent(normally it contains path to Python binaries).
You can also manipulate this search path using sys.path from inside a python file.

Executing Python scripts from windows path variable

I have Python 3.7 on my path (I can execute .py scripts when I am in that local directory in cmd)
I also have a folder of scripts on my path (I can open them from any local directory in cmd i.e. by typing "script.py")
However, I cannot execute these scripts from any local directory explicitly using python, i.e. "python script.py"
Any ideas why this is the case? Thanks
Edit:
The desired folder "scripts" is set in PYTHONPATH variable, and checking within python I see
import sys
sys.path
['', 'C:\Users\benma\Desktop\scripts',...
I can import a file from scripts into python already running, but not execute it directly

Python doesn't search PATH to look for your scripts. You can run the script directly because the shell is searching PATH looking for something that matches.
PYTHONPATH won't help when executing from the shell. It is only used by Python when importing modules:
Augment the default search path for module files.
I don't think you're going to get exactly what you're after. The closest is probably executable modules.

Multiple user created modules from a directory not in PATH using just command window (Anaconda 3)

Up to this point I have organized my projects in such a way that everything I'm working on is in the same folder, so to play around/debug I have just launched Python from that folder like this:
C:\Users\Username\Dropbox\Projects\MyShinyProject>Python
>>>
However, I want to start organizing things better. I have created some "Utilities" classes that I expect I'll use over and over again. So they should be in their own folder.
So now, say I have a Projects folder (in Windows) with lots of subfolders of things I have been working on:
Projects
Sandbox
Sandbox1
Sandbox2
MyUtilities
__init__.py
Utility1.py
MyShinyProject
__init__.py
ImportantClass.py
I would like to be able to go into the command prompt and use classes/functions from both the MyUtilities folder and from the MyShinyProject folder. However, if I launch Python from inside MyShinyProject, I don't have access to MyUtilities (or vice versa). I've tried doing a relative import like this:
>>>import ..MyUtilities.Utility1
But that doesn't work:
import ..MyUtilities.Utility1
^
SyntaxError
If it matters: I don't use an IDE. I just use Notepad++ and the command prompt. Also, I added the __init__.py files to the folders because I read somewhere you're supposed to do that when you make modules, but I don't understand how to get all of this working correctly, or if I'm even close to doing it right.
I also tried adding my Projects folder to the PATH variable in the Windows environment table, but that doesn't seem to work. Even after adding it importing doesn't work, and when I do this:
import sys
for x in sys.path:
print(x)
...the folder I added to PATH does not appear (I tried adding it to the beginning and the end).
How can I use several of my user created modules together using the command prompt to import them?

Assuming you have __init__.py in your Projects folder, in the console you can do this:
import sys
sys.path.append("C:\Users\Username\Dropbox\Projects")
import Projects.MyUtilities.Utility1
Or if you want to add your desired directory permanently to the python path, you can append your directory to the value of the environment variable called PYTHONPATH.

Python import files mac

I have an exercise for a course in Python I have to complete. I have saved my methods/defs in one file and just need to figure out how to run it. I was hoping you could explain to me how to import files (I know the syntax"import filename"). When ever I do this I get an error. How do I change the file path of the import to the file on my desktop? I am using a mac and running IDLE 2.7.3

If the files are in the same directory as that file you can just use
import <filename> #(without the <>)
However, if you are referring to the files in a separate directory use imp
import imp
module = imp.load_source('module.name', '/path/to/file.py')
module.SomeClass()

On Mac OS X, there are two basic ways to launch IDLE. One is by double-clicking on an IDLE icon from the Applications folder in the Finder. (I'll call that IDLE.app) The second is to launch IDLE from a terminal session shell with something like:
idle2.7
I'll call that bin/idle because it refers to file in one of your system's bin directories.
When launching bin/idle, IDLE will inherit the current working directory of the shell and will add that directory to the front of the list of directories Python searches for imports. You can examine that list of directories in the IDLE shell window:
import sys
sys.path
When you launch IDLE.app, however, it is not associated with any shell so there is no opportunity to tell IDLE which working directory to use. In this case, IDLE.app uses your Documents folder/directory as its working directory: in shell notation, ~/Documents, also spelled /Users/your_login_name/Documents.
You can manually manipulate sys.path in your Python program to add another directory for imports:
import sys
sys.path.insert(0, '/path/to/directory')
Another way to manipulate sys.path is to use the PYTHONPATH environment variable. But, again, when using IDLE.app, there is no (easy) way to pass environment variables to it, unlike with bin/idle
So if you want to use IDLE.app, the simplest approach is to put into your Documents directory all of the files and packages directories that you want to be able to import. Otherwise, use a terminal session, set the desired working directory and/or PYTHONPATH variable and launch bin/idle. So something like:
cd ~/MyProject
export PYTHONPATH=~/"AnotherPackageDirectory"
idle2.7
Yet another approach is to install your modules in the default locations that Python searches and use Distutils or easy_install or pip. But that's something to learn about later when you have a finished project.