This is a seemingly simple problem but it proves to be harder than expected.
I've created a project in pycharm in the following layout
bin
main
helpers
userhelper
models
user
session
tests
userTest
in my main I run the code that calls everything and this works like a charm in pycharm. Now I want to run this on a server and start it with cron. How do I start this from cron while keeping all the module references in place?
I guess I need to add the root of my project to the python path. To do this I added the following bash script to invoke my project with:
PYTHONPATH="${PYTHONPATH}:/home/steven/projectX"
export PYTHONPATH
python bin/main.py
But this does not seem to do anything, what would be the best way to periodically run the bin/main.py within this project and have all my modules and things like 'ConfigParser.RawConfigParser().read(os.path.abspath("../configuration.cfg"))' in place relative to my project?
EDIT: I am not trying to fix my imports or debugging my code, I have a large project in pycharm that runs a simulation that I want to invoke on the server an maintain within my development setup. The question is how do I run this in the same way pycharm does?
It sounds like you're interested in making a distributable Python package. You should read through the tutorial here. Ultimately, you're going to want to write a setup.py (sure you could call it something else, but it's like renaming self -- why do it?) that will configure your project. Now a word of advice since I've seen many people go down a wrong path here. You NEVER want to modify your PYTHONPATH directly. It might be the quickest solution to get something up and working, but it WILL cause lasting problems.
So I've found the way of dealing with mu issue here. The best option would obviously be to create a distributable python package and use this. However since I am developing a simulation that has loads of features and desired outcomes there is a lot of tuning going on and having to create a package every time I want to run the project is a bit out of the scope for my project.
What I do want to do is being able to run the files in a similar way to how I am doing this on my development machine with PyCharm. The way I resolved this is to have my main.py in the root of my project and to have all references relative to the executed file. This means my userhelper looks for the datastore as follows:
path = os.path.join(os.path.dirname(os.path.dirname(__file__)),
"resources/" + settings['datastore_filename']
This resolves my issue of not being able to run my project on the server the same way it runs on my PC.
Related
I'm learning how to execute python files and functions through node. I've been following this tutorial https://medium.com/swlh/run-python-script-from-node-js-and-send-data-to-browser-15677fcf199f which shows a very simple setup.
I had no problem doing this. Now, when executing a python file i have in a python project i created using Pycharm, i get a stderr like this 'ModuleNotFoundError: No module named 'requests''.
Inside the project i import a lot of modules and have a venv folder that Pycharm created as i wanted to use a venv to avoid problems with my dependencies. Please correct me, but i understand this error is caused because when node executes the file, it is not using the venv of my Pycharm project, so fails because it obviously never installed those packages.
So my question is, how can y execute the file from node, but in the context of the venv, so whenever it sees an import statemnt in python it knows the package, since it's inside the venv, does that make sense?
My current spawn statement looks like this: const python = spawn("python", ["main.py"], {cwd: "path/to/file/dir"});
I've read lots of questions in the forum, but none helped me to solve this, help is much appreciated!
*Attached is a picture showing how this venv folder Pycharm creates looks like.
I am trying to learn Python coming from R. I am used to the Project functionality provided by RStudio, that I can create a new Project and start a new session loading the file with '.Rproj'. The new session automatically uses the folder as the working directory and then I can set all the paths relatively. No need to change anything for different computers etc.
I tried asking people more familiar with python but usually nobody really understood my question. I heard, this would not be a problem once i started using jupyter or other stuff correctly when I were to use my files on another session.
My questions is then either if there is something like the Project functionality from RStudio or how do I better manage my projects across different computer. I could not any good links on that.
I am not so much familiar with R studio, but the functionality that you described are available on spyder IDE. You have dedicated project window and you and switch between project. There are many more functionalities (lie an always-on python console, variable/function display).
Please check
https://www.spyder-ide.org/
You can further use the python os package to make your code os independent.
I have the same issue. Though the main thing I use RStudio Projects for is to manage the working directory, and open RStudio in the right place. The best I've come up with is very specific to my problem - it doesn't do any of the other stuff .rproj and RStudio do, and isn't terribly portable.
I'm on Windows, using anaconda and JupyterLab in (it's not quite RStudio, but seems a good option to me) and I've found it a pain to navigate to the right location with the right environment for each project.
My understanding of python isn't great, but it looks like with jupyter lab .ipynb files the default working directory is the directory the file is in (so doesn't suffer from the chronic setwd() issue that .rproj helps to eliminate).
So I've created a .bat file (called pyproj.bat in this case) in the root of my project folder structure, to open jupyterlab in the right python environment:
call <anaconda_dir>/Scripts/activate.bat
call conda activate <project_environment_name>
call jupyter lab
cmd /k
My first line of that script is call %USERPROFILE%/Anaconda3/Scripts/activate.bat which at least gives it a chance of being portable between users in my organisation.
Would be nice if there were something a bit more automatic that could handle this - e.g. better searching for the activate script, and perhaps the environment management too (perhaps write the .yml and offer to build the environment if it's not present)
(Borrowed some the .bat idea from this question: How to make batch files run in anaconda prompt)
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
Originally the requirements were to have an outward facing application and we were going to use a server. However, now all the code is done and long term vision has changed... The requirements are to have an inward facing application without a server (physical or remote).
The team that will be using the application will be 5 users on the same network and each is to have the app individually installed on their desktop.
My question is what is the best way to deploy this application to everyone's machine as a package they can easily install themselves. How can the application always be running on everyone's machine without typing python manage.py runserver?
I had to do this exact same thing. My solution was to use PyInstaller, which bundles all your packages and files into one neat distribution folder with an executable file in it.
First, look at this link for a tutorial. To be honest, this alone did not work for me, but it could for you. So I will document some things to look out for down below and continue from here as if it runs perfectly for you.
After PyInstaller runs on your manage.py file, it will create three new objects for you: a dist folder, a build folder, and a myAppName.spec file. In the dist folder is an executable for the project. To deploy this project to other places, the only thing needed is that dist folder (or even just the myAppName folder inside that). Simply zip it up and send it off.
The executable inside the dist folder does not run just by clicking on it though, it still requires you to type python manage.py runserver. To get around this, I made a shortcut to the executable and modified it's target. So if your project's path is C:\path\to\proj, the target of the shortcut will be C:\path\to\proj runserver. This should allow you to double click the shortcut and run the local server. If you also wanted to open a web browser on the same click, make a batch file. If you don't like having the console open, try TrayIt.
If you want to re-run PyInstaller on your project, get rid of those dist and build folders for speed, but keep that .SPEC file and run pyinstaller on that instead. It can help you be more specific about the process if you care.
Issues to look out for:
As mentioned above, when I ran PyInstaller on my Django app I had a few of problems to work through.
Whenever I ran my new executable, the console complained about "Module not found" errors. Here is my solution to these errors. I eventually had to add more hidden imports than my answer shows. Ask and you shall receive.
PyInstaller was supposed to package all templates (html files), but it did not. I had to manually add all files to the dist\myAppName\django\contrib\admin\templates folder.
PyInstaller was also supposed to grab all outside modules you use, but it did not. To use the missing module, I had to grab the module's folder from where I downloaded it and manually add it to my dist/myAppName folder. Just pasting it there worked out fine.
PyInstaller is also supposed to get rid of unused imports, but it did not. I had to add a lot of modules to the excludes list in my .SPEC file so that my dist folder wasn't huge.
Let me know if your PyInstaller experiences are better than mine. I think my issues may have been isolated instances.
You could use py2exe for Windows or py2app for OSX to convert your python application to an executable to start your Django application.
I wrote a small python program and again I am struggling with producing a good structure.
A few days ago I read a blog and many other websites, which advise against from a import b imports and recommend to use always import a.b "project absolute" path (the whole path from the project root to what I want to import) imports. That as a background.
I included a __main__.py so that I can run my program with python -m <directory>, which was another recommendation of someone on stackoverflow in one of the hundreds of python import questions. It is supposed to help keeping code runnable and testable with the same import structure, which was a problem in another project of mine.
Now what I want it, that from anywhere in my system, I can run python -m <dir of my code> and not only from one directory up the RSTCitations directory.
How can I achieve that, without:
python path manipulations (which are a dirty hack)
changing my imports somehow and getting a not recommended import structure
doing other dark magic to my code
I want to stick to best practices in organizing my code but still want it to be runnable from wherever I am in the terminal.
Example of fail
When I run the program as described from another directory completely unrelated to my program, I get for example the following error:
/home/user/development/anaconda3/bin/python: No module named /home/user/development/rst-citations-to-raw-latex/RSTCitations
However the path is correct. That is exactly where the code is.
You can :
install your program with pip ( see Installing Python packages from local file system folder with pip )
put your module in the python import path (and/or edit PYTHONPATH in the environment for the users that need to use it)
If you don't need other users to 'import' your library but just use it as a standalone program, you can also just put a symlink/script to your program, makeing it runnable from a directory which is in your PATH.
I want to distribute my Python application to co-workers for them to use. The application will on be run on Linux systems, but the users do not have admin privileges so cannot install my application's module dependencies. I would the users to be able to untar my application and then run my main.py script. Running another one-time 'install'-type script is okay, but not much else.
PyInstaller is close to what I want. Except I would like to distribute the source code of my application as well. So the application should be stand-alone and self-contained (with or without the python interpreter is fine, preferably with), but users should be able to make small changes to the code and rerun the application. My ideal solution is to create some sort of compressed/compiled archive of all my applications module dependencies and distribute that with my application. It doesn't have to be all dependencies, but at least the non-standard packages. The application will then import modules from this archive instead of the user's PYTHONPATH.
I tried virtualenv, but having the users source the activate script was a little too much. I've been looking into numerous other solutions, but I can't find one that works for me.
Why don't you create a directory with the interpreter you want to use, add in any modules etc. Then drop in a bash script, say run.sh which calls the program. It can launch your chosen interpretter with your python files, arguments etc.
Any source files can remain this way and be edited in place. You could tar and distribute the whole directory, or put in something like git.
One approach is to use virtualenv. It is designed to create isolated python environment and does a good job at it. It should be possible (link to package the virtualenv with your app with some effort. However virtualenv is not designed for that so it's not as easy as it could be.
package-virtualenv GitHub project might also help.