I made a website in node.js and I have used child process to run python scripts and use their results in my website. But when I host my application on heroku, heroku is unable to run the python scripts.
I have noticed that heroku was able to run python scripts which only had inbuilt python packages like sys, json but it was failing for scripts which were using external packages like requests, beautifulsoup.
Attempt-1
I made a requirements.txt file inside the project folder and pushed my code to heroku again. But it was still not working. I noticed that heroku uses the requirements.txt file only for python based web applications but not for node.js applications.
Attempt-2
I made a python virtual env inside the project folder and imported the required python packages into the venv. I then removed gitignore from the venv and pushed the whole venv folder to heroku. It still didn't work.
Please let me know if anyone came across a way to handle this.
You can use a Procfile to specify a command to install the python dependencies then start the server.
Procfile:
web: pip install -r requirements.txt && npm start
Place Procfile in your project's root directory and deploy to heroku.
Solution:
Have a requirements.txt file in your project folder.
After you create a heroku app using "heroku create", heroku will identify your app as python based and will not download any of the node dependencies.
Now, go to your heroku profile and go to your app's settings. There is an option named "Add Buildpack" in there. Click on that and add node.js as one of the buildpacks.
Go back to your project and push it to heroku. This time heroku will identify your app as both python and node.js based and will download all the required files.
Related
I'm working on deploying my first dash app via heroku. I already did the example deployment given here: https://dash.plotly.com/deployment
which was quick and easy. Now I'm running through the same process with my own dash app, which is much more involved than the example app in the link, but which runs fine locally.
In step 2 given in the link, you initialize an empty git repo and run a new virtualenv, okay. Then you have to reinstall all the app's dependencies into the new virtual env.
This is my question, what are my app dependencies? Are those the modules I install at the top of my apps .py file? such as pandas, numpy, sklearn, etc?
Yes, that's correct. They must be defined in the requirements.txt file in the root of your project.
You can create the file manually or running this command
pip freeze > requirements.txt
The file looks something like this
pandas==1.2.0
numpy==1.19.4
This is my first experience deploying a Flask web app to Azure.
I followed this tutorial.
The default demo app they have works fine for me.
Afterwards, I pushed my Flask app via git. The log shows deployment was successful. However, when I browse the hosted app via link provided in "Application Properties", I get a 500 error as follows:
The page cannot be displayed because an internal server error has
occurred.
Most likely causes: IIS received the request; however, an internal
error occurred during the processing of the request. The root cause of
this error depends on which module handles the request and what was
happening in the worker process when this error occurred. IIS was not
able to access the web.config file for the Web site or application.
This can occur if the NTFS permissions are set incorrectly. IIS was
not able to process configuration for the Web site or application. The
authenticated user does not have permission to use this DLL. The
request is mapped to a managed handler but the .NET Extensibility
Feature is not installed.
The only off-base thing I can see by browsing the wwwroot via KUDU is that none of the packages I have installed in my local virtual environment are installed on Azure despite the existence of the "requirements.txt" file in wwwroot.
My understanding is that Azure would pip install any non existent package that it finds in the requirements.txt upon GIT successful push. But it doesn't seem to be happening for me.
Am I doing something wrong and the missing packages is just a symptom or could it be the cause the issue?
Notes:
My Flask app works fine locally (linux) and on a 3rd party VPS
I redeployed several times starting from scratch to no avail (I use local GIT method)
I cloned the Azure Flask demo app locally, changed just the app folder and pushed back to Azure, yet no success.
Azure is set to Python 2.7 same as my virtual env locally
As suggested in the tutorial linked above, I deleted the "env" folder and redeployed to trick Azure to reinstall the virtual env. It did but with its own default packages not the one in my requirements.txt.
My requirements.txt has the following:
bcrypt==3.1.0 cffi==1.7.0 click==6.6 Flask==0.11.1 Flask-Bcrypt==0.7.1
Flask-Login==0.3.2 Flask-SQLAlchemy==2.1 Flask-WTF==0.12
itsdangerous==0.24 Jinja2==2.8 MarkupSafe==0.23 pycparser==2.14
PyMySQL==0.7.7 python-http-client==1.2.3 six==1.10.0 smtpapi==0.3.1
SQLAlchemy==1.0.14 Werkzeug==0.11.10 WTForms==2.1
As Azure Web Apps will run a deploy.cmd script as the deployment task to control which commands or tasks will be run during the deployment.
You can use the command of Azure-CLI azure site deploymentscript --python to get the python applications' deployment task script.
And you can find the following script in this deploy.cmd sciprt:
IF NOT EXIST "%DEPLOYMENT_TARGET%\requirements.txt" goto postPython
IF EXIST "%DEPLOYMENT_TARGET%\.skipPythonDeployment" goto postPython
echo Detected requirements.txt. You can skip Python specific steps with a .skipPythonDeployment file.
So the .skipPythonDeployment will skip all the following steps in deployment task, including creating virtual environment.
You can try to remove .skipPythonDeployment from your application, and try again.
Additionally, please refer to https://github.com/projectkudu/kudu/wiki/Custom-Deployment-Script for more info.
I am having issues deploying my Pyramid app on Heroku. It runs fine locally but as soon as I try to launch it I receive this error "pkg_resources.DistributionNotFound: mymedaproject". mymedaproject is the name of my project and is not a python library which is why I am confused. I followed the instructions from this recipe to get to this point:
http://pyramid-cookbook.readthedocs.org/en/latest/deployment/heroku.html
Any ideas?
May be you forgot to put your python project mymedaproject in development mode. What follows is the relevant part of the cookbook recipe.
Create a Procfile
$ echo "web: ./run" > Procfile
Create run with the following:
#!/bin/bash
python setup.py develop
python runapp.py
The first line puts your python project in development mode and enables Paste to load it using your INI file. Make sure Procfile, run, runapp.py and setup.py are in same directory.
References
Getting Started with Python on Heroku
Process Types and the Procfile
Optimization
running a script using a Procfile should work without making it executable
$ echo "web: sh ./run" > Procfile
Check your .gitignore file that it isn't blocking any egg or egg-info information.
If it is, Heroku won't be receiving the egg for your application.
I recently made some changes to the structure of my Flask app hosted on heroku and now heroku has decided to detect it as a Node.js app intead of a Python app. My application uses both python (Flask) for the backend api and javascript for the front end.
The changes I made included integrating npm and bower into my application to streamline the javascript development of the app.
The problem was introduced when I added a package.json to my root directory when I started using npm. It seems that the build detection script runs the nodejs detection first (here) which leads to this code: if [ -f $1/package.json ]; then
echo "Node.js" && exit 0 executing and Heroku thinks it's a nodejs app and exits before the python detection has a chance to run.
To solve this I had to manually tell Heroku that I wanted a python build using this command
heroku config:set BUILDPACK_URL=https://github.com/heroku/heroku-buildpack-python.
The package.json file is causing Heroku to detect it as a node.js app. To prevent this, add the file name to a .slugignore file:
echo 'package.json' >> .slugignore
git add .slugignore
.slugignore is like .gitignore. It resides in the root directory of your repository and should contain a list of filenames and wildcard patterns. The matching files remain in your git repository, but are deleted from the slug after you push to Heroku. The deletion occurs before the buildpacks run, so the node.js buildpack won't find package.json and the app won't be misidentified as a node.js app.
I'm following this guide and trying to develop a Flask app to run on the Google App Engine. I followed the guide to the letter but when I launch the dev app server from the Launcher and go to http://localhost:8080/, I get a HTTP 500 error.
I check the logs and it says No module named flask. Then I check the interactive console in the admin console by running import flask and I get the same error message. I can import flask in any other python file without error.
Is there a way to fix this?
Working a bit with GAE and Flask I have realized this:
Running directly with Python
To run the app with python directly (python app.py) you need have the dependents packages installed in your environment using the command: pip install flask
Running with dev_appserver.py
To run the app with the dev_appserver.py provided by GAE SDK you need have all dependent packages inside your project, as: Flask, jinja2... Look in my another answer a example how to configure this packages : https://stackoverflow.com/a/14248647/1050818
UPDATED
Running Python, Virtualenv, Flask and GAE on Windows
Install Python
Install Python http://www.python.org/ftp/python/2.7.2/python-2.7.2.msi
Click in Windows Start button and search by "Edit the system environment" and open
Go to the tab Advanced and click on button "Environment Variables…"
When the Environment Variables window opens, choose Path from the System variables list and click Edit…
Add this ;C:\Python27;C:\Python27\Scripts at the end of the value and save
Install setuptools MS Windows installer (Necessary to install PIP on Windows)
Choose the correct installer for you in this page http://pypi.python.org/pypi/setuptools#files( I used this one: http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11.win32-py2.7.exe#md5=57e1e64f6b7c7f1d2eddfc9746bbaf20)
Download the installar and Install that
Install PIP
Download PIP http://pypi.python.org/pypi/pip#downloads
Extract it to any folder
From that directory, type python setup.py install
Install Virtualenv
Execute pip install virtualenv
Execute this mkdir c:\virtualenvs to create a folder to the Virtual Envs
Execute this cd c:\virtualenvs to access that folder
Execute virtualenv flaskdemo to create a virtualenv for you project
Active the virtualenv c:\virtualenvs\flaskdemo\scripts\activate
Install Google App Engine SDK
Install the SDK https://developers.google.com/appengine/downloads
Create the project
Create a directory for your project
Create the main of your application https://github.com/maxcnunes/flaskgaedemo/blob/master/main.py
Create the configuration of your appliction for Google App Engine https://github.com/maxcnunes/flaskgaedemo/blob/master/app.yaml
Create a file to let GAE initialize your application https://github.com/maxcnunes/flaskgaedemo/blob/master/initialize_gae.py
(Look a example of the code here: https://github.com/maxcnunes/flaskgaedemo )
Install Flask to run Locally
Execute pip install flask
Install Flask to run on the GAE
Download Flask https://github.com/mitsuhiko/flask/archive/0.9.zip and extract the folder flask inside your project
Download Werkzeug https://github.com/mitsuhiko/werkzeug/archive/0.8.3.zip and extract the folder werkzeug inside your project
Download Jinja2 https://github.com/mitsuhiko/jinja2/archive/2.6.zip and extract the folder jinja2 inside your project
Download Simple Json https://github.com/simplejson/simplejson/archive/v3.0.5.zip and extract the folder simplejson inside your project
Running the application with GAE SDK
Open Google App Engine Launcher
Add a new application
Run the application
Click in Browse button to open your application on browser
Finally click on Deploy button to deploy your application
Usually, templates come with a requirements.txt. If not, add your dependencies there and then run pip install -t lib -r requirements.txt to force the libraries to be saved in the lib folder.
Make sure you've added lib to appengine_config.py with vendor.add('lib') if it's not already there.
I was also facing the same issue and after spending 1 day on it have found out my silly mistake actually while refactoring my flask app I have changed
appengine_config.py to some other name.
Ideally appengine_config.py should look like below if you are having all your dependencies in lib folder only
from google.appengine.ext import vendor
#Add any libraries installed in the "lib" folder.
vendor.add('lib')
And because it was not able to find and execute appengine_config.py so lib folder was not registered as a dependency folder. To check you can try printing something in appengine_config.py to check if it's being executed on server startup.
tldr: use appengine_config.py and copy your virtualenv to a folder called lib, then make SURE you are running the app via dev_appserver.py
(the below is via bash in ubuntu)
SO after a long battle, I find that virtual env and gcloud dont play nice -
I copied everything from my virtual env dir
.../.virtualenvs/nimble/local/lib/python2.7/site-packages
into
[projectdir]/lib
and my appengine_config.py finally worked locally like it does in the cloud, but I absolutely HAVE to run
dev_appserver.py [my proj dir here]
or the google.appengine module wont load. did not know I should be using dev server. I feel very dumb.
for reference, heres the appengine_config.py
"""`appengine_config` gets loaded when starting a new application instance."""
print 'running app config yaya!'
from google.appengine.ext import vendor
vendor.add('lib')
print 'I am the line after adding lib, it should have worked'
import os
print os.getcwd()
Do you have Extra Libraries component for Python installed? It can be installed with
gcloud components install app-engine-python-extras
After installing this extra library you should be able to use built-in flask library without a problem. For more information, refer to this page
Source