I'm trying to run a hobby project on bluemix that is a combination of nodejs and python, that expects the runtimes to be collocated.
At first, I didn't even know there was a python dependency. So I deployed based on the node SDK starter app.
There is a "requirements.txt" for python dependencies, but I can see it's not really being used. Is there something I can do to get deployment to recognize that the app is a hybrid like this, IOW to process the requirements.txt so when python is invoked my deps are there?
In general, I would recommend splitting the application up so they do not have this dependency. But if you can't, I can think of two options:
Use multi-buildpack. Create the .buildpacks file in the application root directory and push your application with -b https://github.com/ddollar/heroku-buildpack-multi.git option. During staging, the buildpacks specified in that file will be called on your application.
Write your own custom buildpack. It's not that difficult and you can only do the minimum your application needs.
Related
I'm deploying my Azure Function app using a CI/CD pipeline in Azure DevOps. The function invokes three console applications that are included in the package. One of the console applications is a stand alone .exe, it works without issue. The other two have dependencies to a number of dll:s that are also included in the package. This setup works well on my local machine, and when deployed using WebDeploy.
When instead deploying using Run From Package to a freshly created Function App Service, the function app itself loads fine as well as the standalone .exe console app, but both console apps that have dll dependencies fail to run, and both return with exit code 0xC0000135 to my function app (indicating that a dll failed to load).
Now, if I deploy once using Webdeploy and then deploy again using Run From Package, I get the latest build installed - and the console apps now work (!). I think this might be due to the .exe not being able to access the virtual file system when loading the dll:s, is this correct?
I could stick with WebDeploy but I really want to use the package deploy since the cold start time is much faster during scale-out (will need 100+ instances in production). I am also concerned that this way, the app actually needs to copy both the zip package, and the site structure under wwwroot, causing additional overhead.
What is the best way to include dependencies such as dll:s in a package when using Run From Package with Azure Functions?
(The function app is v3, built using .NET Core 3.1)
This question is mostly about the technical details + some best practices of how to efficiently deploy a python web app that's built using platter.
Taking Django for instance, I have a project that's already built into a tarball distribution. This includes all wheels of all deps + the package of the app itself.
My repo directory also contains some other files that need to be distributed with the deployed code, such as: manage.py, a fabfile package with fabric utils, and some configuration files (for supervisor, nginx, etc).
So my questions are:
How can I wrap these extra files into the distribution that contains the project?
If I simply use git to clone/pull the project on the server I have these files, but then I have duplicate of the source code being both in the project and zipped in the tarball. How can I avoid that? Committing the tarball into a separate repo?
Perhaps the duplication is not so bad, and I'll end up with multiple tarballs in my dist/ directory and only one symlinked to the current from which I deploy?
Same goes for a Tornado based app.
My first rule of deployment is "whatever works". Every production environment has different requirements. But to give opinions on your questions:
Not everything should be in your Python project. Perhaps there is a way to do it, but I think it's using the wrong hammer.
You can create a separate Git repo that handles configuration and asset files for your production deployment (this does not even be managed by Git if you don't care about old, irrelevant configuration files). This does not have to be a Python project, just the files for the production deployment. You may optionally put a Python script or two in here (or just a README.txt or fab files or a Buildout config) to automate tasks such as unpacking your platter or copying config files around.
It's tempting (and possible) to put production config things in your main Git repo. This is even suggested by apps that create boilerplate files for development and production configuration. This doesn't mean it's the best way to do things though.
My rule is that the main Git repo is "development only". It's cloned by developers who are setting up and working in development environments. It conflates a Python project far too much to try and be an Python application and also be a place to manage a production system, IMHO.
Production is managed separately. Sometimes by people different from the developers or at least the developer is wearing a different hat when thinking about a production deployment. This way you can also have a small, clean repo that tracks just changes to your production system.
Playing with symlinks within a single deployment that represents different builds is an extra layer of confusion. And the impetus to do so comes from trying to do everything from a single Python project.
Deploy your python application to something like /var/myapp/build-2015-10-29/. Then create a symlink at /var/myapp/current/ that points to this location. This way you can create a full deployment at /var/myapp/build-2015-11-05/ and tweak the config to start on a separate port, bring the app up and ensure everything works, then just switch from the symlink from the old build to the new build with minimal downtime.
Can anyone help to point us to instructions on how to get Django >1.5 working on Google App Engine? I have seen a number of people claim they have Django 1.6 working. We'd like to get 1.6 or 1.7 running. I have searched here for instructions on how to set this up. No luck so far.
Update:
In our development machine we have Django 1.7 installed (both /user/local and on virtualenv). However, if we modify GAE yaml to use Django 1.7 we get the following error messages:
google.appengine.api.yaml_errors.EventError: django version "1.7" is not supported, use one of: "1.2", "1.3", "1.4", "1.5" or "latest" ("latest" recommended for development only) in "./app.yaml",
The version 1.9.12 GoogleAppEngine sdk install in our /Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib directory shows the following Django versions listed:
django-0.96 django-1.2 django-1.3 django-1.4 django-1.5
My question is related to how to get our development environment setup correctly for Django 1.7 on Google App Engine and how to make sure we successfully deploy our app with Django 1.7 when we deploy to Google App Engine in production. How do we get the Django 1.7 to deploy on GAE when we deploy our app?
You can use any pure Python third party libraries in your Google App Engine application. In order to use a third party library, simply include the files in your application's directory, and they will be uploaded with your application when you deploy it to our system. You can import the files as you would any other Python files with your application.
I have application using Django 1.7 this way and everything is working fine. However, sometimes you may need to sort of hack something due to the App Engine limitations and its specifics. But it depends on your use cases.
I would also suggest to use virtual environment for your project. Install each library that is not supported by App Engine directly via pip and then create a symlink in your application directory pointing to the given library.
This way you can keep all required packages in a file (e.g. requirements.txt) that can be stored in SCM system (e.g. Git) along with your source files and other team members can quite easily replicate your working environment.
Provided that you use virtual environment and install all needed libraries (Django, ...) via pip, here is the directory layout that should work for you.
virtual-env-root
.Python
bin
include
lib
app-engine-project-root
app.yaml
django-project-root
django-app-root
symlink-to-django -> lib/python2.7/site-packages/django
symlink-to-another-lib -> lib/python2.7/site-packages/...
Such a layout can be easily deployed with the below command.
$ appcfg.py update app-engine-project-root
Or tested with App Engine development server.
$ dev_appserver.py app-engine-project-root
UPDATE
Since App Engine Python SDK version 1.9.15 you can use the vendoring mechanism to set up third party libraries. You do not have to create symlinks in your application directory pointing to the Python lib folder anymore.
Create lib directory directly in your application root and tell your app how to find libraries in this directory by means of appengine_config.py file.
from google.appengine.ext import vendor
# Add any libraries installed in the "lib" folder.
vendor.add('lib')
New directory layout follows.
virtual-env-root
.Python
bin
include
lib
app-engine-project-root
lib
app.yaml
appengine_config.py
django-project-root
django-app-root
Use pip with the -t lib flag to install libraries in this directory.
$ pip install -t lib [lib-name]
Or
$ pip install -t lib -r requirements.txt
You cannot - GAE only supports 1.5, and even that is marked as experimental. If you need django 1.7, perhaps you should use Google Compute Engine, which is Google's brand name for virtual machines that you can spool up.
If you are not married to Google App Engine, Heroku supports django 1.7 without issues.
Do you have specific a guide on how to move a Django 1.7 project to
Google Compute Engine? There is a bunch of Google stuff without any
guides on how to make them work.
Here are the steps, but they are the same had you deployed on any other server because GCE just gives you a linux instance:
First, make sure your developer account has a billing method attached to it.
Go to the developer console
Create a new project by clicking on Projects, then Create Project.
Wait as the project is being created (you'll see a progress window on the bottom right of your screen).
Once the project is finished creating, the console will automatically shift to that project's settings:
You can create a new instance, or deploy a ready-made template from the second column. You can see there are popular stacks and software applications for which templates are created.
As there is no django template yet, you will start by creating an instance.
Billing is controlled on a per-project basis, so you'll have enable billing at this point if you haven't done so already.
The next page is where you configure the instance. The fields are self-explanatory. You set the type of machine you like (how many virtual CPUs and memory), where (physically) you prefer the machine to be located, if you want both HTTP and HTTPS ports open, and then a disk image from which the instance will boot:
Once you have configured the machine, it will be brought online booted up and then you'll have access to the terminal via SSH.
From this point forward, you should treat this instance like any linux server. Install whatever you need to make your project work using the normal packaging tools; upload your files, etc.
For Amazon, the process is a bit simpler as there is a large library of AMIs that you can use for a one-click deployment process. AMI is Amazon Machine Image - a template from which you can deploy an instance.
For Heroku, as its a PaaS, you don't have to worry about the hardware components; however as with most PaaS platforms, you don't have write access to the filesystem. So to manage your static assets you have to do some extra work. The easiest option is to create a S3 bucket on Amazon and use that with django-storages. The official django tutorial at heroku suggests the use of dj-static to serve files directly from Heroku. This works fine for testing, but if you want to start uploading files, then you need to handle those correctly.
However, once you sort that out the steps are even simpler:
Pre-requisites:
git
heroku toolbelt
dj-database-url Python package
gunicorn Python package
The basic steps:
Create a git repository (if you have not done already) in your source code directory with git init.
Create a requirements.txt at the root of your project. pip freeze > requirements.txt should do it if you are using a virtual environment. Otherwise, you can create a text file and list the packages you need.
Adjust your settings.py, by adding this line at the very bottom: import dj_database_url
DATABASES['default'] = dj_database_url.config()
Create a Procfile (case is important). This is how you tell Heroku what kind of dyno (process) you need for your application. For django, you need a web dyno so in this file the following line should do: web: gunicorn yourproject.wsgi --log-file -
Create an app on Heroku and deploy. You should run these commands from your source code directory:
heroku create --buildpack https://github.com/heroku/heroku-buildpack-python
heroku addons:add heroku-postgresql:dev
git push heroku master
heroku run python yourproject/manage.py migrate --noinput
heroku run python web/manage.py collectstatic
You only do the first two steps once, then whenever you need to update your application simply git push heroku master to create a new revision on Heroku.
App Engine's Python environment currently knows how to provide Django up to version 1.5 via the libraries: configuration mechanism. This doesn't mean that later versions of Django won't work, only that they aren't yet built in. (I'm not sure why the latest built-in version is 1.5. It may have something to do with AE's historical policy of bundling each supported version of Django with the SDK, which probably needs to be revised to keep the SDK from getting too large.)
You can try to include Django 1.7 with your application files. I haven't tried this with 1.7 specifically yet, but it's worked with previous versions. Some adjustments to sys.path will be needed in your main.py.
Note that there is a limit of 10,000 application files. If you're concerned about this limit, one option is to use Python's zipimport and include Django as a zip archive. https://docs.python.org/2/library/zipimport.html
I'm deploying an app to AWS Elastic Beanstalk using the API:
https://elasticbeanstalk.us-east-1.amazon.com/?ApplicationName=SampleApp
&SourceBundle.S3Bucket=amazonaws.com
&SourceBundle.S3Key=sample.war
...
My impression from reading around a bit is that Java deployments use .war, .zips are supported (docs) and that one can use .git (but only with PHP or using eb? doc).
Can I use the API to create an application version from a .git for a Python app? Or are zips the only type supported?
(Alternatively, can I git push to AWS without using the commandline tools?)
There are two ways to deploy to AWS:
The API backend, where it is basically a .zip file referenced from S3. When deploying, the Instance will unpack and run some custom scripts (which you can override from your AMI, or via Custom Configuration Files, which are the recommended way). Note that in order to create and deploy a new version in an AWS Elastic Beanstalk Environment, you need three calls: upload to s3, Create Application Version, and UpdateEnvironment.
The git endpoint, which works like this:
You install the AWS Elastic Beanstalk DevTools, and run a setup script on your git repo
When ran, the setup script patches your .git/config in order to support git aws.push and in particular, git aws.remote (which is not documented)
git aws.push simply takes your keys, builds a custom URL (git aws.remote), and does a git push -f master
Once AWS receives this (url is basically <api>/<app>/<commitid>(/<envname>), it creates the s3 .zip file (from the commit contents), then the application version on <app> for <commitid> and if <envname> is present, it also issues a UpdateEnvironment call. Your AWS ids are hashed and embedded into the URL just like all AWS calls, but sent as username / password auth tokens.
(full reference docs)
I've ported that as a Maven Plugin a few months ago, and this file show how it is done in plain Java. It actually covers a lot of code (since it actually builds a custom git repo - using jgit, calculates the hashes and pushes into it)
I'm strongly considering backporting as a ant task, or perhaps simply make it work without a pom.xml file present, so users only use maven to do the deployment.
Historically, only the first method was supported, while the second grew up in importance. Since the second is actually far easier (in beanstalk-maven-plugin, you have to call three different methods while a simply git push does all the three), we're supporting git-based deployments, and even published an archetype for it (you see a sample project here, especially the README.md in particular).
(btw, if you're using .war files, my elastic beanstalk plugin support both ways, and we're actually in favor of git, since it allows us to some incremental deployments)
So, you wanna still implement it?
There are three files I suggest you read:
FastDeployMojo.java is the main façade
RequestSigner does the real magic
This is a testcase for RequestSigner
Wanna do in
Python? I'd look for Dulwich
C#? The powershell version is based in it
Ruby? The Linux is based on it
Java? Use mine, it uses jgit
I am trying to create my first site in Django and as I'm looking for example apps out there to draw inspiration from, I constantly stumble upon a term called "reusable apps".
I understand the concept of an app that is reusable easy enough, but the means of reusing an app in Django are quite lost for me. Few questions that are bugging me in the whole business are:
What is the preferred way to re-use an existing Django app? Where do I put it and how do I reference it?
From what I understand, the recommendation is to put it on your "PYTHONPATH", but that breaks as soon as I need to deploy my app to a remote location that I have limited access to (e.g. on a hosting service).
So, if I develop my site on my local computer and intend to deploy it on an ISP where I only have ftp access, how do I re-use 3rd party Django apps so that if I deploy my site, the site keeps working (e.g. the only thing I can count on is that the service provider has Python 2.5 and Django 1.x installed)?
How do I organize my Django project so that I could easily deploy it along with all of the reusable apps I want to use?
In general, the only thing required to use a reusable app is to make sure it's on sys.path, so that you can import it from Python code. In most cases (if the author follows best practice), the reusable app tarball or bundle will contain a top-level directory with docs, a README, a setup.py, and then a subdirectory containing the actual app (see django-voting for an example; the app itself is in the "voting" subdirectory). This subdirectory is what needs to be placed in your Python path. Possible methods for doing that include:
running pip install appname, if the app has been uploaded to PyPI (these days most are)
installing the app with setup.py install (this has the same result as pip install appname, but requires that you first download and unpack the code yourself; pip will do that for you)
manually symlinking the code directory to your Python site-packages directory
using software like virtualenv to create a "virtual Python environment" that has its own site-packages directory, and then running setup.py install or pip install appname with that virtualenv active, or placing or symlinking the app in the virtualenv's site-packages (highly recommended over all the "global installation" options, if you value your future sanity)
placing the application in some directory where you intend to place various apps, and then adding that directory to the PYTHONPATH environment variable
You'll know you've got it in the right place if you can fire up a Python interpreter and "import voting" (for example) without getting an ImportError.
On a server where you have FTP access only, your only option is really the last one, and they have to set it up for you. If they claim to support Django they must provide some place where you can upload packages and they will be available for importing in Python. Without knowing details of your webhost, it's impossible to say how they structure that for you.
An old question, but here's what I do:
If you're using a version control system (VCS), I suggest putting all of the reusable apps and libraries (including django) that your software needs in the VCS. If you don't want to put them directly under your project root, you can modify settings.py to add their location to sys.path.
After that deployment is as simple as cloning or checking out the VCS repository to wherever you want to use it.
This has two added benefits:
Version mismatches; your software always uses the version that you tested it with, and not the version that was available at the time of deployment.
If multiple people work on the project, nobody else has to deal with installing the dependencies.
When it's time to update a component's version, update it in your VCS and then propagate the update to your deployments via it.