So I have a bit of a issue, I want to use Heroku to host my flask web app, and then I also want to use Heroku pipeline to link to the GitHub repository where I am housing this project. The issue is that on my website I allow the user to upload files to the server, but I feel that If I were to update the GitHub repository I will lose all the files the user uploaded when the server reloads the new GitHub. I would like to know if this is a real issue and if so is there some way I could fix this?
Storing user-uploaded files to Heroku isn't a good idea because Heroku provides ephemeral filesystem.
The Heroku filesystem is ephemeral - that means that any changes to the filesystem whilst the dyno is running only last until that dyno is shut down or restarted. Each dyno boots with a clean copy of the filesystem from the most recent deploy. This is similar to how many container based systems, such as Docker, operate.
So even if you just restart your app, Users will lose their files. But they provide some alternate options to store these. As you are using python this Addon may help you.
Read More - https://help.heroku.com/K1PPS2WM/why-are-my-file-uploads-missing-deleted
Related
I'm using Python/Django on Heroku (Cedar Stack) and I've got a management command that I need to write that will pull a file out of an S3 bucket and process it. I'm not sure I understand how to use the ephemeral filesystem. Are there only certain directories that are writeable? I found an other article that implied that there were only certain folders that were writable (but, it doesn't seem to apply to the Cedar stack). I found this dev article but it doesn't go into much detail (note: I do understand that it's just temporary. I only need to unzip the file and process the file). Can I just create a folder anywhere under the application's root? And how would I get that? It seems like I could probably just use $HOME. I did a bit of testing by connecting to via
$ heroku run bash
and running:
$ echo #HOME
returns:
/app
and running:
$ mkdir $HOME/tmp
creates a folder in the app's root and gives with the same user and group as the other files and folders.
So... anything I'm missing here? A better way to do it? Is there an OS environment variable for this? I've run "env" and I don't see a better one.
To really understand the ephemeral filesystem, you need to understand what a dyno is. You can read more about how dynos work. In a nutshell, though, a process runs on Heroku in a virtual machine with its own filesystem. That virtual machine can stop for a number of reasons, taking the filesystem along with it.
The underlying filesystem will be destroyed when an app is restarted, reconfigured (e.g. heroku config ...), scaled, etc. For example, if you have two web dynos, write some files to the ephemeral filesystem, and scale to three dynos, those files will be destroyed because your app is running on new dynos.
In general, the ephemeral filesystem works just like any filesystem. Directories you have permission to write to, such as $HOME and /tmp, you can write files to. Any files that require permanence should be written to S3, or a similar durable store. S3 is preferred as Heroku runs on AWS and S3 offers some performance advantages. Any files that can be recreated at will can be stored on the dyno's ephemeral store.
You can create a file under the '/tmp' directory, and that file will be destroyed after the request is complete. I'm doing this on Cedar, and I haven't had any problems.
I have an e-commerce Django website hosted in Heroku (free acc). I dynamically upload the image and price through the Django admin page. The images were showing up for one day, but from the next day, I am getting an "image not found (404)" error. What's the reason for this error?
Failed to load resource: the server responded with a status of 404 (Not Found)
You can't save (persistantly) media files into Heroku's local filesystem.
The Heroku filesystem is ephemeral - that means that any changes to
the filesystem whilst the dyno is running only last until that dyno is
shut down or restarted. Each dyno boots with a clean copy of the
filesystem from the most recent deploy. This is similar to how many
container based systems, such as Docker, operate.
In addition, under normal operations dynos will restart every day in a
process known as "Cycling".
These two facts mean that the filesystem on Heroku is not suitable for
persistent storage of data. In cases where you need to store data we
recommend using a database addon such as Postgres (for data) or a
dedicated file storage service such as AWS S3 (for static files). If
you don't want to set up an account with AWS to create an S3 bucket we
also have addons here that handle storage and processing of static
assets https://elements.heroku.com/addons
Reference: https://help.heroku.com/K1PPS2WM/why-are-my-file-uploads-missing-deleted
I am new to server side operations and have a personal project that I am building. I am running Python3, Django2 with mysql (which will be changed to postgres for live usage) and currently have my static files stored inside the virtualenv on my local drive. It's also built locally on Ubuntu 16.04 if this changes anything, for example: Maybe git deployment would be easiest?
My first question is: I don't expect lot's of traffic (under 1000 a day), and databases are very simple. Only an admin can upload static content, such as posts, images, tags and categories. Essentially a blog format. Where should I store these files? Amazon s3, azure, google, or anything you suggest
2. Second question is: Where should I host my web app, and how will it affect where I store my static files?
I'd like to note that I am an entrepreneur doing this on my own so an inexpensive and simple setup is what I am aiming for as I don't have much experience in server side tech, but willing to learn.
Thank you in advance for your time and sorry if I left out some required information, I'll update the Q as needed.
We store our static files on s3 using s3utils. You should also check out the this post, it describes how to set up an S3 bucket with the proper permissions.
We host our webapps on AWS Ec2 with Nginx and Gunicorn. Check out How To Set Up Django with Postgres, Nginx, and Gunicorn on Ubuntu 16.04
I am saving some information to a textfile that is stored in my Heroku app. It can be updated by post requests from a user using an IOS device. It all works and it stores the information. But as you all know the Heroku app goes idle after an hour. So after the server goes idle and i make a GET request, the information previously put is lost?
There is a link in my heroku apps like afternoon-springs.... /ResetAllInfo but that link is never accessed. I watched the heroku logs to see.
Any ideas?
It seems like Heroku does not support write to filesystem: Heroku Documentation and here:
Each dyno gets its own ephemeral filesystem, with a fresh copy of the most recently deployed code. During the dyno’s lifetime its running processes can use the filesystem as a temporary scratchpad, but no files that are written are visible to processes in any other dyno and any files written will be discarded the moment the dyno is stopped or restarted.
So it's just not possible. Heroku suggests:
Use the provided PostgreSQL database instead of a filesystem database
Is there a way to use Sqlite3 with Django on Heroku?
The cedar stack's filesystem is not readonly.
However, you still mustn't store any data on it because the filesystem is ephemeral.
Any time your application restarts, whatever you had written to your application's filesystem disappears forever.
Any time you add a dyno, the two dynos each have their own ephemeral system; any data stored by one dyno to its ephemeral filesystem is not available to the other dyno or to any additional dynos you may add later.
Sqlite3 writes data to the local filesystem. You cannot use Sqlite3 with Heroku.
Heroku provides a default PostgreSQL installation, which Heroku manages. You can use that.
You can also use any third-party-managed cloud database system, such as Amazon RDS' or Xeround's MySQL, MongoHQ's or MongoLab's MongoDB, or Cloudant's CouchDB - all of which are available as Heroku addons.
I'm not sure when this answer became out of date, but as of at least 21 Nov 2013, sqlite3 CAN be used on heroku: https://devcenter.heroku.com/articles/sqlite3
It will work fine if you're just doing a tiny demo app, e.g. running 1 dyno and don't care that the database gets wiped at least once every 24 hours. If not, the heroku help article suggests migrating to Postgres.
Make sure the .db file is in your git directory somewhere and not in /tmp/ though, as it would be if for instance you were following the Flask tutorial app, flaskr.