I am beginner with django.
On my website I would like to create a library which allows downloads of executable files which I created myself. I would like to count how many times each file has been downloaded.
I thought to use a middleware, knowing that I am able to make a middleware which counts and displays the number of times a page has been viewed :
def stats_middleware (get_response):
def middleware (request):
try :
p = Stat.objects.get(url = request.path)
p.views_number = F('views_number')+1
p.save
except Stat.DoesNotExist :
p = Stat.objects.create(url= request.path)
response = get_response(request)
response.content += bytes(
"cette page a été vue {} fois.".format(p.views_number),
"utf8"
)
return response
return middleware
I thought that if I managed to open the download in a new page, I could count the number of times it appears and thus the number of downloads of the file, but I did not manage to open the download in another tab.
How can I do this?
You do not need to use middleware. Serving files through django is bad idea and is very ineffective. To control access to files and so on the servers creates special mechanism. X-Sendfile in Apache and X-Accel-Redirect in Ngnix. You need only create special response header in your View. In same View you can count downloads.
You car read more here: Django - Understanding X-Sendfile
And try to use this package: https://github.com/johnsensible/django-sendfile
Sample code:
from sendfile import sendfile
def download(request):
# here increment counter of download
return sendfile(request, file_name)
I have a Python script that is running periodically on an AWS EC2 Ubuntu machine.
This script reads data from some files and sometimes changes data in them.
I want to download these files from OneDrive, do my own thing with them, and upload them back to OneDrive.
I want this to be done automatically, without the need for a user to approve any login or credentials. I'm ok with doing it once (i.e. approving the login on the first run) but the rest has to run automatically, without asking ever again for approvals (unless the permissions change, of course).
What is the best way to do this?
I've been reading the documentation on Microsoft Graph API but I'm struggling with the authentication part. I've created an application in Azure AAD, gave the sample permissions (to test) and created a secret credential.
I managed to do it. I'm not sure if it's the best way but it is working now. It's running automatically every hour and I don't need to touch it.
I followed the information on https://learn.microsoft.com/en-gb/azure/active-directory/develop/v2-oauth2-auth-code-flow
This is what I did.
Azure Portal
Create an application. Azure Active Directory -> App Registrations -> Applications from personal account
In Supported account types, choose the one that has personal Microsoft accounts.
In Redirect URI, choose Public client/native. We'll add the specific URI later.
In the application details, in the section Overview, take note of the Application (client) ID. We'll need this later.
In the section Authentication, click Add a Platform and choose Desktop + devices. You can use your own, I chose one of the suggested: https://login.microsoftonline.com/common/oauth2/nativeclient
In the section API permissions, you have to add all the permissions that your app will use. I added User.Read, Files.ReadWrite and offline_access. The offline_access is to be able to get the refresh token, which will be crucial to keep the app running without asking the user to login.
I did not create any Certificate or Secret.
Web
Looks like to get a token for the first time we have to use a browser or emulate something like that.
There must be a programmatic way to do this, but I had no idea how to do it. I also thought about using Selenium for this, but since it's only one time and my app will request tokens every hour (keeping the tokens fresh), I dropped that idea.
If we add new permissions, the tokens that we have will become invalid and we have to do this manual part again.
Open a browser and go to the URL below. Use the Scopes and the Redirect URI that you set up in Azure Portal.
https://login.microsoftonline.com/common/oauth2/v2.0/authorize?client_id=your_app_client_id&response_type=code&redirect_uri=https%3A%2F%2Flogin.microsoftonline.com%2Fcommon%2Foauth2%2Fnativeclient&response_mode=query&scope=User.Read%20offline_access%20Files.ReadWrite
That URL will redirect you to the Redirect URI that you set up and with a code=something in the URL. Copy that something.
Do a POST request with type FORM URL Encoded. I used https://reqbin.com/ for this.
Endpoint: https://login.microsoftonline.com/common/oauth2/v2.0/token
Form URL: grant_type=authorization_code&client_id=your_app_client_id&code=use_the_code_returned_on_previous_step
This will return an Access Token and a Refresh Token. Store the Refresh Token somewhere. I'm saving it in a file.
Python
# Build the POST parameters
params = {
'grant_type': 'refresh_token',
'client_id': your_app_client_id,
'refresh_token': refresh_token_that_you_got_in_the_previous_step
}
response = requests.post('https://login.microsoftonline.com/common/oauth2/v2.0/token', data=params)
access_token = response.json()['access_token']
new_refresh_token = response.json()['refresh_token']
# ^ Save somewhere the new refresh token.
# I just overwrite the file with the new one.
# This new one will be used next time.
header = {'Authorization': 'Bearer ' + access_token}
# Download the file
response = requests.get('https://graph.microsoft.com/v1.0/me/drive/root:' +
PATH_TO_FILE + '/' + FILE_NAME + ':/content', headers=header)
# Save the file in the disk
with open(file_name, 'wb') as file:
file.write(response.content)
So basically, I have the Refresh Token always updated.
I call the Token endpoint using that Refresh Token, and the API gives me an Access Token to use during the current session and a new Refresh Token.
I use this new Refresh Token the next time I run the program, and so on.
I've just published a repo which does this. Contributions and pull requests welcome:
https://github.com/stevemurch/onedrive-download
I am using Cloud 9 IDE to build a website. My goal is to serve a static website from the site root '/index.html' and so on. The content in this site will be regenerated on a schedule (daily in this example). At the '/admin' and '/api' endpoints I want to serve a couple of flask apps.
Because this is being built on the Cloud 9 IDE, I do not have access to the proxy server configuration. I have to serve everything to one port using the HTML protocol. uWSGI is capable of doing exactly this. I am struggling with my configuration file though:
#uwsgi.ini
[uwsgi]
static-index = index.html
static-map2 = /=/home/ubuntu/workspace/generated-site
static-map2 = /static=/home/ubuntu/workspace/static-assets
mount = /admin=admin.py
mount = /api=api.py
manage-script-name = true
master = true
processes = 5
socket=0.0.0.0:8080
protocol=http
Requests to /admin and /api work as expected returning a result or 404 error.
Requests to / and /index.html both return generated-site/index.html as expected.
A request to /no_exist.html returns 404 Not Found as expected.
My problem is with the second static-map2. A request to /static/test.html came back 404 Not Found (I put an html file there to test).
static-map2 keep the path portion of the URL for its search so the request /static/test.html will be mapped to the file
/home/ubuntu/workspace/static-assets/static/test.html
You most probably want the simple static-map which strips the path from the URL before mapping to the file-system. So a request to /static/test.html will search for the file
/home/ubuntu/workspace/static-assets/test.html
My question is about how to serve multiple urls.py (like urls1.py, urls2.py and so on) files in a single Django project.
I am using Win7 x64, django 1.4.1, python 2.7.3 and as a server django dev-server tool.
I have decided to use a method which i found by google from
http://effbot.org/zone/django-multihost.htm
I have created a multihost.py file and put in to the django middleware folder:
C:\python27\Lib\site-packages\django\middleware\multihost.py
With the following code:
from django.conf import settings
from django.utils.cache import patch_vary_headers
class MultiHostMiddleware:
def process_request(self, request):
try:
host = request.META["HTTP_HOST"]
if host[-3:] == ":80":
host = host[:-3] # ignore default port number, if present
request.urlconf = settings.HOST_MIDDLEWARE_URLCONF_MAP[host]
except KeyError:
pass # use default urlconf (settings.ROOT_URLCONF)
def process_response(self, request, response):
if getattr(request, "urlconf", None):
patch_vary_headers(response, ('Host',))
return response
Also in my project setting.py file i have added a mapping dictionary like the link above shows:
# File: settings.py
HOST_MIDDLEWARE_URLCONF_MAP = {
"mysite1.com": "urls1",
#"mysite2.com": "urls2"
}
I did not yet implemented the error handling like described by the link above.
My hosts file includes the follwoing:
127.0.0.1 mysite1.com
Project structure is the following:
effbot django project folder:
+ effbot
--- settings.py
--- view.py
--- wsgi.py
--- urls.py
--- urls1.py
+ objex
--- models.py
--- views.py
+ static
--- css
--- images
There is no templates folder, because i dont serve html items from files, they are coming from databsse (i doubt that my problem is in this).
Now the problem is: when i go for the adress
mysite1.com
in my browser with django dev-server launched i get code 301 from the server. And browser shows "cannot display page" message.
Could you please explain me how to use mentioned method? I'm new to django and haven't done any real projects yet. Just have read the docs and launched a couple of sites at home to learn how it works.
I expect that urlconfs will be called in dependance from incoming
request.META["HTTP_HOST"]
The target is to serve different urlconfs for mysite1.com and mysite2.com
in a single django project.
I think this should to work some how.
Thank you for any feedback.
EDIT:
After some research attempts i found that i plugged my multyhost.py incorrectly in settings.
Fixed now. But the same result still.
Also i found out that my django dev-server tool is not reflecting anyhow that it handles any requests from the browser (IE9) except when i do "http://127.0.0.1".
May be i have to try some production server for my task, like nginx?
Your ROOT_URLCONF should be effbot.urls without .pyas you can see in the example in the documentation.
Also, HOST_MIDDLEWARE_URLCONF_MAP should reflect the ROOT_URLCONF so add `effbot. like this:
HOST_MIDDLEWARE_URLCONF_MAP = {
"mysite1.com": "effbot.urls1",
#"mysite2.com": "effbot.urls2"
}
One more thing, please try with another browser (Chrome, Firefox), sometimes I had problems accessing dev server with IE.
I want users on the site to be able to download files whose paths are obscured so they cannot be directly downloaded.
For instance, I'd like the URL to be something like this: http://example.com/download/?f=somefile.txt
And on the server, I know that all downloadable files reside in the folder /home/user/files/.
Is there a way to make Django serve that file for download as opposed to trying to find a URL and View to display it?
For the "best of both worlds" you could combine S.Lott's solution with the xsendfile module: django generates the path to the file (or the file itself), but the actual file serving is handled by Apache/Lighttpd. Once you've set up mod_xsendfile, integrating with your view takes a few lines of code:
from django.utils.encoding import smart_str
response = HttpResponse(mimetype='application/force-download') # mimetype is replaced by content_type for django 1.7
response['Content-Disposition'] = 'attachment; filename=%s' % smart_str(file_name)
response['X-Sendfile'] = smart_str(path_to_file)
# It's usually a good idea to set the 'Content-Length' header too.
# You can also set any other required headers: Cache-Control, etc.
return response
Of course, this will only work if you have control over your server, or your hosting company has mod_xsendfile already set up.
EDIT:
mimetype is replaced by content_type for django 1.7
response = HttpResponse(content_type='application/force-download')
EDIT:
For nginx check this, it uses X-Accel-Redirect instead of apache X-Sendfile header.
A "download" is simply an HTTP header change.
See http://docs.djangoproject.com/en/dev/ref/request-response/#telling-the-browser-to-treat-the-response-as-a-file-attachment for how to respond with a download.
You only need one URL definition for "/download".
The request's GET or POST dictionary will have the "f=somefile.txt" information.
Your view function will simply merge the base path with the "f" value, open the file, create and return a response object. It should be less than 12 lines of code.
For a very simple but not efficient or scalable solution, you can just use the built in django serve view. This is excellent for quick prototypes or one-off work, but as has been mentioned throughout this question, you should use something like apache or nginx in production.
from django.views.static import serve
filepath = '/some/path/to/local/file.txt'
return serve(request, os.path.basename(filepath), os.path.dirname(filepath))
S.Lott has the "good"/simple solution, and elo80ka has the "best"/efficient solution. Here is a "better"/middle solution - no server setup, but more efficient for large files than the naive fix:
http://djangosnippets.org/snippets/365/
Basically, Django still handles serving the file but does not load the whole thing into memory at once. This allows your server to (slowly) serve a big file without ramping up the memory usage.
Again, S.Lott's X-SendFile is still better for larger files. But if you can't or don't want to bother with that, then this middle solution will gain you better efficiency without the hassle.
Just mentioning the FileResponse object available in Django 1.10
Edit: Just ran into my own answer while searching for an easy way to stream files via Django, so here is a more complete example (to future me). It assumes that the FileField name is imported_file
views.py
from django.views.generic.detail import DetailView
from django.http import FileResponse
class BaseFileDownloadView(DetailView):
def get(self, request, *args, **kwargs):
filename=self.kwargs.get('filename', None)
if filename is None:
raise ValueError("Found empty filename")
some_file = self.model.objects.get(imported_file=filename)
response = FileResponse(some_file.imported_file, content_type="text/csv")
# https://docs.djangoproject.com/en/1.11/howto/outputting-csv/#streaming-large-csv-files
response['Content-Disposition'] = 'attachment; filename="%s"'%filename
return response
class SomeFileDownloadView(BaseFileDownloadView):
model = SomeModel
urls.py
...
url(r'^somefile/(?P<filename>[-\w_\\-\\.]+)$', views.SomeFileDownloadView.as_view(), name='somefile-download'),
...
Tried #Rocketmonkeys solution but downloaded files were being stored as *.bin and given random names. That's not fine of course. Adding another line from #elo80ka solved the problem.
Here is the code I'm using now:
from wsgiref.util import FileWrapper
from django.http import HttpResponse
filename = "/home/stackoverflow-addict/private-folder(not-porn)/image.jpg"
wrapper = FileWrapper(file(filename))
response = HttpResponse(wrapper, content_type='text/plain')
response['Content-Disposition'] = 'attachment; filename=%s' % os.path.basename(filename)
response['Content-Length'] = os.path.getsize(filename)
return response
You can now store files in a private directory (not inside /media nor /public_html) and expose them via django to certain users or under certain circumstances.
Hope it helps.
Thanks to #elo80ka, #S.Lott and #Rocketmonkeys for the answers, got the perfect solution combining all of them =)
It was mentioned above that the mod_xsendfile method does not allow for non-ASCII characters in filenames.
For this reason, I have a patch available for mod_xsendfile that will allow any file to be sent, as long as the name is url encoded, and the additional header:
X-SendFile-Encoding: url
Is sent as well.
http://ben.timby.com/?p=149
Try: https://pypi.python.org/pypi/django-sendfile/
"Abstraction to offload file uploads to web-server (e.g. Apache with mod_xsendfile) once Django has checked permissions etc."
You should use sendfile apis given by popular servers like apache or nginx in production. For many years I was using the sendfile api of these servers for protecting files. Then created a simple middleware based django app for this purpose suitable for both development & production purposes. You can access the source code here.
UPDATE: in new version python provider uses django FileResponse if available and also adds support for many server implementations from lighthttp, caddy to hiawatha
Usage
pip install django-fileprovider
add fileprovider app to INSTALLED_APPS settings,
add fileprovider.middleware.FileProviderMiddleware to MIDDLEWARE_CLASSES settings
set FILEPROVIDER_NAME settings to nginx or apache in production, by default it is python for development purpose.
in your class-based or function views, set the response header X-File value to the absolute path of the file. For example:
def hello(request):
# code to check or protect the file from unauthorized access
response = HttpResponse()
response['X-File'] = '/absolute/path/to/file'
return response
django-fileprovider implemented in a way that your code will need only minimum modification.
Nginx configuration
To protect file from direct access you can set the configuration as
location /files/ {
internal;
root /home/sideffect0/secret_files/;
}
Here nginx sets a location url /files/ only access internaly, if you are using above configuration you can set X-File as:
response['X-File'] = '/files/filename.extension'
By doing this with nginx configuration, the file will be protected & also you can control the file from django views
def qrcodesave(request):
import urllib2;
url ="http://chart.apis.google.com/chart?cht=qr&chs=300x300&chl=s&chld=H|0";
opener = urllib2.urlopen(url);
content_type = "application/octet-stream"
response = HttpResponse(opener.read(), content_type=content_type)
response["Content-Disposition"]= "attachment; filename=aktel.png"
return response
Django recommend that you use another server to serve static media (another server running on the same machine is fine.) They recommend the use of such servers as lighttp.
This is very simple to set up. However. if 'somefile.txt' is generated on request (content is dynamic) then you may want django to serve it.
Django Docs - Static Files
Another project to have a look at: http://readthedocs.org/docs/django-private-files/en/latest/usage.html
Looks promissing, haven't tested it myself yet tho.
Basically the project abstracts the mod_xsendfile configuration and allows you to do things like:
from django.db import models
from django.contrib.auth.models import User
from private_files import PrivateFileField
def is_owner(request, instance):
return (not request.user.is_anonymous()) and request.user.is_authenticated and
instance.owner.pk = request.user.pk
class FileSubmission(models.Model):
description = models.CharField("description", max_length = 200)
owner = models.ForeignKey(User)
uploaded_file = PrivateFileField("file", upload_to = 'uploads', condition = is_owner)
I have faced the same problem more then once and so implemented using xsendfile module and auth view decorators the django-filelibrary. Feel free to use it as inspiration for your own solution.
https://github.com/danielsokolowski/django-filelibrary
Providing protected access to static html folder using https://github.com/johnsensible/django-sendfile: https://gist.github.com/iutinvg/9907731
I did a project on this. You can look at my github repo:
https://github.com/nishant-boro/django-rest-framework-download-expert
This module provides a simple way to serve files for download in django rest framework using Apache module Xsendfile. It also has an additional feature of serving downloads only to users belonging to a particular group