Develop Reference

Download files through url with parameters using colab and save it into storage session

i wanna ask my problem here. I have to download several csv file format through url with parameters and store it into google colab storage session. I try it using get request module by this code
report_header = {
'Content-Disposition' : 'attachment'
}
files_url = 'https://server.server/vendors/tasks/report/'
payloads = {"download-format2" : "true", "batch" : "574", "course" : "15"}
r = requests.get(files_url, params = payloads, headers = report_header, allow_redirects=True)
csv_url = r.url
with open('report.csv', 'wb') as zzz:
report.write(r.content)
but the result is 404 not found page. Even though, if i open those url using browser, it will auto download the attachment (csv file). The source need authentication, but it's not problem because the auth successful.
I also do it with wget
!wget https://server.server/vendors/tasks/report/?course=15&batch=574&download-format2=true
but the result also 404
What's wrong with my code? Is there any method that i don't know to do this task?
Thank you!

HTTP 401 error when accessing WebDAV with Python client

I've built a Python application that generates a inventory CSV file, and I want to upload that file to my store through BigCommerce's WebDAV application. I'm using the following Python client to access the WebDAV.
https://pypi.org/project/webdavclient3/
I can access my store and add files to the content folder with CyberDuck, but I get a HTTP 401 error when I try to access it from my Python script. Here is what I'm using to connect with WebDAV.
# webDAV upload to BigCommerce
options = {
'webdav_hostname': "https://mystore.com",
'webdav_login': "email#email.com",
'webdav_password': "password",
'webdav_root': "/dav/",
}
client = Client(options)
print("Exist:", client.check("/content/mytest")) # returns "Exist: False"
print(client.list())
print(client.free())
print("HERE")
I get an error at client.list() that reads
Request to https://mystore.com/dav/ failed with code 401 and message:
<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns"><s:exception>Sabre\\DAV\\Exception\\NotAuthenticated</s:exception><s:message>No 'Authorization: Digest' header found. Either the client didn't send one, or the server is misconfigured</s:message>
</d:error>
I guess it's saying my login and/or password is incorrect or there is no authentication? But how come I could log in through CyberDuck with the same credentials?
I saw someone asking about a similar problem in the following link, and I've tried the suggestions from Karen. None of them worked.
https://support.bigcommerce.com/s/question/0D51B00004G4XfYSAV/unable-to-access-upload-files-or-create-directory-through-webdav-api

I know this is 6 months old, but I figured I'd still post the solution for visibility when others try to do this.
The webdavclient library does not support HTTP Digest Authentication which is required to upload to the WebDAV. You can achieve this using the basic Python Requests library combined with the HTTPDigestAuth library.
Sample code:
import requests
from requests.auth import HTTPDigestAuth
# Below, put the URL of your BigCommerce WebDAV, including the file name you want to create/upload
# If you're uploading a CSV that you want to use as a product import, you will put it in the /dav/import_files/ directory.
url='https://store-abcdefg123.mybigcommerce.com/dav/import_files/products_upload_filename.csv' # example
# Local filename relative to this python script to upload
files = open('products_upload_filename.csv', 'rb')
# BigCommerce WebDAV login credentials.
# Found under Server Settings > File Access (WebDAV)
usern = 'youremail#email.com' # username
passw = 'password123' # password
# Make the file upload request
r = requests.request('PUT', url=url, data=files, auth=HTTPDigestAuth(usern, passw))
r.status_code
print(r.headers)
print(r.status_code)

I had the same issue and it was from the AuthType in the VirtualHost (/etc/httpd/conf.d/webdav.conf). I switched from Digest to Basic to fix it.

Error raised when trying to POST image to php using Requests

I am trying to upload a .jpg file to a server using HTTP POST with the requests library in Python 3.7.
Target URL has some PHP code that handles the upload, taking 'fileToUpload' as the multipart variable.
I tried putting the request into a with-statement, changing the data=files to files=files (as recommended by some example code), or setting the headers to multipart/form-data (which should not be necessary for this library)
import requests
url = 'http://someurl.com/upload/dir/post.php'
files = {'fileToUpload' : open('image.jpg', 'rb')}
r = requests.post(url, data=files)
If I run the script I raise every single error message in the post.php file, while if I take the thing to Insomnia or Postman the upload works just fine, so the server-side seems to be working.

HTTP Error 401: Authorization Required while downloading a file from HTTPS website and saving it

Basically i need a program that given a URL, it downloads a file and saves it. I know this should be easy but there are a couple of drawbacks here...
First, it is part of a tool I'm building at work, I have everything else besides that and the URL is HTTPS, the URL is of those you would paste in your browser and you'd get a pop up saying if you want to open or save the file (.txt).
Second, I'm a beginner at this, so if there's info I'm not providing please ask me. :)
I'm using Python 3.3 by the way.
I tried this:
import urllib.request
response = urllib.request.urlopen('https://websitewithfile.com')
txt = response.read()
print(txt)
And I get:
urllib.error.HTTPError: HTTP Error 401: Authorization Required
Any ideas? Thanks!!

You can do this easily with the requests library.
import requests
response = requests.get('https://websitewithfile.com/text.txt',verify=False, auth=('user', 'pass'))
print(response.text)
to save the file you would type
with open('filename.txt','w') as fout:
fout.write(response.text):
(I would suggest you always set verify=True in the resquests.get() command)
Here is the documentation:

Doesn't the browser also ask you to sign in? Then you need to repeat the request with the added authentication like this:
Python urllib2, basic HTTP authentication, and tr.im
Equally good: Python, HTTPS GET with basic authentication

If you don't have Requests module, then the code below works for python 2.6 or later. Not sure about 3.x
import urllib
testfile = urllib.URLopener()
testfile.retrieve("https://randomsite.com/file.gz", "/local/path/to/download/file")

You can try this solution: https://github.qualcomm.com/graphics-infra/urllib-siteminder
import siteminder
import getpass
url = 'https://XYZ.dns.com'
r = siteminder.urlopen(url, getpass.getuser(), getpass.getpass(), "dns.com")
Password:<Enter Your Password>
data = r.read() / pd.read_html(r.read()) # need to import panda as pd for the second one

Multipart POST request Google Glass

I am trying to add an attachment to my timeline with the multipart encoding. I've been doing something like the following:
req = urllib2.Request(url,data={body}, header={header})
resp = urllib2.urlopen(req).read()
And it has been working fine for application/json. However, I'm not sure how to format the body for multipart. I've also used some libraries: requests and poster and they both return 401 for some reason.
How can I make a multipart request either with a libary(preferably a plug-in to urllib2) or with urllib2 itself (like the block of code above)?
EDIT:
I also would like this to be able to support the mirror-api "video/vnd.google-glass.stream-url" from https://developers.google.com/glass/timeline
For the request using poster library here is the code:
register_openers()
datagen, headers = multipart_encode({'image1':open('555.jpg', 'rb')})
Here it is using requets:
headers = {'Authorization' : 'Bearer %s' % access_token}
files = {'file': open('555.jpg', 'rb')}
r = requests.post(timeline_url,files=files, headers=headers)
Returns 401 -> header
Thank you

There is a working Curl example of a multipart request that uses the streaming video url feature here:
Previous Streaming Video Answer with Curl example
It does exactly what you are trying to do, but with Curl. You just need to adapt that to your technology stack.
The 401 you are receiving is going to prevent you even if you use the right syntax. A 401 response indicates you do not have authorization to modify the timeline. Make sure you can insert a simple hello world text only card first. Once you get past the 401 error and get into parsing errors and format issues the link above should be everything you need.
One last note, you don't need urllib2, the Mirror API team dropped a gem of a feature in our lap and we don't need to be bothered with getting the binary of the video, check that example linked above I only provided a URL in the multipart payload, no need to stream the binary data! Google does all the magic in XE6 and above for us.
Thanks Team Glass!
I think you will find this is simpler than you think. Try out the curl example and watch out for incompatible video types, when you get that far, if you don't use a compatible type it will appear not to work in Glass, make sure your video is encoded in a Glass friendly format.
Good luck!

How to add an attachment to a timeline with multipart encoding:
The easiest way to add attachments with multipart encoding to a timeline is to use the
Google APIs Client Library for Python. With this library, you can simple use the following example code provided in the Mirror API timeline insert documentation (click the Python tab under Examples).
from apiclient.discovery import build
service = build('mirror', 'v1')
def insert_timeline_item(service, text, content_type=None, attachment=None,
notification_level=None):
timeline_item = {'text': text}
media_body = None
if notification_level:
timeline_item['notification'] = {'level': notification_level}
if content_type and attachment:
media_body = MediaIoBaseUpload(
io.BytesIO(attachment), mimetype=content_type, resumable=True)
try:
return service.timeline().insert(
body=timeline_item, media_body=media_body).execute()
except errors.HttpError, error:
print 'An error occurred: %s' % error
You cannot actually use requests or poster to automatically encode your data, because these libraries encode things in multipart/form-data whereas Mirror API wants things in multipart/related.
How to debug your current error code:
Your code gives a 401, which is an authorization error. This means you are probably failing to include your access token with your requests. To include an access token, set the Authorization field to Bearer: YOUR_ACCESS_TOKEN in your request (documentation here).
If you do not know how to get an access token, the Glass developer docs has a page here explaining how to obtain an access token. Make sure that your authorization process requested the following scope for multipart-upload, otherwise you will get a 403 error. https://www.googleapis.com/auth/glass.timeline

This is how I did it and how the python client library does it.
from email.mime.multipart import MIMEMultipart
from email.mime.nonmultipart import MIMENonMultipart
from email.mime.image import MIMEImage
mime_root = MIMEMultipart('related', '===============xxxxxxxxxxxxx==')
headers= {'Content-Type': 'multipart/related; '
'boundary="%s"' % mime_root.get_boundary(),
'Authorization':'Bearer %s' % access_token}
setattr(mime_root, '_write_headers', lambda self: None)
#Create the metadata part of the MIME
mime_text = MIMENonMultipart(*['application','json'])
mime_text.set_payload("{'text':'waddup doe!'}")
print "Attaching the json"
mime_root.attach(mime_text)
if method == 'Image':
#DO Image
file_upload = open('555.jpg', 'rb')
mime_image = MIMENonMultipart(*['image', 'jpeg'])
#add the required header
mime_image['Content-Transfer-Encoding'] = 'binary'
#read the file as binary
mime_image.set_payload(file_upload.read())
print "attaching the jpeg"
mime_root.attach(mime_image)
elif method == 'Video':
mime_video = MIMENonMultipart(*['video', 'vnd.google-glass.stream-url'])
#add the payload
mime_video.set_payload('https://dl.dropboxusercontent.com/u/6562706/sweetie-wobbly-cat-720p.mp4')
mime_root.attach(mime_video)
Mark Scheel I used your video for testing purposes :) Thank you.