I started learning SalesForce and developing apps using django.
I need assistance with uploading a file to salesforce, For that I read simple-salesforce and this that help to upload file using rest and SOAP api.
My question is how do I upload one or more files using simple-salesforce?
Here is the code block I use for uploading files.
def load_attachments(sf, new_attachments):
'''
Method to attach the Template from the Parent Case to each of the children.
#param: new_attachments the dictionary of child cases to the file name of the template
'''
url = "https://" + sf.get_forced_url() + ".my.salesforce.com/services/data/v29.0/sobjects/Attachment/"
bearer = "Bearer " + sf.get_session_id()
header = {'Content-Type': 'application/json', 'Authorization': bearer}
for each in new_attachments:
body = ""
long_name = str(new_attachments[each]).split(sep="\\")
short_name = long_name[len(long_name) - 1]
with open(new_attachments[each], "rb") as upload:
body = base64.b64encode(upload.read())
data = json.dumps({
'ParentId': each,
'Name': short_name,
'body': body
})
response = requests.post(url, headers=header, data=data)
print(response.text)
Basically, to send the file, you need to use the requests module and submit the file via a post transaction. The post transaction requires the URL to which the request is sent, the header information, and the data.
Here, sf is the instance of returned by the simple-salesforce initialization. Since my instance uses custom domains, I had to create my own function in simple-salesforce to handle that; I call it get_forced_url(). Note: The URL is may be different for you depending on which version you are using [the v29.0 portion may change].
Then I set up my bearer and header.
The next thing is a loop that submits a new attachment for each attachment in a map from Parent ID to the File I wish to upload. This is important to note, attachments must have a Parent Object so you need to know the ParentId. For each attachment, I blank out the body, create a long and short name for the attachment. Then the important part. On attachments, the actual data of the file is stored as a base-64 binary array. So the file must be opened as binary, hence the "rb" and then encoded to base-64.
Once the file has been parsed to base-64 binary, I build my json string where ParentId is the object ID of the parent object, the Name is the short name, and the body is the base-64 encoded string of data.
Then the file is submitted to the URL with the headers and data. Then I print the response so I could watch it happening.
To upload files, you only need simple-salesforce
Complete example, including creating Account, Contact and Case. Then attaching the file to Case.
#Create Account, Contact and Case
AccountID = sf.Account.create({'Name':'Test12','Phone':'987654321'})["id"]
ContactID = sf.Contact.create({'LastName':'Smith2','Email':'example3#example.com'})["id"]
CaseID = sf.Case.create({'AccountId':AccountID,'ContactId':ContactID,'Description':'Test4321','Subject':'Test4321'})
#Convert image to Base64
import json, base64
with open('test1.png', mode='rb') as file:
img = file.read()
image = base64.encodebytes(img).decode('utf-8')
#The simple example
sf.Attachment.create({'ParentId': CaseID["id"],'Name':'TestFile1','body': image,'ContentType':'image/png'})
And how to change the 'one-file' example to multiple files
sf.bulk.Attachment.insert([
{'ParentId': CaseID["id"],'Name':'TestFile2','body': image,'ContentType':'image/png'},
{'ParentId': CaseID["id"],'Name':'TestFile3','body': image,'ContentType':'image/png'},
{'ParentId': CaseID["id"],'Name':'TestFile4','body': image,'ContentType':'image/png'},
{'ParentId': CaseID["id"],'Name':'TestFile5','body': image,'ContentType':'image/png'},
{'ParentId': CaseID["id"],'Name':'TestFile6','body': image,'ContentType':'image/png'},
{'ParentId': CaseID["id"],'Name':'TestFile7','body': image,'ContentType':'image/png'},
{'ParentId': CaseID["id"],'Name':'TestFile8','body': image,'ContentType':'image/png'},
{'ParentId': CaseID["id"],'Name':'TestFile9','body': image,'ContentType':'image/png'}],batch_size=1000,use_serial=True)
(you know how to fix the rest)
Related
I have a problem with getting my test running using Robot Framework and robotframework-requests. I need to send a POST request and a binary data in the body. I looked at this question already, but it's not really answered. Here's how my test case looks like:
Upload ${filename} file
Create Session mysession http://${ADDRESS}
${data} = Get Binary File ${filename}
&{headers} = Create Dictionary Content-Type=application/octet-stream Accept=application/octet-stream
${resp} = Post Request mysession ${CGIPath} data=${data} headers=&{headers}
[Return] ${resp.status_code} ${resp.text}
The problem is that my binary data is about 250MB. When the data is read with Get Binary File I see that memory consumption goes up to 2.x GB. A few seconds later when the Post Request is triggered my test is killed by OOM. I already looked at files parameter, but it seems it uses multipart encoding upload, which is not what I need.
My other thought was about passing open file handler directly to underlying requests library, but I guess that would require robotframework-request modification. Another idea is to fall back to curl for this test only.
Am I missing something in my test? What is the better way to address this?
I proceeded with the idea of robotframework-request modification and added this method
def post_request_binary(
self,
alias,
uri,
path=None,
params=None,
headers=None,
allow_redirects=None,
timeout=None):
session = self._cache.switch(alias)
redir = True if allow_redirects is None else allow_redirects
self._capture_output()
method_name = "post"
method = getattr(session, method_name)
with open(path, 'rb') as f:
resp = method(self._get_url(session, uri),
data=f,
params=self._utf8_urlencode(params),
headers=headers,
allow_redirects=allow_redirects,
timeout=self._get_timeout(timeout),
cookies=self.cookies,
verify=self.verify)
self._print_debug()
# Store the last session object
session.last_resp = resp
self.builtin.log(method_name + ' response: ' + resp.text, 'DEBUG')
return resp
I guess I can improve it a bit and create a pull request.
I am downloading file from Hadoop to Django backend and storing the file using the code below:
import shutil
import requests
url = 'http://112.138.0.12:9870/webhdfs/v1/user/username/1.jpg?op=OPEN&user.name=username'
response = requests.get(url, stream=True)
with open('img.png', 'wb') as out_file:
shutil.copyfileobj(response.raw, out_file)
del response
I don't need to store the file in backend local system since I want to send this file to Angular 5 frontend where user will save this file in their local system. I'm getting the following error
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position
0: invalid start byte.
Can someone suggest me what would be the right way to do download large files in a short time?
DJANGO:
views.py:
class DownloadFileView(GenericAPIView):
serializer_class = UserNameSerializer
def get(self, request):
key = request.META.get('HTTP_AUTHORIZATION').split()[1]
user_id = Token.objects.get(key=key).user_id
user_name = User.objects.get(id=user_id).username
response = download_files(user_name)
return Response(response)
def download_files(user_name):
response = requests.get('http://112.138.0.12:9870/webhdfs/v1/user/' + user_name + '/1.jpg?op=OPEN&user.name=username', stream=True)
return response.raw
ANGULAR:
DownloadFile(){
this.userService.DownloadFiles().subscribe((data : any) => {
const blob = new Blob([data], { type: 'application/octet-stream'});
fileUrl = this.sanitizer.bypassSecurityTrustResourceUrl(window.URL.createObjectURL(blob));
}
}
DownloadFiles() {
this.token = localStorage.getItem('userToken')
var reqHeader = new HttpHeaders({ 'Content-Type': 'application/octet-stream', 'Authorization': 'token ' + this.token });
console.log(reqHeader)
return this.http.get(this.rootURL + 'download/', { headers: reqHeader});
}
To begin with your unicode error, it's because:
HttpResponse.init(content='', content_type=None, status=200,
reason=None, charset=None)
Instantiates an HttpResponse
object with the given page content and content type.
content should be an iterator or a string. If it’s an iterator, it
should return strings, and those strings will be joined together to
form the content of the response. If it is not an iterator or a
string, it will be converted to a string when accessed.
I do believe django is having trouble converting the binary data in the file to string. A more common approach when dealing with file downloads is:
response = HttpResponse(content_type="application/jpeg")
response.write(binary_data)
This works because there is a call to make_bytes behind the scenes which handles the binary data correctly.
Having said that, this is not the most efficient way to go about it. Your web app makes a request to a remote server using requests, and then passes that onto the client. Why not get your angular code to fetch the data directly from the end point?
Can't do that because you want authentication you say? Ok, How about checking the authentiation and then sending an HttpResponseDirect like this:
return HttpResponseRedirect('http://112.138.0.12:9870/webhdfs/v1/user/' + user_name + '/1.jpg?op=OPEN&user.name=username')
I'm in the process of writing a very simple Python application for a friend that will query a service and get some data in return. I can manage the GET requests easily enough, but I'm having trouble with the POST requests. Just to get my feet wet, I've only slightly modified their example JSON data, but when I send it, I get an error. Here's the code (with identifying information changed):
import urllib.request
import json
def WebServiceClass(Object):
def __init__(self, user, password, host):
self.user = user
self.password = password
self.host = host
self.url = "https://{}/urldata/".format(self.host)
mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm()
mgr.add_password(None, "https://{}".format(self.host), self.user, self.password)
self.opener = urllib.request.build_opener(urllib.request.HTTPDigestAuthHandler(mgr))
username = "myusername"
password = "mypassword"
service_host = "thisisthehostinfo"
web_service_object = WebServiceClass(username, password, service_host)
user_query = {"searchFields":
{
"First Name": "firstname",
"Last Name": "lastname"
},
"returnFields":["Entity ID","First Name","Last Name"]
}
user_query = json.dumps(user_query)
user_query = user_query.encode("ascii")
the_url = web_service_object.url + "modules/Users/search"
try:
user_data = web_service_object.opener.open(the_url, user_query)
user_data.read()
except urllib.error.HTTPError as e:
print(e.code)
print(e.read())
I got the class data from their API documentation.
As I said, I can do GET requests fine, but this POST request gives me a 500 error with the following text:
Content type 'application/x-www-form-urlencoded;charset=UTF-8' not supported
In researching this error, my assumption has become that the above error means I need to submit the data as multipart/form-data. Whether or not that assumption is correct is something I would like to test, but stock Python doesn't appear to have any easy way to create multipart/form-data - there are modules out there, but all of them appear to take a file and convert it to multipart/form-data, whereas I just want to convert this simple JSON data to test.
This leads me to two questions: does it seem as though I'm correct in my assumption that I need multipart/form-data to get this to work correctly? And if so, do I need to put my JSON data into a text file and use one of those modules out there to turn it into multipart, or is there some way to do it without creating a file?
Maybe you want to try the requests lib, You can pass a files param, then requests will send a multipart/form-data POST instead of an application/x-www-form-urlencoded POST. You are not limited to using actual files in that dictionary, however:
import requests
response = requests.post('http://httpbin.org/post', files=dict(foo='bar'))
print response.status_code
If you want to know more about the requests lib, and specially in sending multipart forms take a look at this:
http://docs.python-requests.org/en/master/
and
http://docs.python-requests.org/en/master/user/advanced/?highlight=Multipart-Encoded%20Files
I'm working on a python script that will:
1) pull GIS metadata from an enterprise database
2) parse the metadata from XML to plain text
3) attach the text files to the corresponding published datasets in Socrata (which are published monthly)
4) The script will also be run monthly, so that any schema changes in the enterprise dataset are reflected in the attached plain text metadata files on Socrata.
I've been able to successfully attach the text metadata files to published Socrata datasets using some code found here. The problem is, each time the script is run, an additional attachment is added. I would like to either delete the existing attachment and add a new one, or overwrite the existing attachment with the contents of the new one.
I've done a fair amount of research on this, and can't seem to find any documentation for managing attachments using the Socrata API. Any suggestions?
I ended up figuring this one out. Had to alter a few lines to empty out the attachments in the attach_file function in the Socrata Python library from:
def attach_file(self, filename):
metadata = self.metadata()
if not metadata.has_key('attachments'):
metadata['attachments'] = []
to:
def attach_file(self, filename):
metadata = self.metadata()
metadata['attachments'] = [] #empty out metadata, regardless of existing metadata
Ended up using the same API and was able to replace attachments using the following code:
def attach_file(self, filename, clear_metadata):
metadata = self.metadata()
if not metadata.has_key('attachments'):
metadata['attachments'] = []
# if the user wants to clear all existing attachments on dataset
if clear_metadata:
metadata['attachments'] = []
response = self.multipart_post('/assets', filename)
if not response.has_key('id'):
print "Error uploading file to assets service: no ID returned: %s" % response
return
attachment = {'blobId': response['id'],
'name': response['nameForOutput'],
'filename': response['nameForOutput']}
metadata['attachments'].append(attachment)
self._request("/views/%s.json" % self.id, 'PUT', {'metadata':metadata})
def multipart_post(self, url, filename, field='file'):
print("Running multipart_post")
authBase64 = base64.encodestring('%s:%s' % (self.username, self.password)).replace('\n', '')
datagen, headers = multipart_encode({field: open(filename, "rb")})
headers['X-App-Token'] = self.app_token
headers['Authorization'] = "Basic %s" % authBase64
print("url=" + url)
request = Request("%s%s" % (self.url, url), datagen, headers)
print(str(Request))
response = urlopen(request).read()
return json.loads(response)
I am trying to upload a file to JIRA via its REST API using the python lib found here: jira python documentation
It seems pretty straight forward I wrote a method that allows me to pass an issue and then it attaches a filename. and one that lets me retrieve an issue from JIRA.
from jira.client import JIRA
class JIRAReport (object):
def attach(self,issue):
print 'Attaching... '
attachment = self.jira.add_attachment(issue, attachment=self.reportpath, filename='Report.xlsx')
print 'Success!'
def getissue(self):
if not self.issue == None:
return self.jira.issue(self.issue)
return None
then in my main script I am getting the issue and attaching the file to an issue I retrieved from JIRA
report = JiraReport()
report.issue = 'ProjectKey-1'
report.reportpath = '../report_upload/tmp/' + filename
issue = report.getissue()
if not issue == None:
report.attach(issue)
else:
print "No Issue with Key Found"
I am able to get the issue/create issues if needed but when using the self.jira.add_attachment() method I am getting 405 Method Not Allowed.
The file exists and is able to be opened.
Here is the add_attachment() method from the source code:
def add_attachment(self, issue, attachment, filename=None):
"""
Attach an attachment to an issue and returns a Resource for it.
The client will *not* attempt to open or validate the attachment; it expects a file-like object to be ready
for its use. The user is still responsible for tidying up (e.g., closing the file, killing the socket, etc.)
:param issue: the issue to attach the attachment to
:param attachment: file-like object to attach to the issue, also works if it is a string with the filename.
:param filename: optional name for the attached file. If omitted, the file object's ``name`` attribute
is used. If you aquired the file-like object by any other method than ``open()``, make sure
that a name is specified in one way or the other.
:rtype: an Attachment Resource
"""
if isinstance(attachment, string_types):
attachment = open(attachment, "rb")
# TODO: Support attaching multiple files at once?
url = self._get_url('issue/' + str(issue) + '/attachments')
fname = filename
if not fname:
fname = os.path.basename(attachment.name)
content_type = mimetypes.guess_type(fname)[0]
if not content_type:
content_type = 'application/octet-stream'
files = {
'file': (fname, attachment, content_type)
}
r = self._session.post(url, files=files, headers=self._options['headers'])
raise_on_error(r)
attachment = Attachment(self._options, self._session, json.loads(r.text)[0])
return attachment
It is mentioned in documentation that as a argument they expect file-like object.
Try to do something like :
file_obj = open('test.txt','rb')
jira.add_attachment(issue,file_obj,'test.txt')
file_obj.close()
Check that the URL that you are specifying for JIRA (if using the on-demand service) is https://instance.atlassian.net.
I just hit this as well, and it sends a POST request to http://instance.atlassian.net and gets redirected to https://instance.atlassian.net, but the client sends a GET request to the redirected address (see: https://softwareengineering.stackexchange.com/questions/99894/why-doesnt-http-have-post-redirect for more information)