I'm using Flask with Python 3.3 and I know support is still experimental but I'm running into errors when trying to test file uploads. I'm using unittest.TestCase and based on Python 2.7 examples I've seen in the docs I'm trying
rv = self.app.post('/add', data=dict(
file=(io.StringIO("this is a test"), 'test.pdf'),
), follow_redirects=True)
and getting
TypeError: 'str' does not support the buffer interface
I've tried a few variations around io.StringIO but can't find anything that works. Any help is much appreciated!
The full stack trace is
Traceback (most recent call last):
File "archive_tests.py", line 44, in test_add_transcript
), follow_redirects=True)
File "/srv/transcript_archive/py3env/lib/python3.3/site-packages/werkzeug/test.py", line 771, in post
return self.open(*args, **kw)
File "/srv/transcript_archive/py3env/lib/python3.3/site-packages/flask/testing.py", line 108, in open
follow_redirects=follow_redirects)
File "/srv/transcript_archive/py3env/lib/python3.3/site-packages/werkzeug/test.py", line 725, in open
environ = args[0].get_environ()
File "/srv/transcript_archive/py3env/lib/python3.3/site-packages/werkzeug/test.py", line 535, in get_environ
stream_encode_multipart(values, charset=self.charset)
File "/srv/transcript_archive/py3env/lib/python3.3/site-packages/werkzeug/test.py", line 98, in stream_encode_multipart
write_binary(chunk)
File "/srv/transcript_archive/py3env/lib/python3.3/site-packages/werkzeug/test.py", line 59, in write_binary
stream.write(string)
TypeError: 'str' does not support the buffer interface
In Python 3, you need to use io.BytesIO() (with a bytes value) to simulate an uploaded file:
rv = self.app.post('/add', data=dict(
file=(io.BytesIO(b"this is a test"), 'test.pdf'),
), follow_redirects=True)
Note the b'...' string defining a bytes literal.
In the Python 2 test examples, the StringIO() object holds a byte string, not a unicode value, and in Python 3, io.BytesIO() is the equivalent.
Related
I having trouble converting dialogflow types such as ListIntentsResponse, EntityType to json. I have researched a lot into this. Converting every entry one by one is a headache thats why I want a workaround.
I have tried using google.protobuf.json_format methods. But it doesnt works. says UNknown field : DESCRIPTOR
from google.protobuf.json_format import *
client = dialogflow.IntentsClient()
request = dialogflow.ListIntentsRequest(
parent=f'projects/{DIALOGFLOW_PROJECT_ID}/agent'
)
response = client.list_intents(request)
# print(response)
print(MessageToJson(response ,descriptor_pool=None))```
**Error==>>>**
Traceback (most recent call last):
File "c:\Users\1150-Bilal\Desktop\chatbot\intents.py", line 12, in <module>
intentlist()
File "c:\Users\1150-Bilal\Desktop\chatbot\intents.py", line 10, in intentlist
print(MessageToJson(response ,descriptor_pool=None))
File "C:\Users\1150-Bilal\Desktop\chatbot\botenv\lib\site-packages\google\protobuf\json_format.py", line 130, in MessageToJson
return printer.ToJsonString(message, indent, sort_keys, ensure_ascii)
File "C:\Users\1150-Bilal\Desktop\chatbot\botenv\lib\site-packages\google\protobuf\json_format.py", line 197, in ToJsonString
js = self._MessageToJsonObject(message)
File "C:\Users\1150-Bilal\Desktop\chatbot\botenv\lib\site-packages\google\protobuf\json_format.py", line 203, in _MessageToJsonObject
message_descriptor = message.DESCRIPTOR
File "C:\Users\1150-Bilal\Desktop\chatbot\botenv\lib\site-packages\google\cloud\dialogflow_v2\services\intents\pagers.py", line 74, in __getattr__
return getattr(self._response, name)
File "C:\Users\1150-Bilal\Desktop\chatbot\botenv\lib\site-packages\proto\message.py", line 747, in __getattr__
raise AttributeError(
AttributeError: Unknown field for ListIntentsResponse: DESCRIPTOR
I've been migrating some Python 2.7.11 code to 3.5.1 after running into trouble with unicode. This was the last straw - since I started using the venv module there's no reason to be on 2.7 just because someone doesn't like 3!
The problem occurs while trying to run a one-way sync (ie. downloading changes only).
Here is the full error message, paths shortened:
Traceback (most recent call last):
File "%SCRIPT%.py", line 209, in <module>
updated_schedules = dbx_sync.One_Way_Sync(config['Dropbox Parameters']['Directory'], config['Dropbox Parameters']['Base Path'])
File "%COMMON_PATH%\modules\dropbox_sync_schedules.py", line 62, in One_Way_Sync
result = client.delta(cursor, base_path)
File "%COMMON_PATH%\env-home\lib\site-packages\dropbox\client.py", line 569, in delta
return self.rest_client.POST(url, params, headers)
File "%COMMON_PATH%\env-home\lib\site-packages\dropbox\rest.py", line 322, in POST
return cls.IMPL.POST(*n, **kw)
File "%COMMON_PATH%\env-home\lib\site-packages\dropbox\rest.py", line 260, in POST
is_json_request=is_json_request)
File "%COMMON_PATH%\env-home\lib\site-packages\dropbox\rest.py", line 235, in request
raise ErrorResponse(r, r.read())
dropbox.rest.ErrorResponse: [400] 'Invalid "cursor" parameter: u"b\'\'"'
Searching for "invalid cursor parameter" wasn't any help, so I thought I'd come here.
u"b\'\'" is the key here. I just couldn't understand how that representation had ended up being sent as a string.
The issue was in reading the old cursor from a file (which for this example is empty): in Python 2 I had opened the file in mode rb - in Python 3 just r is all that's required, and everything works.
Hurrah!
I am attempting to use a Webservice created by one of our developers that allows us to upload files into the system, within certain restrictions.
Using SUDS, I get the following information:
Suds ( https://fedorahosted.org/suds/ ) version: 0.4 GA build: R699-20100913
Service ( ConnectToEFS ) tns="http://tempuri.org/"
Prefixes (3)
ns0 = "http://schemas.microsoft.com/2003/10/Serialization/"
ns1 = "http://schemas.microsoft.com/Message"
ns2 = "http://tempuri.org/"
Ports (1):
(BasicHttpBinding_IConnectToEFS)
Methods (2):
CreateContentFolder(xs:string FileCode, xs:string FolderName, xs:string ContentType, xs:string MetaDataXML, )
UploadFile(ns1:StreamBody FileByteStream, )
Types (4):
ns1:StreamBody
ns0:char
ns0:duration
ns0:guid
My method to using UploadFile is as follows:
def webserviceUploadFile(self, targetLocation, fileName, fileSource):
fileSource = './test_files/' + fileSource
ntlm = WindowsHttpAuthenticated(username=uname, password=upass)
client = Client(webservice_url, transport=ntlm)
client.set_options(soapheaders={'TargetLocation':targetLocation, 'FileName': fileName})
body = client.factory.create('AIRDocument')
body_file = open(fileSource, 'rb')
body_data = body_file.read()
body.FileByteStream = body_data
return client.service.UploadFile(body)
Running this gets me the following result:
Traceback (most recent call last):
File "test_cases.py", line 639, in test_upload_file_invalid_extension
result_string = self.HM.webserviceUploadFile('9999', 'AD-1234-5424__44.exe',
'test_data.pdf')
File "test_cases.py", line 81, in webserviceUploadFile
return client.service.UploadFile(body)
File "build\bdist.win32\egg\suds\client.py", line 542, in __call__
return client.invoke(args, kwargs)
File "build\bdist.win32\egg\suds\client.py", line 595, in invoke
soapenv = binding.get_message(self.method, args, kwargs)
File "build\bdist.win32\egg\suds\bindings\binding.py", line 120, in get_message
content = self.bodycontent(method, args, kwargs)
File "build\bdist.win32\egg\suds\bindings\document.py", line 63, in bodycontent
p = self.mkparam(method, pd, value)
File "build\bdist.win32\egg\suds\bindings\document.py", line 105, in mkparam
return Binding.mkparam(self, method, pdef, object)
File "build\bdist.win32\egg\suds\bindings\binding.py", line 287, in mkparam
return marshaller.process(content)
File "build\bdist.win32\egg\suds\mx\core.py", line 62, in process
self.append(document, content)
File "build\bdist.win32\egg\suds\mx\core.py", line 75, in append
self.appender.append(parent, content)
File "build\bdist.win32\egg\suds\mx\appender.py", line 102, in append
appender.append(parent, content)
File "build\bdist.win32\egg\suds\mx\appender.py", line 243, in append
Appender.append(self, child, cont)
File "build\bdist.win32\egg\suds\mx\appender.py", line 182, in append
self.marshaller.append(parent, content)
File "build\bdist.win32\egg\suds\mx\core.py", line 75, in append
self.appender.append(parent, content)
File "build\bdist.win32\egg\suds\mx\appender.py", line 102, in append
appender.append(parent, content)
File "build\bdist.win32\egg\suds\mx\appender.py", line 198, in append
child.setText(tostr(content.value))
File "build\bdist.win32\egg\suds\sax\element.py", line 251, in setText
self.text = Text(value)
File "build\bdist.win32\egg\suds\sax\text.py", line 43, in __new__
result = super(Text, cls).__new__(cls, *args, **kwargs)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 10: ordinal
not in range(128)
After much research and talking with the developer of the webservice, I modified the body_data = body_file.read() into body_data = body_file.read().decode("UTF-8") which gets me this error:
Traceback (most recent call last):
File "test_cases.py", line 639, in test_upload_file_invalid_extension
result_string = self.HM.webserviceUploadFile('9999', 'AD-1234-5424__44.exe', 'test_data.pdf')
File "test_cases.py", line 79, in webserviceUploadFile
body_data = body_file.read().decode("utf-8")
File "C:\python27\lib\encodings\utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe2 in position 10: invalid
continuation byte
Which is less than helpful.
After more research into the problem, I tried adding 'errors='ignore'' to the UTF-8 encode, and this was the result:
<TransactionDescription>Error in INTL-CONF_France_PROJ_MA_126807.docx: An exception has been thrown when reading the stream.. Inner Exception: System.Xml.XmlException: The byte 0x03 is not valid at this location. Line 1, position 318.
at System.Xml.XmlExceptionHelper.ThrowXmlException(XmlDictionaryReader reader, String res, String arg1, String arg2, String arg3)
at System.Xml.XmlUTF8TextReader.Read()
at System.ServiceModel.Dispatcher.StreamFormatter.MessageBodyStream.Exhaust(XmlDictionaryReader reader)
at System.ServiceModel.Dispatcher.StreamFormatter.MessageBodyStream.Read(Byte[] buffer, Int32 offset, Int32 count). Source: System.ServiceModel</TransactionDescription>
Which pretty much stumps me on what to do. Based on the result stack trace by the webservice, it looks like it wants UTF-8 but I can't seem to get it to the webservice without Python or SUDS throwing a fit, or by ignoring problems in the encoding. The system I'm working on only takes in MicroSoft office type files (doc, xls, and the like), PDFs, and TXT files, so using something that I have more control on the encoding is not an option. I also tried detecting the encoding used by the sample PDF and the sample DOCX, but using what it suggested (Latin-1, ISO8859-x, and several windows XXXX) all were accepted by Python and SUDS, but not by the webservice.
Also note in the example shown, its most frequently referencing a test to an invalid extension. This error applies even in what should be a test of the successful upload, which is the only time really that the final stacktrace ever shows up.
You can use this base64.b64encode(body_file.read()) and this will return the base64 string value. So your request variable must be a string.
I recently started writing a simple client using the Blogger API to do some basic posting I implemented the client in Python and used the example code verbatim from the Blogger Developer's Guide to login, get the blog id, and make a new post. I ran the script and everything went fine until I got to this line:
return blogger_service.Post(entry, '/feeds/%s/posts/default' % blog_id)
I got the error message:
Traceback (most recent call last):
File "cs1121post.py", line 38, in <module>
cs1121post()
File "cs1121post.py", line 33, in cs1121post
return blogger_service.Post(entry, '/feeds/%s/posts/default' % blog_id)
File "/usr/local/lib/python2.7/dist-packages/gdata/service.py", line 1236, in Post
media_source=media_source, converter=converter)
File "/usr/local/lib/python2.7/dist-packages/gdata/service.py", line 1322, in PostOrPut
headers=extra_headers, url_params=url_params)
File "/usr/local/lib/python2.7/dist-packages/atom/__init__.py", line 93, in optional_warn_function
return f(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/atom/service.py", line 176, in request
content_length = CalculateDataLength(data)
File "/usr/local/lib/python2.7/dist-packages/atom/service.py", line 736, in CalculateDataLength
return len(str(data))
File "/usr/local/lib/python2.7/dist-packages/atom/__init__.py", line 377, in __str__
return self.ToString()
File "/usr/local/lib/python2.7/dist-packages/atom/__init__.py", line 374, in ToString
return ElementTree.tostring(self._ToElementTree(), encoding=string_encoding)
File "/usr/local/lib/python2.7/dist-packages/atom/__init__.py", line 369, in _ToElementTree
self._AddMembersToElementTree(new_tree)
File "/usr/local/lib/python2.7/dist-packages/atom/__init__.py", line 331, in _AddMembersToElementTree
member._BecomeChildElement(tree)
File "/usr/local/lib/python2.7/dist-packages/atom/__init__.py", line 357, in _BecomeChildElement
self._AddMembersToElementTree(new_child)
File "/usr/local/lib/python2.7/dist-packages/atom/__init__.py", line 342, in _AddMembersToElementTree
ExtensionContainer._AddMembersToElementTree(self, tree)
File "/usr/local/lib/python2.7/dist-packages/atom/__init__.py", line 224, in _AddMembersToElementTree
tree.text = self.text.decode(MEMBER_STRING_ENCODING)
AttributeError: 'list' object has no attribute 'decode'
By which I'm taking it that ElementTree is at fault here. I installed ElementTree via
sudo python setup.py install
in case it matters. Is there some known incompatibility between ElementTree and Python v2.7.1? Has this happened to anybody else and how did you get it working? If you need any additional information, please reply to the thread. All the source code that is relevant is basically just the example code from the Developers Guide mentioned above. I haven't modified that at all (not even the variable names). Any input is greatly appreciated.
The stacktrace is actually pretty clear about this: You're calling decode() on a list instead of a tree element. Try getting the first element from the list and calling decode() on that:
firsttext = self.text[0].decode(MEMBER_STRING_ENCODING)
I am consuming a webservice (written in java) - that basically returns a byte[] array (the SOAP equivalent is base64 encoded binary data).
I am using the python suds library and the following code works for me on my mac (and on cygwin under windows), but the decoding does not work on vanilla windows (python 2.6.5). I am primarily a java developer so any help will be really helpful.
from suds.client import Client
import base64,os,shutil,tarfile,StringIO
u = "user"
p = "password"
url = "https://xxxx/?wsdl"
client = Client(url, username=u, password=p)
bin = client.service.getTargz("test")
f = open("tools.tar.gz", "w")
f.write(base64.b64decode(bin.encode('ASCII')))
f.close()
print "finished writing"
tarfile.open("tools.tar.gz").extractall()
Works great on a mac - but on windows gives me this error:
C:\client>python client.py
xml
Getting the sysprep file from the webservice
finished writing
Traceback (most recent call last):
File "client.py", line 28, in
tarfile.open("tools.tar.gz").extractall()
File "C:\Python26\lib\tarfile.py", line 1653, in open
return func(name, "r", fileobj, **kwargs)
File "C:\Python26\lib\tarfile.py", line 1720, in gzopen
**kwargs)
File "C:\Python26\lib\tarfile.py", line 1698, in taropen
return cls(name, mode, fileobj, **kwargs)
File "C:\Python26\lib\tarfile.py", line 1571, in __init__
self.firstmember = self.next()
File "C:\Python26\lib\tarfile.py", line 2317, in next
tarinfo = self.tarinfo.fromtarfile(self)
File "C:\Python26\lib\tarfile.py", line 1235, in fromtarfile
buf = tarfile.fileobj.read(BLOCKSIZE)
File "C:\Python26\lib\gzip.py", line 219, in read
self._read(readsize)
File "C:\Python26\lib\gzip.py", line 271, in _read
uncompress = self.decompress.decompress(buf)
zlib.error: Error -3 while decompressing: invalid distance too far back
Try
f = open("tools.tar.gz", "wb")
It's crucial to tell Python that it's a binary file (in Py3, it also becomes crucial on Unixy systems, but in Py2 it's not strictly needed on them, which is why your code works on MacOSX): the default is text, which, on Windows, translates each \n written into \r\n on disk upon writing.