I am trying to download e-mail attachments using Google's API-s but in this process, I bumped into some interesting errors:
Using Google's example (from here) for users.messages.attachments: get method , I've got a KeyError saying that there is no such keyword as data.
Using the method proposed in Download Attachments from gmail using Gmail API , TypeError: 'Resource' object is not callable is raised at the line:
att=gmail_service.users().messages().attachments().get(userId=user_id, messageId=msg_id,id=att_id).execute()
The relevant part of my code is:
import codecs
message = gmail_service.users().threads().get(id=thread['id'],userId='me',format='metadata').execute()['messages']
themessage = gmail_service.users().messages().get(userId='me', id=message[0]['id']).execute()
attachmentid = themessage['payload']['parts'][1]['body']['attachmentId'].encode('utf-8')
attachment = gmail_service().users().messages().attachments().get(userId='me',messageId=message[0]['id'], id=attachmentid).execute()
Thank you very much in advance and sorry for violating any kind of convention. I am a hobby programmer, my major is not even related to CS.
And the raw information of the e-mail:
{u'historyId': u'87616', u'id': u'14cc2903d8201ce5', u'snippet': u'Lorem ipsum dolor sit amet, consectetur adipiscing elit. Maecenas dapibus leo auctor vulputate', u'sizeEstimate': 12503, u'threadId': u'14cc2903d8201ce5', u'labelIds': [u'INBOX', u'IMPORTANT', u'CATEGORY_PERSONAL', u'UNREAD'], u'payload': {u'mimeType': u'multipart/mixed', u'headers': [{u'name': u'Delivered-To', u'value': u'nagytomi1992#gmail.com'}, {u'name': u'Received', u'value': u'by 10.27.172.12 with SMTP id v12csp3301331wle; Thu, 16 Apr 2015 07:09:35 -0700 (PDT)'}, {u'name': u'X-Received', u'value': u'by 10.194.47.165 with SMTP id e5mr59439281wjn.128.1429193375163; Thu, 16 Apr 2015 07:09:35 -0700 (PDT)'}, {u'name': u'Return-Path', u'value': u'<nghuytungi#windowslive.com>'}, {u'name': u'Received', u'value': u'from DUB004-OMC2S3.hotmail.com (dub004-omc2s3.hotmail.com. [157.55.1.142]) by mx.google.com with ESMTPS id k8si15998291wia.75.2015.04.16.07.09.35 for <nagytomi1992#gmail.com> (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 16 Apr 2015 07:09:35 -0700 (PDT)'}, {u'name': u'Received-SPF', u'value': u'pass (google.com: domain of nghuytungi#windowslive.com designates 157.55.1.142 as permitted sender) client-ip=157.55.1.142;'}, {u'name': u'Authentication-Results', u'value': u'mx.google.com; spf=pass (google.com: domain of nghuytungi#windowslive.com designates 157.55.1.142 as permitted sender) smtp.mail=nghuytungi#windowslive.com'}, {u'name': u'Received', u'value': u'from DUB122-W21 ([157.55.1.137]) by DUB004-OMC2S3.hotmail.com over TLS secured channel with Microsoft SMTPSVC(7.5.7601.22751);\t Thu, 16 Apr 2015 07:09:34 -0700'}, {u'name': u'X-TMN', u'value': u'[u8z/PkebpkoUTRJcKA0Traso7CgFYEaU5ZPsEYJzyy0=]'}, {u'name': u'X-Originating-Email', u'value': u'[nghuytungi#windowslive.com]'}, {u'name': u'Message-ID', u'value': u'<DUB122-W210C6AC460DD3A13E54414D0E40#phx.gbl>'}, {u'name': u'Return-Path', u'value': u'nghuytungi#windowslive.com'}, {u'name': u'Content-Type', u'value': u'multipart/mixed; boundary="_7a3a4cda-38d8-405f-985d-255abada5722_"'}, {u'name': u'From', u'value': u'Tung Nguyen Huy <nghuytungi#windowslive.com>'}, {u'name': u'To', u'value': u'"nagytomi1992#gmail.com" <nagytomi1992#gmail.com>'}, {u'name': u'Subject', u'value': u'Chicago'}, {u'name': u'Date', u'value': u'Thu, 16 Apr 2015 16:09:34 +0200'}, {u'name': u'Importance', u'value': u'Normal'}, {u'name': u'MIME-Version', u'value': u'1.0'}, {u'name': u'X-OriginalArrivalTime', u'value': u'16 Apr 2015 14:09:34.0855 (UTC) FILETIME=[F77AD970:01D0784E]'}],
u'parts': [{u'mimeType': u'multipart/alternative', u'headers': [{u'name': u'Content-Type', u'value': u'multipart/alternative; boundary="_2c3faa5d-3517-45ad-9ae0-82a26eb7b6e5_"'}], u'parts': [{u'mimeType': u'text/plain', u'headers': [{u'name': u'Content-Type', u'value': u'text/plain; charset="iso-8859-1"'}, {u'name': u'Content-Transfer-Encoding', u'value': u'quoted-printable'}], u'body': {u'data': u'TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdC4gTWFlY2VuYXMgZGFwaWJ1cyBsZW8gYXVjdG9yIHZ1bHB1dGF0ZSBzb2RhbGVzLiBOdWxsYSBjb25ndWUgYXVndWUgdmVsIGRpZ25pc3NpbSBjb25zZXF1YXQuIFNlZCBzb2xsaWNpdHVkaW4gZmV1Z2lhdCBsaWd1bGEsIHZpdGFlIHNvbGxpY2l0dWRpbiBlcm9zIHRlbXB1cyBpZC4gTnVsbGEgZWZmaWNpdHVyIG1hZ25hIHB1bHZpbmFyIHZ1bHB1dGF0ZSBwdWx2aW5hci4gTnVsbGFtIHBoYXJldHJhIGV1aXNtb2Qgb3JjaS4gRnVzY2UgbW9sbGlzLCBleCBhIGZpbmlidXMgZWxlaWZlbmQsIGVuaW0gZW5pbSB2ZWhpY3VsYSBvZGlvLCBhIG1hdHRpcyBkaWFtIG1hZ25hIHF1aXMgbGVvLiBQcmFlc2VudCBub24gbmVxdWUgdmVzdGlidWx1bSwgcmhvbmN1cyBudWxsYSBzZWQsIGhlbmRyZXJpdCBsZW8uIFNlZCB0aW5jaWR1bnQgbGVvIGlkIG1hc3NhIGVsZWlmZW5kLCBlZ2V0IGNvbmRpbWVudHVtIGRvbG9yIGludGVyZHVtLlZpdmFtdXMgYXQgaGVuZHJlcml0IGRpYW0uIEFlbmVhbiBpbnRlcmR1bSBjb25zZXF1YXQgcG9ydGEuIE5hbSBhIGF1Z3VlIHNlZCBpcHN1bSB2ZXN0aWJ1bHVtIHByZXRpdW0gcXVpcyBpZCBkdWkuIEFlbmVhbiBpZCByaXN1cyBlbmltLiBEdWlzIHRlbXB1cyBlcm9zIGdyYXZpZGEsIHZlbmVuYXRpcyBtYXNzYSBwb3J0dGl0b3IsIHRlbXBvciB0ZWxsdXMuIEV0aWFtIGNvbnNlcXVhdCBsaWJlcm8gc2l0IGFtZXQgb2RpbyB2dWxwdXRhdGUsIGV1IGVnZXN0YXMgbGlndWxhIHBlbGxlbnRlc3F1ZS4gRHVpcyB1cm5hIHRlbGx1cywgcHJldGl1bSBub24gZXVpc21vZCBub24sIHZpdmVycmEgZWdldCBkdWkuIEFsaXF1YW0gZXQgZXggbWV0dXMuIEluIGV4IGxvcmVtLCBmaW5pYnVzIGlkIHZvbHV0cGF0IGluLCBsb2JvcnRpcyB2ZWwgb3JjaS4gVXQgdnVscHV0YXRlIGx1Y3R1cyBwdXJ1cyBub24gdmVoaWN1bGEuIFByYWVzZW50IG1hZ25hIHNhcGllbiwgc29sbGljaXR1ZGluIG5lYyBtYXNzYSBhYywgcnV0cnVtIGF1Y3RvciBhdWd1ZS4gU2VkIGVzdCBudW5jLCBjb21tb2RvIGhlbmRyZXJpdCBmYWNpbGlzaXMgYWMsIG1vbGVzdGllIGRpZ25pc3NpbSBhcmN1LiBOdWxsYW0gdmVsIGVyb3MgbmVxdWUuIEFsaXF1YW0gZnJpbmdpbGxhIHZlbmVuYXRpcyBlcm9zLCB2ZWwgcGVsbGVudGVzcXVlIG1pIGNvbmd1ZSBzZWQuDQpRdWlzcXVlIGNvbnNlcXVhdCBmYXVjaWJ1cyBlc3QsIGV1IGVmZmljaXR1ciBhdWd1ZSB2aXZlcnJhIGJpYmVuZHVtLiBDcmFzIHZhcml1cyBtb2xlc3RpZSBzYXBpZW4gdXQgdmFyaXVzLiBBZW5lYW4gdml0YWUgcHVydXMgdXJuYS4gRXRpYW0gY29uc2VjdGV0dXIgZXVpc21vZCBsZWN0dXMgc2VkIG1heGltdXMuIE51bmMgbmVjIHRlbGx1cyBkdWkuIFBlbGxlbnRlc3F1ZSBtYWxlc3VhZGEsIGlwc3VtIG5vbiB2ZWhpY3VsYSB1bGxhbWNvcnBlciwgbmliaCBmZWxpcyBmYWNpbGlzaXMgYXVndWUsIGlkIHZlbmVuYXRpcyBtYXNzYSBuaXNsIGluIGZlbGlzLiBGdXNjZSBhbGlxdWFtIGNvbnZhbGxpcyBmZXJtZW50dW0uIEluIG1hbGVzdWFkYSwgYW50ZSB2ZWwgbWFsZXN1YWRhIHNjZWxlcmlzcXVlLCBuZXF1ZSBlcmF0IG1hdHRpcyBtYWduYSwgbm9uIGZhY2lsaXNpcyBtYXVyaXMgdGVsbHVzIG5vbiBlcm9zLiBBZW5lYW4gdml0YWUgZXJvcyBsdWN0dXMgbGlndWxhIHZpdmVycmEgZmluaWJ1cyBub24gcXVpcyBlc3QuIFNlZCBldCBjb25ndWUgc2VtLCB2b2x1dHBhdCBtb2xlc3RpZSBvZGlvLiBOYW0gZXJhdCBzYXBpZW4sIGZpbmlidXMgdmVsIHZvbHV0cGF0IHZlbCwgcG9zdWVyZSBpZCBlbGl0LiBOdWxsYW0gZnJpbmdpbGxhLCBtYXVyaXMgbmVjIHNhZ2l0dGlzIGVmZmljaXR1ciwgbG9yZW0gdXJuYSBtb2xlc3RpZSB0ZWxsdXMsIG5vbiBiaWJlbmR1bSBvcmNpIGlwc3VtIHV0IHF1YW0uIE1hdXJpcyBxdWlzIGxvcmVtIHNvZGFsZXMsIHBoYXJldHJhIHNlbSBpbiwgbW9sZXN0aWUgdG9ydG9yLg0KRG9uZWMgYXQgYXJjdSBzYWdpdHRpcywgcHVsdmluYXIgbGFjdXMgYWMsIHZlbmVuYXRpcyBtaS4gSW4gaGFjIGhhYml0YXNzZSBwbGF0ZWEgZGljdHVtc3QuIFF1aXNxdWUgcGhhcmV0cmEgZHVpIGFjIHBvc3VlcmUgbGFjaW5pYS4gRHVpcyBzY2VsZXJpc3F1ZSBwaGFyZXRyYSB1cm5hLCB1dCB1bHRyaWNlcyB2ZWxpdCB1bHRyaWNpZXMgaW4uIFByYWVzZW50IHZhcml1cyBlc3QgYWMgY29uZ3VlIGxvYm9ydGlzLiBNYWVjZW5hcyBpbXBlcmRpZXQgbW9sZXN0aWUgcHVydXMsIGEgZWZmaWNpdHVyIGRvbG9yIHZvbHV0cGF0IHZlbC4gRG9uZWMgdmVsaXQganVzdG8sIGZhY2lsaXNpcyBlZ2V0IGV4IHNpdCBhbWV0LCBiaWJlbmR1bSBhY2N1bXNhbiBuaXNsLiBJbiBpbiBsaWd1bGEgaWQgZGlhbSBsYWNpbmlhIGx1Y3R1cyB2aXRhZSBxdWlzIHZlbGl0LiBDcmFzIGFsaXF1YW0gZGlhbSB2aXRhZSBlcm9zIGZlcm1lbnR1bSBibGFuZGl0LiBBZW5lYW4gYXQgcmhvbmN1cyBvZGlvLiBTZWQgdWx0cmljZXMgYmliZW5kdW0gZXJhdCwgYWMgcGxhY2VyYXQgdHVycGlzIHNjZWxlcmlzcXVlIGlkLiBOdW5jIGNvbmRpbWVudHVtIHRvcnRvciBub24gZXN0IGV1aXNtb2QsIGEgbW9sZXN0aWUgbWkgZXVpc21vZC4gUGhhc2VsbHVzIHV0IHZlbGl0IHZlbCBqdXN0byBzY2VsZXJpc3F1ZSBhY2N1bXNhbiB2ZWwgdXQgbmlzbC4gQ3VyYWJpdHVyIGZyaW5naWxsYSBleCB1dCBzYXBpZW4gYWNjdW1zYW4gY29uZ3VlLiBVdCBtb2xsaXMgYXQgbGlndWxhIGVnZXQgZmluaWJ1cy4NClZlc3RpYnVsdW0gZWdlc3RhcyBsYWN1cyBsZW8sIGF0IG1vbGVzdGllIGF1Z3VlIHByZXRpdW0gaWQuIEludGVnZXIgc2NlbGVyaXNxdWUgZWZmaWNpdHVyIG5pc2kgc2l0IGFtZXQgY29uc2VjdGV0dXIuIE51bGxhIHJpc3VzIG1pLCBmZXVnaWF0IGVnZXQgaXBzdW0gYXQsIGZpbmlidXMgdGluY2lkdW50IG1pLiBQcm9pbiB2ZWwgdGVsbHVzIGEgbG9yZW0gc2FnaXR0aXMgbWF4aW11cy4gVXQgdnVscHV0YXRlIGVyYXQgYWxpcXVldCBpcHN1bSBydXRydW0gbGFjaW5pYS4gQ2xhc3MgYXB0ZW50IHRhY2l0aSBzb2Npb3NxdSBhZCBsaXRvcmEgdG9ycXVlbnQgcGVyIGNvbnViaWEgbm9zdHJhLCBwZXIgaW5jZXB0b3MgaGltZW5hZW9zLiBEb25lYyBpZCBtb2xsaXMgc2VtLiBNYWVjZW5hcyBmZXVnaWF0IHJpc3VzIHZlbCBmZXVnaWF0IGVmZmljaXR1ci4gRnVzY2UgYW50ZSB0ZWxsdXMsIGJpYmVuZHVtIGF0IHByZXRpdW0gYXVjdG9yLCBtYXR0aXMgZWdldCBpcHN1bS4gTnVuYyBzYWdpdHRpcyB0b3J0b3IgcXVpcyBhcmN1IHZpdmVycmEsIHZlbCBsb2JvcnRpcyBsYWN1cyBldWlzbW9kLiBRdWlzcXVlIGRpY3R1bSB0ZW1wb3IgYmliZW5kdW0uIFNlZCBtYXhpbXVzIGxpYmVybyBzaXQgYW1ldCBqdXN0byBmcmluZ2lsbGEgdHJpc3RpcXVlLiAJCSAJICAgCQkgIA==', u'size': 3451}, u'partId': u'0.0', u'filename': u''}, {u'mimeType': u'text/html', u'headers': [{u'name': u'Content-Type', u'value': u'text/html; charset="iso-8859-1"'}, {u'name': u'Content-Transfer-Encoding', u'value': u'quoted-printable'}], u'body': {u'data': u'PGh0bWw-DQo8aGVhZD4NCjxzdHlsZT48IS0tDQouaG1tZXNzYWdlIFANCnsNCm1hcmdpbjowcHg7DQpwYWRkaW5nOjBweA0KfQ0KYm9keS5obW1lc3NhZ2UNCnsNCmZvbnQtc2l6ZTogMTJwdDsNCmZvbnQtZmFtaWx5OkNhbGlicmkNCn0NCi0tPjwvc3R5bGU-PC9oZWFkPg0KPGJvZHkgY2xhc3M9J2htbWVzc2FnZSc-PGRpdiBkaXI9J2x0cic-PHAgc3R5bGU9ImNvbG9yOiByZ2IoMzQsIDM0LCAzNCk7IGZvbnQtZmFtaWx5OiBhcmlhbCwgc2Fucy1zZXJpZjsgZm9udC1zaXplOiAxMi44MDAwMDAxOTA3MzQ5cHg7IHRleHQtYWxpZ246IGp1c3RpZnk7IG1hcmdpbi1ib3R0b206IDE0cHg7IGJhY2tncm91bmQtY29sb3I6IHJnYigyNTUsIDI1NSwgMjU1KTsiPjxmb250IGNvbG9yPSIjMDAwMDAwIiBmYWNlPSJBcmlhbCwgSGVsdmV0aWNhLCBzYW5zIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOiAxMXB4OyBsaW5lLWhlaWdodDogMTRweDsiPkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0LCBjb25zZWN0ZXR1ciBhZGlwaXNjaW5nIGVsaXQuIE1hZWNlbmFzIGRhcGlidXMgbGVvIGF1Y3RvciB2dWxwdXRhdGUgc29kYWxlcy4gTnVsbGEgY29uZ3VlIGF1Z3VlIHZlbCBkaWduaXNzaW0gY29uc2VxdWF0LiBTZWQgc29sbGljaXR1ZGluIGZldWdpYXQgbGlndWxhLCB2aXRhZSBzb2xsaWNpdHVkaW4gZXJvcyB0ZW1wdXMgaWQuIE51bGxhIGVmZmljaXR1ciBtYWduYSBwdWx2aW5hciB2dWxwdXRhdGUgcHVsdmluYXIuIE51bGxhbSBwaGFyZXRyYSBldWlzbW9kIG9yY2kuIEZ1c2NlIG1vbGxpcywgZXggYSBmaW5pYnVzIGVsZWlmZW5kLCBlbmltIGVuaW0gdmVoaWN1bGEgb2RpbywgYSBtYXR0aXMgZGlhbSBtYWduYSBxdWlzIGxlby4gUHJhZXNlbnQgbm9uIG5lcXVlIHZlc3RpYnVsdW0sIHJob25jdXMgbnVsbGEgc2VkLCBoZW5kcmVyaXQgbGVvLiBTZWQgdGluY2lkdW50IGxlbyBpZCBtYXNzYSBlbGVpZmVuZCwgZWdldCBjb25kaW1lbnR1bSBkb2xvciBpbnRlcmR1bS48L3NwYW4-PC9mb250PjwvcD48cCBzdHlsZT0iY29sb3I6IHJnYigzNCwgMzQsIDM0KTsgZm9udC1mYW1pbHk6IGFyaWFsLCBzYW5zLXNlcmlmOyBmb250LXNpemU6IDEyLjgwMDAwMDE5MDczNDlweDsgdGV4dC1hbGlnbjoganVzdGlmeTsgbWFyZ2luLWJvdHRvbTogMTRweDsgYmFja2dyb3VuZC1jb2xvcjogcmdiKDI1NSwgMjU1LCAyNTUpOyI-PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZTogMTFweDsgbGluZS1oZWlnaHQ6IDE0cHg7IGNvbG9yOiByZ2IoMCwgMCwgMCk7IGZvbnQtZmFtaWx5OiBBcmlhbCwgSGVsdmV0aWNhLCBzYW5zOyI-Vml2YW11cyBhdCBoZW5kcmVyaXQgZGlhbS4gQWVuZWFuIGludGVyZHVtIGNvbnNlcXVhdCBwb3J0YS4gTmFtIGEgYXVndWUgc2VkIGlwc3VtIHZlc3RpYnVsdW0gcHJldGl1bSBxdWlzIGlkIGR1aS4gQWVuZWFuIGlkIHJpc3VzIGVuaW0uIER1aXMgdGVtcHVzIGVyb3MgZ3JhdmlkYSwgdmVuZW5hdGlzIG1hc3NhIHBvcnR0aXRvciwgdGVtcG9yIHRlbGx1cy4gRXRpYW0gY29uc2VxdWF0IGxpYmVybyBzaXQgYW1ldCBvZGlvIHZ1bHB1dGF0ZSwgZXUgZWdlc3RhcyBsaWd1bGEgcGVsbGVudGVzcXVlLiBEdWlzIHVybmEgdGVsbHVzLCBwcmV0aXVtIG5vbiBldWlzbW9kIG5vbiwgdml2ZXJyYSBlZ2V0IGR1aS4gQWxpcXVhbSBldCBleCBtZXR1cy4gSW4gZXggbG9yZW0sIGZpbmlidXMgaWQgdm9sdXRwYXQgaW4sIGxvYm9ydGlzIHZlbCBvcmNpLiBVdCB2dWxwdXRhdGUgbHVjdHVzIHB1cnVzIG5vbiB2ZWhpY3VsYS4gUHJhZXNlbnQgbWFnbmEgc2FwaWVuLCBzb2xsaWNpdHVkaW4gbmVjIG1hc3NhIGFjLCBydXRydW0gYXVjdG9yIGF1Z3VlLiBTZWQgZXN0IG51bmMsIGNvbW1vZG8gaGVuZHJlcml0IGZhY2lsaXNpcyBhYywgbW9sZXN0aWUgZGlnbmlzc2ltIGFyY3UuIE51bGxhbSB2ZWwgZXJvcyBuZXF1ZS4gQWxpcXVhbSBmcmluZ2lsbGEgdmVuZW5hdGlzIGVyb3MsIHZlbCBwZWxsZW50ZXNxdWUgbWkgY29uZ3VlIHNlZC48L3NwYW4-PGJyPjwvcD48cCBzdHlsZT0iY29sb3I6IHJnYigzNCwgMzQsIDM0KTsgZm9udC1mYW1pbHk6IGFyaWFsLCBzYW5zLXNlcmlmOyBmb250LXNpemU6IDEyLjgwMDAwMDE5MDczNDlweDsgdGV4dC1hbGlnbjoganVzdGlmeTsgbWFyZ2luLWJvdHRvbTogMTRweDsgYmFja2dyb3VuZC1jb2xvcjogcmdiKDI1NSwgMjU1LCAyNTUpOyI-PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZTogMTFweDsgbGluZS1oZWlnaHQ6IDE0cHg7IGNvbG9yOiByZ2IoMCwgMCwgMCk7IGZvbnQtZmFtaWx5OiBBcmlhbCwgSGVsdmV0aWNhLCBzYW5zOyI-UXVpc3F1ZSBjb25zZXF1YXQgZmF1Y2lidXMgZXN0LCBldSBlZmZpY2l0dXIgYXVndWUgdml2ZXJyYSBiaWJlbmR1bS4gQ3JhcyB2YXJpdXMgbW9sZXN0aWUgc2FwaWVuIHV0IHZhcml1cy4gQWVuZWFuIHZpdGFlIHB1cnVzIHVybmEuIEV0aWFtIGNvbnNlY3RldHVyIGV1aXNtb2QgbGVjdHVzIHNlZCBtYXhpbXVzLiBOdW5jIG5lYyB0ZWxsdXMgZHVpLiBQZWxsZW50ZXNxdWUgbWFsZXN1YWRhLCBpcHN1bSBub24gdmVoaWN1bGEgdWxsYW1jb3JwZXIsIG5pYmggZmVsaXMgZmFjaWxpc2lzIGF1Z3VlLCBpZCB2ZW5lbmF0aXMgbWFzc2EgbmlzbCBpbiBmZWxpcy4gRnVzY2UgYWxpcXVhbSBjb252YWxsaXMgZmVybWVudHVtLiBJbiBtYWxlc3VhZGEsIGFudGUgdmVsIG1hbGVzdWFkYSBzY2VsZXJpc3F1ZSwgbmVxdWUgZXJhdCBtYXR0aXMgbWFnbmEsIG5vbiBmYWNpbGlzaXMgbWF1cmlzIHRlbGx1cyBub24gZXJvcy4gQWVuZWFuIHZpdGFlIGVyb3MgbHVjdHVzIGxpZ3VsYSB2aXZlcnJhIGZpbmlidXMgbm9uIHF1aXMgZXN0LiBTZWQgZXQgY29uZ3VlIHNlbSwgdm9sdXRwYXQgbW9sZXN0aWUgb2Rpby4gTmFtIGVyYXQgc2FwaWVuLCBmaW5pYnVzIHZlbCB2b2x1dHBhdCB2ZWwsIHBvc3VlcmUgaWQgZWxpdC4gTnVsbGFtIGZyaW5naWxsYSwgbWF1cmlzIG5lYyBzYWdpdHRpcyBlZmZpY2l0dXIsIGxvcmVtIHVybmEgbW9sZXN0aWUgdGVsbHVzLCBub24gYmliZW5kdW0gb3JjaSBpcHN1bSB1dCBxdWFtLiBNYXVyaXMgcXVpcyBsb3JlbSBzb2RhbGVzLCBwaGFyZXRyYSBzZW0gaW4sIG1vbGVzdGllIHRvcnRvci48L3NwYW4-PGJyPjwvcD48cCBzdHlsZT0iY29sb3I6IHJnYigzNCwgMzQsIDM0KTsgZm9udC1mYW1pbHk6IGFyaWFsLCBzYW5zLXNlcmlmOyBmb250LXNpemU6IDEyLjgwMDAwMDE5MDczNDlweDsgdGV4dC1hbGlnbjoganVzdGlmeTsgbWFyZ2luLWJvdHRvbTogMTRweDsgYmFja2dyb3VuZC1jb2xvcjogcmdiKDI1NSwgMjU1LCAyNTUpOyI-PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZTogMTFweDsgbGluZS1oZWlnaHQ6IDE0cHg7IGNvbG9yOiByZ2IoMCwgMCwgMCk7IGZvbnQtZmFtaWx5OiBBcmlhbCwgSGVsdmV0aWNhLCBzYW5zOyI-RG9uZWMgYXQgYXJjdSBzYWdpdHRpcywgcHVsdmluYXIgbGFjdXMgYWMsIHZlbmVuYXRpcyBtaS4gSW4gaGFjIGhhYml0YXNzZSBwbGF0ZWEgZGljdHVtc3QuIFF1aXNxdWUgcGhhcmV0cmEgZHVpIGFjIHBvc3VlcmUgbGFjaW5pYS4gRHVpcyBzY2VsZXJpc3F1ZSBwaGFyZXRyYSB1cm5hLCB1dCB1bHRyaWNlcyB2ZWxpdCB1bHRyaWNpZXMgaW4uIFByYWVzZW50IHZhcml1cyBlc3QgYWMgY29uZ3VlIGxvYm9ydGlzLiBNYWVjZW5hcyBpbXBlcmRpZXQgbW9sZXN0aWUgcHVydXMsIGEgZWZmaWNpdHVyIGRvbG9yIHZvbHV0cGF0IHZlbC4gRG9uZWMgdmVsaXQganVzdG8sIGZhY2lsaXNpcyBlZ2V0IGV4IHNpdCBhbWV0LCBiaWJlbmR1bSBhY2N1bXNhbiBuaXNsLiBJbiBpbiBsaWd1bGEgaWQgZGlhbSBsYWNpbmlhIGx1Y3R1cyB2aXRhZSBxdWlzIHZlbGl0LiBDcmFzIGFsaXF1YW0gZGlhbSB2aXRhZSBlcm9zIGZlcm1lbnR1bSBibGFuZGl0LiBBZW5lYW4gYXQgcmhvbmN1cyBvZGlvLiBTZWQgdWx0cmljZXMgYmliZW5kdW0gZXJhdCwgYWMgcGxhY2VyYXQgdHVycGlzIHNjZWxlcmlzcXVlIGlkLiBOdW5jIGNvbmRpbWVudHVtIHRvcnRvciBub24gZXN0IGV1aXNtb2QsIGEgbW9sZXN0aWUgbWkgZXVpc21vZC4gUGhhc2VsbHVzIHV0IHZlbGl0IHZlbCBqdXN0byBzY2VsZXJpc3F1ZSBhY2N1bXNhbiB2ZWwgdXQgbmlzbC4gQ3VyYWJpdHVyIGZyaW5naWxsYSBleCB1dCBzYXBpZW4gYWNjdW1zYW4gY29uZ3VlLiBVdCBtb2xsaXMgYXQgbGlndWxhIGVnZXQgZmluaWJ1cy48L3NwYW4-PGJyPjwvcD48cCBzdHlsZT0iY29sb3I6IHJnYigzNCwgMzQsIDM0KTsgZm9udC1mYW1pbHk6IGFyaWFsLCBzYW5zLXNlcmlmOyBmb250LXNpemU6IDEyLjgwMDAwMDE5MDczNDlweDsgdGV4dC1hbGlnbjoganVzdGlmeTsgbWFyZ2luLWJvdHRvbTogMTRweDsgYmFja2dyb3VuZC1jb2xvcjogcmdiKDI1NSwgMjU1LCAyNTUpOyI-PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZTogMTFweDsgbGluZS1oZWlnaHQ6IDE0cHg7IGNvbG9yOiByZ2IoMCwgMCwgMCk7IGZvbnQtZmFtaWx5OiBBcmlhbCwgSGVsdmV0aWNhLCBzYW5zOyI-VmVzdGlidWx1bSBlZ2VzdGFzIGxhY3VzIGxlbywgYXQgbW9sZXN0aWUgYXVndWUgcHJldGl1bSBpZC4gSW50ZWdlciBzY2VsZXJpc3F1ZSBlZmZpY2l0dXIgbmlzaSBzaXQgYW1ldCBjb25zZWN0ZXR1ci4gTnVsbGEgcmlzdXMgbWksIGZldWdpYXQgZWdldCBpcHN1bSBhdCwgZmluaWJ1cyB0aW5jaWR1bnQgbWkuIFByb2luIHZlbCB0ZWxsdXMgYSBsb3JlbSBzYWdpdHRpcyBtYXhpbXVzLiBVdCB2dWxwdXRhdGUgZXJhdCBhbGlxdWV0IGlwc3VtIHJ1dHJ1bSBsYWNpbmlhLiBDbGFzcyBhcHRlbnQgdGFjaXRpIHNvY2lvc3F1IGFkIGxpdG9yYSB0b3JxdWVudCBwZXIgY29udWJpYSBub3N0cmEsIHBlciBpbmNlcHRvcyBoaW1lbmFlb3MuIERvbmVjIGlkIG1vbGxpcyBzZW0uIE1hZWNlbmFzIGZldWdpYXQgcmlzdXMgdmVsIGZldWdpYXQgZWZmaWNpdHVyLiBGdXNjZSBhbnRlIHRlbGx1cywgYmliZW5kdW0gYXQgcHJldGl1bSBhdWN0b3IsIG1hdHRpcyBlZ2V0IGlwc3VtLiBOdW5jIHNhZ2l0dGlzIHRvcnRvciBxdWlzIGFyY3Ugdml2ZXJyYSwgdmVsIGxvYm9ydGlzIGxhY3VzIGV1aXNtb2QuIFF1aXNxdWUgZGljdHVtIHRlbXBvciBiaWJlbmR1bS4gU2VkIG1heGltdXMgbGliZXJvIHNpdCBhbWV0IGp1c3RvIGZyaW5naWxsYSB0cmlzdGlxdWUuPC9zcGFuPjwvcD4gCQkgCSAgIAkJICA8L2Rpdj48L2JvZHk-DQo8L2h0bWw-', u'size': 5160}, u'partId': u'0.1', u'filename': u''}], u'body': {u'size': 0}, u'filename': u''},
{u'mimeType': u'text/plain', u'headers': [{u'name': u'Content-Type', u'value': u'text/plain'}, {u'name': u'Content-Transfer-Encoding', u'value': u'base64'}, {u'name': u'Content-Disposition', u'value': u'attachment; filename="villamosE.txt"'}], u'body': {u'attachmentId': u'ANGjdJ8D_1cYbyaSlXR4LOH0WEcY5G0wXTmiG4d85fkEfkiVGtaicJ_sEsJBSj8UsRfQv79C2Now8DKKvpuN4uhA1x3WBGATfxtFHd6m8YUOQkWPUZje37eLFeLAHjF1eiGo3dg3PPNRidhowbB6dXw4WW2Nqd9DioX3K_M9JmxTq4q_X_GUUHB3R5X_Y9yE-tvFdzeKMdWHLQG5-ah5sVaRDjS4gaD6U2ibMHV75yLfNj3P17xPfG9zU4igMYSS6QPI60Ry5aM7M1S_C30amy-HfJpA6fi-gEqqBtpGHg', u'size': 748}, u'partId': u'1', u'filename': u'villamosE.txt'}],
u'body': {u'size': 0}, u'filename': u''}}
The not callable error seems to imply the last line of your code should read gmail_service.users(), not gmail_service().users() as it currently does
Related
I am trying to get answer in json format but it simply returns a String.What am i doing wrong?
from flask import Flask, request, Response
import mysql.connector
import json
app = Flask(__name__)
def createConnection():
connection = mysql.connector.connect(user="akshay", password="akshay", host="localhost", database="BOOKS")
return connection
#app.route("/contact")
def getContacts():
print(request)
connection = createConnection()
cursor = connection.cursor()
cursor.execute("SELECT ID, NAME, AUTHOR, SUBJECT, PRICE FROM BOOKS;")
contacts = []
for (ID, NAME, AUTHOR, SUBJECT, PRICE) in cursor:
contacts.append({
'ID': ID,
'NAME': NAME,
"AUTHOR": AUTHOR,
"SUBJECT": SUBJECT,
"PRICE": PRICE
})
cursor.close()
connection.close()
response = Response()
response.content_type ='application/json'
response.data = json.loads(json.dumps(contacts))
return response
app.run('0.0.0.0', 4000)
And this is Output::
[{u'PRICE': 9999.999, u'AUTHOR': u'AKSHAY', u'NAME': u'AKSHAY', u'ID':
4, u'SUBJECT': u'BIG_DATA'}, {u'PRICE': 123.456, u'AUTHOR': u'Yashwant
Kanetkar', u'NAME': u'Exploring C', u'ID': 1001, u'SUBJECT': u'C
Programming'}, {u'PRICE': 371.019, u'AUTHOR': u'Yashwant Kanetkar',
u'NAME': u'Pointers in C', u'ID': 1002, u'SUBJECT': u'C Programming'},
{u'PRICE': 334.215, u'AUTHOR': u'E Balaguruswami', u'NAME': u'ANSI C
Programming', u'ID': 1003, u'SUBJECT': u'C Programming'}, {u'PRICE':
140.121, u'AUTHOR': u'Dennis Ritchie', u'NAME': u'ANSI C Programming', u'ID': 1004, u'SUBJECT': u'C Programming'}, {u'PRICE': 417.764,
u'AUTHOR': u'Herbert Schildt', u'NAME': u'C++ Complete Reference',
u'ID': 2001, u'SUBJECT': u'C++ Programming'}, {u'PRICE': 620.665,
u'AUTHOR': u'Stanley Lippman', u'NAME': u'C++ Primer', u'ID': 2002,
u'SUBJECT': u'C++ Programming'}, {u'PRICE': 987.213, u'AUTHOR':
u'Bjarne Stroustrup', u'NAME': u'C++ Programming Language', u'ID':
2003, u'SUBJECT': u'C++ Programming'}, {u'PRICE': 525.121, u'AUTHOR':
u'Herbert Schildt', u'NAME': u'Java Complete Reference', u'ID': 3001,
u'SUBJECT': u'Java Programming'}, {u'PRICE': 575.651, u'AUTHOR': u'Cay
Horstmann', u'NAME': u'Core Java Volume I', u'ID': 3002, u'SUBJECT':
u'Java Programming'}, {u'PRICE': 458.238, u'AUTHOR': u'James Gosling',
u'NAME': u'Java Programming Language', u'ID': 3003, u'SUBJECT': u'Java
Programming'}, {u'PRICE': 567.391, u'AUTHOR': u'Peter Galvin',
u'NAME': u'Operatig System Concepts', u'ID': 4001, u'SUBJECT':
u'Operating Systems'}, {u'PRICE': 421.938, u'AUTHOR': u'Mauris J
Bach', u'NAME': u'Design of UNIX Operating System', u'ID': 4002,
u'SUBJECT': u'Operating Systems'}, {u'PRICE': 352.822, u'AUTHOR':
u'Uresh Vahalia', u'NAME': u'UNIX Internals', u'ID': 4003, u'SUBJECT':
u'Operating Systems'}, {u'PRICE': 872.652, u'AUTHOR': u'Ayn Rand',
u'NAME': u'Atlas Shrugged', u'ID': 8001, u'SUBJECT': u'Novell'},
{u'PRICE': 9999.999, u'AUTHOR': u'AKSHAY', u'NAME': u'AKSHAY', u'ID':
9001, u'SUBJECT': u'BIG_DATA'}]
There is nothing particularly wrong with your code, just remove the call to json.loads() so that the JSON encoded data is used as the response data.
Calling json.loads() will recreate and pass a Python data structure (list of dictionaries in this case) to the response, which will not serialise as valid JSON. In Python 2 it will serialise using the Python repr() format as produced by str() which will not always produce a JSON compatible string (the u'' string prefix for example is not valid JSON). In Python 3 is will probably cause an exception because it won't be able to convert to a bytes object.
You could improve the code by using flask.json.jsonify() which will return a Response object with the correct content type header and a JSON body:
from flask.json import jsonify
#app.route("/contact")
def getContacts():
# create contacts...
return jsonify(contacts)
Ok ,finally got the proper output::
/ 20171119101107
// http://localhost:4000/contact
[
{
"SUBJECT": "BIG_DATA",
"PRICE": 9999.999,
"AUTHOR": "AKSHAY",
"ID": 4,
"NAME": "AKSHAY"
},
{
"SUBJECT": "C Programming",
"PRICE": 123.456,
"AUTHOR": "Yashwant Kanetkar",
"ID": 1001,
"NAME": "Exploring C"
},
{
"SUBJECT": "C Programming",
"PRICE": 371.019,
"AUTHOR": "Yashwant Kanetkar",
"ID": 1002,
"NAME": "Pointers in C"
},
I removed json.loads() and installed json viewer plugin.Thank you very much for help.
I'm having a really hard time to get a track id in Spotify search endpoint.
It is deeply nested.
So, if I do this:
results = sp.search(q='artist:' + 'Nirvava + ' track:' + 'Milk it', type='track')
pprint.pprint(results)
I am able to get:
{u'tracks': {u'href': u'https://api.spotify.com/v1/search?query=artist%3ANirvana+track%3AMilk+it&type=track&offset=0&limit=10',
u'items': [{u'album': {u'album_type': u'album',
u'artists': [{u'external_urls': {u'spotify': u'https://open.spotify.com/artist/6olE6TJLqED3rqDCT0FyPh'},
u'href': u'https://api.spotify.com/v1/artists/6olE6TJLqED3rqDCT0FyPh',
u'id': u'6olE6TJLqED3rqDCT0FyPh',
u'name': u'Nirvana',
u'type': u'artist',
u'uri': u'spotify:artist:6olE6TJLqED3rqDCT0FyPh'}],
u'available_markets': [u'CA',
u'MX',
u'US'],
u'external_urls': {u'spotify': u'https://open.spotify.com/album/7wOOA7l306K8HfBKfPoafr'},
u'href': u'https://api.spotify.com/v1/albums/7wOOA7l306K8HfBKfPoafr',
u'id': u'7wOOA7l306K8HfBKfPoafr',
u'images': [{u'height': 640,
u'url': u'https://i.scdn.co/image/3dd2699f0fcf661c35d45745313b64e50f63f91f',
u'width': 640},
{u'height': 300,
u'url': u'https://i.scdn.co/image/a6c604a82d274e4728a8660603ef31ea35e9e1bd',
u'width': 300},
{u'height': 64,
u'url': u'https://i.scdn.co/image/f52728b0ecf5b6bfc998dfd0f6e5b6b5cdfe73f1',
u'width': 64}],
u'name': u'In Utero - 20th Anniversary Remaster',
u'type': u'album',
u'uri': u'spotify:album:7wOOA7l306K8HfBKfPoafr'},
u'artists': [{u'external_urls': {u'spotify': u'https://open.spotify.com/artist/6olE6TJLqED3rqDCT0FyPh'},
u'href': u'https://api.spotify.com/v1/artists/6olE6TJLqED3rqDCT0FyPh',
u'id': u'6olE6TJLqED3rqDCT0FyPh',
u'name': u'Nirvana',
u'type': u'artist',
u'uri': u'spotify:artist:6olE6TJLqED3rqDCT0FyPh'}],
u'available_markets': [u'CA', u'MX', u'US'],
u'disc_number': 1,
u'duration_ms': 234746,
u'explicit': False,
u'external_ids': {u'isrc': u'USGF19960708'},
u'external_urls': {u'spotify': u'https://open.spotify.com/track/4rtZtLpriBscg7zta3TZxp'},
u'href': u'https://api.spotify.com/v1/tracks/4rtZtLpriBscg7zta3TZxp',
u'id': u'4rtZtLpriBscg7zta3TZxp',
u'name': u'Milk It',
u'popularity': 43,
u'preview_url': None,
u'track_number': 8,
u'type': u'track',
-----> u'uri':u'spotify:track:4rtZtLpriBscg7zta3TZxp'},
QUESTION:
now, how do I fetch the last 'uri' (u'uri': u'spotify:track:4rtZtLpriBscg7zta3TZxp'}, under the name 'Milk It'?
>>> print results['tracks']['items'][0]['uri']
spotify:track:4rtZtLpriBscg7zta3TZxp
For each article that a keyword list is returned for. We want to join all the words using the key --> values into a list, as shown below. I would like to strip out the ‘u’ from the list, before I do the append. Then we want to compare how many common words in both list and return the result.
Example lists returned from dic['keywords']:
Article one returns:
[
{
u'value': u'Dunford, Joseph F Jr',
u'name': u'persons',
u'rank': u'1'
},
{
u'value': u'Afghanistan',
u'name': u'glocations',
u'rank': u'1'
},
{
u'value': u'Afghan National Police',
u'name': u'organizations',
u'rank': u'1'
},
{
u'value': u'Afghanistan War (2001- )',
u'name': u'subject',
u'rank': u'1'
},
{
u'value': u'Defense and Military Forces',
u'name': u'subject',
u'rank': u'2'
}
]
Article two returns:
[
{
u'value': u'Gall, Carlotta',
u'name': u'persons',
u'rank': u'1'
},
{
u'value': u'Gannon, Kathy',
u'name': u'persons',
u'rank': u'2'
},
{
u'value': u'Niedringhaus, Anja (1965-2014)',
u'name': u'persons',
u'rank': u'3'
},
{
u'value': u'Kabul (Afghanistan)',
u'name': u'glocations',
u'rank': u'2'
},
{
u'value': u'Afghanistan',
u'name': u'glocations',
u'rank': u'1'
},
{
u'value': u'Afghan National Police',
u'name': u'organizations',
u'rank': u'1'
},
{
u'value': u'Afghanistan War (2001- )',
u'name': u'subject',
u'rank': u'1'
}
]
Desired Output:
List1 = ['Dunford, Joseph F Jr',’ Afghanistan’, ‘Afghan National Police’, ‘: Afghanistan War (2001- )’, ‘Defense and Military Forces’]
List2 = [‘Gall, Carlotta'’,’ u'Gannon, Kathy',’ Niedringhaus, Anja (1965-2014)’,’Afghanistan’]
Keywords in common: 2
My Code is as follows:
from flask import Flask, render_template, request, session, g, redirect, url_for
from nytimesarticle import articleAPI
api = articleAPI('X')
articles = api.search( q = 'Afghan War',
fq = {'headline':'', 'source':['Reuters','AP', 'The New York Times']},
begin_date = 20111231 )
def parse_articles(articles):
'''
This function takes in a response to the NYT api and parses
the articles into a list of dictionaries
'''
news = []
for i in articles['response']['docs']:
dic = {}
dic['id'] = i['_id']
if i['abstract'] is not None:
dic['abstract'] = i['abstract'].encode("utf8")
dic['headline'] = i['headline']['main'].encode("utf8")
dic['desk'] = i['news_desk']
dic['date'] = i['pub_date'][0:10] # cutting time of day.
dic['section'] = i['section_name']
dic['keywords'] = i['keywords']
print dic['keywords']
if i['snippet'] is not None:
dic['snippet'] = i['snippet'].encode("utf8")
dic['source'] = i['source']
dic['type'] = i['type_of_material']
dic['url'] = i['web_url']
dic['word_count'] = i['word_count']
# locations
locations = []
for x in range(0,len(i['keywords'])):
if 'glocations' in i['keywords'][x]['name']:
locations.append(i['keywords'][x]['value'])
dic['locations'] = locations
# subject
subjects = []
for x in range(0,len(i['keywords'])):
if 'subject' in i['keywords'][x]['name']:
subjects.append(i['keywords'][x]['value'])
dic['subjects'] = subjects
news.append(dic)
return(news)
print(parse_articles(articles))
You can use list comprehension to build lists from the given dict.
d = [{u'value': u'Dunford, Joseph F Jr', u'name': u'persons', u'rank': u'1'}, {u'value': u'Afghanistan', u'name': u'glocations', u'rank': u'1'}, {u'value': u'Afghan National Police', u'name': u'organizations', u'rank': u'1'}, {u'value': u'Afghanistan War (2001- )', u'name': u'subject', u'rank': u'1'}, {u'value': u'Defense and Military Forces', u'name': u'subject', u'rank': u'2'}]
print [v['value'] for v in d] # prints [u'Dunford, Joseph F Jr', u'Afghanistan', u'Afghan National Police', u'Afghanistan War (2001- )', u'Defense and Military Forces']
I am getting the following error for the following code and I am not able to solve the error. The error occures after the loop has created some files correctly.
I am using the following code:
import urllib
import json
symbolslist = open("C:/Users/Asus/Desktop/Python/web scraper/scraping/symbols.txt").read()
symbolslist = [s for s in symbolslist.split("\n") if s]
for symbol in symbolslist:
myfile = open("C:/Users/Asus/Desktop/Python/web scraper/scraping/year_data/" + symbol +".txt" , "w+")
myfile.close()
htmltext = urllib.urlopen("http://www.bloomberg.com/markets/api/bulk-time-series/price/"+symbol+"%3AUS?timeFrame=1_YEAR")
data = json.load(htmltext)
try:
datapoints = data[0]["price"]
except KeyError:
print 'Unknown symbol: {0}'.format(symbol)
continue
myfile = open("C:/Users/Asus/Desktop/Python/web scraper/scraping/year_data/" + symbol +".txt" , "a")
for point in datapoints:
myfile.write(str(symbol +","+ str(point['date']) +","+ str(point['value'])+"\n"))
myfile.close()
A symbol is for example AAPL (for Apple) and this is the link to bloomberg data for Apple:
http://www.bloomberg.com/markets/api/bulk-time-series/price/AAPL%3AUS?timeFrame=1_YEAR
This is the data:
[{u'lastPrice': 10.18, u'price': [{u'date': u'2014-12-26', u'value': 10.01}, {u'date': u'2014-12-30', u'value': 10.01}, {u'date': u'2015-01-07', u'value': 10}, {u'date': u'2015-01-09', u'value': 9.97}, {u'date': u'2015-01-12', u'value': 10}, {u'date': u'2015-01-13', u'value': 10.02}, {u'date': u'2015-01-14', u'value': 10.04}, {u'date': u'2015-01-16', u'value': 10.1}, {u'date': u'2015-01-20', u'value': 10.39}, {u'date': u'2015-01-21', u'value': 9.98}, {u'date': u'2015-01-22', u'value': 9.98}, {u'date': u'2015-01-23', u'value': 9.98}, {u'date': u'2015-01-26', u'value': 10.08}, {u'date': u'2015-01-27', u'value': 10.05}, {u'date': u'2015-01-29', u'value': 10.02}, {u'date': u'2015-02-02', u'value': 10.08}, {u'date': u'2015-02-06', u'value': 9.98}, {u'date': u'2015-02-09', u'value': 9.97}, {u'date': u'2015-03-06', u'value': 10.07}, {u'date': u'2015-03-10', u'value': 10.1}, {u'date': u'2015-03-25', u'value': 10.3}, {u'date': u'2015-03-26', u'value': 10.01}, {u'date': u'2015-03-27', u'value': 10.02}, {u'date': u'2015-03-30', u'value': 10.02}, {u'date': u'2015-03-31', u'value': 10.02}, {u'date': u'2015-04-01', u'value': 10.02}, {u'date': u'2015-04-02', u'value': 10.19}, {u'date': u'2015-04-10', u'value': 10.3}, {u'date': u'2015-04-17', u'value': 10.4}, {u'date': u'2015-05-01', u'value': 10.39}, {u'date': u'2015-05-07', u'value': 10.35}, {u'date': u'2015-05-08', u'value': 10.35}, {u'date': u'2015-05-14', u'value': 10.28}, {u'date': u'2015-06-10', u'value': 10.15}, {u'date': u'2015-06-16', u'value': 10.34}, {u'date': u'2015-06-25', u'value': 10.3}, {u'date': u'2015-06-29', u'value': 10.27}, {u'date': u'2015-07-07', u'value': 10.3599}, {u'date': u'2015-07-08', u'value': 10.36}, {u'date': u'2015-07-15', u'value': 10.37}, {u'date': u'2015-07-30', u'value': 10.27}, {u'date': u'2015-08-13', u'value': 10.1}, {u'date': u'2015-08-14', u'value': 10.1}, {u'date': u'2015-08-24', u'value': 10.14}, {u'date': u'2015-08-25', u'value': 10.14}, {u'date': u'2015-08-27', u'value': 10.17}, {u'date': u'2015-08-31', u'value': 10.2}, {u'date': u'2015-09-01', u'value': 10.25}, {u'date': u'2015-09-03', u'value': 10.24}, {u'date': u'2015-09-04', u'value': 10.35}, {u'date': u'2015-09-15', u'value': 10.2401}, {u'date': u'2015-09-21', u'value': 10.22}, {u'date': u'2015-09-22', u'value': 10.22}, {u'date': u'2015-09-28', u'value': 10.2933}, {u'date': u'2015-10-02', u'value': 10.4}, {u'date': u'2015-10-06', u'value': 10.44}, {u'date': u'2015-10-12', u'value': 10.22}, {u'date': u'2015-10-13', u'value': 10.21}, {u'date': u'2015-10-22', u'value': 10.25}, {u'date': u'2015-10-28', u'value': 10.15}, {u'date': u'2015-11-04', u'value': 10.15}, {u'date': u'2015-12-09', u'value': 10}, {u'date': u'2015-12-16', u'value': 10.18}], u'priceMinDecimals': 2, u'nyTradeEndTime': u'16:30:00.000', u'lastUpdateDate': u'2015-12-16', u'nyTradeStartTime': u'09:30:00.000', u'id': u'TFSCU:US', u'timeZoneOffset': -5, u'dateTimeRanges': {}}]
The problem I found is that there exists no data for some symbols, that means the data shown on the URL is: {u'securityType': u'UNKNOWN'}. This occures for example for following symbols: AGNCB, AGNCP, MTGEP, ASRVP, AMSGP, AGIIL, ARWAU, BANFP, BKEPP, BPFHP, BBEPP.
The Problem still is how I can exclude missing data, without interrupting the loop? And why do I get an ValueError after the loop correctly has created 440 files?
The current Error:
Traceback (most recent call last):
File "<pyshell#27>", line 1, in <module>
execfile("C:\Users\Asus\Desktop\Python\web scraper\scraping\stock_to_file.py")
File "C:\Users\Asus\Desktop\Python\web scraper\scraping\stock_to_file.py", line 11, in <module>
data = json.load(htmltext)
File "C:\Python27\lib\json\__init__.py", line 291, in load
**kw)
File "C:\Python27\lib\json\__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "C:\Python27\lib\json\decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Python27\lib\json\decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
According to your comment, the response you're getting from Bloomberg is:
{u'securityType': u'UNKNOWN'}
Bloomberg will return this if it doesn't recognize the symbol you're requesting. If you want to identify the bad symbols and continue with the script for the good ones, you can do one of:
1. Detect the error response, take any desired action, and move to the next (LBYL)
data = json.load(htmltext)
if u'securityType' in data and data[u'securityType'] == u'UNKNOWN':
print 'Unknown symbol: {0}'.format(symbol)
continue
2. Go ahead with the processing and capture the exception (EAFP)
try:
datapoints = data[0]["price"]
except KeyError:
print 'Unknown symbol: {0}'.format(symbol)
continue
It's a stylistic choice. In either case, the important thing is to continue to go to the next symbol once you detect the bad response.
This question is unlikely to help any future visitors; it is only relevant to a small geographic area, a specific moment in time, or an extraordinarily narrow situation that is not generally applicable to the worldwide audience of the internet. For help making this question more broadly applicable, visit the help center.
Closed 10 years ago.
I have the following variable from,am trying to extract the revision value from it..does any one have any inputs on how this can be done?
u'revision': u'ad0beef66e5890cde6f0961ed03d8bc7e3defc63'
INPUT:-
data ={u'status': u'MERGED', u'topic': u'', u'currentPatchSet': {u'approvals': [{u'by': {u'username': u'lnxbuild', u'name': u'Linux Build Service Account', u'email': u'lnxbuild#localhost'}, u'type': u'CRVW', u'description': u'Code Review', u'value': u'2', u'grantedOn': 1360897100}, {u'type': u'SUBM', u'by': {u'username': u'lnxbuild', u'name': u'Linux Build Service Account', u'email': u'lnxbuild#localhost'}, u'value': u'1', u'grantedOn': 1360897100}, {u'by': {u'username': u'lnxbuild', u'name': u'Linux Build Service Account', u'email': u'lnxbuild#localhost'}, u'type': u'TEST', u'description': u'Developer Verified', u'value': u'1', u'grantedOn': 1360897100}, {u'by': {u'username': u'lnxbuild', u'name': u'Linux Build Service Account', u'email': u'lnxbuild#localhost'}, u'type': u'VRIF', u'description': u'Verified', u'value': u'1', u'grantedOn': 1360897100}], u'createdOn': 1360897100, u'number': u'1', u'parents': [u'354a90cd1ae4ba9ce2a2a4ad38f8efbc8c02b02d', u'e4ffc07a6f78210a49f6ad2e0d705d7054c20ed6'], u'uploader': {u'username': u'lnxbuild', u'name': u'Linux Build Service Account', u'email': u'lnxbuild#localhost'}, u'ref': u'refs/changes/52/249452/1', u'revision': u'ad0beef66e5890cde6f0961ed03d8bc7e3defc63'}, u'url': u'https://review-android.company.com/249452', u'createdOn': 1360897100, u'number': u'249452', u'lastUpdated': 1360897121, u'project': u'platform/vendor/qcom-proprietary/wlan', u'sortKey': u'00231d720003ce6c', u'branch': u'master', u'owner': {u'username': u'lnxbuild', u'name': u'Linux Build Service Account', u'email': u'lnxbuild#localhost'}, u'open': False, u'id': u'Iad0beef66e5890cde6f0961ed03d8bc7e3defc63', u'subject': u'Merge "wlan: Enable ActiveModeOffload by default"'}
OUTPUT:-
ad0beef66e5890cde6f0961ed03d8bc7e3defc63
Simply retrieve it by following the 'currentPatchSet' key then the 'revision' key
data['currentPatchSet']['revision']
Interactive example which shows how this works:
>>> import pprint # pretty print for seeing the layout of the dict
>>> data ={u'status': u'MERGED', u'topic': u'', u'currentPatchSet': {u'approvals': [{u'by': {u'username': u'lnxbuild', u'name': u'Linux Build Service Account', u'email': u'lnxbuild#localhost'}, u'type': u'CRVW', u'description': u'Code Review', u'value': u'2', u'grantedOn': 1360897100}, {u'type': u'SUBM', u'by': {u'username': u'lnxbuild', u'name': u'Linux Build Service Account', u'email': u'lnxbuild#localhost'}, u'value': u'1', u'grantedOn': 1360897100}, {u'by': {u'username': u'lnxbuild', u'name': u'Linux Build Service Account', u'email': u'lnxbuild#localhost'}, u'type': u'TEST', u'description': u'Developer Verified', u'value': u'1', u'grantedOn': 1360897100}, {u'by': {u'username': u'lnxbuild', u'name': u'Linux Build Service Account', u'email': u'lnxbuild#localhost'}, u'type': u'VRIF', u'description': u'Verified', u'value': u'1', u'grantedOn': 1360897100}], u'createdOn': 1360897100, u'number': u'1', u'parents': [u'354a90cd1ae4ba9ce2a2a4ad38f8efbc8c02b02d', u'e4ffc07a6f78210a49f6ad2e0d705d7054c20ed6'], u'uploader': {u'username': u'lnxbuild', u'name': u'Linux Build Service Account', u'email': u'lnxbuild#localhost'}, u'ref': u'refs/changes/52/249452/1', u'revision': u'ad0beef66e5890cde6f0961ed03d8bc7e3defc63'}, u'url': u'https://review-android.company.com/249452', u'createdOn': 1360897100, u'number': u'249452', u'lastUpdated': 1360897121, u'project': u'platform/vendor/qcom-proprietary/wlan', u'sortKey': u'00231d720003ce6c', u'branch': u'master', u'owner': {u'username': u'lnxbuild', u'name': u'Linux Build Service Account', u'email': u'lnxbuild#localhost'}, u'open': False, u'id': u'Iad0beef66e5890cde6f0961ed03d8bc7e3defc63', u'subject': u'Merge "wlan: Enable ActiveModeOffload by default"'}
Let's visualize the data
>>> pprint.pprint(data)
{u'branch': u'master',
u'createdOn': 1360897100,
u'currentPatchSet': {u'approvals': [{u'by': {u'email': u'lnxbuild#localhost',
u'name': u'Linux Build Service Account',
u'username': u'lnxbuild'},
u'description': u'Code Review',
u'grantedOn': 1360897100,
u'type': u'CRVW',
u'value': u'2'},
{u'by': {u'email': u'lnxbuild#localhost',
u'name': u'Linux Build Service Account',
u'username': u'lnxbuild'},
u'grantedOn': 1360897100,
u'type': u'SUBM',
u'value': u'1'},
{u'by': {u'email': u'lnxbuild#localhost',
u'name': u'Linux Build Service Account',
u'username': u'lnxbuild'},
u'description': u'Developer Verified',
u'grantedOn': 1360897100,
u'type': u'TEST',
u'value': u'1'},
{u'by': {u'email': u'lnxbuild#localhost',
u'name': u'Linux Build Service Account',
u'username': u'lnxbuild'},
u'description': u'Verified',
u'grantedOn': 1360897100,
u'type': u'VRIF',
u'value': u'1'}],
u'createdOn': 1360897100,
u'number': u'1',
u'parents': [u'354a90cd1ae4ba9ce2a2a4ad38f8efbc8c02b02d',
u'e4ffc07a6f78210a49f6ad2e0d705d7054c20ed6'],
u'ref': u'refs/changes/52/249452/1',
u'revision': u'ad0beef66e5890cde6f0961ed03d8bc7e3defc63',
u'uploader': {u'email': u'lnxbuild#localhost',
u'name': u'Linux Build Service Account',
u'username': u'lnxbuild'}},
u'id': u'Iad0beef66e5890cde6f0961ed03d8bc7e3defc63',
u'lastUpdated': 1360897121,
u'number': u'249452',
u'open': False,
u'owner': {u'email': u'lnxbuild#localhost',
u'name': u'Linux Build Service Account',
u'username': u'lnxbuild'},
u'project': u'platform/vendor/qcom-proprietary/wlan',
u'sortKey': u'00231d720003ce6c',
u'status': u'MERGED',
u'subject': u'Merge "wlan: Enable ActiveModeOffload by default"',
u'topic': u'',
u'url': u'https://review-android.company.com/249452'}
So the revision appears to be in the currentPatchSet dictionary:
>>> pprint.pprint(data['currentPatchSet'])
{u'approvals': [{u'by': {u'email': u'lnxbuild#localhost',
u'name': u'Linux Build Service Account',
u'username': u'lnxbuild'},
u'description': u'Code Review',
u'grantedOn': 1360897100,
u'type': u'CRVW',
u'value': u'2'},
{u'by': {u'email': u'lnxbuild#localhost',
u'name': u'Linux Build Service Account',
u'username': u'lnxbuild'},
u'grantedOn': 1360897100,
u'type': u'SUBM',
u'value': u'1'},
{u'by': {u'email': u'lnxbuild#localhost',
u'name': u'Linux Build Service Account',
u'username': u'lnxbuild'},
u'description': u'Developer Verified',
u'grantedOn': 1360897100,
u'type': u'TEST',
u'value': u'1'},
{u'by': {u'email': u'lnxbuild#localhost',
u'name': u'Linux Build Service Account',
u'username': u'lnxbuild'},
u'description': u'Verified',
u'grantedOn': 1360897100,
u'type': u'VRIF',
u'value': u'1'}],
u'createdOn': 1360897100,
u'number': u'1',
u'parents': [u'354a90cd1ae4ba9ce2a2a4ad38f8efbc8c02b02d',
u'e4ffc07a6f78210a49f6ad2e0d705d7054c20ed6'],
u'ref': u'refs/changes/52/249452/1',
u'revision': u'ad0beef66e5890cde6f0961ed03d8bc7e3defc63',
u'uploader': {u'email': u'lnxbuild#localhost',
u'name': u'Linux Build Service Account',
u'username': u'lnxbuild'}}
Indeed it is, under the 'revision' key. We retrieve it:
>>> print data['currentPatchSet']['revision']
ad0beef66e5890cde6f0961ed03d8bc7e3defc63
>>>