I am trying to create a script that will process Twitter streams. Unfortunately, the OAuth process has stymied me. Adopting some code I found on the internet, I receive a blank response from https://api.twitter.com/oauth/token. In order to better understand the process, I am trying to do this without special modules. Here is my code, what am missing? Any help would be greatly appreciated.
Code:
import http.client
import urllib
import base64
CONSUMER_KEY = 'yadayadayada'
CONSUMER_SECRET = 'I am really tired today'
encoded_CONSUMER_KEY = urllib.parse.quote(CONSUMER_KEY)
encoded_CONSUMER_SECRET = urllib.parse.quote(CONSUMER_SECRET)
concat_consumer_url = encoded_CONSUMER_KEY + ':' + encoded_CONSUMER_SECRET
host = 'api.twitter.com'
url = '/oauth2/token/'
params = urllib.parse.urlencode({'grant_type' : 'client_credentials'})
req = http.client.HTTPSConnection(host, timeout = 100)
req.set_debuglevel(1)
req.putrequest("POST", url)
req.putheader("Host", host)
req.putheader("User-Agent", "My Twitter 1.1")
req.putheader("Authorization", "Basic %s" % base64.b64encode(b'concat_consumer_url'))
req.putheader("Content-Type", "application/x-www-form-urlencoded;charset=UTF-8")
req.putheader("Content-Length", "29")
req.putheader("Accept-Encoding", "identity")
req.endheaders()
req.send(b'params')
resp = req.getresponse()
print ("{} {}".format(resp.status, resp.reason))
Error message:
C:\Python33>app_only_test_klug.py
Traceback <most recent call last>:
File "C:\Python33\app_only_test_klug.py", line 31, in <module>
resp = req.getresponse()
File "C:\Python33\lib\http\client.py", line 1131, in getresponse
response.being()
File "C:\Python33\lib\http\client.py", line 354, in begin
version, status, reason = self._read_status()
File "C:\Python33\lib\http\client.py", line 324, in _read_status
raise BadStatusLine(line)
http.client.BadStatusLine: ''
Any help would be greatly appreciated.
UPDATE:
After some more tinkering, I believe that the issue lies with my base64 encoding:
req.putheader("Authorization", "Basic %s" % base64.b64encode(b'concat_consumer_url'))
When I decode the resulting encoding of the above, I get "b'concat_consumer_url'" rather than a concatenation of the encoded_CONSUMER_KEY and encoded_CONSUMER_SECRET combined around a colon. How do I get the base64 to b64encode the value that concat_comsumer_url represents rather than the string "concat_consumer_url" so that I can move forward? Thanks in advance.
I believe the issue is there as well--you should just pass the variable to the encoding function, rather than the name of the variable as bytes, like this:
req.putheader("Authorization", "Basic %s" % base64.b64encode(concat_consumer_url))
Try it again with that change.
Related
I am trying to play with the Gemini trading API. I have issued myself an API key and a secret, and after configuring my environment, in which I had a lot of issues setting up and installing requests through pip, I used their example code to create a simple script to read my recent trades. Here is the script, minus my API keys:
#!/usr/bin/env/ python
import requests
import json
import base64
import hmac
import hashlib
import datetime, time
url = "https://api.sandbox.gemini.com"
gemini_api_key = "master-xxx"
gemini_api_secret = "xxx".encode()
t = datetime.datetime.now()
payload_nonce = str(int(time.mktime(t.timetuple())*1000))
payload = {"request": "/v1/mytrades", "nonce": payload_nonce}
encoded_payload = json.dumps(payload).encode()
b64 = base64.b64encode(encoded_payload)
signature = hmac.new(gemini_api_secret, b64, hashlib.sha384).hexdigest()
request_headers = {
'Content-Type': "text/plain",
'Content-Length': "0",
'X-GEMINI-APIKEY': gemini_api_key,
'X-GEMINI-PAYLOAD': b64,
'X-GEMINI-SIGNATURE': signature,
'Cache-Control': "no-cache"
}
response = requests.post(url, headers=request_headers)
my_trades = response.json()
print(my_trades)
Now at first, it would run, but give me an error saying I hadn't specified an account. Then, without changing ANYTHING AT ALL, it suddenly quit working altogether. So while I still have some sort of issue accessing the API, I can't even get to the errors anymore to try to figure out why. Now what I get is a JSON decode error, which looks like the following:
Traceback (most recent call last):
File "c:\Users\david\Desktop\Code Projects\GeminiTrader\GeminiTrader-v0.1.py",
line 33, in <module>
my_trades = response.json()
File "C:\Users\david\AppData\Local\Programs\Python\Python39\lib\site-packages\requests-2.25.1-py3.9.egg\requests\models.py", line 900, in json
return complexjson.loads(self.text, **kwargs)
File "C:\Users\david\AppData\Local\Programs\Python\Python39\lib\json\__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "C:\Users\david\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\david\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
What is causing this json decode issue? Why was it not coming up before, when the API was simply rejecting my request due to an account parameter error? Why did it suddenly change to this error without me modifying anything in the code? How can I fix it? I kept having issues with installing requests and getting it to work, maybe I messed something up in that process?
Certainly after fixing this I will have a new host of issues to fix because the documentation on this API is abysmal. Any help in progressing this project would be greatly appreciated! Thanks!
As you are calling an API, there are chances that your API call fails and returns just string or empty response. I would suggest that you first add a check on the status code of your response something like below and then process the json data.
data = requests.post()
if data.status_code != 200:
raise Exception("Error", data.reason)
json_data = data.json()
I've been tasked with learning Python on the spot and feel like I'm drowning. I am trying to translate what was provided by a coworker but am really struggling. The API I am trying to work with is here: https://dev.skuvault.com/v1.0/reference#getonlinesalestatus
And the code I have is:
import requests, json
# Skuvault URIs and Token
SkuBase = "https://app.skuvault.com/api/sales/getOnlineSaleStatus"
SkuProductsUri = SkuBase + "Products(id)/Attributes('name')"
SkuAuthToken = ""
print "[+] Requesting: " + SkuProductsUri
response = requests.post(SkuProductsUri, headers={'Authorization': 'Bearer ' + SkuAuthToken})
productsJson = json.loads(response.status_code)
print "[*] Status: %d\n[*] Reason: %s\n[*] Message: %s\n[*] Raw: %s\n\n" \
% (response.status_code, response.reason, productsJson['Message'], response.text[:300])
I'm receiving the following error when trying to run the script
[+] Requesting: https://app.skuvault.com/api/sales/getOnlineSaleStatusProducts(id)/Attributes('name')
Traceback (most recent call last):
File "test-api.py", line 11, in
productsJson = json.loads(response.status_code)
File "/usr/local/Cellar/python#2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/init.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/local/Cellar/python#2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer
Like I said, I'd literally never worked with Python before this and feel a bit overwhelmed. Thanks.
You're not calling the API correctly. Notice that the API expects you to POST a request, and it expects the request payload to look like
{
"OrderIds": [
"my-id"
],
"TenantToken": "my-tenant-token",
"UserToken": "my-user-token"
}
If I were doing similar, I might do:
sku_base = "https://app.skuvault.com/api/sales/getOnlineSaleStatus"
response = requests.post(
sku_base,
json={
"OrderIds": [ "my-order-id" ],
"TenantToken": "my-tenant-token",
"UserToken": "my-user-token"
}
)
# NOTE: you are not using the status code here. That's an int and will error.
productsJson = json.loads(response.text)
# This is also valid and will result in the same object:
productsJson = response.json()
print "[*] Status: %d\n[*] Reason: %s\n[*] Message: %s\n[*] Raw: %s\n\n" \
% (response.status_code, response.reason, productsJson['Message'], response.text[:300])
Side note:
Python 2 is no longer maintained. It is recommended that you use Python 3 exclusively.
I have a program that will auto create a Github issue via the API. It works in Python 2.7, but when I run it with Python 3 I get the following error:
Traceback (most recent call last):
File "/home/baal/bin/python/zeus-scanner/var/auto_issue/github.py", line 92, in request_issue_creation
urllib2.urlopen(req, timeout=10).read()
File "/usr/lib/python3.5/urllib/request.py", line 163, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.5/urllib/request.py", line 464, in open
req = meth(req)
File "/usr/lib/python3.5/urllib/request.py", line 1183, in do_request_
raise TypeError(msg)
TypeError: POST data should be bytes or an iterable of bytes. It cannot be of type str.
I have the following method that creates a github issue (successful in python 2 unsuccessful in python 3):
def request_issue_creation():
logger.info(set_color(
"Zeus got an unexpected error and will automatically create an issue for this error, please wait..."
))
def __extract_stacktrace(file_data):
logger.info(set_color(
"extracting traceback from log file..."
))
retval, buff_mode, _buffer = [], False, ""
with open(file_data, "r+") as log:
for line in log:
if "Traceback" in line:
buff_mode = True
if line and len(line) < 5:
buff_mode = False
retval.append(_buffer)
_buffer = ""
if buff_mode:
if len(line) > 400:
line = line[:400] + "...\n"
_buffer += line
return "".join(retval)
logger.info(set_color(
"getting authorization..."
))
encoded = __get_encoded_string()
n = get_decode_num(encoded)
token = decode(n, encoded)
current_log_file = get_latest_log_file(CURRENT_LOG_FILE_PATH)
stacktrace = __extract_stacktrace(current_log_file)
issue_title = stacktrace.split("\n")[-2]
issue_data = {
"title": issue_title,
"body": "Error info:\n```{}````\n\n"
"Running details:\n`{}`\n\n"
"Commands used:\n`{}`\n\n"
"Log file info:\n```{}```".format(
str(stacktrace),
str(platform.platform()),
" ".join(sys.argv),
open(current_log_file).read()
),
}
try:
req = urllib2.Request(
url="https://api.github.com/repos/<API-REPO>/issues", data=json.dumps(issue_data),
headers={"Authorization": "token {}".format(token)}
)
urllib2.urlopen(req, timeout=10).read()
logger.info(set_color(
"issue has been created successfully with the following name '{}'...".format(issue_title)
))
except Exception as e:
logger.exception(set_color(
"failed to auto create the issue, got exception '{}', "
"you may manually create an issue...".format(e), level=50
))
I read online that encoding the string to utf-8 will fix the issue, however I'm not sure if that is possible here? Any help would be greatly appreciated, thank you.
You need to encode your JSON payload:
data = json.dumps(issue_data)
if sys.version_info > (3,): # Python 3
data = data.encode('utf8')
req = urllib2.Request(
url="https://api.github.com/repos/<API-REPO>/issues", data=data,
headers={"Authorization": "token {}".format(token),
"Content-Type": "application/json; charset=utf-8"}
)
I added a Content-Type header with a charset parameter to communicate the codec used to the server. This is not always needed, JSON's default codec is UTF-8. If you don't specify the header, a (wrong) default would be supplied; it depends on the server whether or not that matters.
My script executes perfectly when I run it locally, but when I execute it on Python Anywhere, it throws the following error:
Traceback (most recent call last):
File "/home/ectobiologist7/desiderius.py", line 68, in <module>
asendoff()
File "/home/ectobiologist7/desiderius.py", line 64, in asendoff
body=billiam,
File "/usr/local/lib/python3.4/dist-packages/twilio/rest/resources/messages.py", line 122, in create
return self.create_instance(kwargs)
File "/usr/local/lib/python3.4/dist-packages/twilio/rest/resources/base.py", line 365, in create_instance
data=transform_params(body))
File "/usr/local/lib/python3.4/dist-packages/twilio/rest/resources/base.py", line 200, in request
resp = make_twilio_request(method, uri, auth=self.auth, **kwargs)
File "/usr/local/lib/python3.4/dist-packages/twilio/rest/resources/base.py", line 152, in make_twilio_request
resp = make_request(method, uri, **kwargs)
File "/usr/local/lib/python3.4/dist-packages/twilio/rest/resources/base.py", line 117, in make_request
resp, content = http.request(url, method, headers=headers, body=data)
File "/usr/local/lib/python3.4/dist-packages/httplib2/__init__.py", line 1314, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/usr/local/lib/python3.4/dist-packages/httplib2/__init__.py", line 1064, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/usr/local/lib/python3.4/dist-packages/httplib2/__init__.py", line 987, in _conn_request
conn.connect()
File "/usr/lib/python3.4/http/client.py", line 1223, in connect
super().connect()
File "/usr/lib/python3.4/http/client.py", line 834, in connect
self.timeout, self.source_address)
File "/usr/lib/python3.4/socket.py", line 512, in create_connection
raise err
File "/usr/lib/python3.4/socket.py", line 503, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
Now, I've combed through forums both on Stack Overflow and on Python anywhere itself, and none of the solutions I've found have succeeded. One suggestion was that the script wasn't working because a website I was trying to access wasn't on the whitelist, and this is not the case. I tried using a bit of code I found to force PA to use a different proxy, as suggested by a site dev, and that did not work. I also saw the dev recommending that people with this problem instead use Python 2.7. I did that, and still nothing changed. Are there any other possible solutions? Thanks in advance! My code is below.
from twilio.rest import TwilioRestClient
from datetime import *
import urllib.request
from bs4 import BeautifulSoup
from random import randint
import ast
#use julian date to calculate days left till summer
# put your own credentials here
def dateelle():
#use datetime to state date in Month Day, Year format.
todaylist = date.today().timetuple()
monthdict = {1:"January ", 2:"February ", 3:"March ", 4:"April ", 5:"May ", 6:"June ", 7:"July ", 8:"August ", 9:"September ", 10:"October ", 11:"November ", 12:"December "}
month = monthdict[todaylist[1]]
day = todaylist[2]
year = todaylist[0]
ultidate = str(month) + str(day) + ", " + str(year) + " "
return ultidate
def temple():
url = 'http://api.openweathermap.org/data/2.5/weather?id=5429522&APPID=0f25c0515e643bc2427f665f88a4d3eb'
headers = {}
headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36'
tri1 = urllib.request.Request(url, headers=headers)
tri2 = urllib.request.urlopen(tri1)
quell = BeautifulSoup(tri2)
tagwithtemp = int(ast.literal_eval(quell.text)['main']['temp'] * (9/5) - 459.67)
return tagwithtemp
def daysleft():
todayy = date.today().timetuple()[7]
allthatremains = 147 - todayy
return allthatremains
comp = {0:" You look nice today.",
1:" Have a good day.",
2:" Today will be a great day.",
3:" You are an important person who means much to the world.",
4: " Someone's day today will be brightened by your presence.",
5: " Despite the amount of people in the world, your existence matters. :)",
6: " There may be billions of people on this planet, but you still make a difference."}
billiam = "Good morning! Today is " + dateelle() + "and it is " + str(temple()) + " degrees." + comp[randint(0,6)] + " " + str(daysleft()) + " days left until summer!"
def asendoff():
global billiam
ACCOUNT_SID = "***" #the SID, AUTH token, and my phone number are censored
AUTH_TOKEN = "***"
nummer = ["***"]
client = TwilioRestClient(ACCOUNT_SID, AUTH_TOKEN)
for lexis in nummer:
client.messages.create(
to=lexis,
from_="+15026255861",
body=billiam,
)
if date.today().timetuple()[6] != 6 and date.today().timetuple()[6] != 7:
asendoff()
else:
pass
All external connections from PythonAnywhere free accounts are proxied to prevent abuse.
You need to set the proxy for the connection like this:
from twilio.rest.resources import Connection
from twilio.rest.resources.connection import PROXY_TYPE_HTTP
Connection.set_proxy_info(
"proxy.server",
3128,
proxy_type=PROXY_TYPE_HTTP
)
However, in Python 3, httplib2 (and therefore twilio-python) ignores the proxy settings (see the bottom of this page) and so tries to make a direct connection to Twilio which is blocked.
Twilio developer evangelist here.
There have been previous reports suggesting PA had older versions of the library. A few months ago we made a pretty significant change in our libraries and API to enforce SSL.
If you're using the latest version of the library, there should be no change for you. From the error you posted, I can't exactly tell what version of the library PA are using, but digging a little further, users seem to suggest you need a proxy.
My suggestion here would be to slightly change your approach and perhaps deploy this to Heroku (it's free and you will be able to use the exact library version you want). That way, if you see that it works there, you could contact PA and check which library version they have, and you will be able to tell them whether they need an update or not.
Hope this helps you
So I'm working with a block of code which communicates with the Flickr API.
I'm getting a 'syntax error' in xml.parsers.expat.ExpatError (below). Now I can't figure out how it'd be a syntax error in a Python module.
I saw another similar question on SO regarding the Wikipedia API which seemed to return HTML intead of XML. Flickr API returns XML; and I'm also getting the same error when there shouldn't be a response from Flickr (such as flickr.galleries.addPhoto)
CODE:
def _dopost(method, auth=False, **params):
#uncomment to check you aren't killing the flickr server
#print "***** do post %s" % method
params = _prepare_params(params)
url = '%s%s/%s' % (HOST, API, _get_auth_url_suffix(method, auth, params))
payload = 'api_key=%s&method=%s&%s'% \
(API_KEY, method, urlencode(params))
#another useful debug print statement
#print url
#print payload
return _get_data(minidom.parse(urlopen(url, payload)))
TRACEBACK:
Traceback (most recent call last):
File "TESTING.py", line 30, in <module>
flickr.galleries_create('test_title', 'test_descriptionn goes here.')
File "/home/vlad/Documents/Computers/Programming/LEARNING/curatr/flickr.py", line 1006, in galleries_create
primary_photo_id=primary_photo_id)
File "/home/vlad/Documents/Computers/Programming/LEARNING/curatr/flickr.py", line 1066, in _dopost
return _get_data(minidom.parse(urlopen(url, payload)))
File "/usr/lib/python2.6/xml/dom/minidom.py", line 1918, in parse
return expatbuilder.parse(file)
File "/usr/lib/python2.6/xml/dom/expatbuilder.py", line 928, in parse
result = builder.parseFile(file)
File "/usr/lib/python2.6/xml/dom/expatbuilder.py", line 207, in parseFile
parser.Parse(buffer, 0)
xml.parsers.expat.ExpatError: syntax error: line 1, column 62
(Code from http://code.google.com/p/flickrpy/ under New BSD licence)
UPDATE:
print urlopen(url, payload) == <addinfourl at 43340936 whose fp = <socket._fileobject object at 0x29400d0>>
Doing a urlopen(url, payload).read() returns HTML which is hard to read in a terminal :P but I managed to make out a 'You are not signed in.'
The strange part is that Flickr shouldn't return anything here, or if permissions are a problem, it should return a 99: User not logged in / Insufficient permissions error as it does with the GET function (which I'd expect would be in valid XML).
I'm signed in to Flickr (in the browser) and the program is properly authenticated with delete permissions (dangerous, but I wanted to avoid permission problems.)
SyntaxError normally means an error in Python syntax, but I think here that expatbuilder is overloading it to mean an XML syntax error. Put a try:except block around it, and print out the contents of payload and to work out what's wrong with the first line of it.
My guess would be that flickr is rejecting your request for some reason and giving back a plain-text error message, which has an invalid xml character at column 62, but it could be any number of things. You probably want to check the http status code before parsing it.
Also, it's a bit strange this method is called _dopost but you seem to actually be sending an http GET. Perhaps that's why it's failing.
This seems to fix my problem:
url = '%s%s/?api_key=%s&method=%s&%s'% \
(HOST, API, API_KEY, method, _get_auth_url_suffix(method, auth, params))
payload = '%s' % (urlencode(params))
It seems that the API key and method had to be in the URL not in the payload. (Or maybe only one needed to be there, but anyways, it works :-)