As this is the first time I'm trying this out, I do not know what is wrong with the problem. So it would be great if someone can help me solve this problem
The code I'm using is at the bottom page of this website: https://www.twilio.com/blog/2014/11/build-your-own-pokedex-with-django-mms-and-pokeapi.html
Where it give example on how you can make HTTP request function and retrieve database on your query.
The code on the website is this.
query.py
import requests
import json
BASE_URL = 'http://pokeapi.co'
def query_pokeapi(resource_url):
url = '{0}{1}'.format(BASE_URL, resource_url)
response = requests.get(url)
if response.status_code == 200:
return json.loads(response.text)
return None
charizard = query_pokeapi('/api/v1/pokemon/charizard/')
sprite_uri = charizard['sprites'][0]['resource_uri']
description_uri = charizard['descriptions'][0]['resource_uri']
sprite = query_pokeapi(sprite_uri)
description = query_pokeapi(description_uri)
print
charizard['name']
print
description['description']
print
BASE_URL + sprite['image']
In my edit, I only change these print line at the bottom of this
query.py
print(charizard['name'])
print(description['description'])
print(BASE_URL + sprite['image'])
But i got this error instead
Traceback (most recent call last): File "query2.py", line 46, in
sprite_uri = charizard['sprites'][0]['resource_uri'] TypeError: 'NoneType' object is not subscriptable
query_pokeapi must be returning None, which would mean that your API call is not receiving a 200 HTTP response. I'd check your URL, to make sure it's properly formed. Test it in your web browser.
best practice would be to try-except your API call with an error message letting you know that your API call failed and otherwise routing the thread.
Update: reread and the sub scripting issue could be in any layer of your nested object.
Evaluate charizard['sprites'][0]['resource_uri']
step by step in your debugger.
When you call api requests.get(url) then its response is
More than one resource is found at this URI
you are using charizard['sprites'][0]['resource_uri'] on result and it's raising exception.
When I tried to get response then status code is 300 so
def query_pokeapi(resource_url) returning None value.
'{0}{1}'.format(BASE_URL, resource_url)
Update
it means at {0} BASE_URL will be places and at {1} resource_url will be places.
Complete url will be
url = '{0}{1}'.format(BASE_URL, resource_url)
url = 'http://pokeapi.co/api/v1/pokemon/charizard/'.
update
you can try
import json
charizard = query_pokeapi('/api/v1/pokemon/')
data = json.loads(charizard.content)
print data['objects'][0]['descriptions'][0]
result will be
{u'name': u'ekans_gen_1', u'resource_uri': u'/api/v1/description/353/'}
Update with complete code
import requests
import json
BASE_URL = 'http://pokeapi.co'
def query_pokeapi(resource_url):
url = '{0}{1}'.format(BASE_URL, resource_url)
response = requests.get(url)
if response.status_code == 200:
return json.loads(response.text)
return None
charizard = query_pokeapi('/api/v1/pokemon/')
print charizard['objects'][0]['descriptions'][0]
result will be:
{u'name': u'ekans_gen_1', u'resource_uri': u'/api/v1/description/353/'}
Related
I am unable to connect with the pwnedpasswords API when I define a function and I get error "400", which is a bad request. I am using the following code:
import requests
def request_api_data(char):
url = 'https://api.pwnedpasswords.com/range/' + char
res = requests.get(url)
print(res)
request_api_data('123')
However, when I use this code without function then I get a response of "200" which is a good response.
import requests
url = 'https://api.pwnedpasswords.com/range/' + 'CBFDA'
res = requests.get(url)
print(res)
I don't understand why it becomes a bad request when I try to make it a function.
The problem is with the 123, if you try CBFDA in the first code it will work, it gives this error with 123:
"The hash prefix was not in a valid format"
This happens because 123 is not a valid hash prefix
trying to make post requests in python, i'm getting this error
import requests
token='Bearer aslkdjndskgns'
endpoint = "https://aap.xyz"
phone_number='9177903753951'
data = {"number":phone_number}
datas = str(data)
headers={"authorization":token}
a=(requests.post(endpoint, data=datas, headers=headers).json())
i'm getting this error
invalid character 'p' looking for beginning of value "
Please try below code, see if it resolves your issue.
import requests
token='Bearer aslkdjndskgns'
endpoint = 'https://aap.xyz'
data = {'number':'9177903753951'}
headers={"authorization":token}
a=(requests.post(endpoint, data=data, headers=headers).json())
print(a)
I am building a custom responder for the Cortex/Hive, and I have been unable to get my request response to properly convert into a JSON format. When testing in the local development environment, my code in getCount function works flawlessly, but when adding the cortex responder wrapper to it, my code fails.
Since the responder runs from Cortex, I do not receive an error message beyond "input: null", so I had to write to an error log. Using this log, I determined that the error stems from the line data = json.loads(response.text). I tried using simple json, regex-ing the desired value from response.text, changing encoding methods, and banging my head on the keyboard from the sheer stupidity of it not working.
CODE:
import requests
import json
from cortexutils.responder import Responder
class Search(Responder):
def __init__(self):
# Debug
with open('/xxx/xxx/xxx/xxx/error.log','a') as error_log:
error_log.write('Starting: \n')
error_log.write('\n\n')
# End Debug
Responder.__init__(self)
self.apiuser = self.get_param('config.api_user', None)
self.apikey = self.get_param('config.api_key', None)
self.url = self.get_param('config.api_url', None)
self.ioc_type = self.get_param('data.dataType', None)
self.ioc = self.get_param('data.data', None)
# Debug
with open('/xxx/xxx/xxx/xxx/error.log','a') as error_log:
error_log.write('User ID: \n')
error_log.write(self.apiuser)
error_log.write('\n\nSecret: \n')
error_log.write(self.apikey)
error_log.write('\n\n')
error_log.write('IOC Type: \n')
error_log.write(self.ioc_type)
error_log.write('\n\n')
error_log.write('Value: \n')
error_log.write(self.ioc)
error_log.write('\n\n')
# End Debug
def getCount(self):
with open('/xxx/xxx/xxx/xxx/error.log','a') as error_log:
error_log.write('Starting Count: \n')
error_log.write('\n\n')
url = self.url
headers={'Content-Type': 'application/json'}
params={'type': self.ioc_type, 'value': self.ioc}
response = requests.request("GET", url, headers=headers, params = params, auth=(self.apiuser, self.apikey))
data = json.loads(response.text)
with open('/xxx/xxx/xxx/xxx/error.log','a') as error_log:
error_log.write('Response: ')
error_log.write(data)
deviceCount = data['resources'][0]['device_count']
self.count = deviceCount
def run(self):
Responder.run(self)
self.getCount()
self.operations()
def operations(self):
return [self.build_operation('AddTagToCase', tag= self.count)]
if __name__ == '__main__':
Search().run()
Results from response.text:
{"meta":{"query_time":0.014920091,"trace_id":"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"},
"resources":[{"id":"sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"type":"sha256",
"value":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"device_count":11}],"errors":[]}
Error Logging results:
Starting:
User ID:
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Secret:
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
IOC Type:
sha256
Value:
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
run starting:
Starting Count:
self.count should be equal to device_count, but json.loads fails to format this response. This can be seen in my error log results where Count() starts but abruptly ends before data is written to it.
If you could please provide insight into why this fails to format the response properly, please shed some light.
Thank you
This is my standard way of getting a json response from an API using the requests module
response = requests.get(url, headers=headers, params=params, auth=(self.apiuser, self.apikey)).json()
I've got a function like:
def request_API(request_url): #To test
fail_request = -1
response = requests.get(request_url)
if response.ok:
infos = json.loads(response.text)
if infos.has_key("movie"):
return infos["movie"]
if infos.has_key("tvseries"):
return infos["tvseries"]
print "Allocine API Request Error"
return fail_request
I did a test like:
def test_should_fail_to_request(self):
#GIVEN
response = json.dumps({"error":{"code":0,"$":"No result"}})
requests.get = mock.MagicMock(return_value=response)
#WHEN
response = my_mod.request_allocine_API("") #requests mocked so we don't need an URL
#THEN
self.assertEqual(response, -1, "There should be an error in the API")
But I've got an error:
AttributeError: 'str' object has no attribute 'ok'
I know it come from the fact that when I mock request.get I return a JSON. My question is what is the proper way to do it. Have I to recreate an object requests or is there more simple way to do so.
You are mocking requests.get, which normally returns an Response object, to instead return a plain string. Try having it return an Response object instead:
from mock import patch
from requests import Response
def test_should_fail_to_request(self):
mock_response = Response()
mock_response.status_code = 404
mock_response.body = json.dumps({"error":{"code":0,"$":"No result"}})
with patch('requests.get', return_value=mock_response):
response = my_mod.request_allocine_API("")
self.assertEqual(response, -1, "There should be an error in the API")
I use requests-mock library which works well.
the document is in : https://requests-mock.readthedocs.org/en/latest/
The best feature is supporting for regex.
We use a custom scraper that have to take a separate website for a language (this is an architecture limitation). Like site1.co.uk, site1.es, site1.de etc.
But we need to parse a website with many languages, separated by url - like site2.com/en, site2.com/de, site2.com/es and so on.
I thought about MITMProxy: I could redirect all requests this way:
en.site2.com/* --> site2.com/en
de.site2.com/* --> site2.com/de
...
I have written a small script which simply takes URLs and rewrites them:
class MyMaster(flow.FlowMaster):
def handle_request(self, r):
url = r.get_url()
# replace URLs
if 'blabla' in url:
r.set_url(url.replace('something', 'another'))
But the target host generates 301 redirect with the response from the webserver - 'the page has been moved here' and the link to the site2.com/en
It worked when I played with URL rewriting, i.e. site2.com/en --> site2.com/de.
But for different hosts (subdomain and the root domain, to be precise), it does not work.
I tried to replace the Host header in the handle_request method from above:
for key in r.headers.keys():
if key.lower() == 'host':
r.headers[key] = ['site2.com']
also I tried to replace the Referrer - all of that didn't help.
How can I finally spoof that request from the subdomain to the main domain? If it generates a HTTP(s) client warning it's ok since we need that for the scraper (and the warnings there can be turned off), not the real browser.
Thanks!
You need to replace the content of the response and craft the header with just a few fields.
Open a new connection to the redirected url and craft your response :
def handle_request(self, flow):
newUrl = <new-url>
retryCount = 3
newResponse = None
while True:
try:
newResponse = requests.get(newUrl) # import requests
except:
if retryCount == 0:
print 'Cannot reach new url ' + newUrl
traceback.print_exc() # import traceback
return
retryCount -= 1
continue
break
responseHeaders = Headers() # from netlib.http import Headers
if 'Date' in newResponse.headers:
responseHeaders['Date'] = str(newResponse.headers['Date'])
if 'Connection' in newResponse.headers:
responseHeaders['Connection'] = str(newResponse.headers['Connection'])
if 'Content-Type' in newResponse.headers:
responseHeaders['Content-Type'] = str(newResponse.headers['Content-Type'])
if 'Content-Length' in newResponse.headers:
responseHeaders['Content-Length'] = str(newResponse.headers['Content-Length'])
if 'Content-Encoding' in newResponse.headers:
responseHeaders['Content-Encoding'] = str(inetResponse.headers['Content-Encoding'])
response = HTTPResponse( # from libmproxy.models import HTTPResponse
http_version='HTTP/1.1',
status_code=200,
reason='OK',
headers=responseHeaders,
content=newResponse.content)
flow.reply(response)