I'm pulling data from my database and trying to pass it my web app via the API using python requests. I've done this several times, but I'm having a hard time on this script.
It gives me "ValueError: not enough values to unpack (expected 2, got 1)"
rpt_cursor = rpt_conn.cursor()
sql = """SELECT `field` FROM `db`.`table`;"""
rpt_cursor.execute(sql)
#Creating list of field names.
num_fields = len(rpt_cursor.description)
field_names = [i[0] for i in rpt_cursor.description]
# getting results and defining dict to load them into.
results = rpt_cursor.fetchall()
dictofemails = []
print('These are the SQL results....')
print(results) # These look fine.
# Appending data to the dict.
for row in results:
dictofdata.append(dict(zip(field_names,row)))
print('These are the dict results...')
print(dictofdata) # Once again this looks like a fine dict with a format of [{'field_name' : 'xyz'}, {'field_name' : 'abc'}].
api_request_url = 'https://api.domainname.com/api/list/' + str(target) +'/Api_Key/' + my_apikey
print('api_request_url') # This looks fine.
response = requests.put(api_request_url, headers=headers, data=dictofdata)
print(response)
print(response.content)
Any clues you guys could give me would be appreciated.
Edit:
Traceback as requested....
Traceback (most recent call last):
File "NGM_ListMaker_WeeklyBounceLoad.py", line 306, in <module>
response = requests.put(api_request_url, headers=headers, data=dictofemails)
File "C:\Program Files (x86)\Python36-32\lib\site-packages\requests\api.py", line 126, in put
return request('put', url, data=data, **kwargs)
File "C:\Program Files (x86)\Python36-32\lib\site-packages\requests\api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Program Files (x86)\Python36-32\lib\site-packages\requests\sessions.py", line 494, in request
prep = self.prepare_request(req)
File "C:\Program Files (x86)\Python36-32\lib\site-packages\requests\sessions.py", line 437, in prepare_request
hooks=merge_hooks(request.hooks, self.hooks),
File "C:\Program Files (x86)\Python36-32\lib\site-packages\requests\models.py", line 308, in prepare
self.prepare_body(data, files, json)
File "C:\Program Files (x86)\Python36-32\lib\site-packages\requests\models.py", line 499, in prepare_body
body = self._encode_params(data)
File "C:\Program Files (x86)\Python36-32\lib\site-packages\requests\models.py", line 97, in _encode_params
for k, vs in to_key_val_list(data):
ValueError: not enough values to unpack (expected 2, got 1)
Edit with the Answer:
The API was expecting a string instead of the list that I was passing it. Changing the API call to data=str(dictofdata) was all that was needed. Thanks for the help.
The API was expecting a string instead of the list I was trying to pass.
Change this...
response = requests.put(api_request_url, headers=headers, data=dictofdata)
to this...
response = requests.put(api_request_url, headers=headers, data=str(dictofdata))
Works fine now. Thanks everyone.
Related
ok so im using python 3
i was able to get the data of the api using print(endpoint.json())
but i want to make it readable with pandas, so i can iterate through it easier.
this is the code (keep in mind i discarded my own api key and im using rapid api as a resource (specificly the movie database)
import requests
import json
import pandas
url = "https://movies-tvshows-data-imdb.p.rapidapi.com/"
querystring = {"type":"get-popular-movies","page":"1","year":"2020"}
headers = {
'x-rapidapi-host': "movies-tvshows-data-imdb.p.rapidapi.com",
'x-rapidapi-key': my key
}
response = requests.request("GET", url, headers=headers, params=querystring)
data=response.json()
df=pandas.read_json(data)
print(df)
i get this error
Traceback (most recent call last):
File "c:\Users\Home\Documents\studying\newproject\newproject.py", line 15, in <module>
df=pandas.read_json(data)
File "C:\Users\Home\AppData\Local\Programs\Python\Python38\lib\site-packages\pandas\util\_decorators.py", line 199, in wrapper
return func(*args, **kwargs)
File "C:\Users\Home\AppData\Local\Programs\Python\Python38\lib\site-packages\pandas\util\_decorators.py", line 296, in wrapper
return func(*args, **kwargs)
File "C:\Users\Home\AppData\Local\Programs\Python\Python38\lib\site-packages\pandas\io\json\_json.py", line 593, in read_json
filepath_or_buffer, _, compression, should_close = get_filepath_or_buffer(
File "C:\Users\Home\AppData\Local\Programs\Python\Python38\lib\site-packages\pandas\io\common.py", line 243, in get_filepath_or_buffer
raise ValueError(msg)
ValueError: Invalid file path or buffer object type: <class 'dict'>
In your case data is a dict.
So, try with:
pandas.DataFrame.from_dict(data)
I have a config.ini file that looks like this
[REDDIT]
client_id = 'myclientid23jd934g'
client_secret = 'myclientsecretjf30gj5g'
password = 'mypassword'
user_agent = 'myuseragent'
username = 'myusername'
When I try to use reddit's API praw like this:
import configparser
import praw
class redditImageScraper:
def __init__(self, sub, limit):
config = configparser.ConfigParser()
config.read('config.ini')
self.sub = sub
self.limit = limit
self.reddit = praw.Reddit(client_id=config.get('REDDIT','client_id'),
client_secret=config.get('REDDIT','client_secret'),
password=config.get('REDDIT','password'),
user_agent=config.get('REDDIT','user_agent'),
username=config.get('REDDIT','username'))
def get_content(self):
submissions = self.reddit.subreddit(self.sub).hot(limit=self.limit)
for submission in submissions:
print(submission.id)
def main():
scraper = redditImageScraper('aww', 25)
scraper.get_content()
if __name__ == '__main__':
main()
I get this traceback
Traceback (most recent call last):
File "config.py", line 30, in <module>
main()
File "config.py", line 27, in main
scraper.get_content()
File "config.py", line 22, in get_content
for submission in submissions:
File "C:\Users\Evan\Anaconda3\lib\site-packages\praw\models\listing\generator.py", line 61, in __next__
self._next_batch()
File "C:\Users\Evan\Anaconda3\lib\site-packages\praw\models\listing\generator.py", line 71, in _next_batch
self._listing = self._reddit.get(self.url, params=self.params)
File "C:\Users\Evan\Anaconda3\lib\site-packages\praw\reddit.py", line 454, in get
data = self.request("GET", path, params=params)
File "C:\Users\Evan\Anaconda3\lib\site-packages\praw\reddit.py", line 627, in request
method, path, data=data, files=files, params=params
File "C:\Users\Evan\Anaconda3\lib\site-packages\prawcore\sessions.py", line 185, in request
params=params, url=url)
File "C:\Users\Evan\Anaconda3\lib\site-packages\prawcore\sessions.py", line 116, in _request_with_retries
data, files, json, method, params, retries, url)
File "C:\Users\Evan\Anaconda3\lib\site-packages\prawcore\sessions.py", line 101, in _make_request
params=params)
File "C:\Users\Evan\Anaconda3\lib\site-packages\prawcore\rate_limit.py", line 35, in call
kwargs['headers'] = set_header_callback()
File "C:\Users\Evan\Anaconda3\lib\site-packages\prawcore\sessions.py", line 145, in _set_header_callback
self._authorizer.refresh()
File "C:\Users\Evan\Anaconda3\lib\site-packages\prawcore\auth.py", line 328, in refresh
password=self._password)
File "C:\Users\Evan\Anaconda3\lib\site-packages\prawcore\auth.py", line 138, in _request_token
response = self._authenticator._post(url, **data)
File "C:\Users\Evan\Anaconda3\lib\site-packages\prawcore\auth.py", line 31, in _post
raise ResponseException(response)
prawcore.exceptions.ResponseException: received 401 HTTP response
However when I manually insert the credentials, my code runs exactly as expected. Also, if I run the line
print(config.get('REDDIT', 'client_id'))
I get the output 'myclientid23jd934g' as expected.
Is there some reason that praw won't allow me to pass my credentials using configparser?
Double check what your inputs to praw.Reddit are:
kwargs = dict(client_id=config.get('REDDIT','client_id'),
client_secret=config.get('REDDIT','client_secret'),
password=config.get('REDDIT','password'),
user_agent=config.get('REDDIT','user_agent'),
username=config.get('REDDIT','username')))
print(kwargs)
praw.Reddit(**kwargs)
You're overcomplicating configuration here — PRAW will take care of this for you.
If you rename config.ini to praw.ini, you can replace your whole initialization with just
self.reddit = praw.Reddit('REDDIT')
This is because PRAW will look for a praw.ini file and parse it for you. If you want to give the section a more descriptive name, make sure to update it in the praw.ini as well as in the single parameter passed to Reddit (which specifies the section of the file to use).
See https://praw.readthedocs.io/en/latest/getting_started/configuration/prawini.html.
As this page notes, values like username and password should not have quotation marks around them. For example,
password=mypassword
is correct, but
password="mypassword"
is incorrect.
I'm trying to create a Python Wrapper for an upcoming API, so far I've been getting along well, but I keep bumping my head into the same problem. When making a POST request to add a mod to a game through the API, I feed the following dict item into the POST request
{'name': "Necro's Test Mod", 'name_id': None, 'summary': "This is a test mod submitted through the API using python's request library.", 'description': "I'm only about 43% sure this will actually work, whats more likely is that it will fail and return some messed up error with some even weirder error message before i am contacted by the mod.io developper saying they have personally revoked my api key (This isn't actually true i just need words", 'homepage': 'www.edain.wikia.com', 'metadata_blob': 'None', 'stock': 1, 'tags': ['cool', 'python', 'api'], 'logo': <_io.BufferedReader name='C:\\Users\\Clement\\Pictures\\Background\\7z6cSaI-lord-of-the-rings-wallpaper-hd.jpg'>}
And that dictionnary is added as follows
BASE_PATH = "https://api.test.mod.io/v1"
headers = {
'Authorization': 'Bearer ' + client.access_token,
'Content-Type': 'multipart/form-data',
'Accept': 'application/json'
}
r = requests.post(BASE_PATH + '/games/181/mods'.format(self.id), files = dict, headers = headers)
However it returns the following traceback, leading me to believe that the API returns a blank response
Traceback (most recent call last):
File "C:\Users\Clement\Desktop\mod.io\test.py", line 40, in <module>
new_mod = game.add_mod(newmod)
File "C:\Users\Clement\Desktop\mod.io\modio\game.py", line 83, in add_mod
r = requests.post(BASE_PATH + '/games/{}/mods'.format(self.id), files = mod.__dict__, headers = headers)
File "C:\Users\Clement\AppData\Local\Programs\Python\Python36\lib\site-packages\requests\api.py", line 112, in post
return request('post', url, data=data, json=json, **kwargs)
File "C:\Users\Clement\AppData\Local\Programs\Python\Python36\lib\site-packages\requests\api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\Clement\AppData\Local\Programs\Python\Python36\lib\site-packages\requests\sessions.py", line 494, in request
prep = self.prepare_request(req)
File "C:\Users\Clement\AppData\Local\Programs\Python\Python36\lib\site-packages\requests\sessions.py", line 437, in prepare_request
hooks=merge_hooks(request.hooks, self.hooks),
File "C:\Users\Clement\AppData\Local\Programs\Python\Python36\lib\site-packages\requests\models.py", line 308, in prepare
self.prepare_body(data, files, json)
File "C:\Users\Clement\AppData\Local\Programs\Python\Python36\lib\site-packages\requests\models.py", line 496, in prepare_body
(body, content_type) = self._encode_files(files, data)
File "C:\Users\Clement\AppData\Local\Programs\Python\Python36\lib\site-packages\requests\models.py", line 159, in _encode_files
fdata = fp.read()
AttributeError: 'NoneType' object has no attribute 'read'
Here's the documentation for the particular POST request I am attempting.
The files argument is meant for file uploads. You probably want to use data instead here.
r = requests.post(
BASE_PATH + '/games/181/mods'.format(self.id),
data = dict,
headers = headers
)
You get this error because requests expects the value of name_id to be a file pointer with a read() method. But in your payload it's None.
And don't use dict as a variable name.
I am trying to use Face API Python 2.7. I wanted to make gender recognition by photo but always getting the error. This is my code:
from six.moves import urllib
import httplib
import json
params = urllib.urlencode({
'returnFaceId': 'true',
'returnFaceAttributes': 'true',
'returnFaceAttributes': '{string}',
})
headers = {
'ocp-apim-subscription-key': "ee6b8785e7504dfe91efb96d37fc7f51",
'content-type': "application/octet-stream"
}
img = open("d:/Taylor.jpg", "rb")
conn = httplib.HTTPSConnection("api.projectoxford.ai")
conn.request("POST", "/vision/v1.0/tag?%s" % params, img, headers)
res = conn.getresponse()
data = res.read()
conn.close()
I've got this error :
Traceback (most recent call last):
File "<ipython-input-314-df31294bc16f>", line 3, in <module>
res = conn.getresponse()
File "d:\Anaconda2\lib\httplib.py", line 1136, in getresponse
response.begin()
File "d:\Anaconda2\lib\httplib.py", line 453, in begin
version, status, reason = self._read_status()
File "d:\Anaconda2\lib\httplib.py", line 409, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File "d:\Anaconda2\lib\socket.py", line 480, in readline
data = self._sock.recv(self._rbufsize)
File "d:\Anaconda2\lib\ssl.py", line 756, in recv
return self.read(buflen)
File "d:\Anaconda2\lib\ssl.py", line 643, in read
v = self._sslobj.read(len)
error: [Errno 10054]
If I use link instead photo:
img_url='https://raw.githubusercontent.com/Microsoft/Cognitive-Face-Windows/master/Data/detection1.jpg'
conn = httplib.HTTPSConnection("api.projectoxford.ai")
conn.request("POST", "/vision/v1.0/tag?%s" % params, img_url, headers)
res = conn.getresponse()
data = res.read()
conn.close()
I get:
data
Out[330]: '{ "statusCode": 401, "message": "Access denied due to invalid subscription key. Make sure to provide a valid key for an active subscription." }'
If I use:
KEY = 'ee6b8785e7504dfe91efb96d37fc7f51'
CF.Key.set(KEY)
img_url = 'https://raw.githubusercontent.com/Microsoft/Cognitive-Face-Windows/master/Data/detection1.jpg'
result = CF.face.detect(img_url)
all works fine:
result
[{u'faceId': u'52c0d1ac-f041-49cd-a587-e81ef67be2fb',
u'faceRectangle': {u'height': 213,
u'left': 154,
u'top': 207,
u'width': 213}}]
But in this case I don't know how to use method returnFaceAttribute (for gender detection)
and also if I use img in result = CF.face.detect(img_url) instead img_url
I get an error: status_code: 400
response: {"error":{"code":"InvalidImageSize","message":"Image size is too small or too big."}}
Traceback (most recent call last):
File "<ipython-input-332-3fe2623ccadc>", line 1, in <module>
result = CF.face.detect(img)
File "d:\Anaconda2\lib\site-packages\cognitive_face\face.py", line 41, in detect
data=data)
File "d:\Anaconda2\lib\site-packages\cognitive_face\util.py", line 84, in request
error_msg.get('message'))
CognitiveFaceException: Image size is too small or too big.
This happends with all sorts of img sizes.
Could anyone explain how to solve these problems?
It seems like you need to use your API key when connecting to Face API. That's why your second example works and the others don't.
Code 401 means that you're unauthorised, i.e. at that point you didn't log in with your key.
Maybe it'd be easier for you to try and use requests instead of urllib.
The first and error is caused by sending file object instead of file content. You should send image.read() instead of image.
The second error is caused by lacking of subscription key in the request header.
For the last error, I suspect you may have read the image file object once and thus reading it again will return empty result. You can try to send file path instead and it should work with the helper in Face API Python SDK.
I'm trying to send a couple of files to my backend:
class AccountsImporterTestCase(APITestCase):
def test(self):
data = [open('accounts/importer/accounts.csv'), open('accounts/importer/apartments.csv')]
response = self.client.post('/api/v1/accounts/import/', data, format='multipart')
self.assertEqual(response.status_code, status.HTTP_202_ACCEPTED)
But I'm getting an error:
Error
Traceback (most recent call last):
File "/vagrant/conjuntos/accounts/tests/cases.py", line 128, in test
response = self.client.post('/api/v1/accounts/import/', data, format='multipart')
File "/vagrant/venv/lib/python3.4/site-packages/rest_framework/test.py", line 168, in post
path, data=data, format=format, content_type=content_type, **extra)
File "/vagrant/venv/lib/python3.4/site-packages/rest_framework/test.py", line 89, in post
data, content_type = self._encode_data(data, format, content_type)
File "/vagrant/venv/lib/python3.4/site-packages/rest_framework/test.py", line 64, in _encode_data
ret = renderer.render(data)
File "/vagrant/venv/lib/python3.4/site-packages/rest_framework/renderers.py", line 757, in render
return encode_multipart(self.BOUNDARY, data)
File "/vagrant/venv/lib/python3.4/site-packages/django/test/client.py", line 156, in encode_multipart
for (key, value) in data.items():
AttributeError: 'list' object has no attribute 'items'
I know I'm not preparing the data correctly but is it possible to do it?, how?. Thanks!
Update: Trying #Kevin Brown solution
def test(self):
data = QueryDict('', mutable=True)
data.setlist('files', [open('accounts/importer/accounts.csv'), open('accounts/importer/apartments.csv')])
response = self.client.post('/api/v1/accounts/import/', data, format='multipart')
self.assertEqual(response.status_code, status.HTTP_202_ACCEPTED)
Got the following:
Error
Traceback (most recent call last):
File "/vagrant/conjuntos/accounts/tests/cases.py", line 130, in test
response = self.client.post('/api/v1/accounts/import/', data, format='multipart')
File "/vagrant/venv/lib/python3.4/site-packages/rest_framework/test.py", line 168, in post
path, data=data, format=format, content_type=content_type, **extra)
File "/vagrant/venv/lib/python3.4/site-packages/rest_framework/test.py", line 89, in post
data, content_type = self._encode_data(data, format, content_type)
File "/vagrant/venv/lib/python3.4/site-packages/rest_framework/test.py", line 64, in _encode_data
ret = renderer.render(data)
File "/vagrant/venv/lib/python3.4/site-packages/rest_framework/renderers.py", line 757, in render
return encode_multipart(self.BOUNDARY, data)
File "/vagrant/venv/lib/python3.4/site-packages/django/test/client.py", line 182, in encode_multipart
return b'\r\n'.join(lines)
TypeError: sequence item 4: expected bytes, bytearray, or an object with the buffer interface, str found
You are sending a list of files to the view, but you aren't sending them correctly. When you send data to a view, whether it is a Django view or DRF view, you are supposed to send it as a list of key-value pairs.
{
"key": "value",
"file": open("/path/to/file", "rb"),
}
To answer your question...
is it possible to do it?
It does not appear to be possible to upload multiple files using the same key (in tests), but it is possible to spread them out across multiple keys to achieve the same goal. Alternatively, you could set up your views to only handle a single file, and have multiple tests covering different test cases (apartments.csv, accounts.csv, etc.).
Your exception is being triggered because you are passing a single list instead of a dictionary, and Django cannot parse them correctly.
You might have some luck by directly forming the request dictionary using a QueryDict which is the internal representation of form data used by Django.
data = QueryDict(mutable=True)
data.setlist("files", [
open('accounts/importer/accounts.csv', 'rb'),
open('accounts/importer/apartments.csv', 'rb')
])
As this would more closely represent the data sent in through the browser. This has not been tested, but it's the way to send multiple non-file-values in one key.
Check if this solves your problem as from the errors its clearly says that the data should not be a list
data = {"account_csv": open('accounts/importer/accounts.csv'), "apartments_csv": open('accounts/importer/apartments.csv')}
This link you might find useful Uploading multiple files in a single request using python requests module