Send HTTP Headers as a Float in Python Scrapy - python

I am trying at scrape an API that accepts some value in headers only in float form, when I send it in string form, it gives 400 Bad Request, and when I try to send headers in float form scrapy gives Error like this:
self.headers = Headers(headers or {}, encoding=encoding)
File "C:\Python27\lib\site-packages\scrapy\http\headers.py", line 12, in __init__
super(Headers, self).__init__(seq)
File "C:\Python27\lib\site-packages\scrapy\utils\datatypes.py", line 193, in __init__
self.update(seq)
File "C:\Python27\lib\site-packages\scrapy\utils\datatypes.py", line 229, in update
super(CaselessDict, self).update(iseq)
File "C:\Python27\lib\site-packages\scrapy\utils\datatypes.py", line 228, in <genexpr>
iseq = ((self.normkey(k), self.normvalue(v)) for k, v in seq)
File "C:\Python27\lib\site-packages\scrapy\http\headers.py", line 27, in normvalue
return [self._tobytes(x) for x in value]
File "C:\Python27\lib\site-packages\scrapy\http\headers.py", line 40, in _tobytes
raise TypeError('Unsupported value type: {}'.format(type(x)))
TypeError: Unsupported value type: <type 'float'>
None
Anyone have any solution or faced similer kind of problem?

First of all, headers are always sent as string. There is no data-type for headers like int, bool, float.
I might send an Api a header X-RELOAD-TIME 2.0003355 but that doesn't mean i need 2.0003355 as float. And that is what the library is complaining
So in your headers make sure
headers["Name-Of-Float-Header"] = str(float_value)
and then call should be able to go through

Related

web3.py createFilter : TypeError: Unsupported type: '<class 'list'>'. Must be one of: bool, str, bytes, bytearrayor int

I write the contract,
pragma solidity >=0.7.0 <0.9.0;
contract voteEXample{
event getVoterAndVoted(address voter,bytes32 voted);
mapping(bytes32 => uint8) public receivedVotes;
function vote(bytes32 toVote) public{
require(check(toVote));
receivedVotes[toVote] += 1;
emit getVoterAndVoted(msg.sender,toVote);
}
}
And through web3.py, I deployed the contract successfully. But, I failed in using the createFilter method.
greeter = w3.eth.contract(address=address,abi = abi)
transfer_filter = greeter.events.getVoterAndVoted.createFilter(fromBlock="0x0")
The createFilter occurs the following error:
Traceback (most recent call last):
File "D:/SEAC/experiment/VoteTest.py", line 103, in <module>
transfer_filter = greeter.events.getVoterAndVoted.createFilter(fromBlock="0x0")
File "C:\Users\jinpp\AppData\Roaming\Python\Python37\site-packages\eth_utils\decorators.py", line 20, in _wrapper
return self.method(objtype, *args, **kwargs)
File "C:\Users\jinpp\AppData\Roaming\Python\Python37\site-packages\web3\contract.py", line 1231, in createFilter
log_filter = filter_builder.deploy(self.web3)
File "C:\Users\jinpp\AppData\Roaming\Python\Python37\site-packages\web3\_utils\events.py", line 408, in deploy
log_filter = cast("LogFilter", w3.eth.filter(self.filter_params))
File "C:\Users\jinpp\AppData\Roaming\Python\Python37\site-packages\web3\_utils\events.py", line 393, in filter_params
"topics": self.topics,
File "C:\Users\jinpp\AppData\Roaming\Python\Python37\site-packages\web3\_utils\events.py", line 381, in topics
return normalize_topic_list(cons(to_hex(self.event_topic), arg_topics))
File "C:\Users\jinpp\AppData\Roaming\Python\Python37\site-packages\eth_utils\decorators.py", line 76, in wrapper
return to_wrap(*args, **kwargs)
File "C:\Users\jinpp\AppData\Roaming\Python\Python37\site-packages\eth_utils\conversions.py", line 50, in to_hex
"or int.".format(repr(type(primitive)))
TypeError: Unsupported type: '<class 'list'>'. Must be one of: bool, str, bytes, bytearrayor int.
I don't know why this problem happens. Can anyone tell me. Thanks a lot!!!
I'm having the same problem, except with a contract function call. I think that it's happening when the addresses are converted to hex, as the last thing I see before the type error is:
File "/home/user/.local/lib/python3.9/site-packages/eth_utils/conversions.py", line 48, in to_hex
raise TypeError(...
I have a hunch it's because we're not converting the addresses to checksum, as I know needs to be done in web3.py. I'm gonna try that first in my work and post back. I'm using Python 3.9 on Ubuntu 20.04 (I think, just wiped my old MacBook and threw it on there so all this would work).

appending input to the end of api call in python

def i(bot,update,args):
coin=args
infoCall =requests.get("https://api.coingecko.com/api/v3/coins/").json()
coinId = infoCall ['categories']
update.message.reply_text(coinId)
I would like to add to the end of the api request the args declared in coins=args so that it retrieves the info my user requests but this is the error i get
coinId = infoCall ['categories']
KeyError: 'categories'
which my guess is because its not formating the request correctly so the api is giving a 404 and not the info being requested
def i(bot,update,args):
coin=args
infoCall =requests.get("https://api.coingecko.com/api/v3/coins/").json()
infoCall = json.loads(infoCall)+str(coins)
coinId = infoCall['categories']
update.message.reply_text(str (coinId))
after adding this, this is the new error i get
Traceback (most recent call last):
File "C:\Users\Matthew\AppData\Local\Programs\Python\Python37-32\lib\site-packages\telegram\ext\dispatcher.py", line 279, in process_update
handler.handle_update(update, self)
File "C:\Users\Matthew\AppData\Local\Programs\Python\Python37-32\lib\site-packages\telegram\ext\commandhandler.py", line 173, in handle_update
return self.callback(dispatcher.bot, update, **optional_args)
File "C:/Users/Matthew/Desktop/coding_crap/CryptoBotBetav2.py", line 78, in i
infoCall = json.loads(infoCall)+str(coins)
File "C:\Users\Matthew\AppData\Local\Programs\Python\Python37-32\lib\json\__init__.py", line 341, in loads
raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not list
Basically you are not appending the args param to the api point that's why you were getting the error. You need to append the 'bitcoin' to the api point before you make the request rather than on the output.
A typical example would be as follow. I have removed the update and other unused variables. You can put them as you need.
import requests
def i(args):
coin=args
infoCall =requests.get("https://api.coingecko.com/api/v3/coins/"+ args).json()
coinId = infoCall ['categories']
print(coinId)
# update.message.reply_text(coinId)
i('bitcoin')
Output:
['Cryptocurrency']

What am I doing wrong with requests in python: ValueError: Expecting value: line 1 column 1 (char 0)?

I'm not sure even how to ask the question, as it seems it would require quite a lot of code to get into the details. Rather than show the code, I will discuss the behavior when I run.
I am using requests to grab information from an online database. When I run a for loop to go through all of my entries, I get an error like the one below on one of the first 20 entries (usually the first, but not always). The entries in the list are all alike (just different ID numbers). I am using sleep() to ensure that I do not go beyond my rate limit (I have tried increasing sleep to ridiculous wait times, but still get the error). What really surprises me is that it works some, and then gets stuck.... what could cause that?
Also, the code was working before, then I made a large number of edits to other code in the same file, but I didn't think I edited anything related to this.
Traceback (most recent call last):
File "C:/Users/Mark/PycharmProjects/Riot_API_Challenger_Stats/Main.py", line 233, in <module>
main()
File "C:/Users/Mark/PycharmProjects/Riot_API_Challenger_Stats/Main.py", line 212, in main
match_histories=get_match_histories(challenger_Ids+master_Ids)
File "C:/Users/Mark/PycharmProjects/Riot_API_Challenger_Stats/Main.py", line 62, in get_match_histories
match_histories[summoner_Ids[i]]=api.get_match_history_data(summoner_Ids[i])
File "C:\Users\Mark\PycharmProjects\Riot_API_Challenger_Stats\RiotAPI.py", line 52, in get_match_history_data
return self._request(api_url)
File "C:\Users\Mark\PycharmProjects\Riot_API_Challenger_Stats\RiotAPI.py", line 25, in _request
return response.json()
File "C:\Users\Mark\Anaconda3\lib\site-packages\requests\models.py", line 819, in json
return json.loads(self.text, **kwargs)
File "C:\Users\Mark\Anaconda3\lib\json\__init__.py", line 318, in loads
return _default_decoder.decode(s)
File "C:\Users\Mark\Anaconda3\lib\json\decoder.py", line 343, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\Mark\Anaconda3\lib\json\decoder.py", line 361, in raw_decode
raise ValueError(errmsg("Expecting value", s, err.value)) from None
ValueError: Expecting value: line 1 column 1 (char 0)
Here are lines 10-25 of RiotAPI
def _request(self, api_url, params={}):
args = {'api_key':self.api_key}
for key, value in params.items():
if key not in args:
args[key] = value
#requests.get accesses the URL
response = requests.get(
Consts.URL['base'].format(
proxy=self.region,
region=self.region,
url=api_url
),
params=args
)
print(response.url)
return response.json()
Here is the response:
{"matches":[{"matchId":1878534497,"region":"NA","platformId":"NA1","matchMode":"CLASSIC","matchType":"MATCHED_GAME","matchCreation":1436223958539,"matchDuration":2097,"queueType":"RANKED_SOLO_5x5","mapId":11,"season":"SEASON2015","matchVersion":"5.12.0.348","participants":[{"teamId":200,"spell1Id":4,"spell2Id":7,"championId":15,"highestAchievedSeasonTier":"UNRANKED","timeline":{"creepsPerMinDeltas":{"zeroToTen":5.699999999999999,"tenToTwenty":6.9,"twentyToThirty":7.1},"xpPerMinDeltas":{"zeroToTen":358.5,"tenToTwenty":350.0,"twentyToThirty":364.20000000000005},"goldPerMinDeltas":{"zeroToTen":365.3,"tenToTwenty":337.5,"twentyToThirty":287.5},"csDiffPerMinDeltas":{"zeroToTen":-0.7,"tenToTwenty":-1.7000000000000004,"twentyToThirty":1.0999999999999999},"xpDiffPerMinDeltas":{"zeroToTen":-0.9000000000000057,"tenToTwenty":-114.75,"twentyToThirty":-121.19999999999999},"damageTakenPerMinDeltas":{"zeroToTen":480.5,"tenToTwenty":565.3,"twentyToThirty":1258.6},"damageTakenDiffPerMinDeltas":{"zeroToTen":-147.49999999999994,"tenToTwenty":-134.69999999999996,"twentyToThirty":15.0},"role":"DUO_CARRY","lane":"BOTTOM"},"masteries":[{"masteryId":4112,"rank":4},{"masteryId":4114,"rank":1},{"masteryId":4122,"rank":3},{"masteryId":4124,"rank":1},{"masteryId":4132,"rank":1},{"masteryId":4134,"rank":3},{"masteryId":4142,"rank":2},{"masteryId":4144,"rank":1},{"masteryId":4151,"rank":1},{"masteryId":4152,"rank":3},{"masteryId":4162,"rank":1},{"masteryId":4211,"rank":2},{"masteryId":4212,"rank":2},{"masteryId":4221,"rank":1},{"masteryId":4222,"rank":3},{"masteryId":4232,"rank":1}],"stats":{"winner":false,"champLevel":14,"item0":3031,"item1":0,"item2":3142,"item3":3035,"item4":1053,"item5":3250,"item6":3342,"kills":4,"doubleKills":1,"tripleKills":0,"quadraKills":0,"pentaKills":0,"unrealKills":0,"largestKillingSpree":3,"deaths":12,"assists":5,"totalDamageDealt":184710,"totalDamageDealtToChampions":27477,"totalDamageTaken":30740,"largestCriticalStrike":684,"totalHeal":2952,"minionsKilled":237,"neutralMinionsKilled":1,"neutralMinionsKilledTeamJungle":1,"neutralMinionsKilledEnemyJungle":0,"goldEarned":12074,"goldSpent":12065,"combatPlayerScore":0.....etc.}}]}]}
It'd be helpful if you can include a snippet of your code to see how you've actually used it.
However, from the exception, I think you're trying to do a
return response.json()
at line 25 of C:\Users\Mark\PycharmProjects\Riot_API_Challenger_Stats\RiotAPI.py
but the response is not in JSON format.
You can see the raw response with
print response.text
to see the string version of the response and check that the string is in JSON format.
It would be nice if you could post the actual code. But if you are unable to do so due to confidential reasons we can surmise some information from the stack trace.
You are using some HTTP protocol (SOAP/ResT) API to get a number( or series of numbers) in JSON format. One of these ID numbers has a character that is not expected or the JSON is invalid itself. Try and print the JSON request you receive before you pass it to see which one fails. Then create a unit test and run it try to analyze it with breakpoints.
Could be some sort of hyphenated or foreign character based on the database.

ValueError: unknown url type

The title pretty much says it all. Here's my code:
from urllib2 import urlopen as getpage
print = getpage("www.radioreference.com/apps/audio/?ctid=5586")
and here's the traceback error I get:
Traceback (most recent call last):
File "C:/Users/**/Dropbox/Dev/ComServ/citetest.py", line 2, in <module>
contents = getpage("www.radioreference.com/apps/audio/?ctid=5586")
File "C:\Python25\lib\urllib2.py", line 121, in urlopen
return _opener.open(url, data)
File "C:\Python25\lib\urllib2.py", line 366, in open
protocol = req.get_type()
File "C:\Python25\lib\urllib2.py", line 241, in get_type
raise ValueError, "unknown url type: %s" % self.__original
ValueError: unknown url type: www.radioreference.com/apps/audio/?ctid=5586
My best guess is that urllib can't retrieve data from untidy php URLs. if this is the case, is there a work around? If not, what am I doing wrong?
You should first try to add 'http://' in front of the url. Also, do not store the results in print, as it is binding the reference to another (non callable) object.
So this line should be:
page_contents = getpage("http://www.radioreference.com/apps/audio/?ctid=5586")
This returns a file like object. To read its contents you need to use different file manipulation methods, like this:
for line in page_contents.readlines():
print line
You need to pass a full URL: ie it must begin with http://.
Simply use http://www.radioreference.com/apps/audio/?ctid=5586 and it'll work fine.
In [24]: from urllib2 import urlopen as getpage
In [26]: print getpage("http://www.radioreference.com/apps/audio/?ctid=5586")
<addinfourl at 173987116 whose fp = <socket._fileobject object at 0xa5eb6ac>>

Python - ValueError: need more than 1 value to unpack

Please check this question Python oauth2 - making request I am working with vimeo integration in my web application.
Initially I got an oauth_signature and I had no problems(no errors), I tried those things once again from the first and Now I'm getting ValueError: need more than 1 value to unpack while making this request
>>> r = request.get(url, headers=headers)
You can check out my code here https://gist.github.com/2949182
The error is
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/requests-0.10.1-py2.7.egg/requests/api.py", line 51, in get
return request('get', url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests-0.10.1-py2.7.egg/requests/api.py", line 39, in request
return s.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests-0.10.1-py2.7.egg/requests/sessions.py", line 159, in request
headers[k] = header_expand(v)
File "/usr/local/lib/python2.7/dist-packages/requests-0.10.1-py2.7.egg/requests/utils.py", line 152, in header_expand
for i, (value, params) in enumerate(headers):
ValueError: need more than 1 value to unpack
Thanks!
UPDATE
>>> headers
{'Authorization': u'oauth_body_hash=XXXXXXXXXXXXXXXXXXXXXXXXXX,oauth_nonce=3454768,oauth_timestamp=1340035585,oauth_consumer_key=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,oauth_signature_method=HMAC-SHA1,oauth_version=1.0,oauth_signature=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX,oauth_callback=http://127.0.0.1:8000/information/vimeo'}
to be able to unpack a dictionary you would have to use the .items(), so the code would be like this:
for i, (value, params) in enumerate(headers.items()):
now since that is not your code and you can't change it, what the error is telling you is that the headers should not be a dictionary but a tuple (or a list), if you pass the header like this:
headers = [("Authorization", "Values")]
it should work.
EDIT: This doesn't works. Now the dictionary version {"Authorization": "Values"} works for me, maybe updating requests will help.

Categories

Resources