I have a problem with the etherscan api on ropsten testnetwork, the output of the code is: expecting value line 1 column 1 (char 0)
the code:
import requests, json
ADD = "0xfbb61B8b98a59FbC4bD79C23212AddbEFaEB289f"
KEY = "HERE THE API KEY"
REQ = requests.get(f"https://api-ropsten.etherscan.io/api?module=account&action=balance&address={str(ADD)}&tag=latest&apikey={str(KEY)}")
CONTENT = json.loads(REQ.content)
BALANCE = int(CONTENT['result'])
print(BALANCE)
When I try to do a request it gives back <Response [403]>
Some websites don't allow Python scripts to access their website. You can get around this by adding a user agent in you request.
the code would look something like this:
import requests, json
ADD = "0xfbb61B8b98a59FbC4bD79C23212AddbEFaEB289f"
KEY = "HERE THE API KEY"
LINK = f"https://api-ropsten.etherscan.io/api?module=account&action=balance&address={str(ADD)}&tag=latest&apikey={str(KEY)}"
headers = {"HERE YOUR USER-AGENT"}
REQ = requests.get(LINK, headers = headers)
CONTENT = json.loads(REQ.content)
BALANCE = int(CONTENT['result'])
print(BALANCE)
To find your user agent simply type in google: my user agent
Related
Attempting to write a put request in Python that keeps getting denied - the issue seems to be that the request isn't accepting the list as value for the dict.
Any suggestions on how I can get this to be accepted?
import requests
key = 'Bearer abc123'
url = 'www.my_url.com'
headers = {'Content-Type':'application/x-www-form-urlencoded',
'Accept':'application/json',
'Authorization':key}
data = {}
data['first_name']='John'
data['last_name']='Doe'
data['employee_job[roles]'] = [{'name':'CEO'}]
r = requests.put(url,data=data,headers=headers)
if the server accept put request,then the problem maybe json format
you could try this,put the data as json directly,not the form encoded.
need more information to detect the problem,can you provide the api document?
import requests
key = 'Bearer abc123'
url = 'www.my_url.com'
headers = {
'Accept':'application/json',
'Authorization':key}
data = {}
data['first_name']='John'
data['last_name']='Doe'
data['employee_job'] = {'roles':[{'name':'CEO'}]}
r = requests.put(url,json=data,headers=headers)
I am going to extract posts in a forum, named positive wellbeing during isolation" in HealthUnlocked.com
I can extract posts without login, but I cannot extract posts with logging. I used " url = 'https://solaris.healthunlocked.com/posts/positivewellbeing/popular?pageNumber={0}'.format(page)" to extract pots, but I don't know how I can connect it to login as the URL is in JSON format.
I would appreciate it if you could help me.
import requests, json
import pandas as pd
from bs4 import BeautifulSoup
from time import sleep
url = "https://healthunlocked.com/private/programs/subscribed?user-id=1290456"
payload = {
"username" : "my username goes here",
"Password" : "my password goes hereh"
}
s= requests.Session()
p= s.post(url, data = payload)
headers = {"user-agent": "Mozilla/5.0"}
pages =2
data = []
listtitles=[]
listpost=[]
listreplies=[]
listpostID=[]
listauthorID=[]
listauthorName=[]
for page in range(1,pages):
url = 'https://solaris.healthunlocked.com/posts/positivewellbeing/popular?pageNumber=
{0}'.format(page)
r = requests.get(url,headers=headers)
posts = json.loads(r.text)
for post in posts:
sleep(3.5)
listtitles.append(post['title'])
listreplies.append(post ["totalResponses"])
listpostID.append(post["postId"])
listauthorID.append(post ["author"]["userId"])
listauthorName.append(post ["author"]["username"])
url = 'https://healthunlocked.com/positivewellbeing/posts/{0}'.format(post['postId'])
r = requests.get(url,headers=headers)
soup = BeautifulSoup(r.text, 'lxml')
listpost.append(soup.select_one('div.post-body').get_text('|', strip=True))
## save to CSV
df=pd.DataFrame(list(zip(*
[listpostID,listtitles,listpost,listreplies,listauthorID,listauthorName]))).add_prefix('Col')
df.to_csv('out1.csv',index=False)
print(df)
sleep(2)
For most websites, you have to first get a token by logging in. Most of the time, this is a cookie. Then, in authorized requests, you can send that cookie along. Open the network tab in developer tools and then log in with your username and password. You'll be able to see how the request is formatted and where it is too. From there, just try to replicate it in your code.
I wanted to know how can I get data using QID URL. I have some names, I use falcon 2.0 entity linker curl command( change it into python script) to get information of its QID. Now I want to use that QID to access information about the persons gender( male or female) or alias or other information. Can someone give an idea how it should be approached. The code to get QID URL is given below. the link to falcon 2.0 is https://github.com/SDM-TIB/Falcon2.0.
import requests
import json
response_list=[]
person_names=[]
if __name__ == '__main__':
limit=100
with open(filename, 'r') as in_file:
in_reader = in_file.readlines()
for data in in_reader:
if limit > 0:
person_names.append(data.rstrip())
limit -=1
else :
break
"""
Url of post request and header of type json create linking against each line of text.
"""
url="https://labs.tib.eu/falcon/falcon2/api?mode=long"
headers = {'Content-type': 'application/json'}
for name in person_names:
data = {"text":name }
data_json = json.dumps(data)
response = requests.post(url, data=data_json, headers=headers)
print(response.content)
It gives output as http://www.wikidata.org/entity/Q42493 for the entity.
You can convert URLs of the form http://www.wikidata.org/entity/Q42493 to https://www.wikidata.org/wiki/Special:EntityData/Q42493.json to get a JSON payload with the information that you seek, but first you should make sure that the entity resolution algorithm is giving you accurate results so that you have the correct QID to start with.
Firstly, I should add that you can find this request by doing the following:
1- Go to [airline site][1]
2- Type in "From" = "syd"
3- Type in "To" = "sin"
4- Make the departure date sep.3 and click one-way and search
5- On the search result page check your network get request when you click on an available seat option radio button
I'm trying to use the requests module to get the response for example from this site
And this is what I'm trying:
url = "http://www.singaporeair.com/chooseFlightJson.form?"
payload = {'selectedFlightIdDetails[0]':amount_data,'hid_flightIDs':'','hid_recommIDs':'','tripType':"O",'userPreferedCurrency':""}
response = requests.get(url, params=payload)
print response.json()
The response is supposed to be:
{"price":{"total":"595.34","currency":{"code":"AUD","label":""},"adult":{"count":1,"label":"Adult","cost":"328.00","total":"328.00"},"child":{"count":0,"label":"Child","cost":"0.00","total":"0.00"},"infant":{"count":0,"label":"Infant","cost":"0.00","total":"0.00"},"addOns":[{"label":"Airport / Government taxes ","cost":"83.24"},{"label":"Carrier Surcharges","cost":"184.10"}],"disclaimer":"Prices are shown in Canadian Dollar(CAD)","rate":"595.34 AUD \u003d 913.80 CAD","ratehint":"Estimated","unFormattedTotal":"595.34"},"messages":{"O3FF11SYD":"A few seats left","O1FF31SYD":" ","R0FF31SYD":"A few seats left","O2FF31SYD":"A few seats left","O0FF31SYD":" ","O1FF11SYD":"A few seats left","O0FF21SYD":" ","O2FF21SYD":" ","O3FF21SYD":" ","O1FF21SYD":" "},"cabinClass":{"onwardCabin":["Economy"]}}
The response is the value None, encoded in JSON; the server returns null\r\n, which means the same as None in Python.
The content type is wrong here; it is set to text/html, but the response.json() return value is entirely correct for what the server sent:
>>> import requests
>>> url = "http://www.singaporeair.com/chooseFlightJson.form?"
>>> amount_data = 0
>>> payload = {'selectedFlightIdDetails[0]':amount_data,'hid_flightIDs':'','hid_recommIDs':'','tripType':"O",'userPreferedCurrency':""}
>>> response = requests.get(url, params=payload)
>>> response
<Response [200]>
>>> response.headers['content-type']
'text/html; charset=ISO-8859-1'
>>> response.text
'null\r\n'
>>> response.json() is None
True
change protocol from http to https.
url = "https://www.singaporeair.com/chooseFlightJson.form?"
The solution is to use requests session, like so:
session = requests.Session()
Then to call all of the urls you need to simply do:
response = session.get(url)
This sets the cookies and session variables which are necessary to retrieve the data.
I have seen jsessionid used in different ways, in the url, or in this case in a cookie. But there are probably other session info that are required, which is taken care of by a requests session object.
while making http request make sure that response_type set to exact use case you are trying with.
In my case response_type='object' worked to eliminate None type in response.
I'm trying to write a simple script to log into Wikipedia and perform some actions on my user page, using the Mediawiki api. However, I never seem to get past the first login request (from this page: https://en.wikipedia.org/wiki/Wikipedia:Creating_a_bot#Logging_in). I don't think the session cookie that I set is being sent. This is my code so far:
import Cookie, urllib, urllib2, xml.etree.ElementTree
url = 'https://en.wikipedia.org/w/api.php?action=login&format=xml'
username = 'user'
password = 'password'
user_data = [('lgname', username), ('lgpassword', password)]
#Login step 1
#Make the POST request
request = urllib2.Request(url)
data = urllib.urlencode(user_data)
login_raw_data1 = urllib2.urlopen(request, data).read()
#Parse the XML for the login information
login_data1 = xml.etree.ElementTree.fromstring(login_raw_data1)
login_tag = login_data1.find('login')
token = login_tag.attrib['token']
cookieprefix = login_tag.attrib['cookieprefix']
sessionid = login_tag.attrib['sessionid']
#Set the cookies
cookie = Cookie.SimpleCookie()
cookie[cookieprefix + '_session'] = sessionid
#Login step 2
request = urllib2.Request(url)
session_cookie_header = cookieprefix+'_session='+sessionid+'; path=/; domain=.wikipedia.org; HttpOnly'
request.add_header('Set-Cookie', session_cookie_header)
user_data.append(('lgtoken', token))
data = urllib.urlencode(user_data)
login_raw_data2 = urllib2.urlopen(request, data).read()
I think the problem is somewhere in the request.add_header('Set-Cookie', session_cookie_header) line, but I don't know for sure. How do I use these python libraries to send cookies in the header with every request (which is necessary for a lot of API functions).
The latest version of requests has support for sessions (as well as being really simple to use and generally great):
with requests.session() as s:
s.post(url, data=user_data)
r = s.get(url_2)