I'm working on a project on physionet.I have physionet credentials also.
I'm trying to get csv file directly into my python notebook instead of downloading it into my local machine.
I tried many ways but all are giving me 403.
I tried with User-Agent as well but it didn't work.
Anyone can suggest someway to do it.
from requests.auth import HTTPDigestAuth
url = 'https://physionet.org/files/nch-sleep/3.1.0/'
requests.get(url, auth=HTTPDigestAuth('username', 'pass'))
+++++++++++++++++++++++++++
if this is an HTML login, i dont think you would be able to mimic with requests call unless you know all the request header parameters, even though you provide username&password. if you want really an API call for your login, you need to hit the actual login api endpoint with basic auth credentials and tokens and with the response token, hit your actual endpoint to read your csv files.
Related
I am trying to scrape a website to get the shipping information for my company. I am able to log in to the website using Python's request library. The issue I am facing is that after I log in and try to navigate to a different URL that has the information I need the cookies change and logs me out.
When I look at the network in the dev tools I see that the cookies that it changes to are the response cookies. When I use .cookies to see if it was getting picked up, it only shows the request cookies.
I tried setting up a persistent sessions but that did not help. I then tried saving the cookies and got nowhere with that. I am not sure what else to do.
url = 'http://website/login'
creds = {'_method':'****','username':'*****','password':'*****'}
response = requests.post(url,data=creds)
token = response.cookies
response = requests.get('http://webiste/reports/view/17',cookies=token)
You can try token = response.headers['Set-Cookie'].
I am writing a python script which calls the API of Luno.com. It will form part of a larger python script(testing the new API call in a separate script) in which I have already made successful calls to a different site's API, however it did not require authentication. Luno documentation says it needs an API key (which I have) and to use the id and key secret as the username and password in a normal HTTP authentication, being new to python I googled how to do basic HTTP authentication and found this:
from requests.auth import HTTPBasicAuth
result = requests.get('url', auth=('username', 'password'))
I tried using this, but can not seem to get it right no matter what I try,I get the following error when printing the response code and the json output:
<Response [404]>
{'error': 'Cannot find that market', 'error_code': 'ErrMarketNotFound', 'error_action': {}}
I will include my python script and also links to the luno API sections which are applicable. The URL I want to access is https://api.luno.com/api/1/ticker , used to get the currency pair.
https://www.luno.com/en/developers/api
https://www.luno.com/en/developers/api#operation/getTicker
Screenshots if you don't want to click the links:
Python Code:
import json
import requests
from requests.auth import HTTPBasicAuth
urlLuno = 'https://api.luno.com/api/1/ticker'
statsAuthLuno = requests.get(urlLuno, auth=('idhere', 'secretkeyhere'))
print(statsAuthLuno)
print(statsAuthLuno.json())
the ticker API does not require authentication but it does require the market pair as seen here
requests.get(urlLuno, params={"pair": "XBTMYR"})
should get you the details you need.
Or you can get all tickers
I am trying to download a csv file from an authorized website.
I am able to get respond code of 200 with url https://workspace.xxx.com/abc/ (click in this web page to download the csv) but respond code of 401 at url = 'https://workspace.xxx.com/abc/abc.csv'
This is my code:
import requests
r = requests.get(url, auth=('myusername', 'mybasicpass'))
I tried adding header and using session but still get respond code of 401.
First of all, you have to investigate how the website accepts the password.
They might be using HTTP authentication or Authorization header in the request.
You can log in using their website and then download the file .study how they are passing authorization.
I am sure they are not accepting plain passwords in authorization they might be encoding it in base64 or another encoding scheme.
My advice to you is to open the developer console and study their requests in network tab. You can post more information so one could help you more.
How to get Authorization token from a webpage using python requests, i have used requests basicAuth to login, it was worked, but subsequent pages are not accpting te basicAuth, it returns "Authuser is not validated"
There is a login url where i have successfully logged in using python requests's basicAuth. then succeeding pages didn't accept basicAuth credential but it needed authorization header. after looking into browser inspect tool, found out that, this authorization header's value is generated as a part of session local storage. is there any way to get this session value without using webdriver API?
Sounds like what you need is a requests persistent session
import requests
s=requests.Session()
#then simply make the request like you already are
r=s.get(r'https://stackoverflow.com/')
#the cookies are persisted
s.cookies.get_dict()
>{'prov':......}
i can't really get more specific without more info about the site you're using.
I am using the following script:
import requests
import json
import os
COOKIES = json.loads("") #EditThisCookie export here (json) to send requests
COOKIEDICTIONARY = {}
for i in COOKIES:
COOKIEDICTIONARY[i['name']] = i['value']
def follow(id):
post = requests.post("https://instagram.com/web/friendships/" + id + "/follow/", cookies=COOKIEDICTIONARY)
print(post.text)
follow('309438189')
os.system("pause")
This script is supposed to send a follow request to the user, '3049438189' on Instagram. However, if the code is run, the post.text outputs some HTML code, including
"This page could not be loaded. If you have cookies disabled in your
browser, or you are browsing in Private Mode, please try enabling
cookies or turning off Private Mode, and then retrying your action."
It's supposed to append the cookies to the variable, COOKIEDICTIONARY in a "requests" module readable format. If you print the array (I don't know what it's called in Python), it replies with all of the cookies and their values.
The cookies put in are valid and the requests syntax (I believe to be) is correct.
I have fixed it. The problem was certain headers that I needed were not present, such as Origin (I will get the full list soon). For anybody who wants to imitate any instagram post request, you need those headers or it will error.