So I've been trying to use requests so I can log in to replit.com to access files from my account and pull them to me, however on the log in part it says 'Expected X-Requested-With header', and for the status code it keeps giving me 403 (I think that means forbidden). Also to mention that requests.get() works completely fine, however the method I use has to be for a field where the method is POST. I tried to see the HTML response, however I think there is only the Expected X-Requested With header. Can anyone tell me what is wrong with my code? (down below)
response = requests.post("https://replit.com/login", auth = (email, password), allow_redirects = True)
Also even if the requests.post is only the URL it still doesn't work. Any help?
Edit:
I figured out how to do the headers, it turned out the website had its own header so I used that, but now I can't bypass recaptcha.
Related
I know how this may read to you but I am seriously out of ideas on this one.
I've written something in Python that can download stuff from Imgur using their API. I have an Authorization clientID and everything, and this thing works.
But sometimes I am getting a 400 HTML response status code with empty text when requesting a direct link so I can save the file. According to the doc this error will come back at you when you're missing a required parameter or if a part of your request is incorrect; this feels like a weird error to come back at me, when other requests get through just fine. The weirder part is that I can send 100 requests in 100 seconds, all will come back with 400 None. But once I open the image in my browser and view it, the request suddently changes to 200, everything works and this link will never throw a 400 ever again.
My friend suggestet that maybe it had something to do with my IP adress (getting flagged as a spam bot or whatever), so I opened the URLs via my phone on cellular, but as soon as the image loaded on my phone, the requests were successful. Also, when I tried finding more links with this strange behaviour, I tested a bunch of images from the imgur front page, and all worked fine. Combining this with the fact that I got the problematic links from very old reddit threads leaves me with the only idea that it has something to do with the age of the files or rather their last view date.
The requests happened via a code similar to:
headers = {'Authorization': f"Client-ID <id>"}
url = f"https://api.imgur.com/3/image/{fileID}"
requests.get(url, headers=headers)
I'm asking this here mainly because the Imgur API page says the best way to get help about the API is to post the problem here on stackoverflow. Maybe one of their engineers sees this and can answer my question or maybe someone else has an idea what may be going on here. In any case, I would be grateful for any useful input ^^
I'm trying to make one simple request:
ua=UserAgent()
req = requests.get('https://www.casasbahia.com.br/' , headers={'User-Agent':ua.random})
I would understand if I received <Response [403] or something like that, but instead, a recive nothing, the code keep runing with no response.
using logging I see:
I know I could use a timeout to avoid keeping the code running, but I just want to understand why I don't get an response
thanks in advance
I never used this API before, but from what I researched on here just now, there are sites that can block requests from fake users.
So, for reproducing this example on my PC, I installed fake_useragent and requests modules on my Python 3.10, and tried to execute your script. It turns out that with my Authentic UserAgent string, the request can be done. When printed on the console, req.text shows the entire HTML file received from the request.
But if I try again with a fake user agent, using ua.random, it fails. The site was probably developed to detect and reject requests from fake agents (or bots).
Though again, this is just theory. I have no ways to access this site's server files to comprove it.
Very simple problem. When I go to my Web site in any browser, it redirects. When I do a requests.get(my_url) on the exact same URL and protocol, I get a 200 response. Why? It doesn't matter if I set allow_redirects to True or False, I still get the same behavior. I tried setting a header so requests pretends to be Firefox, I still get the same behavior.
How can I get requests.get to give me the redirect that a browser gets on the exact same URL?
#!/usr/bin/env/python
import requests
result = requests.get("https://bgjkfgbjfgghbjdfbhdgfjkh.fake")
print(result.status_code)
The above prints "200".
from all your comments, it sounds like the status 200 is not signifying that the end page was fetched correctly (you say response.text is completely different in both cases - i.e. it isn't the end page you expect).
In this case, either:
You are getting a simple 'click this to redirect' page, and the automated redirect that you experience from firefox is being implemented in javascript (which requets.get can't handle).
Or the website is giving another 'error' page (without a 400 series error), due to something else missing, such as a cookie.
The fact that response.history is empty even when allow_redirects is True is yet more evidence that you are never getting the end page that you see in firefox, and that the 200 you are getting is not the end page, but a 'click here' or error page.
I'm using Python requests 2.19.1 .
But I'm facing an intermittent issue where I get no response at all when I post to a specific url.
I'm trying to check if the LDAP is giving me the expected output for invalid credentials.
Here's the format:
requests.post('https://oxhp-member.uhc.com/Member/MemberPortal/j_acegi_security_check',
credentials_payload)
that I'm posting
Almost everytime, it works fine. But sometimes, it doesn't give any response for that. Even network issues gives us some response. Right? Why am I not getting any response for the above call.
Is there any existing bug in requests?
Somebody please point me in correct direction.
requests is not responsible for "giving back response". The server you are using requests to post to is.
To see the response you have to keep it in a variable and handle it somehow.
resp = requests.post('https://oxhp-member.uhc.com/Member/MemberPortal/j_acegi_security_check',
credentials_payload)
print(resp.status_code)
print(resp.content)
Whatever resp contains is the responsibility of the server.
UPDATE: Solved. I was indeed making a basic mistake, among other things. My usage of the session.headers.update function was incorrect; instead of session.headers.update = {foo:bar}, I needed to be doing session.headers.update({foo:bar}). Additionally, I changed the following section of code:
From this:
payload = urllib.parse.urlencode({
"grant_type":"urn:ietf:params:oauth:grant-type:jwt-bearer",
"assertion": str(token)
})
To this:
payload = "grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&assertion=" + token.decode("utf-8")
The code now works as intended.
Original question below
I've seen several hits on SO and Google about this problem; none of them have helped, although I've certainly made sure to double-check my code to make sure I'm not guilty of the same problems they detail. The problems people tend to have involve passing the POST data as parameters or POSTing to the wrong URL, which I'm not doing, as far as I can tell. Additionally, most of the hits I've found involve 3-legged OAuth2 involving users; I've found comparatively few hits pertaining to service accounts and JWTs, which differ enough from the user flow that I'm concerned about how relevant they are to my problem.
I'm trying to get an access token from the Google Authentication server for a service account. I've generated my JWT and now want to POST to the server to receive back my access token. I've set the headers according to the documentation described here, under "Making the access token request," and as far as I can tell, my request is up to spec, but Google responds back with a 400 response, and the following JSON:
{'error': 'invalid_request', 'error_description': 'Required parameter is missing: grant_type'}
Here's the code causing the problem:
# Returns the session, now with the Host and Authorization headers set.
def gAuthenticate(session):
token = createJWT()
session.headers.update = {
"Host": "www.googleapis.com",
"Content-Type": "application/x-www-form-urlencoded"
}
payload = urllib.parse.urlencode({
"grant_type":"urn:ietf:params:oauth:grant-type:jwt-bearer",
"assertion": str(token)
})
response = session.post("https://www.googleapis.com/oauth2/v4/token", data = payload)
session.headers.update = {"Authorization": "Bearer " + response.json()["access_token"]}
return session
I'm having a lot of strange issues with this code. First of all, if I don't urllib.parse.urlencode my dictionary (i.e. simply payload = {dictionary}), I get only a Bad Request / 'invalid_request' error, which I assume from the less specific error message means that this is less acceptable than what I'm currently doing. Why do I have to do this? Isn't Requests supposed to encode my data for me? I've never had this problem when POSTing with Requests before.
Second, examining the prepared request before it's sent reveals that my headers aren't being correctly set, despite the header update. Neither of the headers I've added to the request are being transmitted.
I've examined the request body and it looks to be identical (except of course the content of the JWT) to the one that Google provides as an example in the documentation.
All of this leads me to believe that I'm making a very basic error somewhere, but I haven't had any success finding it. What am I doing wrong here? Links to any helpful documentation would be extremely appreciated; thanks for your time and attention.
Try "grant_type": "authorization_code". And add grant type as header.
Also, check this link - Accessing Google oAuth giving invalid grant_type (Bad request)