It is bizarre activity, I get different response from cURL and Python with the requests module. Also, Python is very slow: cURL takes about 6 seconds, but Python is 2 minutes.
My cURL command is:
curl -L -s -k --request GET \
--url "https://$1/api/maintenance/maintenance/dual_image_config" \
--header 'Cache-Control: no-cache' \
--header 'Accept-Encoding: gzip, deflate, sdch' \
--header 'Content-Type: application/json' \
--header 'Connection: keep-alive'
The cURL response is:
{ "state_1": "Active", "state_2": "stand-by", "current_active_image": "Image-1", "id": 1}
My Python code is:
import requests, sys
DEFAULT_HEADER = {"Accept-Encoding": "gzip, deflate, br",
"Cache-Control": "no-cache",
"Connection": "keep-alive",
"Content-Type": "application/json",
"X-Requested-With": "XMLHttpRequest"}
url = "https://{0}/api/maintenance/maintenance/dual_image_config".format(sys.argv[1])
response = requests.get(url="", data=None, headers=DEFAULT_HEADER , verify=False)
Python response response.text is:
'0121\r\n{ "state_1": "Active", "state_2": "stand-by", "current_active_image": "Image-1", "id": 1}\r\n0\r\n\r\n'
It will have 0121\r\n & \r\n0\r\n\r\n in start and begin, I have try to change request header & encoding, but it not work, why it is got different response, and so slow ?
Related
The site that I am trying to scrape must follow 302 observed by the browser network tools. I've copied the network request as Curl and its working fine, but when I convert it to python requests its just returning 200.
Curl:
curl -v 'https://my-site.com/CardAuthentication.aspx' \
-H 'Connection: keep-alive' \
-H 'Pragma: no-cache' \
-H 'Cache-Control: no-cache' \
-H 'Upgrade-Insecure-Requests: 1' \
-H 'Origin: https://my-site.com' \
-H 'Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryBN2XwBbcAwvRWZzk' \
-H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36' \
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
-H 'Sec-GPC: 1' \
-H 'Sec-Fetch-Site: same-origin' \
-H 'Sec-Fetch-Mode: navigate' \
-H 'Sec-Fetch-User: ?1' \
-H 'Sec-Fetch-Dest: document' \
-H 'Referer: https://my-site.com/CardAuthentication.aspx' \
-H 'Accept-Language: en-US,en;q=0.9' \
-H 'Cookie: ASP.NET_SessionId=...; TS0192fa71=...; ServiceProvider=UID=...; .ASPXAUTH=...' \
--data-raw $'------WebKitFormBoundaryBN2XwBbcAwvRWZzk\r\ DATA \r\n' \
--compressed
this is returning:
* Trying IP:443...
........
........
< HTTP/1.1 302 Found
< Cache-Control: no-cache
< Pragma: no-cache
< Content-Type: text/html; charset=utf-8
< Expires: -1
< Location: my-site.com/MemberDetails.aspx
< Date: Mon, 07 Feb 2022 09:54:49 GMT
< Content-Length: 61211
< Set-Cookie: TS0192fa71=...; Path=/; Domain=.my-site.com; Secure
Python:
import requests
import logging
logging.basicConfig(level=logging.DEBUG)
cookies = {
"ASP.NET_SessionId": ".....",
"TS0192fa71": ".....",
"ServiceProvider": ".....",
".ASPXAUTH": ".....",
}
headers = {
......
"Content-Type": "multipart/form-data; boundary=----WebKitFormBoundaryBN2XwBbcAwvRWZzk",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36",
"Referer": "https://my-site.com/CardAuthentication.aspx",
"Accept-Language": "en-US,en;q=0.9",
}
data = {
"------WebKitFormBoundaryBN2XwBbcAwvRWZzk\r\nContent-Disposition: form-data; name": '"__EVENTTARGET"\r\n\r\n\r\n-- DATA
}
response = requests.post(
"https://my-site.com/CardAuthentication.aspx",
headers=headers,
cookies=cookies,
data=data,
)
And the return code is 200, with response history empy.
Is there something wrong with requests library or in the way that request library is processing the data? How can I solve this?
Is there something wrong with requests library or in the way that
request library is processing the data? How can I solve this?
This is default requests behavior, you need to set allow_redirects to False if you want to not follow in case of 3xx response code, example
import requests
r = requests.get("http://github.com", allow_redirects=False)
print(r.status_code) # 301
If you want to know more read request.request docs.
For some reasons my requests to an API is failing. This is the code:
def create_alert(alert_dict)
url = host + '/api/alerts'
headers = {
'Content-Type': 'application/json',
'Authorization': 'Bearer tokengen'
}
response = requests.request('POST', url, headers = headers, data=json.dumps(alert_dict))
return response
alert_dict = {
"priority": "1",
"to": to,
"from": from_
}
response = create_alert(alert_dict)
print(response)
The way I'm doing it is saving everything to a dictionary, then making the POST request with it. But for some reason im getting a 500 error.
This is the curl that is working
curl --location --request POST "http://newhost.com/api/alerts" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer tokengen" \
--data "{
\"to\": \"sample#email.com\",
\"from\": \"from#email.com\",
\"priority\": \"1\",
}"
How should I convert my dict in a way that it would work just like the curl --data when requested
I have the code below in Python which is making a POST request for an OAuth2 token. It uses basic authentication.
The code works fine, but I would like to "translate it" to curl.
The code:
#Authorization: Basic c29tZV91c2VyOnBhc3M=
#some_user:pass = base64decode('c29tZV91c2VyOnBhc3M=')
def get_access_token():
burp0_url = "https://myurl:443/api/oauth/token"
burp0_headers = {"Accept": "application/json", "Authorization": "Basic c29tZV91c2VyOnBhc3M=", "Content-Type": "application/x-www-form-urlencoded", "Connection": "close", "Accept-Encoding": "gzip, deflate", "User-Agent": "okhttp/3.0.1"}
burp0_data={"grant_type": "client_credentials"}
return json.loads(requests.post(burp0_url, headers=burp0_headers,
data=burp0_data).text)['access_token']
My guess was be that it would look something like this:
curl -v -XPOST -H 'Authorization: Basic c29tZV91c2VyOnBhc3M=' --header 'Accept: application/json' --header 'Connection: close' --header 'Accept-Encoding: gzip, deflate' --header 'User-Agent: okhttp/3.0.1' --data '{"grant_type": "client_credentials"}' https://myurl:443/api/oauth/token
However I keep getting a HTTP/1.1 400 and the following
* Failed writing body (0 != 10)
* Failed writing data
* stopped the pause stream!
* Closing connection 0
Can you help me?
Looks like you forgot to copy the "Content-Type": "application/x-www-form-urlencoded" header into your curl command.
This would also suggest the data is not submitted as a JSON string as you're currently doing, but as regular form data. You can probably use -F 'grant_type=client_credentials' for that and drop the --data argument.
I use requests as following:
headers = {
'Authorization': f'Bearer {self.__yandex_cloud_manager.get_iam_token().token}',
'Transfer-Encoding': 'chunked',
}
data = {
'voice': self.voice,
'emotion': 'neutral',
'lang': self.language,
'speed': self.temp,
'folderId': self.__get_folder_id(),
'format': 'oggopus',
'text': text,
}
response = requests.post(
url='https://tts.api.cloud.yandex.net/speech/v1/tts:synthesize',
headers=headers,
data=data)
It sends this data:
send: b'POST /speech/v1/tts:synthesize HTTP/1.1\r\nHost: tts.api.cloud.yandex.net\r\nUser-Agent: python-requests/2.21.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nAuthorization: Bearer <removed>\r\nTransfer-Encoding: chunked\r\nContent-Length: 135\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\n'
send: b'voice=alyss&emotion=neutral&lang=ru-RU&speed=1.0&folderId=<removed>&format=oggopus&text=%D0%9F%D1%80%D0%B8%D0%B2%D0%B5%D1%82'
And I get 400 response.
But when I dump request like this:
req = response.request
command = "curl -X {method} -H {headers} -d '{data}' '{uri}'"
method = req.method
uri = req.url
data = req.body
headers = ['"{0}: {1}"'.format(k, v) for k, v in req.headers.items()]
headers = " -H ".join(headers)
print(command.format(method=method, headers=headers, data=data, uri=uri))
It gives the following command:
curl -X POST -H "User-Agent: python-requests/2.21.0" -H "Accept-Encoding: gzip, deflate" -H "Accept: */*" -H "Connection: keep-alive" -H "Authorization: Bearer <removed>" -H "Transfer-Encoding: chunked" -H "Content-Length: 135" -H "Content-Type: application/x-www-form-urlencoded" -d 'voice=alyss&emotion=neutral&lang=ru-RU&speed=1.0&folderId=<removed>&format=oggopus&text=%D0%9F%D1%80%D0%B8%D0%B2%D0%B5%D1%82' 'https://tts.api.cloud.yandex.net/speech/v1/tts:synthesize'
And request with this command executes successfully.
What could be wroing?
Well, it works if I remove "Transfer-Encoding: chunked" header.
I'm trying to use pycurl for the first time and I'm not sure how to use parameters with it.
curl "https://mytest.com/mdb.json"
-H "Cookie: JSESSIONID=6CCB148AEE7318BD08EFC869E0FD33AB; user=testuser; wmUserPrincipal="%"7B"%"22username"%"22"%"3A"%"22testuser"%"22"%"2C"%"22roles"%"22"%"3A"%"5B"%"5D"%"7D; mf_user=322383eec941db6c72f3f2c7d58b7a80"
-H "Content-Type: application/json-rpc"
-H "Accept: */*"
-H "X-Requested-With: XMLHttpRequest"
-H "Connection: keep-alive" --data-binary "{""params"":[""1.2.1"",""instance7"",""1.2"",4,{}],""method"":""getMatrix"",""id"":250}" --compressed
As seen in the curl I have:
--data-binary "{""params"":[""1.2.1"",""instance7"",""1.2"",4,{}],""method"":""getMatrix"",""id"":250}" --compressed
and I cannot figure out what to do with them in pycurl. Maybe it is not even possible or maybe there is a simpler solution than using pycurl.
Thank you!
Got it running with requests which is a simpler and exactly what I needed.
cookie = {
'JSESSIONID': '6CCB148AEE7318BD08EFC869E0FD33AB',
'user': 'testuser',
'wmUserPrincipal': '%7B%22username%22%3A%22testuser%22%2C%22roles%22%3A%5B%5D%7D',
'mf_user': '322383eec941db6c72f3f2c7d58b7a80',
}
head = {
'Content-Type': 'application/json-rpc',
'Accept': '*/*',
'X-Requested-With': 'XMLHttpRequest',
'Connection': 'keep-alive',
}
data = '{"params":["1.2.1","instance7","1.2",4,{}],"method":"getMatrix","id":250}'
requests.post('https://mytest.com/mdb.json', headers=head, cookies=cookie, data=data)