I have a JOOMLA 3 system with Community Builder plugin. I need to update Users from a CSV file.
Here is the code I have tried:
if __name__ == "__main__":
s = fn_login()
r = s.get(cb_users)
print(r)
with open(IMPORT_FILE, 'rb') as f:
files = {'files': f}
r = s.post(post_url, files=files)
The session is ok. I have tried to figure out the POST message, but something may be missing. Nothing happens when I send the post. I have got response 200. What else do I need to consider?
Here is the request header:
curl 'https://3joomla.domain/administrator/index.php'
-X 'POST'
-H 'authority: 3joomla.domain'
-H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9'
-H 'accept-language: en-US,en;q=0.9,hu;q=0.8'
-H 'cache-control: max-age=0'
-H 'content-length: 6710'
-H 'content-type: multipart/form-data; boundary=----WebKitFormBoundaryALBLy00y6zolw7hC'
-H 'cookie: ....
-H 'origin: https://3joomla.domain'
-H 'referer: https://3joomla.domain/administrator/index.php'
-H 'sec-ch-ua: "Not?A_Brand";v="8", "Chromium";v="108", "Google Chrome";v="108"'
-H 'sec-ch-ua-mobile: ?0'
-H 'sec-ch-ua-platform: "Linux"'
-H 'sec-fetch-dest: document'
-H 'sec-fetch-mode: navigate'
-H 'sec-fetch-site: same-origin'
-H 'sec-fetch-user: ?1'
-H 'upgrade-insecure-requests: 1'
-H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36'
--compressed
Related
The site that I am trying to scrape must follow 302 observed by the browser network tools. I've copied the network request as Curl and its working fine, but when I convert it to python requests its just returning 200.
Curl:
curl -v 'https://my-site.com/CardAuthentication.aspx' \
-H 'Connection: keep-alive' \
-H 'Pragma: no-cache' \
-H 'Cache-Control: no-cache' \
-H 'Upgrade-Insecure-Requests: 1' \
-H 'Origin: https://my-site.com' \
-H 'Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryBN2XwBbcAwvRWZzk' \
-H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36' \
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
-H 'Sec-GPC: 1' \
-H 'Sec-Fetch-Site: same-origin' \
-H 'Sec-Fetch-Mode: navigate' \
-H 'Sec-Fetch-User: ?1' \
-H 'Sec-Fetch-Dest: document' \
-H 'Referer: https://my-site.com/CardAuthentication.aspx' \
-H 'Accept-Language: en-US,en;q=0.9' \
-H 'Cookie: ASP.NET_SessionId=...; TS0192fa71=...; ServiceProvider=UID=...; .ASPXAUTH=...' \
--data-raw $'------WebKitFormBoundaryBN2XwBbcAwvRWZzk\r\ DATA \r\n' \
--compressed
this is returning:
* Trying IP:443...
........
........
< HTTP/1.1 302 Found
< Cache-Control: no-cache
< Pragma: no-cache
< Content-Type: text/html; charset=utf-8
< Expires: -1
< Location: my-site.com/MemberDetails.aspx
< Date: Mon, 07 Feb 2022 09:54:49 GMT
< Content-Length: 61211
< Set-Cookie: TS0192fa71=...; Path=/; Domain=.my-site.com; Secure
Python:
import requests
import logging
logging.basicConfig(level=logging.DEBUG)
cookies = {
"ASP.NET_SessionId": ".....",
"TS0192fa71": ".....",
"ServiceProvider": ".....",
".ASPXAUTH": ".....",
}
headers = {
......
"Content-Type": "multipart/form-data; boundary=----WebKitFormBoundaryBN2XwBbcAwvRWZzk",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36",
"Referer": "https://my-site.com/CardAuthentication.aspx",
"Accept-Language": "en-US,en;q=0.9",
}
data = {
"------WebKitFormBoundaryBN2XwBbcAwvRWZzk\r\nContent-Disposition: form-data; name": '"__EVENTTARGET"\r\n\r\n\r\n-- DATA
}
response = requests.post(
"https://my-site.com/CardAuthentication.aspx",
headers=headers,
cookies=cookies,
data=data,
)
And the return code is 200, with response history empy.
Is there something wrong with requests library or in the way that request library is processing the data? How can I solve this?
Is there something wrong with requests library or in the way that
request library is processing the data? How can I solve this?
This is default requests behavior, you need to set allow_redirects to False if you want to not follow in case of 3xx response code, example
import requests
r = requests.get("http://github.com", allow_redirects=False)
print(r.status_code) # 301
If you want to know more read request.request docs.
This question already has answers here:
How to upload file with python requests?
(9 answers)
Closed 1 year ago.
I upload picture to imgbb.com site and show the curl command using firefox
curl "https://imgbb.com/json" -H "User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0" -H "Accept: application/json" -H "Accept-Language: en-US,en;q=0.5" --compressed -H "Content-Type: multipart/form-data; boundary=---------------------------270624834321652808912303071412" -H "Origin: https://imgbb.com" -H "Connection: keep-alive" -H "Referer: https://imgbb.com/" -H "Cookie: PHPSESSID=6o18hipgu8uk8orq4pg47of4b5; __gads=ID=40783678319bf866-227f347cf1a6003f:T=1614875864:RT=1614875864:S=ALNI_MbBjogRusDweg41MHFP01KawfJuiw" -H "DNT: 1" -H "Sec-GPC: 1" -H "TE: Trailers" --data-binary "-----------------------------270624834321652808912303071412"^
"Content-Disposition: form-data; name=""source""; filename=""test.png"""^
"Content-Type: image/png"^
""^
"-----------------------------270624834321652808912303071412"^
"Content-Disposition: form-data; name=""type"""^
""^
"file"^
"-----------------------------270624834321652808912303071412"^
"Content-Disposition: form-data; name=""action"""^
""^
"upload"^
"-----------------------------270624834321652808912303071412"^
"Content-Disposition: form-data; name=""timestamp"""^
""^
"1614875900988"^
"-----------------------------270624834321652808912303071412"^
"Content-Disposition: form-data; name=""auth_token"""^
""^
"e47e40461c19ef13a3768c45e2a0a5189f3bf625"^
"-----------------------------270624834321652808912303071412--"^
""
I can see test.png picture that I upload .
How can I send that command again using python with different picture?
You could use the online curl to Python requests generator.
But you should really be using the imgBB API instead.
I am getting a different response from python and curl, although each uses the exact same parameters.
Python:
import requests
headers = {
'Accept-Language': 'en-US,en',
'Accept': 'text/html,application/xhtml+xml,application/xml',
'Authority': 'www.google.com',
'User-Agent': 'SomeAgent',
'Upgrade-Insecure-Requests': '1',
}
response = requests.get('https://www.avvo.com', headers=headers)
# Returns a 403 response
Curl:
import shlex, subprocess
cmd = '''curl -H 'Accept-Language: en-US,en' -H 'Accept: text/html,application/xhtml+xml,application/xml' -H 'Authority: www.google.com' -H 'User-Agent: SomeAgent' -H 'Upgrade-Insecure-Requests: 1' https://www.avvo.com'''
args = shlex.split(cmd)
process = subprocess.Popen(args, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = process.communicate()
# Returns a 200 response
Both requests are being sent from the same IP. It looks like it's a cloudflare issue, is there any way cloudflare can detect a request coming from the python requests library vs a direct curl command?
I left the website in the code in case its useful to run. Here is the curl command directly:
curl -H 'Accept-Language: en-US,en' -H 'Accept: text/html,application/xhtml+xml,application/xml' -H 'Authority: www.google.com' -H 'User-Agent: SomeAgent' -H 'Upgrade-Insecure-Requests: 1' https://www.avvo.com/administrative-law-lawyer/ny.html
I have a csv file that I want to send via curl to a Django Rest Framework API View I have developed. The csv file itself contains only foo,bar and this is the curl command I'm using:
curl URL -H 'Accept:
application/json' -H 'Referer: http://localhost:5011/ost:5011' -H
'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94
Safari/537.36' -H 'Authorization: JWT TOKEN -H 'Content-Type:
multipart/form-data' -F upload=#testcsv.csv
When it hits my API View in Django, and I run
request.data.get('file').read()
the output is some metadata around the file and not the file contents itself:
(Pdb) b'--------------------------001ec735bfc1a929\r\nContent-
Disposition: form-data; name="upload";
filename="testcsv.csv"\r\nContent-Type: application/octet-
stream\r\n\r\n\r\n--------------------------001ec735bfc1a929--\r\n'
How can I access the actual file itself through this method? My APIview is using
class FileUploadView(APIView):
parser_classes = (MultiPartParser,)
Thanks!
f = request.FILES["filefield_name"]
Here is the curl command:
curl 'http://ecard.bupt.edu.cn/User/ConsumeInfo.aspx'
-H 'Cookie: ASP.NET_SessionId=4mzmi0whscqg4humcs5fx0cf; .NECEID=1; .NEWCAPEC1=$newcapec$:zh-CN_CAMPUS; .ASPXAUTSSM=A91571BB81F620690376AF422A09EEF8A7C4F6C09978F36851B8DEDFA56C19C96A67EBD8BBD29C385C410CBC63D38135EFAE61BF49916E0A6B9AB582C9CB2EEB72BD9DE20D64A51C09474E676B188D937B84E601A981C02F51AA9C85A08EABCC1D30D7B9299E02A45D427B08A44AEBB0'
-H 'Origin: http://ecard.bupt.edu.cn'
-H 'Accept-Encoding: gzip, deflate'
-H 'Accept-Language: zh-CN,zh;q=0.8,en;q=0.6,zh-TW;q=0.4'
-H 'Upgrade-Insecure-Requests: 1'
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36'
-H 'Content-Type: application/x-www-form-urlencoded'
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'
-H 'Cache-Control: max-age=0' -H 'Referer: http://ecard.bupt.edu.cn/User/ConsumeInfo.aspx'
-H 'Connection: keep-alive'
--data '__EVENTTARGET=&__EVENTARGUMENT=&__VIEWSTATE=%2FwEPDwUINDEyMzA5NDkPFgIeCFNvcnRUeXBlBQNBU0MWAmYPZBYCAgMPZBYCAgMPZBYCAgQPPCsAEQMADxYEHgtfIURhdGFCb3VuZGceC18hSXRlbUNvdW50ZmQBEBYAFgAWAAwUKwAAFgJmD2QWAmYPZBYCAgEPZBYCAgEPDxYEHghDc3NDbGFzcwUKU29ydEJ0X0FzYx4EXyFTQgICZGQYAQUiY3RsMDAkQ29udGVudFBsYWNlSG9sZGVyMSRncmlkVmlldw88KwAMAQhmZBb11RlvPBlQn5fbuqe6uh8BRZJ2jUZ5U17xEnRM%2BnbF&__VIEWSTATEGENERATOR=D99777FC&__EVENTVALIDATION=%2FwEdAAgtbl4l0SdCQqYpyy3Ex39BEoHXgPeD%2Fa3eEKPr2bG0rY8WyuyQajjUbeopYM%2Fne68pyn2l0BPEWPPnyd6MfoDZVQGbFRIKjb%2FOTPkkCDIIaJ3X6W3VKO%2FV036Pdf6jf06OTFulzGhY80%2FZ1HbCJJ6LR5Lg6mLp72ibUCB3VipRJuK11qmW%2BSDe3IOvbK4oNUdV5%2FxmkXw25tTwfJ8P8OS2&ctl00%24ContentPlaceHolder1%24rbtnType=0&ctl00%24ContentPlaceHolder1%24txtStartDate=2016-09-17&ctl00%24ContentPlaceHolder1%24txtEndDate=2016-09-24&ctl00%24ContentPlaceHolder1%24btnSearch=%E6%9F%A5++%E8%AF%A2' --compressed
I want to use request to get the return message to analyse, but i'm confused about this.
Thanks!
You can use python requests module. The answer would be something like:
response = requests.get('http://ecard.bupt.edu.cn/User/ConsumeInfo.aspx',
headers={'Origin': 'http://ecard.bupt.edu.cn'},
data={'__EVENTTARGET': '', 'rbtnType':0},
cookies={'.NECEID': 1}
)
I didn't put all parameters there but I think this example is enough to figure out how to send exact the same request you want. The only thing I am not sure is --compressed flag - as I understand from docs it is something more specific to curl(correct me if I'm wrong)
Here is a useful website to convert the curl to requests:
curl to request
Curl Command:
curl 'http://js.t.sinajs.cn/open/api/js/api/bundle.js?version=20150130.02' -H 'Pragma: no-cache' -H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: zh-CN,zh;q=0.8,en;q=0.6,zh-TW;q=0.4' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.87 Safari/537.36' -H 'Accept: */*' -H 'Referer: http://www.liaoxuefeng.com/wiki/001374738125095c955c1e6d8bb493182103fac9270762a000/001375502200090e998439175cc4268b0ea703b3b4ed55e000' -H 'Connection: keep-alive' -H 'Cache-Control: no-cache' --compressed
Python requests:
import requests
headers = {
'Pragma': 'no-cache',
'Accept-Encoding': 'gzip, deflate, sdch',
'Accept-Language': 'zh-CN,zh;q=0.8,en;q=0.6,zh-TW;q=0.4',
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.87 Safari/537.36',
'Accept': '*/*',
'Referer': 'http://www.liaoxuefeng.com/wiki/001374738125095c955c1e6d8bb493182103fac9270762a000/001375502200090e998439175cc4268b0ea703b3b4ed55e000',
'Connection': 'keep-alive',
'Cache-Control': 'no-cache',
}
requests.get('http://js.t.sinajs.cn/open/api/js/api/bundle.js?version=20150130.02', headers=headers)