Develop Reference

How do I copy cookies to get an appropriate info during the parsing?

I have a site with multiple cities. The task is to parse the price from the certain city. How do I get cookies from browser and how to pass it into requests.get(url=url, headers=headers, cookies=cookies?

Usually, cookies are sent to the server via the cookie header, which is part of the HTTP request.
Here is an example of some python code that sends a cookie to the server.
import requests
cookies = {'key': 'value'}
r = requests.get("https://example.com", cookies=cookies)

How to get information from xml after login with requests python

I am trying to get some data from an xml of a website which requires to login first. I try to get python requests Session and I am able to login using get method (shows 200). After that I try to get access to the xml, but it gets me 401.
So far I know that the server checks with each call whether the client is sending a JSESSIONID cookie and whether the VALUE matches the current session.
I manage to get the corresponding cookies and tried to send it via post method but still get 401.
Maybe I think to complicated. I also do not need to achieve this with requests. I would be glad to just get the information from the xml.
import requests
login_url = 'https://www.example.com/login
USER_NAME = 'user'
PASSWORD = 'pass'
xml = 'https://www.example.com/channel_index?123&p=web'
with requests.Session() as s:
response = s.get(login_url,auth = (USER_NAME, PASSWORD))
print(response)
r = s.get(xml)
cookies = s.cookies.get_dict()
r = s.post(xml, cookies = cookies)
print(r)

Get cookies from selenium to requests

I can login to a website with selenium and i can receive all cookies.
But then I have to quickly submit a request to the site. Meanwhile, selenium stays very slow.
That's why I want to receive cookies with selenium and send requests via the request module.
My Selenium Code (First I log in to the website and received all cookies with the code below.)
browser.get('https://www.example.com/login')
cookiem1 = browser.get_cookies()
print(cookiem1)
2nd stage, I will go to another page of the website and make a request.
s = requests.Session()
for cookie in cookiem1:
s.cookies.set(cookie['name'], cookie['value'])
r = s.get("https://example.com/postcomment')
print(r.content)
I use cookies in this way, but when I send the url via request module, the site does not autohorize my user.
My error:
"errorMessage": "Unauthorized user",\r\n "errorDetails": "No cookie"
Probably with this code the site doesn't unauthorized my session
Thanks in advance

try this
import requests as re
ck = browser.get_cookies()
s = re.Session()
c = [s.cookies.set(c['name'], c['value']) for c in ck]
response = s.get("https://example.com/postcomment")

Web Scraping using Requests - Python

I am trying to get data using the Resquest library, but I’m doing something wrong. My explanation, manual search:
URL - https://www9.sabesp.com.br/agenciavirtual/pages/template/siteexterno.iface?idFuncao=18
I fill in the “Informe o RGI” field and after clicking on the Prosseguir button (like Next):
enter image description here
I get this result:
enter image description here
Before I coding, I did the manual search and checked the Form Data:
enter image description here
And then I tried it with this code:
import requests
data = { "frmhome:rgi1": "0963489410"}
url = "https://www9.sabesp.com.br/agenciavirtual/block/send-receive-updates"
res = requests.post(url, data=data)
print(res.text)
My output is:
<session-expired/>
What am I doing wrong?
Many thanks.

When you go to the site using the browser, a session is created and stored in a cookie on your machine. When you make the POST request, the cookies are sent with the request. You receive an session-expired error because you're not sending any session data with your request.
Try this code. It requests the entry page first and stores the cookies. The cookies are then sent with the POST request.
import requests
session = requests.Session() # start session
# get entry page with cookies
response = session.get('https://www9.sabesp.com.br/agenciavirtual/pages/home/paginainicial.iface', timeout=30)
cks = session.cookies # save cookies with Session data
print(session.cookies.get_dict())
data = { "frmhome:rgi1": "0963489410"}
url = "https://www9.sabesp.com.br/agenciavirtual/block/send-receive-updates"
res = requests.post(url, data=data, cookies=cks) # send cookies with request
print(res.text)

How do I get the cookies of the active session to give it to the post request with python?

I have a question about cookies from a website.
If I do a request manually on the Website, it hands over the header with cookies.
cookie is for example this:
cookie: laravel_session=eyJpdiI6IjBCMnNTdFNmV3lCawererwadsf0SXc9PSIsInZhbHVlIjoiTWtZUDduSkFXdzRobERtQ0YxdkxreEkzYzJtWFwvUEVyMHgyT2ljN2EyeXg3XasdfweUUwQzhGRWowVEp5VygsdgfewTQ2MGJmNjEzZmYyNmU4YjdhYTQyMzkyNTYzYjFjMTQ4MDQ5NjlkNDQ0ZGQ4NGRiZDE5YjNhYTRhMzA5ODMifQ%3D%3D
but how can I get that cookie with my request? I want to send a post request, with the data in it, but I dont know how to send this cookie with the request Header or how to generate it?I dont want to put it in manually it should get it from the website. or does it automatically send this cookie with the header?
Thank you!
Elias
I dont get the thing with the laravel_session cookie- where can i get it from?

This should help you with your question because this how send cookies with a python request. Even if this is from wikipedia it is an example to send cookies with a request.
import requests
cookies = {'enwiki_session': '17ab96bd8ffbe8ca58a78657a918558'}
r = requests.post('http://wikipedia.org', cookies=cookies)

If I understand your question correctly, you want to get a cookie from the current session, then send it along with a request. This should do that.
import requests
req_url = 'http://foo.bar.com'
req_data = {'some': 'data'}
session = requests.Session()
response = requests.get(req_url)
cookie_to_send = session.cookies.getdict()['cookie_name']
requests.post(url=req_url, data=req_data, cookie={'cookie_name': cookie_to_send})