I saw this post - Passing csrftoken with python Requests
I've been working through it trying to make it work for Greenhouse. I'm trying to build a script that will automate profile creation.
I can fetch data using GET and cookies, but I think I'm I'm getting stuck with X-CSRF. I downloaded the Live HTTP headers plugin for Mozilla to get the CSRF token, but I'm unsure how to pass it in.
So far what I have:
csrf = 'some_csrf_token'
cookie = 'some_cookie_id'
data = dict('person_first_name'='Morgan') ## this is submitting my name on the form
url = 'https://app.greenhouse.io/people/new?hiring_plan_id=24047' ##submission form page
headers = {'Cookie':cookie}
r = requests.post(url, data=data, headers=headers)
Any thoughts how I should construct my requests.post?
If you want requests to handle the cookies for you, you should set a session.
session = requests.session()
logindata = {'authenticity_token': 'whatevertokenis',
'user[email]': 'your#loginemail.com',
'user[password]': 'yourpassword',
'user[remember_me]': '0'}
login = session.post('https://app.greenhouse.io/users/sign_in', data=logindata) #this should log in in, i don't have an account there to test.
data = dict('person_first_name'='Morgan')
url = 'https://app.greenhouse.io/people/new?hiring_plan_id=24047'
r = session.post(url, data=data) #unless you need to set a user agent or referrer address you may not need the header to be added.
Related
I am trying to get some data from an xml of a website which requires to login first. I try to get python requests Session and I am able to login using get method (shows 200). After that I try to get access to the xml, but it gets me 401.
So far I know that the server checks with each call whether the client is sending a JSESSIONID cookie and whether the VALUE matches the current session.
I manage to get the corresponding cookies and tried to send it via post method but still get 401.
Maybe I think to complicated. I also do not need to achieve this with requests. I would be glad to just get the information from the xml.
import requests
login_url = 'https://www.example.com/login
USER_NAME = 'user'
PASSWORD = 'pass'
xml = 'https://www.example.com/channel_index?123&p=web'
with requests.Session() as s:
response = s.get(login_url,auth = (USER_NAME, PASSWORD))
print(response)
r = s.get(xml)
cookies = s.cookies.get_dict()
r = s.post(xml, cookies = cookies)
print(r)
I am trying to get data using the Resquest library, but I’m doing something wrong. My explanation, manual search:
URL - https://www9.sabesp.com.br/agenciavirtual/pages/template/siteexterno.iface?idFuncao=18
I fill in the “Informe o RGI” field and after clicking on the Prosseguir button (like Next):
enter image description here
I get this result:
enter image description here
Before I coding, I did the manual search and checked the Form Data:
enter image description here
And then I tried it with this code:
import requests
data = { "frmhome:rgi1": "0963489410"}
url = "https://www9.sabesp.com.br/agenciavirtual/block/send-receive-updates"
res = requests.post(url, data=data)
print(res.text)
My output is:
<session-expired/>
What am I doing wrong?
Many thanks.
When you go to the site using the browser, a session is created and stored in a cookie on your machine. When you make the POST request, the cookies are sent with the request. You receive an session-expired error because you're not sending any session data with your request.
Try this code. It requests the entry page first and stores the cookies. The cookies are then sent with the POST request.
import requests
session = requests.Session() # start session
# get entry page with cookies
response = session.get('https://www9.sabesp.com.br/agenciavirtual/pages/home/paginainicial.iface', timeout=30)
cks = session.cookies # save cookies with Session data
print(session.cookies.get_dict())
data = { "frmhome:rgi1": "0963489410"}
url = "https://www9.sabesp.com.br/agenciavirtual/block/send-receive-updates"
res = requests.post(url, data=data, cookies=cks) # send cookies with request
print(res.text)
Using Python requests, I am trying to log in a portal url which forwards to another url. I have read several posting on this subject and even tried:
1) create a session with request get and retrieve csrf token
2) use the csrf token from previous step and do a post on portal URL with payload being the username/password / and csrf token (the same info passed when I use developer tools .
3) after step 2 I still get p.text below as enter username and password and does not show the page behind login and indicates it is looking for me to login
s = requests.Session()
g = s.get("myPortalURL")
resp = g.text
for item in resp.split("\n"):
if "csrf_token" in item:
print (item)
csrfToken = item.strip().split("value=")[1].replace("\"", "").replace("/>","")
data = {'Username': self.pythonDataDict["portalUsername"], 'Password': self.pythonDataDict["portalPassword"],
'csrf_token': csrfToken}
print ("payload= ", data)
headers = {'content-type': 'application/x-www-form-urlencoded'}
p = s.post("myPortalURL", headers=headers, data=data)
soup = BeautifulSoup(p.text)
print (p.text)
Here are some possible issues with your code:
From personal experience; the URL to the login page should be different to the URL for the main page.
There is information missing in data, ALL of the required data needs to be entered in the form,
You may need to include additional request headers
Your code to get the csrf_token could be prone to returning incorrect data depending on the nature of that token (although all tokens vary so it may not)
Also as a side note, it is generally more idiomatic to use a with statement when using a session, e.g.
with requests.Session() as s:
initial_response = s.get("login_url")
# def createForm()...
main_response = s.get("target_url", data = data, headers = headers)
# rest of code...
It would also be extremely helpful if you could include the URL of the page you are trying to access if that is possible.
I tried to connect HP QC using python to create defects and attach files, but I am not able to connect with HP QC. Here is my code:
domain='DEFAULT_773497139'
project='773497139_DEMO'
import requests
url = "https://almalm1250saastrial.saas.hpe.com/qcbin/"
querystring = {"username":"user#gmail.com","password":"password"}
headers = {
'cache-control': "no-cache",
'token': "5d33d0b7-1d04-4989-3349-3005b847ab7f"
}
response = requests.request("POST", url, headers=headers, params=querystring)
#~ print(response.text)
print response.headers
new_header = response.headers
new_url = url+ u'rest/domains/'+domain+u'/projects/'+project
new_querystring = {
"username":"user#gmail.com",
"password":"password",
"domain":'DEFAULT_773497139',
"project":'773497139_DEMO'
}
print new_url
response = requests.request("POST", new_url, headers=new_header, params=new_querystring)
print(response.text)
Now login works fine, but when try other API it asks for, I would get this message:
Authentication failed. Browser based integrations - to login append '?login-form-required=y' to the url you tried to access
If the parameter has been added, then it goes back to login page.
Seems that your urls are not well builded:
base_url ='https://server.saas.hpe.com/qcbin/'
base_url + '/qcbin/rest/domains/
you will get:
..../qcbin/qcbin/...
qcbin twice
The way I do it is to based on python request Sessions. First I create a session, then post my credentials to ../authentication-point/alm-authenticate/ (or sth like this, you should check it) and then using this session I can get, post or do whatever I want.
So:
s = requests.Session()
s.post(`../authentication-point/alm-authenticate/`, data=credentials)
# now session object is authenticated and recognized
# you can s.post, s.get or whatever
I think it's a good url, but I can't check it right now :)
Session issue has beensolved by LWSSO cookie (LWSSO_COOKIE_KEY).
Just send a unicode string to your server and use the header for the basic Authorization as specified by the HP REST API:
login_url = u'https://almalm1250saastrial.saas.hpe.com/qcbin/authentication-point/authenticate'
username,password = user,passwd
logs = base64.b64encode("{0}:{1}".format(username, password))
header['Authorization'] = "Basic {}".format(logs)
POST by using the requests module in python is quite easy:
requests.post(login_url, headers=header)
That's it...now you are authenticated and you can proceed with next action :-) To check on that you can "GET" on:
login_auth = u'https://almalm1250saastrial.saas.hpe.com/qcbin/rest/is-authenticated
you should get a code 200 --> That means you are authenticated.
Hope this help you. Have a nice day and let me know if something is still not clear.
p.s.: to send REST msg in python I am using requests module. It is really easy! You can create a session if you want to send multiple actions--> then work with that sessions--> ALM = requests.session(), then use ALM.post(whatever) and so on :-)
I've tried solving the problem using Mechanize, but I couldn't get it to work.
A website only allows access to the data if cookies are sent after login. I need to do the following:
Log in using POST to a page
Store cookies
Access protected page
You can use a Session in Requests. From the documentation:
The Session object allows you to persist certain parameters across
requests. It also persists cookies across all requests made from the
Session instance.
Here's how a log in and subsequent request might look:
import requests
s = requests.Session(verify='my_cert_file.crt')
r = s.post('https://secure-site.com/login', data={
'username': my_username,
'password': my_password,
})
# logging in sets a cookie which the session remembers
print s.cookies
r = s.get('https://secure-site.com/secure-data')
print r.json()
I came up with the following solution using Mechanize. Cookies are managed by mechanize.Browser.
br = mechanize.Browser()
resp = br.open('https://xxxxxxxxxxxxxxxxxxx')
br.select_form(nr=0)
br['username'] = username
br['password'] = password
response = br.submit()
time.sleep(1)
resp_second = br.open('https://secretwebpage')
print resp_second.read()