I'm still bit new to Python and trying to understand. If you could help me out it would be appreciated!
My issue is that I want to grab my authorization token from twitter.com that is stored in my local storage. Is there a way in Python to obtain the auth_token from localstorage and save it to a text file?
(I know how to write things to a text file but I'm having issues grabbing the auth token. I'm using playwright async and already tried it through the cookies and pasting them in a JSON file, but this results in that the "auth_token" is sometimes placed in the JSON file as:
['cookies'][9]['value']
or:
['cookies'][7]['value'] ['cookies'][8]['value']
Is there a way that I can find this in an easier way? The format looks like:
{
"cookies": [{
"name": "auth_token",
"value": "22b23d52e7c639f123456ed451dfe9ebd9d439d3",
"domain": ".twitter.com",
"path": "/",
"expires": 1816242826,
"httpOnly": true,
"secure": true,
"sameSite": "None"
}, {
"name": "ct0",
"value": "1547663d5b6a5b5c857b726964d9e10c7eb4654c1b210c345d008d28d526f43e7c5a8f4dcfaaead4281bac844cfee5a642fa5a7e7e9824405817de778bbd970f712f5de0cf01bf352de94989da6eb349",
"domain": ".twitter.com",
"path": "/",
"expires": 1816242827,
"httpOnly": false,
"secure": true,
"sameSite": "Lax"
}, {
"name": "twid",
"value": "u%3D1550750937400360960",
"domain": ".twitter.com",
"path": "/",
"expires": 1690098828,
"httpOnly": false,
"secure": true,
"sameSite": "None"
}, {
"name": "_s",
"value": "CgdiXZmky9MlkvLhFqnr4TxEU0eoZnJT4Eir8QAH%2FZ4SZENccyKmnFwtUXTz9BKd",
"domain": ".app.link",
"path": "/",
"expires": 1690098784,
"httpOnly": false,
"secure": true,
"sameSite": "None"
}]
}
Issue is that the "auth_token" is stored in [7/8/9] which is different every time. Is there a way to do it like
['cookies']['auth_token']['value']
right now I have:
with open('t.json') as auth_obtainer:
authfile =json.load(auth_obtainer)
auth_token = json.dumps(authfile['cookies'][9]['value']).replace('"',"")
print(auth_token)
but sometimes it's located differently in the JSON file so it gives me the wrong value
Use a for loop to iterate over dictionary contained in cookies and check if it contains name which has value equal to "auth_token" or not, if it does then access the "value" key of that dictionary.
with open('t.json') as f:
content = json.load(f)
cookies = content['cookies']
for data in cookies:
if data['name'] == 'auth_token':
auth_token = data['value']
print(auth_token)
Related
I have cookie.txt with content like below
[
{
"domain": "example.com",
"expirationDate": 1683810439,
"hostOnly": false,
"httpOnly": false,
"name": "__adroll_fpc",
"path": "/",
"sameSite": "lax",
"secure": false,
"session": false,
"storeId": null,
"value": "2123213-1651041941056"
},
{
"domain": "example.com",
"expirationDate": 1715324838,
"hostOnly": false,
"httpOnly": false,
"name": "_ga",
"path": "/",
"sameSite": null,
"secure": false,
"session": false,
"storeId": null,
"value": "12332.1651041940"
}
]
I'm trying to access each object of that txt like below
def initCookies(self):
with open('cookie.txt', encoding='utf8') as f:
cookies = f.readlines()
mystring = ' '.join([str(item) for item in cookies])
data = json.loads(mystring)
print(type(data))
for cookie in data:
print(cookie)
but it seems print(cookie) has the whole content.
how to access each object within {} ?
I should be able to access them like this cookie.get('name', ''), cookie.get('value', '')
You can simplify your code, even though it already works...
import json
with open('cookie.txt', encoding='utf8') as f:
cookies = json.load(f)
for cookie in cookies:
print(cookie.get("name")) # '__adroll_fpc', '_ga'
why you convert the list to a big string first ? You could just do it directly with json.load instead of json.loads
with open('cookie.txt', encoding='utf8') as f:
data = json.load(f)
data # list of 2 dictionaries
for dic in data:
print(dic.get('name'))
Output:
__adroll_fpc
_ga
Iam gathering cookies from a selenium browser and then writing them to a json file. I want to then read them and only extract the name and value keys and values. However, I dont know how to convert them back into a dictionary without changing the cookies in any way, which I see as a common answer on here.
Here are the cookies below
[{"domain": ".instagram.com", "httpOnly": true, "name": "rur", "path": "/", "secure": true, "value": "PRN"}, {"domain": ".instagram.com", "expiry": 1610372060, "httpOnly": true, "name": "shbid", "path": "/", "secure": true, "value": "2630"}, {"domain": ".instagram.com", "expiry": 1610372060, "httpOnly": true, "name": "shbts", "path": "/", "secure": true, "value": "1609767261.111102"}, {"domain": ".instagram.com", "expiry": 1641303259, "httpOnly": true, "name": "sessionid", "path": "/", "secure": true, "value": "5973912167%3A79EfmCoMj2hdDd%3A29"}, {"domain": ".instagram.com", "expiry": 1641216860, "httpOnly": false, "name": "csrftoken", "path": "/", "secure": true, "value": "z1i7aU8fUnFkO7jXS8eOcVzmzQVfCFTP"}, {"domain": ".instagram.com", "expiry": 1672839253, "httpOnly": false, "name": "mid", "path": "/", "secure": true, "value": "X_MZVgALAAFI1JXJOTF_ZG-E-Cny"}, {"domain": ".instagram.com", "expiry": 1617543260, "httpOnly": false, "name": "ds_user_id", "path": "/", "secure": true, "value": "5973912167"}, {"domain": ".instagram.com", "expiry": 1672839253, "httpOnly": true, "name": "ig_did", "path": "/", "secure": true, "value": "AD703D6F-E24A-4485-A1EC-9736E40C19C3"}, {"domain": ".instagram.com", "httpOnly": true, "name": "urlgen", "path": "/", "secure": true, "value": "\"{\\\"86.15.149.131\\\": 5089}:1kwQ05:r7Bsx0VcOwuIw_rejyFuGdmgAIo\""}, {"domain": "www.instagram.com", "expiry": 4765462453, "httpOnly": false, "name": "ig_cb", "path": "/", "secure": false, "value": "2"}]
I don't really know if this is exactly what you want but I extracted all the domain and value keys and value and put them back in a dictionary and put all the dictionary in a list
var = *YOUR COOKIES LIST*
liste = []
for i in range(len(var)) :
#print(var[i]["domain"])
#print(var[i]["value"])
dic = {"domain":var[i]["domain"],"value":var[i]["value"]}
liste.append(dic)
print(liste)
maybe this would have since you saved it as a file:
import json
with open('data.json', 'r') as f:
datas = json.load(f)
for data in datas:
print(data["name"], " : ", data["value"])
resulting in
rur : PRN
shbid : 2630
shbts : 1609767261.111102
sessionid : 5973912167%3A79EfmCoMj2hdDd%3A29
csrftoken : z1i7aU8fUnFkO7jXS8eOcVzmzQVfCFTP
mid : X_MZVgALAAFI1JXJOTF_ZG-E-Cny
ds_user_id : 5973912167
ig_did : AD703D6F-E24A-4485-A1EC-9736E40C19C3
urlgen : "{\"86.15.149.131\": 5089}:1kwQ05:r7Bsx0VcOwuIw_rejyFuGdmgAIo"
ig_cb : 2
const puppeteer = require('puppeteer-extra')
// add stealth plugin and use defaults (all evasion techniques)
const StealthPlugin = require('puppeteer-extra-plugin-stealth')
puppeteer.use(StealthPlugin())
puppeteer.launch({ headless: true,ignoreHTTPSErrors:true, args: [ '--no-sandbox', '--proxy-server=51.158.111.242:8811' ] }).then(async browser => {
console.log('Running tests..')
const page = await browser.newPage()
await page.goto('https://google.com/')
const cookies = await page.cookies();
console.log(cookies);
await fs.writeFile('./cookies.json', JSON.stringify(cookies, null, 2));
await page.screenshot({ path: 'testresult.png', fullPage: true })
await browser.close()
})
after saving the cookies into the file, it saves multiple cookies dictionary in file
[
{
"name": "_gat",
"value": "1",
"domain": "www.google.com",
"path": "/",
"expires": 1582024891,
"size": 5,
"httpOnly": false,
"secure": false,
"session": false
},
{
"name": "_gid",
"value": "GA1.2.1936512649.1582024831",
"domain": "www.google.com",
"path": "/",
"expires": 1582111231,
"size": 31,
"httpOnly": false,
"secure": false,
"session": false
},
{
"name": "_ga",
"value": "GA1.2.1830413277.1582024831",
"domain": "www.google.com",
"path": "/",
"expires": 1645096831,
"size": 30,
"httpOnly": false,
"secure": false,
"session": false
},
{
"name": "__cfduid",
"value": "d9ccc472957afbae818db60dff47cc5e01582024830",
"domain": "www.google.com",
"path": "/",
"expires": 1584616830.613503,
"size": 51,
"httpOnly": true,
"secure": true,
"session": false,
"sameSite": "Lax"
}
]
I am not able to decide which one should use for making a request in the python-requests module?
I have some cookies in python stored like this:
cookie = [
{"""
"domain": ".justdial.com",
"expirationDate": 1577653041.993055,
"hostOnly": false,
"httpOnly": true,
"name": "_ctk",
"path": "/",
"sameSite": "no_restriction",
"secure": false,
"session": false,
"storeId": "0",
"value": "893b0b69e25c0359d6e1fd88f16fea90a4bd2e0e8f8356e80bfc572e7f7e1343",
"id": 1"""
},
{"""
"domain": ".justdial.com",
"expirationDate": 1546136368,
"hostOnly": false,
"httpOnly": false,
"name": "_fbp",
"path": "/",
"sameSite": "no_restriction",
"secure": false,
"session": false,
"storeId": "0",
"value": "fb.1.1546114608524.1389346931",
"id": 2"""
}
]
requests.post(URL, cookies=cookie)
I am trying to send these cookies using Requests, but that does not work. Is the format wrong, or the way that I am sending it?
Thanks for the help! Using RequestsCookieJar worked, but I found another way: I saved it to a separate file, then, using the json library I got it in the right format and was able to send the cookies.
In your code, cookie is a list. You need to send a dict, or you can use the requests.cookies.RequestsCookieJar() object:
From the docs:
>>> jar = requests.cookies.RequestsCookieJar()
>>> jar.set('tasty_cookie', 'yum', domain='httpbin.org', path='/cookies')
>>> jar.set('gross_cookie', 'blech', domain='httpbin.org', path='/elsewhere')
>>> url = 'https://httpbin.org/cookies'
>>> r = requests.get(url, cookies=jar)
>>> r.text
'{"cookies": {"tasty_cookie": "yum"}}'
how can get those two values utc_last_updated and name given the following json ?
I used requests, to get to fetch the content, and then I used BeautifulSoup to make it like it is now. But now I just want to extract the two values that I have shown.
"data": [
{
"scm": "hg",
"has_wiki": false,
"last_updated": "2016-03-23T14:05:27.433",
"no_forks": false,
"created_on": "2016-03-18T22:55:52.705",
"owner": "user",
"email_mailinglist": "",
"is_mq": false,
"size": 420034,
"read_only": false,
"fork_of": null,
"mq_of": null,
"state": "available",
"utc_created_on": "2016-03-18 21:55:52+00:00",
"website": "",
"description": "",
"has_issues": false,
"is_fork": false,
"slug": "store",
"is_private": true,
"name": "store",
"language": "python",
"utc_last_updated": "2016-03-23 13:05:27+00:00",
"no_public_forks": true,
"creator": null,
"resource_uri": "/1.0/repositories/my_url"
},
{
"scm": "hg",
"has_wiki": false,
"last_updated": "2016-03-18T12:26:22.261",
"no_forks": false,
"created_on": "2016-03-18T12:19:08.262",
"owner": "user",
"email_mailinglist": "",
"is_mq": false,
"size": 173137,
"read_only": false,
"fork_of": null,
"mq_of": null,
"state": "available",
"utc_created_on": "2016-03-18 11:19:08+00:00",
"website": "",
"description": "",
"has_issues": false,
"is_fork": false,
"name": 'foo'
"is_private": true,,
"language": "python",
"utc_last_updated": "2016-03-18 11:26:22+00:00",
"no_public_forks": true,
"creator": null,
"resource_uri": "/1.0/repositories/my_rl"
},
}
I will appreciate any help.
You've got a JSON response, not HTML - parse it with json module:
import json
data = json.loads(response)
for item in data["data"]:
print(item["utc_last_updated"])