Python: Update value in dictionary while looping

Python: Update value in dictionary while looping - python

I would like to loop through a list and would like that with each loop the value in the dictionary update according to the value in the loop. Something like
The value I want to update is "categoryId":"2030110" which is in body.
Categoryid = [2040831,2029816,2029818,2029814,2084696]
body = {"operationName":"GetCategoryForFilterPage","variables":{"categoryId":"2030110","facets":[],"page":1,"pageSize":1000,"searchPhrase":"","sortBy":"RELEVANCE"},"query":"query GetCategoryForFilterPage($categoryId: String!, $searchPhrase: String, $sortBy: SortBy!, $page: Int!, $pageSize: Int!, $facets: [FacetParameterInput!]) {\n category(categoryId: $categoryId) {\n id\n seoTitle\n name\n namePath\n pathSlug\n seoText\n navigation {\n id\n path {\n ...NavigationItem\n __typename\n }\n siblings {\n ...NavigationItem\n __typename\n }\n childrenSortedByHits {\n ...ChildNavigationItem\n __typename\n }\n __typename\n }\n productSearch(searchPhrase: $searchPhrase, sortBy: $sortBy, page: $page, pageSize: $pageSize, facets: $facets) {\n facets {\n name\n type\n displayName\n facetOpened\n nameInUrl\n isTranslated\n unitText\n values {\n displayName\n name\n hits\n min\n max\n __typename\n }\n __typename\n }\n productHits\n productReferences {\n ...ProductReference\n __typename\n }\n ads {\n ...Ads\n __typename\n }\n __typename\n }\n __typename\n }\n}\n\nfragment NavigationItem on CategoryNavigationItem {\n categoryId\n hits\n selected\n category {\n name\n pathSlug\n categoryImageFileName\n __typename\n }\n __typename\n}\n\nfragment ChildNavigationItem on ProductCategoryChildNavigationItem {\n categoryId\n hits\n selected\n category {\n name\n pathSlug\n categoryImageFileName\n __typename\n }\n __typename\n}\n\nfragment ProductReference on ProductReference {\n id\n ticket\n searchResultProduct {\n ...ProductForList\n __typename\n }\n __typename\n}\n\nfragment Ads on AdContainer {\n banner {\n imageUrl\n url\n text\n displayMode\n buttonText\n __typename\n }\n productReferences {\n ...ProductReference\n __typename\n }\n ticket\n __typename\n}\n\nfragment ProductForList on SearchResultProduct {\n id\n displayName\n primaryImageId\n manufacturerName\n manufacturerProductIdentifier\n displaySpecifications\n oneLiner\n internalProduct\n nameSlug\n hasPrimaryServiceProduct\n primaryServiceProductErpIdentifier\n energyClass\n energyLabelImageId\n reviewScore\n isNewProduct\n promotedSpecifications(take: 4) {\n name\n value\n __typename\n }\n price {\n campaignPercentage\n campaignType\n formatted {\n price\n originalPrice\n __typename\n }\n isBestSeller\n isBid\n isCampaign\n isRecommendedProduct\n price\n priceExcludingVat\n originalPrice\n serviceDurationMonths\n __typename\n }\n categoryName\n availability {\n externalStock {\n leadTime\n quantity\n __typename\n }\n availabilityDetails {\n formattedDateUtc\n key\n __typename\n }\n availabilityStatus\n internalStock {\n quantity\n showAlwaysInStock\n __typename\n }\n isAvailableForSale\n isPreOrder\n productLifeCycleState\n showAvailabilityQuantities\n showEtaDate\n __typename\n }\n __typename\n}\n"}
for var in Categoryid:
print(body.update({"categoryId":var}))

Solution by Vishal Singh works:
body['variables']['categoryId'] = xxxx

Related

Can't produce a result without hardcoding cookies within the headers of the following script

I'm trying to scrape the zestimate for this address, 1205 Clover St, Accokeek, MD 20607 from this webpage using requests module. When I use that address in the inputbox of that webpage, I get this zpid 37374749 number, which, when I use within params and issue a post request, I can get the zestimate.
The problem is the script works only when I hardcode cookies from dev tools within the headers of the requests; otherwise, it fails miserably and throws a JSONDecodeError.
import requests
from pprint import pprint
from bs4 import BeautifulSoup
url = 'https://www.zillow.com/graphql/'
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
'referer': 'https://www.zillow.com/how-much-is-my-home-worth/',
'cookie': '_px3=00d7bf4fc2e331a793b7ebf9f9e8059013b86b39ead67dfc15fd89fe4e1b459c:GrviXjUIrsA7lYSeSBB8Y8yQedr9ng0ByK7xkaB6CLDFCGYjxBL2r8dPy6MwnUk0Eqv75g0t1fcIa4c8ksIHoA==:1000:1qeUILwk/42tR6FaZ0ybaI63rAWDBAc7fsAeLcH02MyzXKEEoT0SDnk/zMCLA3Zj1BpBmuqO/4RQDntxiOgtsCGPF1VuxfpmXAeMPcxegynJ/PGXWYE3OLIa9vJ9XkVkGaSoH9knD3Ls7nHyuP/0DhapZbThlzDLoQ//Vflzi0eGv12lCSX/2msW+rCeFgO0RhsBMNSN93EUhmuMBXcBMg==;'
}
params = {"operationName":"HowMuchIsMyHomeWorthReviewQuery","variables":{"zpid":37374749},"query":"query HowMuchIsMyHomeWorthReviewQuery($zpid: ID!) {\n property(zpid: $zpid) {\n streetAddress\n city\n state\n zipcode\n bedrooms\n bathrooms\n livingArea\n zestimate\n homeStatus\n photos(size: XL) {\n url\n __typename\n }\n ...OmpHomeWorthUpsell_property\n isConfirmedClaimedByCurrentSignedInUser\n isVerifiedClaimedByCurrentSignedInUser\n ...UARequiredPropertyDimensions_property\n ...ContactAgentForm_property\n ...HomeInfo_property\n __typename\n }\n viewer {\n ...ContactAgentForm_viewer\n __typename\n }\n abTests {\n ...OmpHomeWorthUpsell_abTests\n ...UARequiredPropertyDimensions_abTests\n ...ContactAgentForm_abTests\n __typename\n }\n}\n\nfragment OmpHomeWorthUpsell_property on Property {\n zpid\n onsiteMessage(placementNames: [\"HMIMHWTopSlot\"]) {\n ...onsiteMessage_fragment\n __typename\n }\n __typename\n}\n\nfragment onsiteMessage_fragment on OnsiteMessageResultType {\n eventId\n decisionContext\n messages {\n skipDisplayReason\n shouldDisplay\n isGlobalHoldout\n isPlacementHoldout\n placementName\n testPhase\n bucket\n placementId\n passThrottle\n lastModified\n eventId\n decisionContext\n selectedTreatment {\n id\n name\n component\n status\n renderingProps\n lastModified\n __typename\n }\n qualifiedTreatments {\n id\n name\n status\n lastModified\n __typename\n }\n __typename\n }\n __typename\n}\n\nfragment OmpHomeWorthUpsell_abTests on ABTests {\n HMIMHW_ZO_NFS_UPSELL_ONSITE_MESSAGING: abTest(\n trial: \"HMIMHW_ZO_NFS_UPSELL_ONSITE_MESSAGING\"\n )\n __typename\n}\n\nfragment UARequiredPropertyDimensions_property on Property {\n currency\n featuredListingTypeDimension\n hasPublicVideo\n hdpTypeDimension\n listingTypeDimension\n price\n propertyTypeDimension\n standingOffer {\n isStandingOfferEligible\n __typename\n }\n zpid\n isZillowOwned\n zillowOfferMarket {\n legacyName\n __typename\n }\n ...ShouldShowVideo_property\n __typename\n}\n\nfragment ShouldShowVideo_property on Property {\n homeStatus\n isZillowOwned\n hasPublicVideo\n primaryPublicVideo {\n sources {\n src\n __typename\n }\n __typename\n }\n richMediaVideos {\n mp4Url\n hlsUrl\n __typename\n }\n __typename\n}\n\nfragment UARequiredPropertyDimensions_abTests on ABTests {\n ZO_HDP_HOUR_ONE_VIDEO: abTest(trial: \"ZO_HDP_HOUR_ONE_VIDEO\")\n __typename\n}\n\nfragment ContactAgentForm_property on Property {\n streetAddress\n state\n city\n zipcode\n zpid\n homeStatus\n homeType\n zestimate\n homeType\n isInstantOfferEnabled\n zillowOfferMarket {\n name\n code\n __typename\n }\n __typename\n}\n\nfragment ContactAgentForm_viewer on Viewer {\n name\n email\n zuid\n __typename\n}\n\nfragment ContactAgentForm_abTests on ABTests {\n SHOW_PL_LEAD_FORM: abTest(trial: \"SHOW_PL_LEAD_FORM\")\n __typename\n}\n\nfragment HomeInfo_property on Property {\n streetAddress\n city\n state\n zipcode\n bedrooms\n bathrooms\n livingArea\n homeStatus\n homeType\n contingentListingType\n photos(size: XL) {\n url\n __typename\n }\n listing_sub_type {\n is_newHome\n is_FSBO\n is_bankOwned\n is_foreclosure\n is_forAuction\n is_comingSoon\n __typename\n }\n __typename\n}\n"}
with requests.Session() as s:
s.headers.update(headers)
res = s.post(url,json=params)
pprint(res.json()['data']['property']['zestimate'])
How can I find success without hardcoding cookies within the headers?

To get zpid dynamically, you have to first submit a request where you put the address. Then use this zpid in second request:
import requests
api_url = "https://www.zillowstatic.com/autocomplete/v3/suggestions/"
graphql_url = "https://www.zillow.com/graphql/"
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36",
"referer": "https://www.zillow.com/how-much-is-my-home-worth/",
}
with requests.session() as s:
s.headers.update(headers)
params = {
"q": "1205 Clover St, Accokeek, MD 20607",
"resultTypes": "allAddress",
"resultCount": "5",
}
data = s.get(api_url, params=params).json()
zpid = data["results"][0]["metaData"]["zpid"]
payload = {
"operationName": "HowMuchIsMyHomeWorthReviewQuery",
"query": 'query HowMuchIsMyHomeWorthReviewQuery($zpid: ID!) {\n property(zpid: $zpid) {\n streetAddress\n city\n state\n zipcode\n bedrooms\n bathrooms\n livingArea\n zestimate\n homeStatus\n photos(size: XL) {\n url\n __typename\n }\n ...OmpHomeWorthUpsell_property\n isConfirmedClaimedByCurrentSignedInUser\n isVerifiedClaimedByCurrentSignedInUser\n ...UARequiredPropertyDimensions_property\n ...ContactAgentForm_property\n ...HomeInfo_property\n __typename\n }\n viewer {\n ...ContactAgentForm_viewer\n __typename\n }\n abTests {\n ...OmpHomeWorthUpsell_abTests\n ...UARequiredPropertyDimensions_abTests\n ...ContactAgentForm_abTests\n __typename\n }\n}\n\nfragment OmpHomeWorthUpsell_property on Property {\n zpid\n onsiteMessage(placementNames: ["HMIMHWTopSlot"]) {\n ...onsiteMessage_fragment\n __typename\n }\n __typename\n}\n\nfragment onsiteMessage_fragment on OnsiteMessageResultType {\n eventId\n decisionContext\n messages {\n skipDisplayReason\n shouldDisplay\n isGlobalHoldout\n isPlacementHoldout\n placementName\n testPhase\n bucket\n placementId\n passThrottle\n lastModified\n eventId\n decisionContext\n selectedTreatment {\n id\n name\n component\n status\n renderingProps\n lastModified\n __typename\n }\n qualifiedTreatments {\n id\n name\n status\n lastModified\n __typename\n }\n __typename\n }\n __typename\n}\n\nfragment OmpHomeWorthUpsell_abTests on ABTests {\n HMIMHW_ZO_NFS_UPSELL_ONSITE_MESSAGING: abTest(\n trial: "HMIMHW_ZO_NFS_UPSELL_ONSITE_MESSAGING"\n )\n __typename\n}\n\nfragment UARequiredPropertyDimensions_property on Property {\n currency\n featuredListingTypeDimension\n hasPublicVideo\n hdpTypeDimension\n listingTypeDimension\n price\n propertyTypeDimension\n standingOffer {\n isStandingOfferEligible\n __typename\n }\n zpid\n isZillowOwned\n zillowOfferMarket {\n legacyName\n __typename\n }\n ...ShouldShowVideo_property\n __typename\n}\n\nfragment ShouldShowVideo_property on Property {\n homeStatus\n isZillowOwned\n hasPublicVideo\n primaryPublicVideo {\n sources {\n src\n __typename\n }\n __typename\n }\n richMediaVideos {\n mp4Url\n hlsUrl\n __typename\n }\n __typename\n}\n\nfragment UARequiredPropertyDimensions_abTests on ABTests {\n ZO_HDP_HOUR_ONE_VIDEO: abTest(trial: "ZO_HDP_HOUR_ONE_VIDEO")\n __typename\n}\n\nfragment ContactAgentForm_property on Property {\n streetAddress\n state\n city\n zipcode\n zpid\n homeStatus\n homeType\n zestimate\n homeType\n isInstantOfferEnabled\n zillowOfferMarket {\n name\n code\n __typename\n }\n __typename\n}\n\nfragment ContactAgentForm_viewer on Viewer {\n name\n email\n zuid\n __typename\n}\n\nfragment ContactAgentForm_abTests on ABTests {\n SHOW_PL_LEAD_FORM: abTest(trial: "SHOW_PL_LEAD_FORM")\n __typename\n}\n\nfragment HomeInfo_property on Property {\n streetAddress\n city\n state\n zipcode\n bedrooms\n bathrooms\n livingArea\n homeStatus\n homeType\n contingentListingType\n photos(size: XL) {\n url\n __typename\n }\n listing_sub_type {\n is_newHome\n is_FSBO\n is_bankOwned\n is_foreclosure\n is_forAuction\n is_comingSoon\n __typename\n }\n __typename\n}\n',
"variables": {"zpid": zpid},
}
data = s.post(graphql_url, json=payload).json()
print(data["data"]["property"]["zestimate"])
Prints:
444700

How to prevent httpx timeout in python?

I am trying to scrape tokopedia here. When I use raw codes, it works and the json is returned. However when I tried to use it as a variable, it reads time out.
website:https://www.tokopedia.com/samudrasembako/regal-marie-roll-230-gram?extParam=ivf%3Dfalse&src=topads
Here's the code
!pip install fake_useragent
!pip install httpx
import requests
from fake_useragent import UserAgent
import httpx
tokopedia=['https://www.tokopedia.com/samudrasembako/regal-marie-roll-230-gram?extParam=ivf%3Dfalse&src=topads']
for url in tokopedia:
ua = UserAgent().random
product_key=url.split(".com")[1].split("/")[2].split("?")[0]
shopdomain=url.split(".com")[1].split("/")[1].split("?")[0]
payload={
"operationName":"PDPGetLayoutQuery",
"variables":
{"shopDomain":f"{shopdomain}",
"productKey":f"{product_key}",
"layoutID":"",
"apiVersion":1,
"userLocation":
{"cityID":"176",
"addressID":"0",
"districtID":"2274",
"postalCode":"",
"latlon":""},
"extParam":""},
"query":"fragment ProductVariant on pdpDataProductVariant {\n errorCode\n parentID\n defaultChild\n sizeChart\n totalStockFmt\n variants {\n productVariantID\n variantID\n name\n identifier\n option {\n picture {\n urlOriginal: url\n urlThumbnail: url100\n __typename\n }\n productVariantOptionID\n variantUnitValueID\n value\n hex\n stock\n __typename\n }\n __typename\n }\n children {\n productID\n price\n priceFmt\n optionID\n optionName\n productName\n productURL\n picture {\n urlOriginal: url\n urlThumbnail: url100\n __typename\n }\n stock {\n stock\n isBuyable\n stockWordingHTML\n minimumOrder\n maximumOrder\n __typename\n }\n isCOD\n isWishlist\n campaignInfo {\n campaignID\n campaignType\n campaignTypeName\n campaignIdentifier\n background\n discountPercentage\n originalPrice\n discountPrice\n stock\n stockSoldPercentage\n startDate\n endDate\n endDateUnix\n appLinks\n isAppsOnly\n isActive\n hideGimmick\n isCheckImei\n minOrder\n __typename\n }\n thematicCampaign {\n additionalInfo\n background\n campaignName\n icon\n __typename\n }\n __typename\n }\n __typename\n}\n\nfragment ProductMedia on pdpDataProductMedia {\n media {\n type\n urlOriginal: URLOriginal\n urlThumbnail: URLThumbnail\n urlMaxRes: URLMaxRes\n videoUrl: videoURLAndroid\n prefix\n suffix\n description\n variantOptionID\n __typename\n }\n videos {\n source\n url\n __typename\n }\n __typename\n}\n\nfragment ProductHighlight on pdpDataProductContent {\n name\n price {\n value\n currency\n __typename\n }\n campaign {\n campaignID\n campaignType\n campaignTypeName\n campaignIdentifier\n background\n percentageAmount\n originalPrice\n discountedPrice\n originalStock\n stock\n stockSoldPercentage\n threshold\n startDate\n endDate\n endDateUnix\n appLinks\n isAppsOnly\n isActive\n hideGimmick\n __typename\n }\n thematicCampaign {\n additionalInfo\n background\n campaignName\n icon\n __typename\n }\n stock {\n useStock\n value\n stockWording\n __typename\n }\n variant {\n isVariant\n parentID\n __typename\n }\n wholesale {\n minQty\n price {\n value\n currency\n __typename\n }\n __typename\n }\n isCashback {\n percentage\n __typename\n }\n isTradeIn\n isOS\n isPowerMerchant\n isWishlist\n isCOD\n isFreeOngkir {\n isActive\n __typename\n }\n preorder {\n duration\n timeUnit\n isActive\n preorderInDays\n __typename\n }\n __typename\n}\n\nfragment ProductCustomInfo on pdpDataCustomInfo {\n icon\n title\n isApplink\n applink\n separator\n description\n __typename\n}\n\nfragment ProductInfo on pdpDataProductInfo {\n row\n content {\n title\n subtitle\n applink\n __typename\n }\n __typename\n}\n\nfragment ProductDetail on pdpDataProductDetail {\n content {\n title\n subtitle\n applink\n showAtFront\n isAnnotation\n __typename\n }\n __typename\n}\n\nfragment ProductDataInfo on pdpDataInfo {\n icon\n title\n isApplink\n applink\n content {\n icon\n text\n __typename\n }\n __typename\n}\n\nfragment ProductSocial on pdpDataSocialProof {\n row\n content {\n icon\n title\n subtitle\n applink\n type\n rating\n __typename\n }\n __typename\n}\n\nquery PDPGetLayoutQuery($shopDomain: String, $productKey: String, $layoutID: String, $apiVersion: Float, $userLocation: pdpUserLocation, $extParam: String) {\n pdpGetLayout(shopDomain: $shopDomain, productKey: $productKey, layoutID: $layoutID, apiVersion: $apiVersion, userLocation: $userLocation, extParam: $extParam) {\n requestID\n name\n pdpSession\n basicInfo {\n alias\n createdAt\n isQA\n id: productID\n shopID\n shopName\n minOrder\n maxOrder\n weight\n weightUnit\n condition\n status\n url\n needPrescription\n catalogID\n isLeasing\n isBlacklisted\n menu {\n id\n name\n url\n __typename\n }\n category {\n id\n name\n title\n breadcrumbURL\n isAdult\n isKyc\n minAge\n detail {\n id\n name\n breadcrumbURL\n isAdult\n __typename\n }\n __typename\n }\n txStats {\n transactionSuccess\n transactionReject\n countSold\n paymentVerified\n itemSoldFmt\n __typename\n }\n stats {\n countView\n countReview\n countTalk\n rating\n __typename\n }\n __typename\n }\n components {\n name\n type\n position\n data {\n ...ProductMedia\n ...ProductHighlight\n ...ProductInfo\n ...ProductDetail\n ...ProductSocial\n ...ProductDataInfo\n ...ProductCustomInfo\n ...ProductVariant\n __typename\n }\n __typename\n }\n __typename\n }\n}\n"
}
headers={
'origin': 'https://www.tokopedia.com',
'referer': f'{url}',
'sec-ch-ua': '"Not?A_Brand";v="8", "Chromium";v="108", "Google Chrome";v="108"',
'sec-ch-ua-mobile': '?0',
'sec-ch-ua-platform': "Windows",
'sec-fetch-dest': 'empty',
'sec-fetch-mode': 'cors',
'sec-fetch-site': 'same-site',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36',
'x-device': 'desktop',
'x-source': 'tokopedia-lite',
'x-tkpd-akamai': 'pdpGetLayout',
'x-tkpd-lite-service': 'zeus',
'x-version': '53ac990'
}
client= httpx.Client()
resp=client.post("https://gql.tokopedia.com/graphql/PDPGetLayoutQuery",json=payload,headers=headers)
Can someone please help? So that the above code runs and returns the json.

A Python script works fine on Windows but throws an error on Mac

I wish to get different product information from this webpage using requests module. I've created a script in Python to get a JSON response by issuing post requests with appropriate parameters.
The script works fine on windows but throws this error JSONDecodeError: Expecting value: line 1 column 1 (char 0) on mac.
Here is how I've tried:
import requests
start_url = 'https://www.jumbo.com/producten/'
link = 'https://www.jumbo.com/api/graphql'
payload = {"operation":"searchResult","variables":{"searchTerms":"","sortOption":"","showMoreIds":"","offSet":0,"pageSize":24,"categoryUrl":"/producten/"},"query":"\n fragment productFields on Product {\n id: sku\n brand\n badgeDescription\n category\n subtitle: packSizeDisplay\n title\n image\n inAssortment\n availability {\n availability\n isAvailable\n label\n }\n isAvailable\n isSponsored\n link\n status\n retailSet\n prices: price {\n price\n promoPrice\n pricePerUnit {\n price\n unit\n }\n }\n crossSellSkus\n quantityDetails {\n maxAmount\n minAmount\n stepAmount\n defaultAmount\n unit\n }\n quantityOptions {\n maxAmount\n minAmount\n stepAmount\n unit\n }\n primaryBadge: primaryBadges {\n alt\n image\n }\n secondaryBadges {\n alt\n image\n }\n promotions {\n id\n group\n isKiesAndMix\n image\n tags {\n text\n inverse\n }\n start {\n dayShort\n date\n monthShort\n }\n end {\n dayShort\n date\n monthShort\n }\n attachments{\n type\n path\n }\n }\n }\n\n query searchResult(\n $searchTerms: String\n $filters: String\n $offSet: Int\n $showMoreIds: String\n $sortOption: String\n $pageSize: Int\n $categoryUrl: String\n ) {\n searchResult(\n searchTerms: $searchTerms\n filters: $filters\n offSet: $offSet\n showMoreIds: $showMoreIds\n sortOption: $sortOption\n pageSize: $pageSize\n categoryUrl: $categoryUrl\n ) {\n canonicalRelativePath\n categoryIdPath\n categoryTiles {\n id\n label\n imageLink\n navigationState\n siteRootPath\n }\n urlState\n newUrl\n redirectUrl\n shelfDescription\n removeAllAction\n powerFilters {\n displayName\n navigationState\n siteRootPath\n }\n metaData {\n title\n description\n }\n headerContent {\n headerText\n count\n }\n helperText {\n show\n shortBody\n longBody\n header\n linkText\n targetUrl\n messageType\n }\n recipeLink {\n linkText\n targetUrl\n textIsRich\n }\n guidedNavigation {\n ancestors {\n label\n }\n displayName\n dimensionName\n groupName\n name\n multiSelect\n moreLink {\n label\n navigationState\n }\n lessLink {\n label\n navigationState\n }\n refinements {\n label\n count\n multiSelect\n navigationState\n siteRootPath\n }\n }\n selectedRefinements {\n refinementCrumbs {\n label\n count\n multiSelect\n dimensionName\n ancestors {\n label\n navigationState\n }\n removeAction {\n navigationState\n }\n }\n searchCrumbs {\n terms\n removeAction {\n navigationState\n }\n }\n removeAllAction {\n navigationState\n }\n }\n socialLists {\n title\n totalNumRecs\n lists {\n id\n title\n followers\n productImages\n thumbnail\n author\n labels\n isAuthorVerified\n }\n }\n mainContent {\n searchWarning\n searchAdjustments {\n originalTerms\n adjustedSearches {\n key\n terms {\n autoPhrased\n adjustedTerms\n spellCorrected\n }\n }\n }\n }\n productsResultList {\n pagingActionTemplate {\n navigationState\n }\n lastRecNum\n totalNumRecs\n sortOptions {\n navigationState\n label\n selected\n }\n products {\n ...productFields\n retailSetProducts {\n ...productFields\n }\n }\n }\n }\n }\n"}
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36',
'accept': 'application/json, text/plain, */*',
'referer': 'https://www.jumbo.com/producten/',
'origin': 'https://www.jumbo.com',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'en-US,en;q=0.9,bn;q=0.8',
}
with requests.Session() as s:
s.headers.update(headers)
s.get(start_url)
res = s.post(link,json=payload)
print(res.json())
How can I make it work on the Mac?
This video demo represents how the script performs when I execute it on windows.

Your request headers include:
'accept-encoding': 'gzip, deflate, br'
and the post response includes this:
'Content-Encoding': 'br'
So the response is compressed using Brotli compression, which is not automatically handled by the requests module.
Here's how to decode it (note, this is hardcoded to use Brotli, and to be robust it should check the header's Content-Encoding):
import requests
import brotli
import json
import pprint
start_url = 'https://www.jumbo.com/producten/'
link = 'https://www.jumbo.com/api/graphql'
payload = {"operation":"searchResult","variables":{"searchTerms":"","sortOption":"","showMoreIds":"","offSet":0,"pageSize":24,"categoryUrl":"/producten/"},"query":"\n fragment productFields on Product {\n id: sku\n brand\n badgeDescription\n category\n subtitle: packSizeDisplay\n title\n image\n inAssortment\n availability {\n availability\n isAvailable\n label\n }\n isAvailable\n isSponsored\n link\n status\n retailSet\n prices: price {\n price\n promoPrice\n pricePerUnit {\n price\n unit\n }\n }\n crossSellSkus\n quantityDetails {\n maxAmount\n minAmount\n stepAmount\n defaultAmount\n unit\n }\n quantityOptions {\n maxAmount\n minAmount\n stepAmount\n unit\n }\n primaryBadge: primaryBadges {\n alt\n image\n }\n secondaryBadges {\n alt\n image\n }\n promotions {\n id\n group\n isKiesAndMix\n image\n tags {\n text\n inverse\n }\n start {\n dayShort\n date\n monthShort\n }\n end {\n dayShort\n date\n monthShort\n }\n attachments{\n type\n path\n }\n }\n }\n\n query searchResult(\n $searchTerms: String\n $filters: String\n $offSet: Int\n $showMoreIds: String\n $sortOption: String\n $pageSize: Int\n $categoryUrl: String\n ) {\n searchResult(\n searchTerms: $searchTerms\n filters: $filters\n offSet: $offSet\n showMoreIds: $showMoreIds\n sortOption: $sortOption\n pageSize: $pageSize\n categoryUrl: $categoryUrl\n ) {\n canonicalRelativePath\n categoryIdPath\n categoryTiles {\n id\n label\n imageLink\n navigationState\n siteRootPath\n }\n urlState\n newUrl\n redirectUrl\n shelfDescription\n removeAllAction\n powerFilters {\n displayName\n navigationState\n siteRootPath\n }\n metaData {\n title\n description\n }\n headerContent {\n headerText\n count\n }\n helperText {\n show\n shortBody\n longBody\n header\n linkText\n targetUrl\n messageType\n }\n recipeLink {\n linkText\n targetUrl\n textIsRich\n }\n guidedNavigation {\n ancestors {\n label\n }\n displayName\n dimensionName\n groupName\n name\n multiSelect\n moreLink {\n label\n navigationState\n }\n lessLink {\n label\n navigationState\n }\n refinements {\n label\n count\n multiSelect\n navigationState\n siteRootPath\n }\n }\n selectedRefinements {\n refinementCrumbs {\n label\n count\n multiSelect\n dimensionName\n ancestors {\n label\n navigationState\n }\n removeAction {\n navigationState\n }\n }\n searchCrumbs {\n terms\n removeAction {\n navigationState\n }\n }\n removeAllAction {\n navigationState\n }\n }\n socialLists {\n title\n totalNumRecs\n lists {\n id\n title\n followers\n productImages\n thumbnail\n author\n labels\n isAuthorVerified\n }\n }\n mainContent {\n searchWarning\n searchAdjustments {\n originalTerms\n adjustedSearches {\n key\n terms {\n autoPhrased\n adjustedTerms\n spellCorrected\n }\n }\n }\n }\n productsResultList {\n pagingActionTemplate {\n navigationState\n }\n lastRecNum\n totalNumRecs\n sortOptions {\n navigationState\n label\n selected\n }\n products {\n ...productFields\n retailSetProducts {\n ...productFields\n }\n }\n }\n }\n }\n"}
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36',
'accept': 'application/json, text/plain, */*',
'referer': 'https://www.jumbo.com/producten/',
'origin': 'https://www.jumbo.com',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'en-US,en;q=0.9,bn;q=0.8',
}
with requests.Session() as s:
s.headers.update(headers)
s.get(start_url)
res = s.post(link,json=payload)
body = brotli.decompress(res.content)
pprint.pprint(json.loads(body))
A better solution is to simply remove br from accept-encoding. The requests module automatically handles gzip and deflate for you, but not br.

Python requests post a query to graphql with variables

I'm trying to get a list of offers for an item sold on opensea.io
def getHighestOffer(self):
query = """query OrdersQuery(\n $cursor: String\n $count: Int = 10\n $excludeMaker: IdentityInputType\n $isExpired: Boolean\n $isFilled: Boolean\n $isValid: Boolean\n $maker: IdentityInputType\n $makerArchetype: ArchetypeInputType\n $makerAssetIsPayment: Boolean\n $takerArchetype: ArchetypeInputType\n $takerAssetCategories: [CollectionSlug!]\n $takerAssetCollections: [CollectionSlug!]\n $takerAssetIsOwnedBy: IdentityInputType\n $takerAssetIsPayment: Boolean\n $sortAscending: Boolean\n $sortBy: OrderSortOption\n $makerAssetBundle: BundleSlug\n $takerAssetBundle: BundleSlug\n) {\n ...Orders_data_2g7x2d\n}\n\nfragment AccountLink_data on AccountType {\n address\n chain {\n identifier\n id\n }\n user {\n publicUsername\n id\n }\n ...ProfileImage_data\n ...wallet_accountKey\n}\n\nfragment AskPrice_data on OrderV2Type {\n dutchAuctionFinalPrice\n openedAt\n priceFnEndedAt\n makerAssetBundle {\n assetQuantities(first: 30) {\n edges {\n node {\n ...quantity_data\n id\n }\n }\n }\n id\n }\n takerAssetBundle {\n assetQuantities(first: 1) {\n edges {\n node {\n ...AssetQuantity_data\n id\n }\n }\n }\n id\n }\n}\n\nfragment AssetCell_assetBundle on AssetBundleType {\n assetQuantities(first: 2) {\n edges {\n node {\n asset {\n collection {\n name\n id\n }\n name\n ...AssetMedia_asset\n ...asset_url\n id\n }\n relayId\n id\n }\n }\n }\n name\n slug\n}\n\nfragment AssetMedia_asset on AssetType {\n animationUrl\n backgroundColor\n collection {\n description\n displayData {\n cardDisplayStyle\n }\n imageUrl\n hidden\n name\n slug\n id\n }\n description\n name\n tokenId\n imageUrl\n}\n\nfragment AssetQuantity_data on AssetQuantityType {\n asset {\n ...Price_data\n id\n }\n quantity\n}\n\nfragment Orders_data_2g7x2d on Query {\n orders(after: $cursor, excludeMaker: $excludeMaker, first: $count, isExpired: $isExpired, isFilled: $isFilled, isValid: $isValid, maker: $maker, makerArchetype: $makerArchetype, makerAssetIsPayment: $makerAssetIsPayment, takerArchetype: $takerArchetype, takerAssetCategories: $takerAssetCategories, takerAssetCollections: $takerAssetCollections, takerAssetIsOwnedBy: $takerAssetIsOwnedBy, takerAssetIsPayment: $takerAssetIsPayment, sortAscending: $sortAscending, sortBy: $sortBy, makerAssetBundle: $makerAssetBundle, takerAssetBundle: $takerAssetBundle) {\n edges {\n node {\n closedAt\n isFulfillable\n isValid\n oldOrder\n openedAt\n orderType\n maker {\n address\n ...AccountLink_data\n ...wallet_accountKey\n id\n }\n makerAsset: makerAssetBundle {\n assetQuantities(first: 1) {\n edges {\n node {\n asset {\n assetContract {\n account {\n address\n chain {\n identifier\n id\n }\n id\n }\n id\n }\n id\n }\n id\n }\n }\n }\n id\n }\n makerAssetBundle {\n assetQuantities(first: 30) {\n edges {\n node {\n ...AssetQuantity_data\n ...quantity_data\n id\n }\n }\n }\n id\n }\n relayId\n side\n taker {\n ...AccountLink_data\n ...wallet_accountKey\n id\n address\n }\n takerAssetBundle {\n assetQuantities(first: 1) {\n edges {\n node {\n ...AssetQuantity_data\n ...quantity_data\n asset {\n ownedQuantity(identity: {})\n decimals\n symbol\n relayId\n assetContract {\n account {\n address\n id\n }\n id\n }\n id\n }\n quantity\n id\n }\n }\n }\n id\n }\n ...AskPrice_data\n ...orderLink_data\n makerAssetBundleDisplay: makerAssetBundle {\n ...AssetCell_assetBundle\n id\n }\n takerAssetBundleDisplay: takerAssetBundle {\n ...AssetCell_assetBundle\n id\n }\n id\n __typename\n }\n cursor\n }\n pageInfo {\n endCursor\n hasNextPage\n }\n }\n}\n\nfragment Price_data on AssetType {\n decimals\n imageUrl\n symbol\n usdSpotPrice\n assetContract {\n blockExplorerLink\n id\n }\n}\n\nfragment ProfileImage_data on AccountType {\n imageUrl\n address\n chain {\n identifier\n id\n }\n}\n\nfragment asset_url on AssetType {\n assetContract {\n account {\n address\n chain {\n identifier\n id\n }\n id\n }\n id\n }\n tokenId\n}\n\nfragment orderLink_data on OrderV2Type {\n makerAssetBundle {\n assetQuantities(first: 30) {\n edges {\n node {\n asset {\n externalLink\n collection {\n externalUrl\n id\n }\n id\n }\n id\n }\n }\n }\n id\n }\n}\n\nfragment quantity_data on AssetQuantityType {\n asset {\n decimals\n id\n }\n quantity\n}\n\nfragment wallet_accountKey on AccountType {\n address\n chain {\n identifier\n id\n }\n}\n"""
variables = {"cursor":None,"count":10,"excludeMaker":None,"isExpired":False,"isFilled":None,"isValid":True,"maker":None,"makerArchetype":None,"makerAssetIsPayment":True,"takerArchetype":{"assetContractAddress":"0x7c40c393dc0f283f318791d746d894ddd3693572","tokenId":"7722"},"takerAssetCategories":None,"takerAssetCollections":None,"takerAssetIsOwnedBy":None,"takerAssetIsPayment":None,"sortAscending":None,"sortBy":"MAKER_ASSETS_USD_PRICE","makerAssetBundle":None,"takerAssetBundle":None}
response = requests.post('https://api.opensea.io/graphql/', json={'query': query},data=variables)
print(response.text)
(In variables the "assetContractAddress" and "tokenId" are unique to the item.
However when I run this I get:
{"errors":[{"message":"Must provide query string."}]}
And if I don't use data=variables in requests.post I get:
{"errors":[{"message":"[400] One of taker_asset_categories, taker_asset_collections, maker, taker, maker, include_maker_assets, include_taker_assets, maker_assets, taker_assets, maker_asset_is_owned_by, taker_asset_is_owned_by, exclude_maker, maker_asset_bundle, taker_asset_bundle needs to be defined.","locations":[{"line":118,"column":3}],"path":["orders"]}],"data":{"orders":null}}
How can I use requests.post with the query and variables to get the proper response?
Thanks!

You need to send the variables in the json data like this :
{
"query": "your query",
"variables": {
"var1": "value1"
}
}
checkout this
Example:
import requests
query = """query OrdersQuery(\n $cursor: String\n $count: Int = 10\n $excludeMaker: IdentityInputType\n $isExpired: Boolean\n $isFilled: Boolean\n $isValid: Boolean\n $maker: IdentityInputType\n $makerArchetype: ArchetypeInputType\n $makerAssetIsPayment: Boolean\n $takerArchetype: ArchetypeInputType\n $takerAssetCategories: [CollectionSlug!]\n $takerAssetCollections: [CollectionSlug!]\n $takerAssetIsOwnedBy: IdentityInputType\n $takerAssetIsPayment: Boolean\n $sortAscending: Boolean\n $sortBy: OrderSortOption\n $makerAssetBundle: BundleSlug\n $takerAssetBundle: BundleSlug\n) {\n ...Orders_data_2g7x2d\n}\n\nfragment AccountLink_data on AccountType {\n address\n chain {\n identifier\n id\n }\n user {\n publicUsername\n id\n }\n ...ProfileImage_data\n ...wallet_accountKey\n}\n\nfragment AskPrice_data on OrderV2Type {\n dutchAuctionFinalPrice\n openedAt\n priceFnEndedAt\n makerAssetBundle {\n assetQuantities(first: 30) {\n edges {\n node {\n ...quantity_data\n id\n }\n }\n }\n id\n }\n takerAssetBundle {\n assetQuantities(first: 1) {\n edges {\n node {\n ...AssetQuantity_data\n id\n }\n }\n }\n id\n }\n}\n\nfragment AssetCell_assetBundle on AssetBundleType {\n assetQuantities(first: 2) {\n edges {\n node {\n asset {\n collection {\n name\n id\n }\n name\n ...AssetMedia_asset\n ...asset_url\n id\n }\n relayId\n id\n }\n }\n }\n name\n slug\n}\n\nfragment AssetMedia_asset on AssetType {\n animationUrl\n backgroundColor\n collection {\n description\n displayData {\n cardDisplayStyle\n }\n imageUrl\n hidden\n name\n slug\n id\n }\n description\n name\n tokenId\n imageUrl\n}\n\nfragment AssetQuantity_data on AssetQuantityType {\n asset {\n ...Price_data\n id\n }\n quantity\n}\n\nfragment Orders_data_2g7x2d on Query {\n orders(after: $cursor, excludeMaker: $excludeMaker, first: $count, isExpired: $isExpired, isFilled: $isFilled, isValid: $isValid, maker: $maker, makerArchetype: $makerArchetype, makerAssetIsPayment: $makerAssetIsPayment, takerArchetype: $takerArchetype, takerAssetCategories: $takerAssetCategories, takerAssetCollections: $takerAssetCollections, takerAssetIsOwnedBy: $takerAssetIsOwnedBy, takerAssetIsPayment: $takerAssetIsPayment, sortAscending: $sortAscending, sortBy: $sortBy, makerAssetBundle: $makerAssetBundle, takerAssetBundle: $takerAssetBundle) {\n edges {\n node {\n closedAt\n isFulfillable\n isValid\n oldOrder\n openedAt\n orderType\n maker {\n address\n ...AccountLink_data\n ...wallet_accountKey\n id\n }\n makerAsset: makerAssetBundle {\n assetQuantities(first: 1) {\n edges {\n node {\n asset {\n assetContract {\n account {\n address\n chain {\n identifier\n id\n }\n id\n }\n id\n }\n id\n }\n id\n }\n }\n }\n id\n }\n makerAssetBundle {\n assetQuantities(first: 30) {\n edges {\n node {\n ...AssetQuantity_data\n ...quantity_data\n id\n }\n }\n }\n id\n }\n relayId\n side\n taker {\n ...AccountLink_data\n ...wallet_accountKey\n id\n address\n }\n takerAssetBundle {\n assetQuantities(first: 1) {\n edges {\n node {\n ...AssetQuantity_data\n ...quantity_data\n asset {\n ownedQuantity(identity: {})\n decimals\n symbol\n relayId\n assetContract {\n account {\n address\n id\n }\n id\n }\n id\n }\n quantity\n id\n }\n }\n }\n id\n }\n ...AskPrice_data\n ...orderLink_data\n makerAssetBundleDisplay: makerAssetBundle {\n ...AssetCell_assetBundle\n id\n }\n takerAssetBundleDisplay: takerAssetBundle {\n ...AssetCell_assetBundle\n id\n }\n id\n __typename\n }\n cursor\n }\n pageInfo {\n endCursor\n hasNextPage\n }\n }\n}\n\nfragment Price_data on AssetType {\n decimals\n imageUrl\n symbol\n usdSpotPrice\n assetContract {\n blockExplorerLink\n id\n }\n}\n\nfragment ProfileImage_data on AccountType {\n imageUrl\n address\n chain {\n identifier\n id\n }\n}\n\nfragment asset_url on AssetType {\n assetContract {\n account {\n address\n chain {\n identifier\n id\n }\n id\n }\n id\n }\n tokenId\n}\n\nfragment orderLink_data on OrderV2Type {\n makerAssetBundle {\n assetQuantities(first: 30) {\n edges {\n node {\n asset {\n externalLink\n collection {\n externalUrl\n id\n }\n id\n }\n id\n }\n }\n }\n id\n }\n}\n\nfragment quantity_data on AssetQuantityType {\n asset {\n decimals\n id\n }\n quantity\n}\n\nfragment wallet_accountKey on AccountType {\n address\n chain {\n identifier\n id\n }\n}\n"""
variables = {"cursor": None, "count": 10, "excludeMaker": None, "isExpired": False, "isFilled": None, "isValid": True, "maker": None, "makerArchetype": None, "makerAssetIsPayment": True, "takerArchetype": {"assetContractAddress": "0x7c40c393dc0f283f318791d746d894ddd3693572",
"tokenId": "7722"}, "takerAssetCategories": None, "takerAssetCollections": None, "takerAssetIsOwnedBy": None, "takerAssetIsPayment": None, "sortAscending": None, "sortBy": "MAKER_ASSETS_USD_PRICE", "makerAssetBundle": None, "takerAssetBundle": None}
response = requests.post('https://api.opensea.io/graphql/',
json={"query": query, "variables": variables}
)
print(response.text)

How to paginate through all the pages of a website using requests and beautifulsoup in python

So i was trying to scrape names and prices for the shower curtains from this site. The site has above 200 pages, but this code works only for the first 100 pages and then it repeats scraping the same 100 pages again.
import requests
from bs4 import BeautifulSoup
import re
import csv
site = "https://ih1.redbubble.net/image.{}/ur,shower_curtain_closed,square,600x600.1.jpg"
firstrow = ['No.', 'Name', 'Price', 'Image Url']
with open('curtains.csv', 'a', newline='') as csvFile:
writer = csv.writer(csvFile)
writer.writerow(firstrow)
csvFile.close()
def main(url):
count = 0
for page in range(1,205):
print('\n','*'*10 , 'Scraping Page # {}'.format(page) , '*'*10)
print('Link # {}'.format(url.format(page)))
final_url = url.format(page)
r = requests.get(final_url)
soup = BeautifulSoup(r.content, 'html.parser')
target = soup.select("img[class*=styles__rounded--1lyoH]")
imgs = [img.group(1) for img in re.finditer(r'\.(\d+\.\d{4})', r.text)]
goal = list(dict.fromkeys(imgs))
for tar, go in zip(target, goal):
count += 1
name = tar['alt']
price = tar.find_all_next('span')[3].text
img = site.format(go)
print('*'*20 , count , '*'*20)
print('Name: {}'.format(name))
print('Price: {}'.format(price))
print('Image Url: {}'.format(img))
row = [count, name, price, img]
with open('curtains.csv', 'a', newline='' , encoding='utf-8') as csvFile:
writer = csv.writer(csvFile)
writer.writerow(row)
csvFile.close()
main("https://www.redbubble.com/shop/shower-curtains?page={}")

import requests
import csv
data = {
"operationName": "withSearchResults",
"query": "query withSearchResults($query: String!, $queryParams: QueryParams, $locale: String!, $country: String!, $currency: String!, $previewTypeIds: [String!], $experience: String) {\n searchResults(query: $query, queryParams: $queryParams, locale: $locale, country: $country, currency: $currency, previewTypeIds: $previewTypeIds, experience: $experience) {\n ...Results\n ...TrendingResults\n ...Metadata\n ...Filters\n ...Pagination\n ...LandingPage\n __typename\n }\n}\n\nfragment Results on SearchResults {\n results {\n inventoryItem(locale: $locale, country: $country, currency: $currency, previewTypeIds: $previewTypeIds) {\n id\n description\n productTypeId\n productPageUrl\n blankItemId\n price {\n id\n amount\n currency\n __typename\n }\n previewSet {\n id\n previews {\n previewTypeId\n url\n __typename\n }\n __typename\n }\n gaCode\n gaCategory\n attributes {\n name\n value\n attributes {\n name\n value\n __typename\n }\n __typename\n }\n volumeDiscount {\n id\n thresholds {\n percentOff\n quantity\n __typename\n }\n __typename\n }\n experiencesProductCard {\n name\n value\n __typename\n }\n __typename\n }\n work(locale: $locale) {\n id\n title\n artistName\n isMatureContent\n tags\n __typename\n }\n defaultPreviewTypeId\n groupId\n rank\n __typename\n }\n __typename\n}\n\nfragment TrendingResults on SearchResults {\n trendingResults {\n inventoryItem(locale: $locale, country: $country, currency: $currency, previewTypeIds: $previewTypeIds) {\n id\n description\n productPageUrl\n productTypeId\n price {\n id\n amount\n currency\n __typename\n }\n previewSet {\n id\n previews {\n previewTypeId\n url\n __typename\n }\n __typename\n }\n volumeDiscount {\n id\n thresholds {\n percentOff\n quantity\n __typename\n }\n __typename\n }\n gaCode\n gaCategory\n attributes {\n name\n value\n attributes {\n name\n value\n __typename\n }\n __typename\n }\n experiencesProductCard {\n name\n value\n __typename\n }\n __typename\n }\n work(locale: $locale) {\n id\n title\n artistName\n isMatureContent\n tags\n __typename\n }\n defaultPreviewTypeId\n rank\n __typename\n }\n __typename\n}\n\nfragment Metadata on SearchResults {\n metadata {\n title\n searchContext {\n category\n __typename\n }\n resultCount\n topic\n searchBar {\n iaCode\n pillLabel\n keywords\n __typename\n }\n __typename\n }\n __typename\n}\n\nfragment Filters on SearchResults {\n filters {\n resetUrl\n staticFilters {\n type\n label\n options {\n name\n label\n applied\n url\n options {\n name\n label\n applied\n url\n __typename\n }\n __typename\n }\n __typename\n }\n filters {\n type\n label\n experiences {\n name\n value\n __typename\n }\n options {\n name\n label\n applied\n disabled\n url\n hexColor\n imageUrl\n __typename\n }\n __typename\n }\n appliedCount\n appliedPath\n resets {\n label\n url\n __typename\n }\n __typename\n }\n __typename\n}\n\nfragment Pagination on SearchResults {\n pagination {\n currentPage\n perPage\n showPreviousPageLink\n showNextPageLink\n paginationLinks {\n namedLinks {\n previousPage {\n rel\n url\n __typename\n }\n nextPage {\n rel\n url\n __typename\n }\n __typename\n }\n __typename\n }\n fromNumber\n toNumber\n total\n __typename\n }\n __typename\n}\n\nfragment LandingPage on SearchResults {\n metadata {\n formattedQuery\n landingPage {\n hero {\n pitch\n title\n image\n color\n __typename\n }\n bubbles {\n title\n items {\n title\n image\n realisticImage\n url\n isExternal\n __typename\n }\n hasImages\n __typename\n }\n seoMetadata {\n pageDescription\n robots\n canonicalURL\n searchTitle\n seoImage\n alternatePageVersions {\n href\n locale\n __typename\n }\n relatedTagLinks {\n title\n href\n text\n __typename\n }\n __typename\n }\n footer {\n text\n breadcrumbs {\n name\n url\n __typename\n }\n __typename\n }\n __typename\n }\n relatedTopics {\n title\n url\n __typename\n }\n relatedProducts {\n id\n url\n productTitle\n fullTitle\n __typename\n }\n searchPageType\n resultCount\n searchUUID\n __typename\n }\n __typename\n}\n",
"variables": {"country": "EG", "currency": "USD", "experience": "srp", "locale": "en",
"previewTypeIds": ["product_close", "alternate_product_close", "artwork"],
"query": "shower-curtains", "queryParams": {"page": 0}
}
}
def main(url):
with requests.Session() as req:
with open("data.csv", 'w', newline="", encoding="UTF-8") as f:
writer = csv.writer(f)
writer.writerow(["Name", "Price", "IMG"])
for item in range(1, 11):
print(f"Extracting Page# {item}")
data['variables']['queryParams']['page'] = item
r = req.post(url, json=data).json()
for item in r['data']['searchResults']['results']:
writer.writerow([
item['work']['title'],
item['inventoryItem']['price']['amount'],
item['inventoryItem']['previewSet']['previews'][0]['url']
])
main("https://www.redbubble.com/boom/graphql")
Output: view-online
Sample:

Develop Reference

Python is a programming language that lets you work quickly and integrate systems more effectively.

Python: Update value in dictionary while looping - python

Solution by Vishal Singh works: body['variables']['categoryId'] = xxxx

Related

Can't produce a result without hardcoding cookies within the headers of the following script

How to prevent httpx timeout in python?

A Python script works fine on Windows but throws an error on Mac

Python requests post a query to graphql with variables

How to paginate through all the pages of a website using requests and beautifulsoup in python

Categories

Resources