Read JSON Direct from API into DataFrame via PySpark/Databricks - python

I'm attempting to improve a current process that we have. Presently the process is:
API Call Made > JSON Saved as File > API Calls are iterated ending up with multiple files > Files are then read into a Databricks Dataframe.
I am attempting to remove the need to save the JSON as a file and then using read.json the files themselves into a dataframe before I then iterate through the data.
Is there a way I can read the json response into a string and then read it directly into a data frame?
My attempt is below but it keeps failing:
payload={}
headers = {
'Authorization': 'Basic ==',
'Cookie': 'JSESSIONID='
}
response = requests.request("GET", apipath, headers=headers, data=payload)
jsonData = json.dumps(response.text)
jsonDataList = []
jsonDataList.append(jsonData)
jsonRDD = sc.parallelize(jsonDataList)
df = spark.read.json(jsonRDD)
However I do a df.printSchema() and get told its "corrupt" data.
I've also tried to do the following:
payload={}
headers = {
'Authorization': 'Basic ==',
'Cookie': 'JSESSIONID='
}
response = requests.request("GET", apipath, headers=headers, data=payload)
jsonData = json.dumps(response.text)
#jsonDataList = []
#jsonDataList.append(jsonData)
#jsonRDD = sc.parallelize(jsonDataList)
df = spark.read.json(jsonData)
But get told relative path in URI and I'm guessing that is because its not reading directly from a file.
Any assistance would be really appreciated.


I reproduced same in my environment. I got below result:
To Resolve above error please follow this code:
import requests
resp = requests.get('https://reqres.in/api/users?page=1,name,href')
db1 = spark.sparkContext.parallelize([resp.text])
df2 = spark.read.json(db1)
df2.show()
Output:

Related

How to access certain items from a python requests response?

I am new to python requests and I am unsure how to access one singular item from the response rather than all of them. Any help would be appreciated. I would like to be able to print the kdRatio, there is multiple different kdRatios in the response but I was looking for the one following lifetime and All and Properties. Thanks
import requests as rq
header = {
'x-rapidapi-host': "call-of-duty-modern-warfare.p.rapidapi.com",
'x-rapidapi-key': "0beff1d7acmsh29065741543208dp152ecbjsnb56e8654cc77"
}
response = rq.get('https://call-of-duty-modern-
warfare.p.rapidapi.com/multiplayer/%25C7%259Ep%25C3%25ABx%25231878/battle', headers=header)
print(response)
print(response.text)
Here is the response I am getting.
<Response [200]>
{"title":"mw","platform":"battle","username":"Ǟpëx#1878","type":"mp","level":55,"maxLevel":0,"levelXpRemainder":3000,"levelXpGained":7000,"prestige":0,"prestigeId":0,"maxPrestige":0,"totalXp":960000,"paragonRank":0,"paragonId":0,"s":1,"p":1,"lifetime":{"all":{"properties":{"recordLongestWinStreak":14,"recordXpInAMatch":29276,"accuracy":0.2253633737564087,"losses":456,"totalGamesPlayed":1216,"score":2853122,"winLossRatio":1.094298243522644,"totalShots":286484,"bestScoreXp":0,"gamesPlayed":1216,"bestSquardKills":0,"bestSguardWave":0,"bestConfirmed":19,"deaths":13057,"wins":499,"bestSquardCrates":0,"kdRatio":1.495213270187378,"bestAssists":27,"bestFieldgoals":0,"bestScore":16625,"recordDeathsInAMatch":55,"scorePerGame":2346.3174342105262,"bestSPM":2450,"bestKillChains":0,"recordKillsInAMatch":113,"suicides":79,"wlRatio":1.094298243522644,"currentWinStreak":2,"bestMatchBonusXp":0,"bestMatchXp":0,"bestSguardWeaponLevel":0,"bestKD":16,"kills":19523,"bestKillsAsInfected":3,"bestReturns":0,"bestStabs":0,"bestKillsAsSurvivor":10,"timePlayedTotal":488334,"bestDestructions":1,"headshots":3303,"bestRescues":5,"assists":3237,"ties":1,"recordKillStreak":29,"bestPlants":1,"misses":221921,"bestDamage":0,"bestSetbacks":0,"bestTouchdowns":0,"scorePerMinute":350.5537603361634,"bestDeaths":55,"bestMedalXp":0,"bestDefends":41,"bestSquardRevives":0,"bestKills":113,"bestDefuses":0,"bestCaptures":8,"hits":64563,"bestKillStreak":29,"bestDenied":8}},"mode":{"gun":{"properties":{"kills":12,"score":1262,"timePlayed":311,"kdRatio":0.9230769230769231,"setBacks":0,"scorePerMinute":243.47266881028938,"stabs":0,"deaths":13}},"dom":{"properties":{"kills":3986,"score":466680,"timePlayed":92224,"kdRatio":1.619008935824533,"captures":228,"defends":412,"scorePerMinute":303.617279666898,"deaths":2462}},"war":{"properties":{"kills":1292,"score":165815,"timePlayed":25846,"kdRatio":1.453318335208099,"assists":271,"scorePerMinute":384.92996982124896,"deaths":889}},"hq":{"properties":{"kills":4111,"score":441585,"timePlayed":99696,"kdRatio":1.6563255439161966,"captures":93,"defends":27,"scorePerMinute":265.7589070775157,"deaths":2482}},"hc_dom":{"properties":{"kills":942,"score":103870,"timePlayed":23342,"kdRatio":1.3792093704245973,"captures":46,"defends":81,"scorePerMinute":266.9951160997344,"deaths":683}},"hc_conf":{"properties":{"kills":73,"score":8300,"timePlayed":1367,"kdRatio":1.2372881355932204,"confirms":15,"scorePerMinute":364.3013899049012,"denies":20,"deaths":59}},"koth":{"properties":{"kills":1311,"score":187420,"timePlayed":27342,"kdRatio":1.4219088937093276,"defends":37,"objTime":697,"scorePerMinute":411.27935044985736,"deaths":922}},"conf":{"properties":{"kills":1042,"score":146180,"timePlayed":19643,"kdRatio":1.507959479015919,"confirms":273,"scorePerMinute":446.5102071984931,"denies":104,"deaths":691}},"hc_hq":{"properties":{"kills":166,"score":17735,"timePlayed":2726,"kdRatio":1.2296296296296296,"captures":2,"defends":1,"scorePerMinute":390.35216434336024,"deaths":135}},"arena":{"properties":{"kills":815,"score":143855,"timePlayed":41493,"damage":97577,"kdRatio":1.1256906077348066,"assists":193,"scorePerMinute":208.0182199407129,"deaths":724}},"br_dmz":{"properties":{"wins":0,"kills":7,"kdRatio":3.5,"downs":9,"topTwentyFive":0,"topTen":0,"contracts":0,"revives":0,"topFive":0,"score":3312,"timePlayed":1108,"gamesPlayed":1,"tokens":0,"scorePerMinute":179.35018050541518,"cash":0,"deaths":2}},"br":{"properties":{"wins":5,"kills":736,"kdRatio":1.6919540229885057,"downs":789,"topTwentyFive":117,"topTen":54,"contracts":104,"revives":100,"topFive":27,"score":713500,"timePlayed":169249,"gamesPlayed":146,"tokens":0,"scorePerMinute":252.94093318128913,"cash":0,"deaths":435}},"sd":{"properties":{"kills":146,"score":71070,"timePlayed":44172,"kdRatio":1.4174757281553398,"plants":0,"scorePerMinute":96.5362673186634,"defuses":2,"deaths":103}},"grnd":{"properties":{"kills":0,"score":1875,"timePlayed":292,"kdRatio":0,"defends":0,"objTime":0,"scorePerMinute":385.2739726027398,"deaths":0}},"cyber":{"properties":{"kills":70,"score":37335,"timePlayed":20866,"kdRatio":1.206896551724138,"plants":1,"scorePerMinute":107.35646506278157,"revives":0,"deaths":58}},"hc_war":{"properties":{"kills":0,"score":625,"timePlayed":222,"kdRatio":0,"assists":0,"scorePerMinute":168.9189189189189,"deaths":0}},"br_all":{"properties":{"wins":5,"kills":743,"kdRatio":1.700228832951945,"downs":798,"topTwentyFive":117,"topTen":54,"contracts":104,"revives":100,"topFive":27,"score":716812,"timePlayed":170357,"gamesPlayed":147,"tokens":0,"scorePerMinute":252.46229975874195,"cash":0,"deaths":437}},"hc_sd":{"properties":{"kills":0,"score":0,"timePlayed":0,"kdRatio":0,"plants":0,"scorePerMinute":0,"defuses":0,"deaths":0}},"arm":{"properties":{"kills":526,"score":76725,"timePlayed":25625,"kdRatio":1.1311827956989247,"captures":27,"defends":16,"scorePerMinute":179.64878048780488,"deaths":465}},"hc_cyber":{"properties":{"kills":0,"score":0,"timePlayed":0,"kdRatio":0,"plants":0,"scorePerMinute":0,"revives":0,"deaths":0}},"infect":{"properties":{"kills":67,"score":28425,"infected":17,"timePlayed":4649,"kdRatio":0.9305555555555556,"scorePerMinute":366.85308668530865,"time":2641,"deaths":72}}},"map":{},"itemData":{"weapon_assault_rifle":{"iw8_ar_tango21":{"properties":{"hits":41,"kills":8,"kdRatio":8,"headshots":1,"accuracy":0.16141732283464566,"shots":254,"deaths":0}},"iw8_ar_mike4":{"properties":{"hits":11685,"kills":2576,"kdRatio":1.4015233949945594,"headshots":471,"accuracy":0.2240136497833672,"shots":52162,"deaths":1838}},"iw8_ar_valpha":{"properties":{"hits":0,"kills":0,"kdRatio":0,"headshots":0,"accuracy":0,"shots":0,"deaths":0}},"iw8_ar_falpha":{"properties":{"hits":54,"kills":14,"kdRatio":1.5555555555555556,"headshots":5,"accuracy":0.18305084745762712,"shots":295,"deaths":9}},"iw8_ar_mcharlie":{"properties":{"hits":870,"kills":141,"kdRatio":1.3689320388349515,"headshots":35,"accuracy":0.223305954825462,"shots":3896,"deaths":103}},"iw8_ar_akilo47":{"properties":{"hits":3741,"kills":781,"kdRatio":1.3259762308998302,"headshots":137,"accuracy":0.26047904191616766,"shots":14362,"deaths":589}},"iw8_ar_asierra12":{"properties":{"hits":456,"kills":151,"kdRatio":1.26890756302521,"headshots":33,"accuracy":0.22233057045343735,"shots":2051,"deaths":119}},"iw8_ar_galima":{"properties":{"hits":3,"kills":1,"kdRatio":1,"headshots":0,"accuracy":0.23076923076923078,"shots":13,"deaths":0}},"iw8_ar_sierra552":{"properties":{"hits":1611,"kills":353,"kdRatio":1.6045454545454545,"headshots":48,"accuracy":0.23719081272084805,"shots":6792,"deaths":220}},"iw8_ar_falima":{"properties":{"hits":2579,"kills":1010,"kdRatio":1.6476345840130506,"headshots":208,"accuracy":0.26316326530612244,"shots":9800,"deaths":613}},"iw8_ar_anovember94":{"properties":{"hits":24,"kills":4,"kdRatio":0.8,"headshots":3,"accuracy":0.13114754098360656,"shots":183,"deaths":5}},"iw8_ar_kilo433":{"properties":{"hits":5034,"kills":1034,"kdRatio":1.267156862745098,"headshots":179,"accuracy":0.2387818992505455,"shots":21082,"deaths":816}},"iw8_ar_scharlie":{"properties":{"hits":326,"kills":91,"kdRatio":1.3582089552238805,"headshots":12,"accuracy":0.24847560975609756,"shots":1312,"deaths":67}}},"weapon_shotgun":{"iw8_sh_mike26":{"properties":{"hits":5,"kills":1,"kdRatio":1,"headshots":0,"accuracy":0.3125,"shots":16,"deaths":0}},"iw8_sh_charlie725":{"properties":{"hits":1538,"kills":1195,"kdRatio":1.3307349665924275,"headshots":187,"accuracy":0.694043321299639,"shots":2216,"deaths":898}},"iw8_sh_oscar12":{"properties":{"hits":973,"kills":441,"kdRatio":1.1221374045801527,"headshots":45,"accuracy":0.43053097345132746,"shots":2260,"deaths":393}},"iw8_sh_romeo870":{"properties":{"hits":58,"kills":28,"kdRatio":0.8,"headshots":4,"accuracy":0.5272727272727272,"shots":110,"deaths":35}},"iw8_sh_dpapa12":{"properties":{"hits":20,"kills":9,"kdRatio":0.6428571428571429,"headshots":0,"accuracy":0.32786885245901637,"shots":61,"deaths":14}}},"weapon_marksman":{"iw8_sn_sbeta":{"properties":{"hits":192,"kills":143,"kdRatio":0.9930555555555556,"headshots":35,"accuracy":0.3742690058479532,"shots":513,"deaths":144}},"iw8_sn_crossbow":{"properties":{"hits":4,"kills":7,"kdRatio":1.4,"headshots":2,"accuracy":0.125,"shots":32,"deaths":5}},"iw8_sn_kilo98":{"properties":{"hits":76,"kills":70,"kdRatio":1.1666666666666667,"headshots":32,"accuracy":0.4342857142857143,"shots":175,"deaths":60}},"iw8_sn_mike14":{"properties":{"hits":52,"kills":27,"kdRatio":1.9285714285714286,"headshots":12,"accuracy":0.23318385650224216,"shots":223,"deaths":14}},"iw8_sn_sksierra":{"properties":{"hits":26,"kills":9,"kdRatio":9,"headshots":5,"accuracy":0.2857142857142857,"shots":91,"deaths":0}}},"weapon_sniper":{"iw8_sn_romeo700":{"properties":{"hits":0,"kills":0,"kdRatio":0,"headshots":0,"accuracy":0,"shots":0,"deaths":0}},"iw8_sn_alpha50":{"properties":{"hits":138,"kills":168,"kdRatio":1.3658536585365855,"headshots":41,"accuracy":0.36507936507936506,"shots":378,"deaths":123}},"iw8_sn_hdromeo":{"properties":{"hits":22,"kills":27,"kdRatio":0.8181818181818182,"headshots":5,"accuracy":0.3188405797101449,"shots":69,"deaths":33}},"iw8_sn_delta":{"properties":{"hits":79,"kills":51,"kdRatio":1,"headshots":15,"accuracy":0.30268199233716475,"shots":261,"deaths":51}},"iw8_sn_xmike109":{"properties":{"hits":1,"kills":1,"kdRatio":1,"headshots":0,"accuracy":1,"shots":1,"deaths":0}}},"tacticals":{"equip_gas_grenade":{"properties":{"extraStat1":1,"uses":19}},"equip_snapshot_grenade":{"properties":{"extraStat1":471,"uses":979}},"equip_decoy":{"properties":{"extraStat1":95,"uses":15}},"equip_smoke":{"properties":{"extraStat1":0,"uses":157}},"equip_concussion":{"properties":{"extraStat1":280,"uses":660}},"equip_hb_sensor":{"properties":{"extraStat1":0,"uses":98}},"equip_flash":{"properties":{"extraStat1":101,"uses":283}},"equip_adrenaline":{"properties":{"extraStat1":142214,"uses":2505}}},"lethals":{"equip_frag":{"properties":{"kills":41,"uses":556}},"equip_thermite":{"properties":{"kills":2,"uses":25}},"equip_semtex":{"properties":{"kills":6,"uses":71}},"equip_claymore":{"properties":{"kills":225,"uses":1509}},"equip_c4":{"properties":{"kills":57,"uses":664}},"equip_at_mine":{"properties":{"kills":285,"uses":2563}},"equip_throwing_knife":{"properties":{"kills":5,"uses":30}},"equip_molotov":{"properties":{"kills":24,"uses":465}}},"weapon_lmg":{"iw8_lm_kilo121":{"properties":{"hits":218,"kills":59,"kdRatio":1.5128205128205128,"headshots":13,"accuracy":0.15900802334062727,"shots":1371,"deaths":39}},"iw8_lm_mkilo3":{"properties":{"hits":441,"kills":105,"kdRatio":1.09375,"headshots":18,"accuracy":0.17904993909866018,"shots":2463,"deaths":96}},"iw8_lm_mgolf34":{"properties":{"hits":64,"kills":20,"kdRatio":1.8181818181818181,"headshots":6,"accuracy":0.17344173441734417,"shots":369,"deaths":11}},"iw8_lm_lima86":{"properties":{"hits":150,"kills":25,"kdRatio":1.25,"headshots":8,"accuracy":0.20080321285140562,"shots":747,"deaths":20}},"iw8_lm_pkilo":{"properties":{"hits":1390,"kills":342,"kdRatio":1.1324503311258278,"headshots":62,"accuracy":0.15873015873015872,"shots":8757,"deaths":302}},"iw8_lm_sierrax":{"properties":{"hits":1875,"kills":429,"kdRatio":1.1230366492146597,"headshots":63,"accuracy":0.22278992395437264,"shots":8416,"deaths":382}},"iw8_lm_mgolf36":{"properties":{"hits":1756,"kills":388,"kdRatio":1.065934065934066,"headshots":88,"accuracy":0.20223425083496488,"shots":8683,"deaths":364}}},"weapon_launcher":{"iw8_la_gromeo":{"properties":{"hits":23,"kills":0,"kdRatio":0,"headshots":0,"accuracy":0.45098039215686275,"shots":51,"deaths":26}},"iw8_la_rpapa7":{"properties":{"hits":3,"kills":51,"kdRatio":1.1590909090909092,"headshots":0,"accuracy":0.02040816326530612,"shots":147,"deaths":44}},"iw8_la_juliet":{"properties":{"hits":8,"kills":0,"kdRatio":0,"headshots":0,"accuracy":0.24242424242424243,"shots":33,"deaths":0}},"iw8_la_kgolf":{"properties":{"hits":0,"kills":0,"kdRatio":0,"headshots":0,"accuracy":0,"shots":0,"deaths":0}},"iw8_la_mike32":{"properties":{"hits":0,"kills":0,"kdRatio":0,"headshots":0,"accuracy":0,"shots":14,"deaths":0}}},"supers":{"super_emp_drone":{"properties":{"kills":0,"misc1":0,"misc2":0,"uses":0}},"super_trophy":{"properties":{"kills":0,"misc1":33,"misc2":0,"uses":32}},"super_ammo_drop":{"properties":{"kills":1,"misc1":207,"misc2":0,"uses":108}},"super_weapon_drop":{"properties":{"kills":0,"misc1":2,"misc2":0,"uses":1}},"super_fulton":{"properties":{"kills":0,"misc1":0,"misc2":0,"uses":0}},"super_armor_drop":{"properties":{"kills":0,"misc1":0,"misc2":0,"uses":24}},"super_select":{"properties":{"kills":0,"misc1":0,"misc2":0,"uses":303}},"super_tac_insert":{"properties":{"kills":0,"misc1":2,"misc2":0,"uses":2}},"super_recon_drone":{"properties":{"kills":0,"misc1":5,"misc2":0,"uses":5}},"super_deadsilence":{"properties":{"kills":616,"misc1":9,"misc2":0,"uses":474}},"super_supply_drop":{"properties":{"kills":0,"misc1":0,"misc2":0,"uses":45}},"super_tac_cover":{"properties":{"kills":0,"misc1":807,"misc2":0,"uses":33}},"super_support_box":{"properties":{"kills":29,"misc1":0,"misc2":0,"uses":34}}},"weapon_pistol":{"iw8_pi_cpapa":{"properties":{"hits":91,"kills":50,"kdRatio":0.7142857142857143,"headshots":11,"accuracy":0.20634920634920634,"shots":441,"deaths":70}},"iw8_pi_mike9":{"properties":{"hits":12,"kills":3,"kdRatio":3,"headshots":1,"accuracy":0.11764705882352941,"shots":102,"deaths":0}},"iw8_pi_mike1911":{"properties":{"hits":91,"kills":18,"kdRatio":0.8571428571428571,"headshots":2,"accuracy":0.1961206896551724,"shots":464,"deaths":21}},"iw8_pi_golf21":{"properties":{"hits":219,"kills":26,"kdRatio":1.1818181818181819,"headshots":4,"accuracy":0.26290516206482595,"shots":833,"deaths":22}},"iw8_pi_decho":{"properties":{"hits":203,"kills":73,"kdRatio":0.7019230769230769,"headshots":15,"accuracy":0.3481989708404803,"shots":583,"deaths":104}},"iw8_pi_papa320":{"properties":{"hits":87,"kills":16,"kdRatio":0.4444444444444444,"headshots":5,"accuracy":0.17864476386036962,"shots":487,"deaths":36}}},"weapon_other":{"iw8_me_riotshield":{"properties":{"hits":0,"kills":2,"kdRatio":0.5,"headshots":0,"accuracy":0,"shots":0,"deaths":4}}},"weapon_smg":{"iw8_sm_mpapa7":{"properties":{"hits":5392,"kills":944,"kdRatio":1.3964497041420119,"headshots":194,"accuracy":0.255315119087078,"shots":21119,"deaths":676}},"iw8_sm_augolf":{"properties":{"hits":6903,"kills":1747,"kdRatio":1.373427672955975,"headshots":290,"accuracy":0.23947131062235483,"shots":28826,"deaths":1272}},"iw8_sm_papa90":{"properties":{"hits":3969,"kills":746,"kdRatio":1.231023102310231,"headshots":180,"accuracy":0.22036533229692964,"shots":18011,"deaths":606}},"iw8_sm_charlie9":{"properties":{"hits":0,"kills":0,"kdRatio":0,"headshots":0,"accuracy":0,"shots":0,"deaths":0}},"iw8_sm_mpapa5":{"properties":{"hits":6885,"kills":1647,"kdRatio":1.5082417582417582,"headshots":283,"accuracy":0.22102728731942214,"shots":31150,"deaths":1092}},"iw8_sm_smgolf45":{"properties":{"hits":24,"kills":5,"kdRatio":5,"headshots":1,"accuracy":0.23300970873786409,"shots":103,"deaths":0}},"iw8_sm_beta":{"properties":{"hits":3053,"kills":686,"kdRatio":1.4503171247357294,"headshots":126,"accuracy":0.2310253499810821,"shots":13215,"deaths":473}},"iw8_sm_victor":{"properties":{"hits":61,"kills":11,"kdRatio":0.6111111111111112,"headshots":2,"accuracy":0.23371647509578544,"shots":261,"deaths":18}},"iw8_sm_uzulu":{"properties":{"hits":3345,"kills":989,"kdRatio":1.356652949245542,"headshots":184,"accuracy":0.21162849550803492,"shots":15806,"deaths":729}}},"weapon_melee":{"iw8_me_akimboblunt":{"properties":{"hits":0,"kills":0,"kdRatio":0,"headshots":0,"accuracy":0,"shots":0,"deaths":0}},"iw8_me_akimboblades":{"properties":{"hits":0,"kills":0,"kdRatio":0,"headshots":0,"accuracy":0,"shots":0,"deaths":0}},"iw8_knife":{"properties":{"hits":0,"kills":9,"kdRatio":0.17307692307692307,"headshots":0,"accuracy":0,"shots":0,"deaths":52}}}},"scorestreakData":{"lethalScorestreakData":{"precision_airstrike":{"properties":{"extraStat1":92,"uses":110,"awardedCount":96}},"cruise_predator":{"properties":{"extraStat1":13,"uses":12,"awardedCount":0}},"manual_turret":{"properties":{"extraStat1":0,"uses":3,"awardedCount":0}},"white_phosphorus":{"properties":{"extraStat1":3,"uses":1,"awardedCount":1}},"hover_jet":{"properties":{"extraStat1":482,"uses":176,"awardedCount":177}},"chopper_gunner":{"properties":{"extraStat1":507,"uses":72,"awardedCount":77}},"gunship":{"properties":{"extraStat1":19,"uses":3,"awardedCount":3}},"sentry_gun":{"properties":{"extraStat1":0,"uses":7,"awardedCount":4}},"toma_strike":{"properties":{"extraStat1":17,"uses":39,"awardedCount":25}},"nuke":{"properties":{"extraStat1":0,"uses":0,"awardedCount":0}},"juggernaut":{"properties":{"extraStat1":0,"uses":0,"awardedCount":0}},"pac_sentry":{"properties":{"extraStat1":0,"uses":0,"awardedCount":0}},"chopper_support":{"properties":{"extraStat1":2,"uses":1,"awardedCount":1}},"bradley":{"properties":{"extraStat1":0,"uses":0,"awardedCount":0}}},"supportScorestreakData":{"airdrop":{"properties":{"extraStat1":0,"uses":48,"awardedCount":55}},"radar_drone_overwatch":{"properties":{"extraStat1":0,"uses":66,"awardedCount":64}},"scrambler_drone_guard":{"properties":{"extraStat1":0,"uses":5,"awardedCount":0}},"uav":{"properties":{"extraStat1":5675,"uses":1194,"awardedCount":1151}},"airdrop_multiple":{"properties":{"extraStat1":0,"uses":0,"awardedCount":0}},"directional_uav":{"properties":{"extraStat1":13,"uses":2,"awardedCount":1}}}},"accoladeData":{"properties":{"classChanges":138,"highestAvgAltitude":225,"killsFromBehind":203,"lmgDeaths":78,"riotShieldDamageAbsorbed":5,"flashbangHits":24,"meleeKills":34,"tagsLargestBank":0,"shotgunKills":109,"sniperDeaths":99,"timeProne":80,"killstreakWhitePhosphorousKillsAssists":1,"shortestLife":80,"deathsFromBehind":112,"higherRankedKills":181,"mostAssists":142,"leastKills":55,"tagsDenied":0,"killstreakWheelsonKills":0,"sniperHeadshots":41,"killstreakJuggernautKills":0,"smokesUsed":15,"avengerKills":129,"decoyHits":6,"killstreakCarePackageUsed":0,"molotovKills":15,"gasHits":0,"comebackKills":77,"lmgHeadshots":49,"smgDeaths":96,"carrierKills":0,"deployableCoverUsed":18,"thermiteKills":2,"arKills":137,"c4Kills":0,"suicides":47,"clutch":4,"survivorKills":0,"killstreakGunshipKills":2,"timeSpentAsPassenger":3,"returns":0,"smgHeadshots":169,"launcherDeaths":3,"oneShotOneKills":100,"ammoBoxUsed":1,"spawnSelectSquad":1,"weaponPickups":128,"pointBlankKills":200,"tagsCaptured":0,"killstreakGroundKills":0,"distanceTraveledInVehicle":2,"longestLife":139,"stunHits":56,"spawnSelectFlag":0,"shotgunHeadshots":92,"bombDefused":5,"snapshotHits":110,"noKillsWithDeath":3,"killstreakAUAVAssists":2,"killstreakPersonalUAVKills":13,"tacticalInsertionSpawns":63,"launcherKills":2,"spawnSelectVehicle":2,"mostKillsLeastDeaths":62,"mostKills":201,"defends":34,"timeSpentAsDriver":1,"bombDetonated":2,"arHeadshots":123,"timeOnPoint":0,"lmgKills":59,"killstreakUAVAssists":179,"carepackagesCaptured":24,"mostKillsLongestStreak":133,"killstreakCruiseMissileKills":6,"longestStreak":221,"destroyedKillstreaks":18,"hipfireKills":192,"stimDamageHealed":137,"skippedKillcams":76,"leastAssists":198,"mostMultikills":207,"highestRankedKills":179,"killstreakAirstrikeKills":34,"distanceTravelled":114,"killstreakKills":119,"semtexKills":6,"penetrationKills":128,"explosionsSurvived":104,"highestMultikill":241,"arDeaths":78,"longshotKills":231,"proximityMineKills":126,"tagsMegaBanked":0,"mostKillsMostHeadshots":95,"firstInfected":1,"killstreakCUAVAssists":3,"throwingKnifeKills":3,"executionKills":1,"lastSurvivor":0,"reconDroneMarks":0,"deadSilenceKills":58,"revengeKills":164,"infectedKills":1,"killEnemyTeam":186,"sniperKills":63,"killstreakCluserStrikeKills":7,"meleeDeaths":57,"timeWatchingKillcams":27,"killstreakTankKills":0,"noKillNoDeath":5,"shotgunDeaths":87,"killstreakChopperGunnerKills":67,"shotsFired":120,"stoppingPowerKills":3,"pistolPeaths":29,"killstreakShieldTurretKills":0,"timeCrouched":39,"noDeathsFromBehind":855,"bombPlanted":4,"setbacks":0,"smgKills":187,"claymoreKills":66,"kills10NoDeaths":1,"pistolHeadshots":20,"killstreakVTOLJetKills":105,"headshots":219,"mostDeaths":68,"adsKills":153,"empDroneHits":0,"defenderKills":157,"launcherHeadshots":2,"timesSelectedAsSquadLeader":2,"killstreakAirKills":124,"assaults":1,"fragKills":19,"killstreakEmergencyAirdropUsed":0,"captures":16,"killstreakChopperSupportKills":1,"spawnSelectBase":0,"noKill10Deaths":0,"leastDeaths":181,"killstreakSentryGunKills":0,"longestTimeSpentOnWeapon":0,"lowerRankedKills":171,"trophySystemHits":6,"clutchRevives":0,"lowestAvgAltitude":55,"pickups":0,"pistolKills":47,"reloads":134}}},"weekly":{"all":{"properties":null},"mode":{},"map":{}},"engagement":null}

Parse the JSON response with response.json(), then access by key:
data = response.json()
kd_ratio = data['lifetime']['all']['properties']['kdRatio']

Here you can take the JSON response instead of a text response.
import requests as rq
header = {
'x-rapidapi-host': "call-of-duty-modern-warfare.p.rapidapi.com",
'x-rapidapi-key': "0beff1d7acmsh29065741543208dp152ecbjsnb56e8654cc77"
}
response = rq.get('https://call-of-duty-modern-
warfare.p.rapidapi.com/multiplayer/%25C7%259Ep%25C3%25ABx%25231878/battle', headers=header)
print(response)
print(response.json()['lifetime']['all']['properties']['kdRatio'])
For better understanding:
data = response.json()
print(data.keys()) # will print all the keys
print(type(data['all'])) # checked the type.
# since it is dict
print(data['all'].keys())
print(data['all']['properties'])
print(data['all']['properties'].keys())
print(data['all']['properties']['kdRatio'])
Try this in ipython you will get more idea about json data

Accessing data from a json array in python JSONDecodeError at /get/

My response looks like:
[
{"_id":"5f6060d0d279373c0017447d","name":"Faizan","email":"faizan#test.com"}
]
I want to get the name in python. I am tryting:
response = requests.get(url)
data = response.json()
The error I am getting is:
JSONDecodeError at /get/

this might help you
import requests
import json
dburl = 'https://postman-9e13.restdb.io/rest/contact'
headers = {'x-apikey': '7267cbb3c251a01dd8563ca447194e78af67d', 'Content-Type': 'application/json'}
params = {"name":"Faizan","email":"faizan#test.com"}
r = requests.get(dburl, params=params, headers=headers)
print(r.json())
after seeing the result, you can pass to your templates like this
geodata = response.json()
return render(request, 'main/get.html', { 'name': geodata[0]['name'] })
or you can change accordingly.

The snippet you shared is correct, however the problem might be with the URL.
response = requests.get(url)
data = response.json()
Here, .json() only works for the response which is in the JSON format only. If it is showing that error that means your URL returning the HTML document.

requests.post returns isSuccess:false even though Postman returns true

I am trying to post some information into an API based on their recommended format. When I use Postman( tool to test APIs), I see that the response has the isSuccess flag set to true. However, when I write the same code in Python using the requests library, I get the isSuccess flag as false
As mentioned about, I verified the headers and the json data object, both are the same yet the results defer
import requests
data = {"AccountNumber":"100007777",
"ActivityID":"78",
"ActivityDT":"2019-08-07 12:00:00",
"ActivityValue":"1"
}
url = "http://<IP>/<API_PATH>"
headers = {
"X-Tenant":"Default",
"Content-Type":"application/json"
}
response = requests.post(url,data=data, headers = headers)
print(response.content)
This code should successfully post the data and I should get a isSuccess:true in my response variable.
Can anyone help me figure out what might be wrong?

Can you try to change;
response = requests.post(url,data=data, headers = headers)
to;
response = requests.post(url,json=data, headers = headers)
or;
response = requests.post(url,body=data, headers = headers)

Can't get full table from requests python

I'm trying to get the whole table from this website: https://br.investing.com/commodities/aluminum-historical-data
But when I send this code:
with requests.Session() as s:
r = s.post('https://br.investing.com/commodities/aluminum-historical-data',
headers={"curr_id": "49768","smlID": "300586","header": "Alumínio Futuros Dados Históricos",
'User-Agent': 'Mozilla/5.0', 'st_date': '01/01/2017','end_date': '29/09/2018',
'interval_sec': 'Daily','sort_col': 'date','sort_ord': 'DESC','action': 'historical_data'})
bs2 = BeautifulSoup(r.text,'lxml')
tb = bs2.find('table',{"id":"curr_table"})
It only returns a piece of the table, not the whole date I just filtered.
I did see the post page below:
Can anyone help me get the whole table I just filtered?

You made two mistakes with your code.
The first one is the url.
You need to use the correct url to request data to investing.com.
Your current url is 'https://br.investing.com/commodities/aluminum-historical-data'
However, when you see inspection and click 'Network' the Request URLis https://br.investing.com/instruments/HistoricalDataAjax.
Your second mistake exists in s.post(blah). As Federico Rubbi referred above, what you coded assigned to headers must be assigned to data instead.
Now, your mistakes are all resolved. You need to do only one step more. You have to add a dictionary {'X-Requested-With': 'XMLHttpRequest'} to your_headers. Seeing from your code, I can see that you have already checked Network tab in HTML inspection. So, you are probably able to see why you need {'X-Requested-With': 'XMLHttpRequest'}.
So the entire code should be as follows.
import requests
import bs4 as bs
with requests.Session() as s:
url = 'https://br.investing.com/instruments/HistoricalDataAjax' # Making up for the first mistake.
your_headers = {'User-Agent': 'Mozilla/5.0'}
s.get(url, headers= your_headers)
c_list = s.cookies.get_dict().items()
cookie_list = [key+'='+value for key,value in c_list]
cookie = ','.join(cookie_list)
your_headers = {**{'X-Requested-With': 'XMLHttpRequest'},**your_headers}
your_headers['Cookie'] = cookie
data= {} # Your data. Making up for the second mistake.
response = s.post(url, data= data, headers = your_headers)

The problem is that you're passing form data as headers.
You have to send data with data keyworded argument in request.Session.post:
with requests.Session() as session:
url = 'https://br.investing.com/commodities/aluminum-historical-data'
data = {
"curr_id": "49768",
"smlID": "300586",
"header": "Alumínio Futuros Dados Históricos",
'User-Agent': 'Mozilla/5.0',
'st_date': '01/01/2017',
'end_date': '29/09/2018',
'interval_sec': 'Daily',
'sort_col': 'date',
'sort_ord': 'DESC',
'action': 'historical_data',
}
your_headers = {} # your headers here
response = session.post(url, data=data, headers=your_headers)
bs2 = BeautifulSoup(response.text,'lxml')
tb = bs2.find('table',{"id":"curr_table"})
I'd also recommend including your headers (especially user-agents) in the POST request because the site could not allow bots. In this case, if you do it, it will be harder to detect the bot.

python requests POST 400 error

I've been looking to many questions similar to mine but I could not find a solution.
I'm using requests to perform a POST request. I've tried a lot of combinations inside the request but nothing returns a 201 ok.
Here is my code:
import json
import requests
if __name__ == '__main__':
headers = {'content-type' : 'application/json'}
url = "http://myserver/ext/v3.1/test_device"
message = {"atribute_a": "value", "atribute_b": "valueb"}
params = {"priority":"normal"}
r = requests.post(url, params=params, headers=headers, data = json.dumps(message) )
print(r)
I've also tried withou json.dumps but it also gives me 400 bad request. I've also tried to add the params directly to the url like: ...?priority=normal but with no success.

The easiest technique is to use json instead of data as followed:
requests.post(url, headers=headers, params=params, json=data)

Based on the comments, your server is actually expecting data as a stringified JSON object.
As far as the params are concerned, it'd most probably help if they're declared as a tuple of tuples (or a dict of dicts)
Try the following -
headers = {
'content-type': 'application/json',
}
params = (
('priority', 'normal'),
)
data = {
"atribute_a": "value",
"atribute_b": false
}
requests.post(url, headers=headers, params=params, data=str(data))

Categories

Resources