Develop Reference

JSON Attribute Error str has no attribute JSON when collecting data off of website

Hello so im making a program to collect specific data off of a website. And im using cloud scraper so i dont get blocked. I have used cloud scraper before and It works perfectly fine. Only thing now is im trying to get the gas guzzlers on etheruem but when i try connecting to opensea I get an error saying.Would anyone know of a solution or an API i can use to get gas guzzlers?
Traceback (most recent call last):
File "main.py", line 9, in <module>
data = response.json()
AttributeError: 'str' object has no attribute 'json'```
Code
import cloudscraper
import json
import re
from json import loads
# Or: scraper = cloudscraper.CloudScraper() # CloudScraper inherits from requests.Session
scraper = cloudscraper.CloudScraper()
response = scraper.get("https://etherscan.io/gastracker").text
data = response.json()
b = json.loads(data)
print(b)````

How can I append a Boolean value to a text file?

I'm trying to add a boolean value to a text file but get this error:
Traceback (most recent call last):
File "/Users/valentinwestermann/Documents/La dieta mediterranea_dhooks.py", line 32, in <module>
f.write(variant["available"])
TypeError: write() argument must be str, not bool
Does anyone have an idea how to fix this? : )
It is supposed to work as a restock monitor and make a text version of the product availability when the bot launches and then constantly compare it and notifiers the user about product restocks.
import bs4 as bs
import urllib.request
import discord
from discord.ext import commands
from dhooks import Webhook
import requests
import json
r = requests.get("https://www.feature.com/products.json")
products = json.loads((r.text))["products"]
for product in products:
print("============================================")
print(product["title"])
print(product["tags"])
print(product["published_at"])
print(product["created_at"])
print(product["product_type"])
for variant in product["variants"]:
print(variant['title'])
print(variant['available'],"\n")
data =("available")
with open("stock_index.txt","w") as f:
for product in products:
for variant in product["variants"]:
if variant['available'] == True:
f.write(product["title"])
f.write(variant['title'])
print(variant["available"])
f.write("--------------------")

You can convert to a string first:
f.write(str(variant["available"]))

python's beautiful soup module giving error

I am using the following code in an attempt to do webscraping .
import sys , os
import requests, webbrowser,bs4
from PIL import Image
import pyautogui
p = requests.get('http://www.goal.com/en-ie/news/ozil-agent-eviscerates-jealous-keown-over-stupid-comments/1javhtwzz72q113dnonn24mnr1')
n = open("exml.txt" , 'wb')
for i in p.iter_content(1000) :
n.write(i)
n.close()
n = open("exml.txt" , 'r')
soupy= bs4.BeautifulSoup(n,"html.parser")
elems = soupy.select('img[src]')
for u in elems :
print (u)
so what I am intending to do is to extract all the image links that is there in the xml response obtained from the page .
(Please correct me If I am wrong in thinking that requests.get returns the whole static html file of the webpage that opens on entering the URL)
However in the line :
soupy= bs4.BeautifulSoup(n,"html.parser")
I am getting the following error :
Traceback (most recent call last):
File "../../perl/webscratcher.txt", line 24, in <module>
soupy= bs4.BeautifulSoup(n,"html.parser")
File "C:\Users\Kanishc\AppData\Local\Programs\Python\Python36-32\lib\site-packages\bs4\__init__.py", line 191, in __init__
markup = markup.read()
File "C:\Users\Kanishc\AppData\Local\Programs\Python\Python36-32\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 24662: character maps to <undefined>
I am clueless about the error and the "Appdata" folder is empty .
How to proceed further ?
Post Trying suggestions :
I changed the extension of the filename to py and this error got removed . However on the following line :
soupy= bs4.BeautifulSoup(n,"lxml") I am getting the following error :
Traceback (most recent call last):
File "C:\perl\webscratcher.py", line 23, in
soupy= bs4.BeautifulSoup(p,"lxml")
File "C:\Users\PREMRAJ\AppData\Local\Programs\Python\Python36-32\lib\site-packages\bs4_init_.py", line 192, in init
elif len(markup) <= 256 and (
TypeError: object of type 'Response' has no len()
How to tackle this ?

You are over-complicating things. Pass the bytes content of a Response object directly into the constructor of the BeautifulSoup object, instead of writing it to a file.
import requests
from bs4 import BeautifulSoup
response = requests.get('http://www.goal.com/en-ie/news/ozil-agent-eviscerates-jealous-keown-over-stupid-comments/1javhtwzz72q113dnonn24mnr1')
soup = BeautifulSoup(response.content, 'lxml')
for element in soup.select('img[src]'):
print(element)

Okay so you you might want to do a review on working with BeautifulSoup. I referenced an old project of mine and this is all you need for printing them. Check the BS documents to find the exact syntax you want with the select method.
This will print all the img tags from the html
import requests, bs4
site = 'http://www.goal.com/en-ie/news/ozil-agent-eviscerates-jealous-keown-over-stupid-comments/1javhtwzz72q113dnonn24mnr1'
p = requests.get(site).text
soupy = bs4.BeautifulSoup(p,"html.parser")
elems = soupy.select('img[src]')
for u in elems :
print (u)

Unable to parse json response with python

So I ran into an issue while importing API data into my code. Any help is much appreciated.
from urllib2 import Request, urlopen, URLError
import json, requests
data = requests.get('https://masari.superpools.net/api/live_stats?update=1522693430318').json()
data_parsed = json.loads(open(data,"r").read())
print data_parsed
I'm still quite new to python, and I ran into this error:
>C:\Users\bot2>python C:\Users\bot2\Desktop\Python_Code\helloworld.py
Traceback (most recent call last):
File "C:\Users\bot2\Desktop\Python_Code\helloworld.py", line 5, in <module>
data_parsed = json.loads(open(data,"r").read())
TypeError: coercing to Unicode: need string or buffer, dict found

data is already received as a json object (which is a dict in this case). Just do the following:
data = requests.get('https://masari.superpools.net/api/live_stats?update=1522693430318').json()
print data
Use data['network'] for example to access nested dictionaries.

How to properly scrape json response from reddit?

I am attempting to save all "author" entries from the json linked below into a list however am very new to python. Can someone kindly point me in the right direction?
the json: https://codebeautify.org/jsonviewer/cb0d0a91
Trying to scrape a reddit thread:
import requests
import json
url ="https://www.reddit.com/r/easternshoremd/comments/72u501/going_to_be_in_the_easton_area_for_work_next_week.json"
r = requests.get(url, headers={'User-agent': 'Chrome'})
d = r.json()
scrapedids = []
for child in d['data']['children']:
scrapedids.append(child['data']['author'])
print (scrapedids)
If I switch the url from a reddit post to the subreddit then it works. For example, if I set
url = ("https://www.reddit.com/r/easternshoremd.json")
I believe the issue is my lack of understanding in the directory/tree (whatever it's called) of json. I've been hung up for a few hours and appreciate any assistance.
The error:
Traceback (most recent call last):
File "/home/usr/PycharmProjects/untitled/delete.py", line 14, in
for child in d['data']['children']:
TypeError: list indices must be integers or slices, not str

You included a link to the JSON, which is good. It shows that the root is an array.
Therefore your code should look more like:
import requests
import json
url ="https://www.reddit.com/r/easternshoremd/comments/72u501/going_to_be_in_the_easton_area_for_work_next_week.json"
r = requests.get(url, headers={'User-agent': 'Chrome'})
listings = r.json()
scrapedids = []
for listing in listings:
for child in listing['data']['children']:
scrapedids.append(child['data']['author'])
print (scrapedids)
Note that I renamed d to listings which relates to the kind attribute ('listing').