I am writing a python code to scrape website data through cURL. I converted cURL into python code using https://curlconverter.com/ . The code works just fine but I want to customize according to my need like in this line of code
data = '{"appDate":{"startDate":"2022-01-05T18:30:00.000Z","endDate":"2022-01-06T18:30:00.000Z"},"page_number":1,"page_size":20,"sort":{"key":"AppointmentStartTime","order":-1}}'
After "startDate" I want to add my variable (startdate) which I created like this
variable code
I tried to add variables like this
data = '{"appDate":{"startDate":'+ startdate +,"endDate":'+ enddate +'},"page_number":1,"page_size":20,"sort":{"key":"AppointmentStartTime","order":-1}}' but this did not work.
Also adding '+ str(startdate) +' did not help.
Please can anyone tell me how this should be done.
You might want to transform the json data string into a dictionary using the json module. Then you can freely manipulate and export your data.
import json
raw_data = '{"appDate":{"startDate":"2022-01-05T18:30:00.000Z","endDate":"2022-01-06T18:30:00.000Z"},"page_number":1,"page_size":20,"sort":{"key":"AppointmentStartTime","order":-1}}'
data = json.loads(raw_data) # load json from string to dict
data['appDate']['startDate'] = 23
data['appDate']['endDate'] = 42
print(json.dumps(data)) # export dict to json string
In the example you have shown, there is probably only a small mistake right after + startdate +, with the apostrophe. Compare carefully:
Your code (with the mistake):
data = '{"appDate":{"startDate":'+ startdate +,"endDate":'+ enddate +'},"page_number":1,"page_size":20,"sort":{"key":"AppointmentStartTime","order":-1}}'
^
SyntaxError: invalid syntax
Fixed code:
data = '{"appDate":{"startDate":'+ startdate +',"endDate":'+ enddate +'},"page_number":1,"page_size":20,"sort":{"key":"AppointmentStartTime","order":-1}}'
Related
I have a json config file where I store my path to data there
The data is bucketed in month and days, so without the json I would use an f-string like:
spark.read.parquet(f"home/data/month={MONTH}/day={DAY}")
Now I want to extract that from json. However, I run into problems with the Month and day variable. I do not want to split the path in the json.
But writing it like this:
{
"path":"home/data/month={MONTH}/day={DAY}"
}
and loading with:
DAY="1"
MONTH="12"
conf_path=pandas.read_json("...")
path=conf_path["path"]
data=spark.read_parquet(f"{path}")
does not really work.
Could you hint me a solution to retrieving a path with variable elements and filling them after reading? How would you store the path or retrieve it without splitting the path? Thanks
------- EDIT: SOLUTION --------
Thanks to Deepak Tripathi answer below, the answer is to use string format.
with the code like this:
day="1"
month="12"
conf_path=pandas.read_json("...")
path=conf_path["path"]
data=spark.read_parquet(path.format(MONTH=month, DAY=day))
you should use string.format() instead of f-strings
Still if you want to use f-strings then you should use eval like this, its unsafe
DAY="1"
MONTH="12"
df = pd.DataFrame(
[{
"path":"home/data/month={MONTH}/day={DAY}"
},
{
"path":"home/data/month={MONTH}/day={DAY}"
}
]
)
a = df['path'][0]
print(eval(f"f'{a}'"))
#home/data/month=12/day=1
I have this very long json here: https://textup.fr/601885q4 and would like to read a data that is in one of the "payment_token_contract" specifically those with "id":1
My problem is that I don't get how to call the specific dictionary as they all have the same name. Is this even possible, I'm not used to manipulating such complex objects as I'm a beginner.
I would have tried something like:
["orders][x]["id":1]["base_price"]
with x being in a for loop that iterates through each "orders" present.
But I can't manage to put it all together. Thanks for your help !
You can use a for loop to iterate over the orders, you can check the value of the payment contract id and if its 1 then print the base price for that order
import json
jdata = "yourjson"
jdict = json.loads(jdata)
for order in jdict["orders"]:
if order['payment_token_contract']['id'] == 1:
print(order["base_price"])
I have omited the json data as its to long but you can image jdata is the string of your json
OUTPUT
149000000000000000000
I'm using an API that is giving me and output formatted as
['{"quote":{"symbol":"AAPL"', '"companyName":"Apple Inc."', '"primaryExchange":"Nasdaq Global Select"', '"sector":"Technology"', '"calculationPrice":"close"', '"open":367.88', '"openTime":1593696600532', '"close":364.11', '"closeTime":1593720000277', '"high":370.47', '"low":363.64', '"latestPrice":364.11'}]
...(it keeps going like this with many more categories.)
I am attempting to pull out only the latest price. What would be the best way to do that?
This is what I have but I get a bunch of errors.
string = (data.decode("utf-8"))
data_v = string.split(',')
for word in data_v[latestPrice]:
if word == ["latestPrice"]:
print(word)
print(data_v)
Judging by the output this is JSON. To parse this easily use the JSON module (see https://docs.python.org/3/library/json.html ).
If I'm correct you got this output from Yahoo Finance, if this indeed the case don't fetch and parse it manually but use the yfinance module (see https://pypi.org/project/yfinance/ )
You will have to use JSON module to parse this JSON string. You can convert it into dictionary then. I have indented the JSON code for ease of understanding. You can use the following approach,
import json
text_to_parse = """
{"quote":
{
"symbol":"AAPL",
"companyName":"Apple Inc.",
"primaryExchange":"Nasdaq Global Select",
"sector":"Technology",
"calculationPrice":"close",
"open":367.88,
"openTime":1593696600532,
"close":364.11,
"closeTime":1593720000277,
"high":370.47,
"low":363.64,
"latestPrice":364.11
}
}
"""
parsed_dict = json.loads(text_to_parse)
print(parsed_dict["quote"]["latestPrice"])
When the program is run, it outputs 364.11
I want to filter the json that operatinSystem are linux ,and I have some problem with it,the part of json in
'' : {
that I don't know how dictionary represent it and
"DQ578CGN99KG6ECF" : {
how can I represent it with wildcard, anyone could help my please.
import json
import urllib2
response=urllib2.urlopen('https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/current/index.json')
url=response.read()
urlj=json.loads(url)
filterx=[x for x in urlj if x['??']['??']["attributes"]["operatingSystem"] == 'linux']
I'm not sure about the wildcard representation. I'll look into it and get back to you. Meanwhile, I have already worked with this json before so I can tell you how to access the information you need.
The information you need can be obtained as follows:
for each_product in urlx['products']:
if urlx['products'][each_product]['attributes']['operatingSystem']=="linux":
#your code here
If you need pricing information from the json you need to take the product code string and look into the priceDimensions field for it. Look at the sample json and code accordingly.
https://aws.amazon.com/blogs/aws/new-aws-price-list-api/
I have below JSON String. Now I want to extract each individual field from that JSON string.
So I decided to create a method parse_json which will accept a variable that I want to extract from the JSON String.
Below is my python script -
#!/usr/bin/python
import json
jsonData = '{"pp": [0,3,5,7,9], "sp": [1,2,4,6,8]}'
def parse_json(data):
jj = json.loads(jsonData)
return jj['+data+']
print parse_json('pp')
Now whenever I an passing pp to parse_json method to extract its value from the JSON String, I always get the error as -
return jj['+data+']
KeyError: '+data+'
Any idea how to fix this issue? As I need to pass the variable which I am supposed to extract from the JSON String?
You probably just want this:
return jj[data]
Your code is trying to look up a key named literally '+data+', when instead what you want to do is look up the key with a name of the function's parameter.
Just use data parameter itself.
Replace following line:
return jj['+data+'] # lookup with `+data+`, not `pp`
with:
return jj[data]