I have some text data
>>> print content
Date,Open,High,Low,Close,Volume,Adj Close
2015-03-17,4355.83,4384.98,4349.69,4375.63,1724370000,4375.63
2015-03-16,4338.29,4371.46,4327.27,4370.47,1713480000,4370.47
2015-03-13,4328.09,4347.87,4289.30,4314.90,1851410000,4314.90
2015-03-12,4302.73,4339.20,4300.87,4336.23,1855110000,4336.23
2015-03-11,4336.05,4342.87,4304.28,4305.38,1846020000,4305.38
Now I want to convert this into a Dict, so that I can load this into a database using the cursor.executemany that allows me to provide dict as an input.
Is there a module to auto convert this into a Dict. I looked at Numpy - loadtext but that requires me to write this first to a file. Is there a way that i can do this without creating a file?
Use csv.DictReader
>>> with open('text.txt') as f:
... dreader = csv.DictReader(f)
... for row in dreader:
... print(row)
...
{'Adj Close': '4375.63', 'High': '4384.98', 'Volume': '1724370000', 'Low': '4349.69', 'Close': '4375.63', 'Open': '4355.83', 'Date': '2015-03-17'}
{'Adj Close': '4370.47', 'High': '4371.46', 'Volume': '1713480000', 'Low': '4327.27', 'Close': '4370.47', 'Open': '4338.29', 'Date': '2015-03-16'}
{'Adj Close': '4314.90', 'High': '4347.87', 'Volume': '1851410000', 'Low': '4289.30', 'Close': '4314.90', 'Open': '4328.09', 'Date': '2015-03-13'}
{'Adj Close': '4336.23', 'High': '4339.20', 'Volume': '1855110000', 'Low': '4300.87', 'Close': '4336.23', 'Open': '4302.73', 'Date': '2015-03-12'}
{'Adj Close': '4305.38', 'High': '4342.87', 'Volume': '1846020000', 'Low': '4304.28', 'Close': '4305.38', 'Open': '4336.05', 'Date': '2015-03-11'}
I looked at Numpy - loadtext but that requires me to write this first to a file. Is there a way that I can do this without creating a file?
You can use a file like object if you do not want to have physical data.
Use tempfile.TemporaryFile
from tempfile import TemporaryFile
with TemporaryFile('w+t') as flike:
flike.write(content)
flike.seek(0)
dreader = csv.DictReader(flike)
for row in dreader:
#do something
Use io.StringIO
import io #python3 or import StringIO in python2
flike = io.StringIO(content)
for row in csv.DictReader(flike)
#do something
Related
I am using this python library to grab a response from the google places API.
Using the places function from the above library returns the object you see below. You can see that is just gives me a boolean on whether the restaurant is open or closed. How can I see the hours it is open for each day of the week? If this library isnt capable of it, can anyone show me an example that is?
Here is the line of code making the request for full context.
import googlemaps # https://googlemaps.github.io/google-maps-services-python/docs/index.html
gmaps = googlemaps.Client(key='apiKey')
response = gmaps.places(query=charlestonBars[idx], location=charleston)
[ { 'business_status': 'OPERATIONAL',
'formatted_address': '467 King St, Charleston, SC 29403, United States',
'geometry': { 'location': {'lat': 32.7890988, 'lng': -79.9386229},
'viewport': { 'northeast': { 'lat': 32.79045632989271,
'lng': -79.93725907010727},
'southwest': { 'lat': 32.78775667010727,
'lng': -79.93995872989272}}},
'icon': 'https://maps.gstatic.com/mapfiles/place_api/icons/v1/png_71/bar-71.png',
'icon_background_color': '#FF9E67',
'icon_mask_base_uri': 'https://maps.gstatic.com/mapfiles/place_api/icons/v2/bar_pinlet',
'name': "A.C.'s Bar & Grill",
------->'opening_hours': {'open_now': True},
'photos': [ { 'height': 1816,
'html_attributions': [ '<a '
'href="https://maps.google.com/maps/contrib/106222166168758671498">Lisa '
'Royce</a>'],
'photo_reference': 'ARywPAKpP_eyNL_y625xWYrQvSjAI91TzEx4XgT1rwCxjFyQjEAwZb2ha9EgE2RcKJalrZhjp0yTWa6QqvPNU9c7GeNBTDtzXVI0rHq2RXtTySGu8sjcB76keFugOmsl1ix4NnDVh0NO0vt_PO3nIZ-R-ytOzzIRhJgPAJd3SxKNQNfEyVp5',
'width': 4032}],
'place_id': 'ChIJk6nSrWt6_ogRE9KiNXVG5KA',
'plus_code': { 'compound_code': 'Q3Q6+JG Charleston, South Carolina',
'global_code': '8742Q3Q6+JG'},
'price_level': 1,
'rating': 4.3,
'reference': 'ChIJk6nSrWt6_ogRE9KiNXVG5KA',
'types': [ 'bar',
'restaurant',
'point_of_interest',
'food',
'establishment'
What you need to use is the Place Details of Places API
Before we proceed with the solution, we must understand that there's a difference in the result of a Place Search and a Place Details request.
According to the documentation:
"Place Search requests and Place Details requests do not return the same fields. Place Search requests return a subset of the fields that are returned by Place Details requests. If the field you want is not returned by Place Search, you can use Place Search to get a place_id, then use that Place ID to make a Place Details request."
Now what you used on your code according to the python library documentation is the places(*args, **kwargs) which is the Place Search.
The Google Maps API documentation you provided on your comment above where you can get the expected result of hours each day is from Place Details which is the place(*args, **kwargs) from the python library documentation.
As quoted above, to request the detail of a place, you need the place_id which you can get by doing a Places Search request like you did on your question. So what you only need to do is get the place_id of the location you want through Place Search, then use that place_id to get the Place Details result that includes the opening_hours field result.
Here's what it looks like on a python code:
# I used pprint to print the result on the console
from pprint import pprint
import googlemaps #import googlemaps python library
# Instantiate the client using your own API key
API_KEY = 'API_KEY'
map_client = googlemaps.Client(API_KEY)
# Store the location you want, in my case, I tried using 'Mall of Asia'
location_name = 'Mall of Asia'
# Instantiate Place Search request using `places(*args, **kwargs)` from the library
# Use the `stored location` as an argument for query
place_search = map_client.places(location_name)
# Get the search results
search_results = place_search.get('results')
# Store the place_id from the result to be used
place_id = (search_results[0]['place_id'])
# Instantiate Place Details request using the `place(*args, **kwargs)` from the library
# Use the stored place_id as an argument for the request
place_details = map_client.place(place_id)
# Get the Place Details result
details_results = place_details.get('result')
# Print the result specifying what you only need which is the `opening_hours` field
pprint(details_results['opening_hours'])
The result of this sample request would be this:
{'open_now': False,
'periods': [{'close': {'day': 0, 'time': '2200'},
'open': {'day': 0, 'time': '1000'}},
{'close': {'day': 1, 'time': '2200'},
'open': {'day': 1, 'time': '1000'}},
{'close': {'day': 2, 'time': '2200'},
'open': {'day': 2, 'time': '1000'}},
{'close': {'day': 3, 'time': '2200'},
'open': {'day': 3, 'time': '1000'}},
{'close': {'day': 4, 'time': '2200'},
'open': {'day': 4, 'time': '1000'}},
{'close': {'day': 5, 'time': '2200'},
'open': {'day': 5, 'time': '1000'}},
{'close': {'day': 6, 'time': '2200'},
'open': {'day': 6, 'time': '1000'}}],
'weekday_text': ['Monday: 10:00\u202fAM\u2009–\u200910:00\u202fPM',
'Tuesday: 10:00\u202fAM\u2009–\u200910:00\u202fPM',
'Wednesday: 10:00\u202fAM\u2009–\u200910:00\u202fPM',
'Thursday: 10:00\u202fAM\u2009–\u200910:00\u202fPM',
'Friday: 10:00\u202fAM\u2009–\u200910:00\u202fPM',
'Saturday: 10:00\u202fAM\u2009–\u200910:00\u202fPM',
'Sunday: 10:00\u202fAM\u2009–\u200910:00\u202fPM']}
That would be all and I hope this helps! Feel free to comment below if this does not meet your expected results.
I want to write this to a Sheet with one Col as "Date" and the other as "Close". How do I go about doing this? New to Python so trying to learn and practice. I have a service account and am able to read from a sheet. This is the data from one sheet that I want to write to another sheet.
[{'Date': '1/4/2021 16:00:00', 'Close': 3186.63},
{'Date': '1/5/2021 16:00:00', 'Close': 3218.51},
{'Date': '1/6/2021 16:00:00', 'Close': 3138.38},
{'Date': '1/7/2021 16:00:00', 'Close': 3162.16},
{'Date': '1/8/2021 16:00:00', 'Close': 3182.7},
{'Date': '1/11/2021 16:00:00', 'Close': 3114.21}]
import pandas as pd
from apiclient import discovery
import pygsheets
from google.oauth2 import service_account
list_data=[
{'Date': '1/4/2021 16:00:00', 'Close': 3186.63},
{'Date': '1/5/2021 16:00:00', 'Close': 3218.51},
{'Date': '1/6/2021 16:00:00', 'Close': 3138.38},
{'Date': '1/7/2021 16:00:00', 'Close': 3162.16},
{'Date': '1/8/2021 16:00:00', 'Close': 3182.7},
{'Date': '1/11/2021 16:00:00', 'Close': 3114.21}
]
df_data=pd.DataFrame(list_data)
SCOPES = ('https://www.googleapis.com/auth/spreadsheets', 'https://www.googleapis.com/auth/drive')
google_credentials= #your credential as a dict
my_credentials = service_account.Credentials.from_service_account_info(google_credentials, scopes=SCOPES)
gc = pygsheets.authorize(custom_credentials=my_credentials)
spreadsheet_key= #the id of the google sheet remember to allow modification
sh = gc.open_by_key(spreadsheet_key)
wks_name='sheet 1'
wks = sh.worksheet_by_title(wks_name)
wks.set_dataframe(df_data, (1, 1), overwrite=True,fit=True)
You can use pandas, pygsheets and google.oath2
Hello I need Output as SILVERM JUN FUT by parsing json response to symbol using Python
follow is the Json Response that I received
{'exchange': 'MCX', 'token': 221599, 'ltp': 71480.0, 'ltt': 1623072465, 'ltq': 1, 'volume': 20227, 'best_bid_price': 71476.0, 'best_bid_quantity': 2, 'best_ask_price': 71487.0, 'best_ask_quantity': 2, 'total_buy_quantity': 2086, 'total_sell_quantity': 2163, 'atp': 71205.04, 'exchange_time_stamp': 1623072466, 'open': 71380.0, 'high': 71550.0, 'low': 70890.0, 'close': 71585.0, 'yearly_high': 71550.0, 'yearly_low': 0.0, 'instrument': Instrument(exchange='MCX', token=221599, symbol='SILVERM JUN FUT', name='', expiry=datetime.date(2021, 6, 30), lot_size=None)}
as Instrument Data is contained in ( ) rounded brackets how can i parse beyond instrument object
update: It was not python dict() as per the person who removed his comment. but a named tuple
following solution worked for me:
P = response['instrument']
print(P.symbol)
I am new to Python. How can I store the following data as a list inside of another list?
inputList = [[{'timestamp': '2017-10-28T00:00:00.000Z', 'open': '0.051430', 'close': '0.051210', 'min': '0.050583', 'max': '0.051955', 'volume': '30953.184', 'volumeQuote': '1584.562468339'}, {'timestamp': '2017-10-29T00:00:00.000Z', 'open': '0.051191', 'close': '0.049403', 'min': '0.048843', 'max': '0.053978', 'volume': '42699.215', 'volumeQuote': '2190.567660769'}],[{'timestamp': '2017-10-28T00:00:00.000Z', 'open': '0.063390', 'close': '0.072991', 'min': '0.062544', 'max': '0.073524', 'volume': '199636.573', 'volumeQuote': '13427.870355674'}, {'timestamp': '2017-10-29T00:00:00.000Z', 'open': '0.072840', 'close': '0.073781', 'min': '0.069449', 'max': '0.090833', 'volume': '284448.623', 'volumeQuote': '21687.962221794'}]]
Output should be:
outputList = [[0.051210, 0.049403],[0.072991, 0.073781]]
and what I have so far is:
[0.051210, 0.049403, 0.072991, 0.073781]
I use the following code:
insideLoop = []
outputList = []
for list in inputList:
for i, v in enumerate(list):
closing = float(v['close'])
insideLoop.append(closing)
outputList.append(insideLoop)
To be noted that the inputList can be several lists long.
Any solution to this?
Thanks a lot!
You can use a simple list comprehension
result = [[x['close'], y['close']] for x, y in inputList]
print(result)
# - > [['0.051210', '0.049403'], ['0.072991', '0.073781']]
Update
For undetermined number of elements in sublist, use nested list comprehension
result = [[x['close'] for x in y] for y in inputList]
print(result)
# - > [['0.051210', '0.049403'], ['0.072991', '0.073781']]
You can use a nested list comprehension:
s = [[{'timestamp': '2017-10-28T00:00:00.000Z', 'open': '0.051430', 'close': '0.051210', 'min': '0.050583', 'max': '0.051955', 'volume': '30953.184', 'volumeQuote': '1584.562468339'}, {'timestamp': '2017-10-29T00:00:00.000Z', 'open': '0.051191', 'close': '0.049403', 'min': '0.048843', 'max': '0.053978', 'volume': '42699.215', 'volumeQuote': '2190.567660769'}],[{'timestamp': '2017-10-28T00:00:00.000Z', 'open': '0.063390', 'close': '0.072991', 'min': '0.062544', 'max': '0.073524', 'volume': '199636.573', 'volumeQuote': '13427.870355674'}, {'timestamp': '2017-10-29T00:00:00.000Z', 'open': '0.072840', 'close': '0.073781', 'min': '0.069449', 'max': '0.090833', 'volume': '284448.623', 'volumeQuote': '21687.962221794'}]]
new_s = [[float(i['close']) for i in b] for b in s]
Output:
[[0.051210, 0.049403], [0.072991, 0.073781]]
Try this:
result = []
for a in inputList:
res = []
for i in a:
res.append(i['close'])
result.append(res)
print(result)
I have this string:
{"type":"summary","symbol":"SPY","open":"267.09","high":"267.22",
"low":"265.6", "prevClose":"266.75","close":"265.66"}
{"type":"quote","symbol":"SPY","bid":265.38,"bidsz":3,
"bidexch":"Q","biddate":"1513293904000","ask":265.42,
"asksz":45,"askexch":"P","askdate":"1513294015000"}
{"type":"summary","symbol":"SPY","open":"267.09",
"high":"267.22","low":"265.6","prevClose":"266.75","close":"265.66"}
If I do:
type(string)
I get:
<type 'unicode'>
If I do:
type(firstString)
where firstString is just the first of the three parts of the string, I get:
<type 'unicode'>
With Python, how can I split it based on the external parentheses, such that from this one string, we obtain three strings, each one having the form "{ ... }"?
If there are no nested objects and newlines are not guaranteed between parts, as you've said, it's as simple as splitting on }:
your_string = '''{"type":"summary","symbol":"SPY","open":"267.09","high":"267.22",
"low":"265.6", "prevClose":"266.75","close":"265.66"}
{"type":"quote","symbol":"SPY","bid":265.38,"bidsz":3,
"bidexch":"Q","biddate":"1513293904000","ask":265.42,
"asksz":45,"askexch":"P","askdate":"1513294015000"}
{"type":"summary","symbol":"SPY","open":"267.09",
"high":"267.22","low":"265.6","prevClose":"266.75","close":"265.66"}'''
substrings = [part.strip() + "}" for part in your_string.split("}") if part.strip()]
# ['{"type":"summary","symbol":"SPY","open":"267.09","high":"267.22",
# "low":"265.6", "prevClose":"266.75","close":"265.66"}',
# '{"type":"quote","symbol":"SPY","bid":265.38,"bidsz":3,
# "bidexch":"Q","biddate":"1513293904000","ask":265.42,
# "asksz":45,"askexch":"P","askdate":"1513294015000"}',
# '{"type":"summary","symbol":"SPY","open":"267.09",
# "high":"267.22","low":"265.6","prevClose":"266.75","close":"265.66"}']
Or you can immediately parse the parts into individual Python dictionaries:
dicts = [json.loads(part + "}") for part in your_string.split("}") if part.strip()]
# [{'open': '267.09', 'high': '267.22', 'prevClose': '266.75', 'type': 'summary',
# 'close': '265.66', 'low': '265.6', 'symbol': 'SPY'},
# {'askdate': '1513294015000', 'bid': 265.38, 'asksz': 45, 'type': 'quote',
# 'ask': 265.42, 'bidsz': 3, 'bidexch': 'Q', 'biddate': '1513293904000',
# 'askexch': 'P', 'symbol': 'SPY'},
# {'open': '267.09', 'high': '267.22', 'prevClose': '266.75', 'type': 'summary',
# 'close': '265.66', 'low': '265.6', 'symbol': 'SPY'}]
You can split the string on the newline character and parse it using the json module.
import json
s = '''{"type":"summary","symbol":"SPY","open":"267.09","high":"267.22","low":"265.6","prevClose":"266.75","close":"265.66"}
{"type":"quote","symbol":"SPY","bid":265.38,"bidsz":3,"bidexch":"Q","biddate":"1513293904000","ask":265.42,"asksz":45,"askexch":"P","askdate":"1513294015000"}
{"type":"summary","symbol":"SPY","open":"267.09","high":"267.22","low":"265.6","prevClose":"266.75","close":"265.66"}'''
[json.loads(line) for line in s.split('\n')]
# returns:
[{'close': '265.66',
'high': '267.22',
'low': '265.6',
'open': '267.09',
'prevClose': '266.75',
'symbol': 'SPY',
'type': 'summary'},
{'ask': 265.42,
'askdate': '1513294015000',
'askexch': 'P',
'asksz': 45,
'bid': 265.38,
'biddate': '1513293904000',
'bidexch': 'Q',
'bidsz': 3,
'symbol': 'SPY',
'type': 'quote'},
{'close': '265.66',
'high': '267.22',
'low': '265.6',
'open': '267.09',
'prevClose': '266.75',
'symbol': 'SPY',
'type': 'summary'}]