I have read numerous StackOverflow threads about looping during try/except statements, using else and finally, if/else statements, and while statements, but none of them address what I want. That or I don't know how to utilise that information to get what I want done.
Basically, I am trying to get adjusted closing stock prices for various companies on a given date. I pasted some dummy data in the code block below to demonstrate (NOTE: you'll have to install pandas and pandas_datareader to get the dummy code to run). The get_stock_adj_close function returns the adj_close price given a ticker and date. The dummy_dataframe contains 4 companies with their tickers and random dates. And the add_days function takes a date and adds any number of days. I would like to append the adjusted close stock prices for each company in the dataframe on the listed date into the stock_prices list.
Because the yahoo stock price database isn't that reliable for older entries and because some dates fall on days when the market is closed, whenever a price isn't available it raises a KeyError: 'Date'. Thus, what I would like to do is keep adding days indefinitely until it finds a date where a price does exist. The problem is it only adds the day once and then raises the same KeyError. I want it to keep adding days until it finds a day where the database has a stock price available and then return back to the dataframe and keep going with the next row. Right now the whole thing breaks on the first GM date (fourth row), which raises the KeyError and the fifth row/second GM date is ignored. Any help is appreciated!
Dummy data:
from datetime import datetime, date, timedelta
import pandas as pd
import pandas_datareader as pdr
from dateutil.relativedelta import relativedelta
def add_days(d, num_days):
return d + timedelta(days=num_days)
def get_stock_adj_close(ticker, chosen_date):
stock_df = pdr.get_data_yahoo(ticker, start = chosen_date, end = chosen_date)
return stock_df.iloc[0]['Adj Close']
d = {'TICKER': ['AMD','AMD','CHTR','GM'], 'DATE': [datetime(2020,2,4), datetime(2019,2,8),datetime(2019,1,31), datetime(2010,4,7)]}
dummy_dataframe = pd.DataFrame(data=d)
stock_prices = []
for i, row in dummy_dataframe.iterrows():
given_date = row['DATE']
try:
stock_price = get_stock_adj_close(row['TICKER'], given_date)
print(stock_price)
stock_prices.append(stock_price)
except KeyError:
given_date = add_days(given_date,1)
stock_price = get_stock_adj_close(row['TICKER'], given_date)
stock_prices.append(stock_price)
print(stock_prices)
I think while loop will help you. For example:
for i, row in dummy_dataframe.iterrows():
given_date = row['DATE']
stock_price_found = False
while not stock_price_found:
try:
stock_price = get_stock_adj_close(row['TICKER'], given_date)
print(stock_price)
stock_prices.append(stock_price)
stock_price_found = False
except KeyError:
given_date = add_days(given_date,1)
Or you can also use while True together with break:
for i, row in dummy_dataframe.iterrows():
given_date = row['DATE']
while True:
try:
stock_price = get_stock_adj_close(row['TICKER'], given_date)
print(stock_price)
stock_prices.append(stock_price)
break
except KeyError:
given_date = add_days(given_date,1)
Don't forget to make sure that you are not stuck in indefinite loop, would be also helpful some other exit conditions from while loop, for example, after 10 failures.
Related
I'm trying to gather dividend yields from multiple stocks via yfinance. I have a loop which creates a CSV-file for each ticker with historical data.
When I've downloaded dividend data via a function previously, it has worked - basically I created a function with a for-loop and then appended a dataframe with the stocks.
However, now I want to do it the same way but with a boolean expression instead, and it's not working.. I'm not getting any errors but I'm not receiving any ticker symbols (which I know satisfy the condition). I've tried to formulate the boolean loop differently, without success.
What am I doing wrong? Below is my code:
import yfinance as yf
import pandas as pd
import os
df = pd.read_csv(r'C:\\Users\Name\Stocks\Trading\teststocks.csv')
tickers = df["Symbol"].tolist()
i=0
listlength = len(tickers)
for ticker in tickers:
i=i+1
print("Downloading data for",ticker,",",i,"of",listlength)
df = yf.download(ticker, period = "max", interval = "1wk", rounding = True)
df.dropna(inplace=True)
df.to_csv(os.path.join("C:\\Users\Name\Stocks\dataset",ticker + ".csv"))
def dividend(df):
info = yf.Ticker(ticker).info
div = info.get("dividendYield")
if div is None:
pass
elif div > 0.04:
return True
else:
return False
for filename in os.listdir("C:\\Users\Name\Stocks\dataset"):
df = pd.read_csv("C:\\Users\Name\Stocks\dataset\{}".format(filename))
if dividend(df):
print("{}".format(filename))
So this function is looping through the ticker symbols from the dataset folder and getting the dividend data from yfinance, however it's not returning with the ticker that satisfy the condition - which in this case is if the dividend yield is higher than 4%. The first dataframe being read is a CSV file with the ticker symbols in the OMXS30 - so for example HM-B.ST should appear from the dividend function..
Another thing that I want to add is that I'm using the same logic for a function for marketcap, which does work. See below:
def marketcap(df):
info = yf.Ticker(ticker).info
mcap = info.get("marketCap")
if mcap is None:
pass
elif mcap > 10000000000:
return True
else:
return False
for filename in os.listdir("C:\\Users\Name\Stocks\dataset"):
df = pd.read_csv("C:\\Users\Name\Stocks\dataset\{}".format(filename))
if marketcap(df):
print("{}".format(filename))
I do not know why the dividend boolean expression does not work, when the marketcap does work.
Thanks in advance.
Neither the function dividend nor marketcap is working as it should. The reason has to do with the following:
for ticker in tickers:
# do stuff
Here you are taking a list of tickers and doing some stuff for each ticker in this list. This means that by the end of your loop, the variable ticker equals the last item in the list. E.g. suppose tickers = ['HM-B.ST','AAPL'], then ticker will at the end equal AAPL.
Now, let's have a look at your function dividend:
def dividend(df):
info = yf.Ticker(ticker).info
div = info.get("dividendYield")
if div is None:
pass
elif div > 0.04:
return True
else:
return False
This function has one argument (df), but it is not actually using it. Instead you are applying yf.Ticker(...).info to a variable ticker, which is no longer being updated at all. If the function is not returning any True values, this must simply mean that the last ticker (e.g. "AAPL") does not represent a dividend stock. So, to fix this you want to change the input for the function: def dividend(ticker). Write something like:
for filename in os.listdir("C:\\Users\Name\Stocks\dataset"):
df = pd.read_csv("C:\\Users\Name\Stocks\dataset\{}".format(filename))
# e.g. with filename like "HM-B.ST.csv", split at "."
# and select only first part
ticker = filename.split('.')[0]
if dividend(ticker):
print("{}".format(filename))
You need to make the same change for your function marketcap. Again, if this function is currently returning True values, this just means that your last list item references a stock has a higher mcap than the threshold.
Edit: Suggested refactored code
import yfinance as yf
import pandas as pd
tickers = ['ABB.ST','TELIA.ST','ELUX-B.ST','HM-B.ST']
def buy_dividend(ticker):
info = yf.Ticker(ticker).info
# keys we need
keys = ['marketCap','trailingPE','dividendYield']
# store returned vals in a `list`. E.g. for 'HM-B.ST':
# [191261163520, 13.417525, 0.0624], i.e. mcap, PE, divYield
vals = [info.get(key) for key in keys]
# if *any* val == `None`, `all()` will be `False`
if all(vals):
# returns `True` if *all* conditions are met, else `False`
return (vals[0] > 1E10) & (vals[1] < 20) & (vals[2] > 0.04)
return False
for ticker in tickers:
# `progress=False` suppresses the progress print
df = yf.download(ticker, period = "max", interval = "1wk",
rounding = True, progress = False)
df.dropna(inplace=True)
if df.empty:
continue
# df.to_csv(os.path.join("C:\\Users\Name\Stocks\dataset",ticker + ".csv"))
# get last close & mean from column `df.Close`
last_close = df.loc[df.index.max(),'Close']
mean = df.Close.mean()
if last_close < mean:
if buy_dividend(ticker):
print("{} is a good buy".format(ticker))
else:
print("{} is not a good buy".format(ticker))
This will print:
TELIA.ST is not a good buy
ELUX-B.ST is a good buy
HM-B.ST is a good buy
# and will silently pass over 'ABB.ST', since `(last_close < mean) == False` here
New function looks like this:
def buy_dividend(ticker):
if df.empty:
pass
else:
last_close = df[-1:]['Close'].values[0]
mean = df["Close"].mean()
if last_close < mean:
info = yf.Ticker(ticker).info
mcap = info.get("marketCap")
if mcap is None:
pass
elif mcap > 1E10:
PE = info.get('trailingPE')
if PE is None:
pass
elif PE < 20:
div = info.get("dividendYield")
if div is None:
pass
elif div > 0.04:
return True
for filename in os.listdir("C:\\Users\Andreas\Aktieanalys\dataset"):
df = pd.read_csv("C:\\Users\Andreas\Aktieanalys\dataset\{}".format(filename))
if buy_dividend(ticker):
print("{} is a good buy".format(filename))
But somehow the dividend yield are messing things up. If the rows containing "div" are being #, then the function works perfect and correctly. Why is that?
I am trying to scrape stock market information from all the stocks concerning the FTSE250.
I use Yahoo_Fin in order to do so. The code works, but I get an error that a stock is delisted.
Hence, I tried to put-in an except exception code. I read the documentation about the try and except libraries, but could not find the correct answer. I don't receive a syntax error, but the except exception code doesn't do anything.
EDIT: Putting two except exceptions works, below is the updated code.
index_df = pdr.get_data_yahoo(index_name, start_date, end_date, progress=False)
index_df['Percent Change'] = index_df['Adj Close'].pct_change()
index_return = (index_df['Percent Change'] + 1).cumprod()[-1]
for ticker in tickers:
# Download historical data as CSV for each stock (makes the process faster)
try:
df = pdr.get_data_yahoo(ticker, start_date, end_date,progress=False)
df.to_csv(f'{ticker}.csv')
except:
except Exception:
if ticker not in tickers:
next(ticker)
for ticker in tickers:
try:
# Calculating returns relative to the market (returns multiple)
df['Percent Change'] = df['Adj Close'].pct_change()
stock_return = (df['Percent Change'] + 1).cumprod()[-1]
returns_multiple = round((stock_return / index_return), 2)
returns_multiples.extend([returns_multiple])
except Exception:
if ticker not in tickers:
next(ticker)
Your exception doesn't make sense to me. You are looping over the tickers, then you check if the ticker is not in tickers. That will always be False, so your "next" statement is never going to be executed and it will just continue on.
Seems what you want is something like this:
for ticker in tickers:
# Download historical data as CSV for each stock (makes the process faster)
try:
df = pdr.get_data_yahoo(ticker, start_date, end_date,progress=False)
df.to_csv(f'{ticker}.csv')
# Calculating returns relative to the market (returns multiple)
df['Percent Change'] = df['Adj Close'].pct_change()
stock_return = (df['Percent Change'] + 1).cumprod()[-1]
returns_multiple = round((stock_return / index_return), 2)
returns_multiples.extend([returns_multiple])
#change the name of the stock index
print (f'Ticker: {ticker}; Returns Multiple against FTSE 250 : {returns_multiple}\n')
time.sleep(1)
except IndexError:
print(f"Error in ticker: {ticker}, skipping...")
Where you pass a proper except condition depending on what the traceback is (can't tell if it's FileNotFoundError or IndexError). But I think all you want to do is process the next ticker in the list?
I have a dataframe which looks like the following (Name of the first dataframe(image below) is relevantdata in the code):
I want the dataframe to be transformed to the following format:
Essentially, I want to get the relevant confirmed number for each Key for all the dates that are available in the dataframe. If a particular date is not available for a Key, we make that value to be zero.
Currently my code is as follows (A try/except block is used as some Keys don't have the the whole range of dates, hence a Keyerror occurs the first time you refer to that date using countrydata.at[date,'Confirmed'] for the respective Key, hence the except block will make an entry of zero into the dictionary for that date):
relevantdata = pandas.read_csv('https://raw.githubusercontent.com/open-covid-19/data/master/output/data_minimal.csv')
dates = relevantdata['Date'].unique().tolist()
covidcountries = relevantdata['Key'].unique().tolist()
data = dict()
data['Country'] = covidcountries
confirmeddata = relevantdata[['Date','Key','Confirmed']]
for country in covidcountries:
for date in dates:
countrydata = confirmeddata.loc[lambda confirmeddata: confirmeddata['Key'] == country].set_index('Date')
try:
if (date in data.keys()) == False:
data[date] = list()
data[date].append(countrydata.at[date,'Confirmed'])
else:
data[date].append(countrydata.at[date,'Confirmed'])
except:
if (date in data.keys()) == False:
data[date].append(0)
else:
data[date].append(0)
finaldf = pandas.DataFrame(data = data)
While the above code accomplished what I want in getting the dataframe in the format I require, it is way too slow, having to loop through every key and date. I want to know if there is a better and faster method to doing the same without having to use a nested for loop. Thank you for all your help.
I am fairly new to python and coding in general.
I have a big data file that provides daily data for the period 2011-2018 for a number of stock tickers (300~).
The data is a .csv file with circa 150k rows and looks as follows (short example):
Date,Symbol,ShortExemptVolume,ShortVolume,TotalVolume
20110103,AAWW,0.0,28369,78113.0
20110103,AMD,0.0,3183556,8095093.0
20110103,AMRS,0.0,14196,18811.0
20110103,ARAY,0.0,31685,77976.0
20110103,ARCC,0.0,177208,423768.0
20110103,ASCMA,0.0,3930,26527.0
20110103,ATI,0.0,193772,301287.0
20110103,ATSG,0.0,23659,72965.0
20110103,AVID,0.0,7211,18896.0
20110103,BMRN,0.0,21740,213974.0
20110103,CAMP,0.0,2000,11401.0
20110103,CIEN,0.0,625165,1309490.0
20110103,COWN,0.0,3195,24293.0
20110103,CSV,0.0,6133,25394.0
I have a function that allows me to filter for a specific symbol and get 10 observations before and after a specified date (could be any date between 2011 and 2018).
import pandas as pd
from datetime import datetime
import urllib
import datetime
def get_data(issue_date, stock_ticker):
df = pd.read_csv (r'D:\Project\Data\Short_Interest\exampledata.csv')
df['Date'] = pd.to_datetime(df['Date'], format="%Y%m%d")
d = df
df = pd.DataFrame(d)
short = df.loc[df.Symbol.eq(stock_ticker)]
# get the index of the row of interest
ix = short[short.Date.eq(issue_date)].index[0]
# get the item row for that row's index
iloc_ix = short.index.get_loc(ix)
# get the +/-1 iloc rows (+2 because that is how slices work), basically +1 and -1 trading days
short_data = short.iloc[iloc_ix-10: iloc_ix+11]
return [short_data]
I want to create a script that iterates a list of 'issue_dates' and 'stock_tickers'. The list (a .csv) looks as following:
ARAY,07/08/2017
ARAY,24/04/2014
ACETQ,16/11/2015
ACETQ,16/11/2015
NVLNA,15/08/2014
ATSG,29/09/2017
ATI,24/05/2016
MDRX,18/06/2013
MDRX,18/06/2013
AMAGX,10/05/2017
AMAGX,14/02/2014
AMD,14/09/2016
To break down my problem and question I would like to know how to do the following:
First, how do I load the inputs?
Second, how do I call the function on each input?
And last, how do I accumulate all the function returns in one dataframe?
To load the inputs and call the function for each row; iterate over the csv file and pass each row's values to the function and accumulate the resulting Seriesin a list.
I modified your function a bit: removed the DataFrame creation so it is only done once and added a try/except block to account for missing dates or tickers (your example data didn't match up too well). The dates in the second csv look like they are day/month/year so I converted them for that format.
import pandas as pd
import datetime, csv
def get_data(df, issue_date, stock_ticker):
'''Return a Series for the ticker centered on the issue date.
'''
short = df.loc[df.Symbol.eq(stock_ticker)]
# get the index of the row of interest
try:
ix = short[short.Date.eq(issue_date)].index[0]
# get the item row for that row's index
iloc_ix = short.index.get_loc(ix)
# get the +/-1 iloc rows (+2 because that is how slices work), basically +1 and -1 trading days
short_data = short.iloc[iloc_ix-10: iloc_ix+11]
except IndexError:
msg = f'no data for {stock_ticker} on {issue_date}'
#log.info(msg)
print(msg)
short_data = None
return short_data
df = pd.read_csv (datafile)
df['Date'] = pd.to_datetime(df['Date'], format="%Y%m%d")
results = []
with open('issues.csv') as issues:
for ticker,date in csv.reader(issues):
day,month,year = map(int,date.split('/'))
# dt = datetime.datetime.strptime(date, r'%d/%m/%Y')
date = datetime.date(year,month,day)
s = get_data(df,date,ticker)
results.append(s)
# print(s)
Creating a single DataFrame or table for all that info may be problematic especially since the date ranges are all different. Probably should ask a separate question regarding that. Its mcve should probably just include a few minimal Pandas Series with a couple of different date ranges and tickers.
I've used:
data = DataReader("yhoo", "yahoo", datetime.datetime(2000, 1, 1),
datetime.datetime.today())
in pandas (python) to get history data of yahoo, but it cannot show today's price (the market has not yet closed) how can I resolve such problem, thanks in advance.
import pandas
import pandas.io.data
import datetime
import urllib2
import csv
YAHOO_TODAY="http://download.finance.yahoo.com/d/quotes.csv?s=%s&f=sd1ohgl1vl1"
def get_quote_today(symbol):
response = urllib2.urlopen(YAHOO_TODAY % symbol)
reader = csv.reader(response, delimiter=",", quotechar='"')
for row in reader:
if row[0] == symbol:
return row
## main ##
symbol = "TSLA"
history = pandas.io.data.DataReader(symbol, "yahoo", start="2014/1/1")
print history.tail(2)
today = datetime.date.today()
df = pandas.DataFrame(index=pandas.DatetimeIndex(start=today, end=today, freq="D"),
columns=["Open", "High", "Low", "Close", "Volume", "Adj Close"],
dtype=float)
row = get_quote_today(symbol)
df.ix[0] = map(float, row[2:])
history = history.append(df)
print "today is %s" % today
print history.tail(2)
just to complete perigee's answer, it cost me quite some time to find a way to append the data.
Open High Low Close Volume Adj Close
Date
2014-02-04 180.7 181.60 176.20 178.73 4686300 178.73
2014-02-05 178.3 180.59 169.36 174.42 7268000 174.42
today is 2014-02-06
Open High Low Close Volume Adj Close
2014-02-05 178.30 180.59 169.36 174.420 7268000 174.420
2014-02-06 176.36 180.11 176.00 178.793 5199297 178.793
Find a way to work around, just use urllib to fetch the data with:
http://download.finance.yahoo.com/d/quotes.csv?s=yhoo&f=sd1ohgl1l1v
then add it to dataframe
This code uses the pandas read_csv method to get the new quote from yahoo, and it checks if the new quote is an update from the current date or a new date in order to update the last record in history or append a new record.
If you add a while(true) loop and a sleep around the new_quote section, you can have the code refresh the quote during the day.
It also has duplicate last trade price to fill in the Close and the Adjusted Close, given that intraday close and adj close are always the same value.
import pandas as pd
import pandas.io.data as web
def get_quote_today(symbol):
url="http://download.finance.yahoo.com/d/quotes.csv?s=%s&f=d1t1ohgl1vl1"
new_quote= pd.read_csv(url%symbol,
names=[u'Date',u'time',u'Open', u'High', u'Low',
u'Close', u'Volume', u'Adj Close'])
# generate timestamp:
stamp = pd.to_datetime(new_quote.Date+" "+new_quote.time)
new_quote.index= stamp
return new_quote.iloc[:, 2:]
if __name__ == "__main__":
symbol = "TSLA"
history = web.DataReader(symbol, "yahoo", start="2014/1/1")
print history.tail()
new_quote = get_quote_today(symbol)
if new_quote.index > history.index[-1]:
if new_quote.index[-1].date() == history.index[-1].date():
# if both quotes are for the first date, update history's last record.
history.iloc[-1]= new_quote.iloc[-1]
else:
history=history.append(new_quote)
history.tail()
So from trying this out and looking at the dataframe, it doesn't look too possible. You tell it to go from a specific day until today, yet the dataframe stops at may 31st 2013. This tells me that yahoo probably has not made it available for you to use in the past couple days or somehow pandas is just not picking it up. It is not just missing 1 day, it is missing 3.
If I do the following:
>>> df = DataReader("yhoo", "yahoo", datetime.datetime(2013, 6, 1),datetime.datetime.today())
>>> len(df)
0
it shows me that there simply is no data to pick up in those days so far. If there is some way around this then I cannot figure it out, but it just seems that the data is not available for you yet, which is hard to believe.
The module from pandas doesn't work anymore, because the google and yahoo doens't provide support anymore. So you can create a function to take the data direct from the Google Finance using the url. Here is a part of a code to do this
import csv
import datetime
import re
import codecs
import requests
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
You can wrote a function to get data from Google Finance using the url, you have to indent the parte bellow.
#You have to indent this part
def get_google_finance_intraday(ticker, period=60, days=1, exchange='NASD'):
"""
Retrieve intraday stock data from Google Finance.
Parameters
----------------
ticker : str
Company ticker symbol.
period : int
Interval between stock values in seconds.
i = 60 corresponds to one minute tick data
i = 86400 corresponds to daily data
days : int
Number of days of data to retrieve.
exchange : str
Exchange from which the quotes should be fetched
Returns
---------------
df : pandas.DataFrame
DataFrame containing the opening price, high price, low price,
closing price, and volume. The index contains the times associated with
the retrieved price values.
"""
# build url
url = 'https://finance.google.com/finance/getprices?p={days}d&f=d,o,h,l,c,v&q={ticker}&i={period}&x={exchange}'.format(ticker=ticker, period=period, days=days, exchange=exchange)
page = requests.get(url)
reader = csv.reader(codecs.iterdecode(page.content.splitlines(), "utf-8"))
columns = ['Open', 'High', 'Low', 'Close', 'Volume']
rows = []
times = []
for row in reader:
if re.match('^[a\d]', row[0]):
if row[0].startswith('a'):
start = datetime.datetime.fromtimestamp(int(row[0][1:]))
times.append(start)
else:
times.append(start+datetime.timedelta(seconds=period*int(row[0])))
rows.append(map(float, row[1:]))
if len(rows):
return pd.DataFrame(rows, index=pd.DatetimeIndex(times, name='Date'), columns=columns)
else:
return pd.DataFrame(rows, index=pd.DatetimeIndex(times, name='Date'))
Now you can just call the function with the ticket that you want, in my case AAPL and the result is a pandas DataFrame containing the opening price, high price, low price, closing price, and volume.
ticker = 'AAPL'
period = 60
days = 1
exchange = 'NASD'
df = get_google_finance_intraday(ticker, period=period, days=days)
df
The simplest way to extract Indian stock price data into Python is to use the nsepy library.
In case you do not have the nsepy library do the following:
pip install nsepy
The following code allows you to extract HDFC stock price for 10 years.
from nsepy import get_history
from datetime import date
dfc=get_history(symbol="HDFCBANK",start=date(2015,5,12),end=date(2020,5,18))
This is so far the easiest code I have found.