I am trying to scrape stock market information from all the stocks concerning the FTSE250.
I use Yahoo_Fin in order to do so. The code works, but I get an error that a stock is delisted.
Hence, I tried to put-in an except exception code. I read the documentation about the try and except libraries, but could not find the correct answer. I don't receive a syntax error, but the except exception code doesn't do anything.
EDIT: Putting two except exceptions works, below is the updated code.
index_df = pdr.get_data_yahoo(index_name, start_date, end_date, progress=False)
index_df['Percent Change'] = index_df['Adj Close'].pct_change()
index_return = (index_df['Percent Change'] + 1).cumprod()[-1]
for ticker in tickers:
# Download historical data as CSV for each stock (makes the process faster)
try:
df = pdr.get_data_yahoo(ticker, start_date, end_date,progress=False)
df.to_csv(f'{ticker}.csv')
except:
except Exception:
if ticker not in tickers:
next(ticker)
for ticker in tickers:
try:
# Calculating returns relative to the market (returns multiple)
df['Percent Change'] = df['Adj Close'].pct_change()
stock_return = (df['Percent Change'] + 1).cumprod()[-1]
returns_multiple = round((stock_return / index_return), 2)
returns_multiples.extend([returns_multiple])
except Exception:
if ticker not in tickers:
next(ticker)
Your exception doesn't make sense to me. You are looping over the tickers, then you check if the ticker is not in tickers. That will always be False, so your "next" statement is never going to be executed and it will just continue on.
Seems what you want is something like this:
for ticker in tickers:
# Download historical data as CSV for each stock (makes the process faster)
try:
df = pdr.get_data_yahoo(ticker, start_date, end_date,progress=False)
df.to_csv(f'{ticker}.csv')
# Calculating returns relative to the market (returns multiple)
df['Percent Change'] = df['Adj Close'].pct_change()
stock_return = (df['Percent Change'] + 1).cumprod()[-1]
returns_multiple = round((stock_return / index_return), 2)
returns_multiples.extend([returns_multiple])
#change the name of the stock index
print (f'Ticker: {ticker}; Returns Multiple against FTSE 250 : {returns_multiple}\n')
time.sleep(1)
except IndexError:
print(f"Error in ticker: {ticker}, skipping...")
Where you pass a proper except condition depending on what the traceback is (can't tell if it's FileNotFoundError or IndexError). But I think all you want to do is process the next ticker in the list?
Related
This is my latest attempt:
Screenshot from my IDE
stockcode = pd.read_csv('D:\Book1.csv')
stockcode = stockcode.Symbol.to_list()
print(len(stockcode))
df = []
for i in stockcode:
df = isec.get_historical_data(
interval=time_interval,
from_date=from_date,
to_date=to_date,
stock_code=i,
exchange_code="NSE",
product_type="cash"
)
df = pd.DataFrame(df["Success"])
df.append(df)
df = pd.concat(df)
I am trying to fetch data of multiple stocks from brokers API but after few mins it throws an error.
how can it restart from the part where it stopped?
I tried some loop exception but of no use and the error thrown was Output exceeds the size limit
Screenshot from my IDE
If I understand you correctly, you want to restart with no looping through items you've already processed? In this case, I'd use manual retry in a code and looping through a copy of your original list without items which are already OK:
number_of_retries = 3
retry_counter = 0
while retry_counter < number_of_retries and not is_success:
try:
stockcode_process = stockcode.copy()
for i in stockcode_process:
# your for-loop body here with the additional last line as following to remove processed item:
stockcode.pop(i)
is_success = true
except Exception:
retry_counter += 1
I'm trying to gather dividend yields from multiple stocks via yfinance. I have a loop which creates a CSV-file for each ticker with historical data.
When I've downloaded dividend data via a function previously, it has worked - basically I created a function with a for-loop and then appended a dataframe with the stocks.
However, now I want to do it the same way but with a boolean expression instead, and it's not working.. I'm not getting any errors but I'm not receiving any ticker symbols (which I know satisfy the condition). I've tried to formulate the boolean loop differently, without success.
What am I doing wrong? Below is my code:
import yfinance as yf
import pandas as pd
import os
df = pd.read_csv(r'C:\\Users\Name\Stocks\Trading\teststocks.csv')
tickers = df["Symbol"].tolist()
i=0
listlength = len(tickers)
for ticker in tickers:
i=i+1
print("Downloading data for",ticker,",",i,"of",listlength)
df = yf.download(ticker, period = "max", interval = "1wk", rounding = True)
df.dropna(inplace=True)
df.to_csv(os.path.join("C:\\Users\Name\Stocks\dataset",ticker + ".csv"))
def dividend(df):
info = yf.Ticker(ticker).info
div = info.get("dividendYield")
if div is None:
pass
elif div > 0.04:
return True
else:
return False
for filename in os.listdir("C:\\Users\Name\Stocks\dataset"):
df = pd.read_csv("C:\\Users\Name\Stocks\dataset\{}".format(filename))
if dividend(df):
print("{}".format(filename))
So this function is looping through the ticker symbols from the dataset folder and getting the dividend data from yfinance, however it's not returning with the ticker that satisfy the condition - which in this case is if the dividend yield is higher than 4%. The first dataframe being read is a CSV file with the ticker symbols in the OMXS30 - so for example HM-B.ST should appear from the dividend function..
Another thing that I want to add is that I'm using the same logic for a function for marketcap, which does work. See below:
def marketcap(df):
info = yf.Ticker(ticker).info
mcap = info.get("marketCap")
if mcap is None:
pass
elif mcap > 10000000000:
return True
else:
return False
for filename in os.listdir("C:\\Users\Name\Stocks\dataset"):
df = pd.read_csv("C:\\Users\Name\Stocks\dataset\{}".format(filename))
if marketcap(df):
print("{}".format(filename))
I do not know why the dividend boolean expression does not work, when the marketcap does work.
Thanks in advance.
Neither the function dividend nor marketcap is working as it should. The reason has to do with the following:
for ticker in tickers:
# do stuff
Here you are taking a list of tickers and doing some stuff for each ticker in this list. This means that by the end of your loop, the variable ticker equals the last item in the list. E.g. suppose tickers = ['HM-B.ST','AAPL'], then ticker will at the end equal AAPL.
Now, let's have a look at your function dividend:
def dividend(df):
info = yf.Ticker(ticker).info
div = info.get("dividendYield")
if div is None:
pass
elif div > 0.04:
return True
else:
return False
This function has one argument (df), but it is not actually using it. Instead you are applying yf.Ticker(...).info to a variable ticker, which is no longer being updated at all. If the function is not returning any True values, this must simply mean that the last ticker (e.g. "AAPL") does not represent a dividend stock. So, to fix this you want to change the input for the function: def dividend(ticker). Write something like:
for filename in os.listdir("C:\\Users\Name\Stocks\dataset"):
df = pd.read_csv("C:\\Users\Name\Stocks\dataset\{}".format(filename))
# e.g. with filename like "HM-B.ST.csv", split at "."
# and select only first part
ticker = filename.split('.')[0]
if dividend(ticker):
print("{}".format(filename))
You need to make the same change for your function marketcap. Again, if this function is currently returning True values, this just means that your last list item references a stock has a higher mcap than the threshold.
Edit: Suggested refactored code
import yfinance as yf
import pandas as pd
tickers = ['ABB.ST','TELIA.ST','ELUX-B.ST','HM-B.ST']
def buy_dividend(ticker):
info = yf.Ticker(ticker).info
# keys we need
keys = ['marketCap','trailingPE','dividendYield']
# store returned vals in a `list`. E.g. for 'HM-B.ST':
# [191261163520, 13.417525, 0.0624], i.e. mcap, PE, divYield
vals = [info.get(key) for key in keys]
# if *any* val == `None`, `all()` will be `False`
if all(vals):
# returns `True` if *all* conditions are met, else `False`
return (vals[0] > 1E10) & (vals[1] < 20) & (vals[2] > 0.04)
return False
for ticker in tickers:
# `progress=False` suppresses the progress print
df = yf.download(ticker, period = "max", interval = "1wk",
rounding = True, progress = False)
df.dropna(inplace=True)
if df.empty:
continue
# df.to_csv(os.path.join("C:\\Users\Name\Stocks\dataset",ticker + ".csv"))
# get last close & mean from column `df.Close`
last_close = df.loc[df.index.max(),'Close']
mean = df.Close.mean()
if last_close < mean:
if buy_dividend(ticker):
print("{} is a good buy".format(ticker))
else:
print("{} is not a good buy".format(ticker))
This will print:
TELIA.ST is not a good buy
ELUX-B.ST is a good buy
HM-B.ST is a good buy
# and will silently pass over 'ABB.ST', since `(last_close < mean) == False` here
New function looks like this:
def buy_dividend(ticker):
if df.empty:
pass
else:
last_close = df[-1:]['Close'].values[0]
mean = df["Close"].mean()
if last_close < mean:
info = yf.Ticker(ticker).info
mcap = info.get("marketCap")
if mcap is None:
pass
elif mcap > 1E10:
PE = info.get('trailingPE')
if PE is None:
pass
elif PE < 20:
div = info.get("dividendYield")
if div is None:
pass
elif div > 0.04:
return True
for filename in os.listdir("C:\\Users\Andreas\Aktieanalys\dataset"):
df = pd.read_csv("C:\\Users\Andreas\Aktieanalys\dataset\{}".format(filename))
if buy_dividend(ticker):
print("{} is a good buy".format(filename))
But somehow the dividend yield are messing things up. If the rows containing "div" are being #, then the function works perfect and correctly. Why is that?
I have read numerous StackOverflow threads about looping during try/except statements, using else and finally, if/else statements, and while statements, but none of them address what I want. That or I don't know how to utilise that information to get what I want done.
Basically, I am trying to get adjusted closing stock prices for various companies on a given date. I pasted some dummy data in the code block below to demonstrate (NOTE: you'll have to install pandas and pandas_datareader to get the dummy code to run). The get_stock_adj_close function returns the adj_close price given a ticker and date. The dummy_dataframe contains 4 companies with their tickers and random dates. And the add_days function takes a date and adds any number of days. I would like to append the adjusted close stock prices for each company in the dataframe on the listed date into the stock_prices list.
Because the yahoo stock price database isn't that reliable for older entries and because some dates fall on days when the market is closed, whenever a price isn't available it raises a KeyError: 'Date'. Thus, what I would like to do is keep adding days indefinitely until it finds a date where a price does exist. The problem is it only adds the day once and then raises the same KeyError. I want it to keep adding days until it finds a day where the database has a stock price available and then return back to the dataframe and keep going with the next row. Right now the whole thing breaks on the first GM date (fourth row), which raises the KeyError and the fifth row/second GM date is ignored. Any help is appreciated!
Dummy data:
from datetime import datetime, date, timedelta
import pandas as pd
import pandas_datareader as pdr
from dateutil.relativedelta import relativedelta
def add_days(d, num_days):
return d + timedelta(days=num_days)
def get_stock_adj_close(ticker, chosen_date):
stock_df = pdr.get_data_yahoo(ticker, start = chosen_date, end = chosen_date)
return stock_df.iloc[0]['Adj Close']
d = {'TICKER': ['AMD','AMD','CHTR','GM'], 'DATE': [datetime(2020,2,4), datetime(2019,2,8),datetime(2019,1,31), datetime(2010,4,7)]}
dummy_dataframe = pd.DataFrame(data=d)
stock_prices = []
for i, row in dummy_dataframe.iterrows():
given_date = row['DATE']
try:
stock_price = get_stock_adj_close(row['TICKER'], given_date)
print(stock_price)
stock_prices.append(stock_price)
except KeyError:
given_date = add_days(given_date,1)
stock_price = get_stock_adj_close(row['TICKER'], given_date)
stock_prices.append(stock_price)
print(stock_prices)
I think while loop will help you. For example:
for i, row in dummy_dataframe.iterrows():
given_date = row['DATE']
stock_price_found = False
while not stock_price_found:
try:
stock_price = get_stock_adj_close(row['TICKER'], given_date)
print(stock_price)
stock_prices.append(stock_price)
stock_price_found = False
except KeyError:
given_date = add_days(given_date,1)
Or you can also use while True together with break:
for i, row in dummy_dataframe.iterrows():
given_date = row['DATE']
while True:
try:
stock_price = get_stock_adj_close(row['TICKER'], given_date)
print(stock_price)
stock_prices.append(stock_price)
break
except KeyError:
given_date = add_days(given_date,1)
Don't forget to make sure that you are not stuck in indefinite loop, would be also helpful some other exit conditions from while loop, for example, after 10 failures.
I am pulling data from an API that allows batch requests, and then storing the data to a Dataframe. When there is an exception with one of the items being looked up via the API, I want to either skip that item entirely, (or write zeroes to the Dataframe) and then go on to the next item.
But my issue is that because the API data is being accessed in bulk (i.e., not looping through each item in the list), an exception for any item in the list breaks the program. So how can I elegantly handle exceptions without looping through each individual item in the tickers list?
Note that removing ERROR from the tickers list will enable the program to run successfully:
import os
from iexfinance.stocks import Stock
import iexfinance
# Set IEX Finance API Token (Sandbox)
os.environ['IEX_API_VERSION'] = 'iexcloud-sandbox'
os.environ['IEX_TOKEN'] = 'Tpk_a4bc3e95d4c94810a3b2d4138dc81c5d'
# List of companies to get data for
tickers = ['MSFT', 'ERROR', 'AMZN']
batch = Stock(tickers, output_format='pandas')
income_ttm = 0
try:
# Get income from last 4 quarters, sum it, and store to temp Dataframe
df_income = batch.get_income_statement(period="year")
print(df_income)
except (iexfinance.utils.exceptions.IEXQueryError, iexfinance.utils.exceptions.IEXSymbolError) as e:
pass
This should do the work
import os
from copy import deepcopy
from iexfinance.stocks import Stock
import iexfinance
def find_wrong_symbol(tickers, err):
wrong_ticker = []
for one_ticker in tickers:
if one_ticker.upper() in err:
wrong_ticker.append(one_ticker)
return wrong_ticker
# Set IEX Finance API Token (Sandbox)
os.environ['IEX_API_VERSION'] = 'iexcloud-sandbox'
os.environ['IEX_TOKEN'] = 'Tpk_a4bc3e95d4c94810a3b2d4138dc81c5d'
# List of companies to get data for
tickers = ['MSFT', 'AMZN', 'failing']
batch = Stock(tickers, output_format='pandas')
income_ttm = 0
try:
# Get income from last 4 quarters, sum it, and store to temp Dataframe
df_income = batch.get_income_statement(period="year")
print(df_income)
except (iexfinance.utils.exceptions.IEXQueryError, iexfinance.utils.exceptions.IEXSymbolError) as e:
wrong_tickers = find_wrong_symbol(tickers, str(e))
tickers_to_get = deepcopy(tickers)
assigning_dict = {}
for wrong_ticker in wrong_tickers:
tickers_to_get.pop(tickers_to_get.index(wrong_ticker))
assigning_dict.update({wrong_ticker: lambda x: 0})
new_batch = Stock(tickers_to_get, output_format='pandas')
df_income = new_batch.get_income_statement(period="year").assign(**assigning_dict)
I create a small function in order to find tickers that are not handle by the API. After deleting the wrong tickers, I recall the API without it and with the assign function add the missing columns with the 0 values (it could be anything, a NaN or another default value).
I'm trying to use pandas to download historical stock data for all Stockholm Large Cap stocks. It works fine but for some stocks it doesn't.
import pandas_datareader.data as pdr
import datetime
import csv
with open('stockholm_largecap.csv', 'rb') as f:
reader = csv.reader(f)
stockholmLargeCap = list(reader)
start = datetime.datetime(1970, 1, 1)
end = datetime.datetime.today();
stockData = {}
for symbol in stockholmLargeCap:
f = pdr.DataReader(symbol, 'yahoo', start, end)
print f
The stockholm_largecap.csv contains all stocks in alphabetical order but once I get to certain stocks I get (for example BETS-B.ST): SymbolWarning: Failed to read symbol: 'BETS-B.ST', replacing with NaN. and the script terminates. Is there some way to continue the program, ignoring the error and what could be the cause of some stocks not working?
raise RemoteDataError(msg.format(self.__class__.__name__))
pandas_datareader._utils.RemoteDataError: No data fetched using 'YahooDailyReader'
use try and except
import pandas_datareader.data as pdr
for symbol in ['SPY', 'holla']:
try:
f = pdr.DataReader(symbol, 'yahoo', "2001-01-01", "2010-01-01")
print f.head(5)
except:
print ('did not find: '+symbol)
Open High Low Close Volume Adj Close
Date
2001-01-02 132.0000 132.1562 127.5625 128.8125 8737500 95.2724
2001-01-03 128.3125 136.0000 127.6562 135.0000 19431600 99.8488
2001-01-04 134.9375 135.4687 133.0000 133.5468 9219000 98.7740
2001-01-05 133.4687 133.6250 129.1875 129.1875 12911400 95.5497
2001-01-08 129.8750 130.1875 127.6875 130.1875 6625300 96.2893
did not find: holla
I had the same problem while I was trying to get stocks from a list , I used exception handling block which continued the execution of the code despite of the symbol warning viz.[ SymbolWarning: Failed to read symbol: 'AXZZW', replacing with NaN.
warnings.warn(msg.format(sym), SymbolWarning)]
from pandas_datareader._utils import RemoteDataError
from pandas_datareader.data import Options
try:
df1 = web.DataReader(rows[i],'yahoo',"2001-01-01","2010-01-01")
print("Downloading",i,"/",len(rows),"............")
print(df1)
except KeyError:
print("Data not found at Ticker %s"%i)
continue
except RemoteDataError:
print("Data not found at Ticker %s"%i)
continue
print("Success!")
Hope this works for you too!