Develop Reference

Python error "Unable to read URL" even though "Requirement already satisfied" in command prompt

I've been trying to work on some beginner/entry level python projects for a resume (working with WIN10) and I've been running into the same constant error. I've been trying to pull info from yahoo finance and I get this error, no matter how I rework my code:
import pandas_datareader as web
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import datetime as dt
start = dt.datetime(2018,1,1)
end = dt.datetime.now()
tickers = ["FB", "GS", "NVDA", "MSFT", "TSLA", "AAPL", "CCL", "BA"]
colnames = []
for ticker in tickers:
data = web.DataReader(ticker, "yahoo", start, end)
if len(colnames) == 0:
combined = data[['Adj Close']].copy()
else:
combined = combined.join(data['Adj Close'])
colnames.append(ticker)
combined.columns = colnames
print(combined)
RemoteDataError: Unable to read URL: https://finance.yahoo.com/quote/FB/history?period1=1514779200&period2=1652673599&interval=1d&frequency=1d&filter=history
I looked around and saw that yahoo finance changed it's structure for python, so I then tried changing the format or my code, still got the error. I tried installing suggested libraries to fix it, and while it says that their installed, I can't use the libraries because it says that module isn't found.
I've looked at other Stack posts and none of the solutions are helping.

Where am I going wrong retrieving stock data from Quandl?

ValueError: The Quandl API key must be provided either through the api_key variable or through the environmental variable QUANDL_API_KEY.
I am trying to retrieve some simple stock data from Quandl. I have put in the actual API key instead of the x in the below example code below but I am still getting errors. Am I missing out on something?
import datetime as dt
import matplotlib.pyplot as plt
from matplotlib import style
import pandas as pd
import pandas_datareader.data as web
style.use('ggplot')
symbol = 'AAPL'
api_key = 'x'
start = dt.datetime(2015, 1, 1)
end = dt.datetime.now()
df = web.DataReader(symbol, 'quandl', start, end, api_key)
print(df.head())

From the quandl docs:
AUTHENTICATION The Quandl Python module is free but you must have a
Quandl API key in order to download data. To get your own API key, you
will need to create a free Quandl account and set your API key.
After importing the Quandl module, you can set your API key with the
following command: quandl.ApiConfig.api_key = "YOURAPIKEY"
So you will need to pip install and import quandl. Then you can set the api_key attribute as above.

If you only want to get the data from Quandl, maybe you can try another approach.
import pandas as pd
import Quandl
api_key = 'yoursuperamazingquandlAPIkey'
df = Quandl.get('heregoesthequandlcode', authtoken = api_key)
print(df.head())

google finance syntax error when printing

import numpy as np
import pandas as pd
import pandas_datareader.data as web
from pandas_datareader import data, wb
goog = web.DataReader('GOOG', data_source='google', start='3/14/2017',end='9/14/2017')
print goog.head()
Why does it shows invalid syntax for goog?
Thank you

Ok I found out because in Python 3 they replace it with a function so the correct type is:
print (goog.head())

Parse requests.get() output into a pandas dataframe

I am following a tutorial an am stuck at parsing the output of requests.get()
My goal is to connect to the API below to pull historical crypto-currency prices and put them into a pandas dataframe for further analysis.
[API: https://www.cryptocompare.com/api/#-api-data-histoday-]
Here's what I have.
import requests
response = requests.get("https://min-api.cryptocompare.com/data/histodayfsym=ETC&tsym=USD&limit=10&aggregate=3&e=CCCAGG")
print(response.text)
Now I want to output into a dataframe...
pd.DataFrame.from_dict(response)
But I get...
PandasError: DataFrame constructor not properly called!

You can use the json package to convert to dict:
import requests
from json import loads
import pandas as pd
response = requests.get("https://min-api.cryptocompare.com/data/histodayfsym=ETC&tsym=USD&limit=10&aggregate=3&e=CCCAGG")
dic = loads(response.text)
print(type(dic))
pd.DataFrame.from_dict(dic)
However as jonrsharpe noted, a much more simple way would be:
import requests
import pandas as pd
response = requests.get("https://min-api.cryptocompare.com/data/histodayfsym=ETC&tsym=USD&limit=10&aggregate=3&e=CCCAGG")
print(type(response.json()))
pd.DataFrame.from_dict(response.json())

python web-scraping yahoo finance

Since Yahoo finance updated their website. some tables seem to be created dynamically and not actually stored in the HTML (I used to get this information using BeautifulSoup, urllib but this won't work anymore). I am after the Analyst tables for example ADP specifically the Earnings Estimates for Year Ago EPS (Current Year Column). You cannot get this information from the API.
I found this link which works well for the Analyst Recommendations Trends. does anyone know how to do something similar for the main table on this page? (LINK:
python lxml etree applet information from yahoo )
I tried to follow the steps taken but frankly its beyond me.
returning the whole table is all I need I can pick out bits from there. cheers

In order to get that data, you need to open Chrome DevTools and select Network tab with XHR filter. If you click on ADP request you can see link in RequestUrl.
You can use Requests library for making a request and parsing json response from the site.
import requests
from pprint import pprint
url = 'https://query1.finance.yahoo.com/v10/finance/quoteSummary/ADP?formatted=true&crumb=ILlIC9tOoXt&lang=en-US&region=US&modules=upgradeDowngradeHistory%2CrecommendationTrend%2CfinancialData%2CearningsHistory%2CearningsTrend%2CindustryTrend%2CindexTrend%2CsectorTrend&corsDomain=finance.yahoo.com'
r = requests.get(url).json()
pprint(r)

further to volds answer above and using the answer in the link I posted above. (credit to saaj). This gives just the dataset I need and is neater when calling the module. I am not sure what the parameter crumb is but, it seems to work ok without it.
import json
from pprint import pprint
from urllib.request import urlopen
from urllib.parse import urlencode
def parse():
host = 'https://query1.finance.yahoo.com'
#host = 'https://query2.finance.yahoo.com' # try if above doesn't work
path = '/v10/finance/quoteSummary/%s' % 'ADP'
params = {
'formatted' : 'true',
#'crumb' : 'ILlIC9tOoXt',
'lang' : 'en-US',
'region' : 'US',
'modules' : 'earningsTrend',
'domain' : 'finance.yahoo.com'
}
response = urlopen('{}{}?{}'.format(host, path, urlencode(params)))
data = json.loads(response.read().decode())
pprint(data)
if __name__ == '__main__':
parse()
Other modules (just add a comma between them):
assetProfile
financialData
defaultKeyStatistics
calendarEvents
incomeStatementHistory
cashflowStatementHistory
balanceSheetHistory
recommendationTrend
upgradeDowngradeHistory
earningsHistory
earningsTrend
industryTrend

In GitHub, c0redumb has proposed a whole solution. You can download the yqd.py. After import it, you can get Yahoo finance data by one line of code, as blew.
import yqd
yf_data = yqd.load_yahoo_quote('GOOG', '20170722', '20170725')
The result 'yf_data' is:
['Date,Open,High,Low,Close,Adj Close,Volume',
'2017-07-24,972.219971,986.200012,970.770020,980.340027,980.340027,3248300',
'2017-07-25,953.809998,959.700012,945.400024,950.700012,950.700012,4661000',
'']