I'm trying to import a text file of all of the NASDAQ symbols from EODData (http://eoddata.com/Data/symbollist.aspx?e=NASDAQ) into a python variable to save as a csv file. When I put the link into a Chrome browser the file downloads, but when I try to import it using urllib2 or pandas it looks like it is reading a website.
It seems to be similar to: EodData wsdl java connection but I'm trying to do it in python.
import urllib2
data = urllib2.urlopen("http://eoddata.com/Data/symbollist.aspx?e=NASDAQ")
for line in data:
print line
It seems that you must log in in order to download the data. You can download your data on chrome because you are automatically logged in. but through python, you have to register/log in. Try to check out the login API of the website and include your credentials in your code.
you have to look at the __init__ methods to fill it out with your credentials :
def __init__(self, username, password,
base_url='http://ws.eoddata.com/data.asmx/',
max_login_retries=3, logger=None):
"""
Args:
username (str): Account username.
password (str): Account password.
base_url (str): Base url of SOAP service
(defaults to `http://ws.eoddata.com/data.asmx/`).
max_login_retries (int): Maximum login retries, increase if there
are several clients working in parallel.
logger (logging.Logger): Client logger.
"""
self._token = ''
self._username = username
self._password = password
self._max_login_retries = max_login_retries
self._base_url = base_url
self.logger = logger or logging.getLogger('eoddata_client')
As per the documentation, you have to replace the username and password with your credentials. Also, you'll have to add the token which can be extracted from your account on the website.
Related
I am working to create a pipeline with the spotify API that logs my streaming history. I am planning to automate it by uploading it as a lambda function and scheduling it to run every few hours. I have everything mostly in order, except for that on the first run the API requires web authentication. Here is my code:
import spotipy
import spotipy.util as util
import urllib3
un = USERNAME
scope = 'user-read-recently-played'
cid = CLIENT_ID
csid = CLIENT_SECRET_ID
redr = r'http://localhost:8888/callback/'
token = util.prompt_for_user_token(un,scope,cid,csid,redr)
When this is run for the first time, this message pops up:
User authentication requires interaction with your
web browser. Once you enter your credentials and
give authorization, you will be redirected to
a url. Paste that url you were directed to to
complete the authorization.
Opened <LINK HERE> in your browser
Enter the URL you were redirected to:
And then I have to copy the link from my browser into that space. I can get the URL that I need to paste using urllib3:
req_adr = ADDRESS_IT_OPENS_IN_BROWSER
http = urllib3.PoolManager()
resp = http.request('GET',req_adr)
redrurl = resp.geturl()
But I don't know how to pass it into the input prompt from the util.prompt_for_user_token response
Any suggestions would be very welcome.
So it turns out there is a workaround. You can run it one time on a local machine and that generates a a file called .cache-USERNAME. If you include that file in your deployment package you don't have to copy/paste the URL and it is able to be automated with a lambda function in AWS.
I'm new to Python, using Python 3.5.1 (windows)
I'm trying to write a code to automate a login module for an API server
The first inputs are an API key and secret which I provide and then I get a URL from the API server and using that we open up the browser pointing to that page which requires us to manually enter the username and password to get authenticated. == so far so good.
Once authenticated the server opens up a webpage and displays a token ref in the url parameters, something like
http://www.xxxx.xx.x.../xxxxxxtoken_ref=a1234xyx
I need to extract the token_ref value and use that for further authentication.
I am trying to use a local server at the redirect IP so I can capture the token_ref using Bottle but I think I am doing something wrong.
My code is below:
from kiteconnect import KiteConnect
from kiteconnect import websocket
from bottle import run , request , route
import webbrowser
login to kite
myapi_key="abcdxyz1234"
myapi_secret = "xxxxyyyyzzzz1234"
kite = KiteConnect(api_key=myapi_key)
url = kite.login_url()
webbrowser.open(url)
get token_request (this is where I'm stuck)
#route('/')
def root():
tk_request = request.query.token_request
return 'the token request is' + tk_request
run()
get access token
user = kite.request_access_token(request_token=tk_request,secret=api_secret)
https://github.com/soundcloud/soundcloud-python
I'm using the Python wrapper above, and I am struggling to get the access token.
import soundcloud
client = soundcloud.Client(
client_id=YOUR_CLIENT_ID,
client_secret=YOUR_CLIENT_SECRET,
redirect_uri='http://yourapp.com/callback'
)
redirect(client.authorize_url())
I am able to reach this point and it successfully allows the user to authorize. However I am lost as to how I am supposed to get the access token.
The documentation says the following:
access_token, expires, scope, refresh_token = client.exchange_token(
code=request.args.get('code'))
render_text("Hi There, %s" % client.get('/me').username)
When I use this, it gives me a 500 error.
On redirect client.authorize_url(), the user will be redirected to a SoundCloud connect screen in their browser and asked to authorize their account with your application.
If the user approves the authorization request, they will be sent to the redirect_uri specified in redirect_uri='http://yourapp.com/callback'. From there, you can extract the code parameter from the query string and use it to obtain an access token.
import soundcloud
# create client object with app credentials
client = soundcloud.Client(client_id='YOUR_CLIENT_ID',
client_secret='YOUR_CLIENT_SECRET',
redirect_uri='http://example.com/callback')
# exchange authorization code for access token
code = params['code']
access_token = client.exchange_token(code)
This is straight from the Server-Side Authentication docs.
You could use selenium webdriver
pip install --upgrade selenium
to open a browser window, log in to the soundcloud account and get the code.
#!/usr/bin/env python
# coding=utf-8
import soundcloud
from selenium import webdriver
driver = webdriver.Firefox(
executable_path='<YOUR_PATH_TO>/geckodriver')
CLIENT_ID='<YOUR_CLIENT_ID>'
CLIENT_SECRET='<YOUR_CLIENT_SECRET>'
REDIRECT_URL='<YOUR_REDIRECT_URL>'
client = soundcloud.Client(
client_id=CLIENT_ID,
client_secret=CLIENT_SECRET,
redirect_uri=REDIRECT_URL)
AUTH_URL = client.authorize_url()
driver.get(AUTH_URL) ## open Firefox to access Soundcloud
code = raw_input("WAITING FOR ACCESS... type ENTER")
code = driver.current_url.replace(REDIRECT_URL+'/?code=','')[:-1]
ACCESS_TOKEN = client.exchange_token(code).access_token
USER = client.get('/me').permalink
FILE = "SOUNDCLOUD.%s.access_token" % USER
FILE_W = open(FILE,'w')
FILE_W.write(ACCESS_TOKEN)
FILE_W.close()
driver.quit()
Maybe you'll need to get geckodriver, you can google it to find the one for your OS.
Note that this access_token is not expiring. You don't need a refresh_token.
You can print the full response object with:
from pprint import pprint
[...]
code = driver.current_url.replace(REDIRECT_URL+'/?code=','')[:-1]
TOKEN_OBJECT = client.exchange_token(code)
pprint (vars(TOKEN_OBJECT))
ACCESS_TOKEN = TOKEN_OBJECT.access_token
[...]
I am trying to get started with the Box.com SDK and I have a few questions.
from boxsdk import OAuth2
oauth = OAuth2(
client_id='YOUR_CLIENT_ID',
client_secret='YOUR_CLIENT_SECRET',
store_tokens=your_store_tokens_callback_method,
)
auth_url, csrf_token = oauth.get_authorization_url('http://YOUR_REDIRECT_URL')
def store_tokens(access_token, refresh_token):
# store the tokens at secure storage (e.g. Keychain)
1)What is the redirect URL and how do I use it? Do I need to have a server running to use this?
2)What sort of code to I need in the store_tokens method?
The redirect URL is only required if you're runng a Web application that needs to respond to user's requests to authenticate. If you're programtically authenticating, you can simply set this as http://localhost. In a scenario where you require the user to manually authenticate, the redirect URL should invoke some function in your web app to store and process the authentication code returned. Do you need a server running? Well, if you want to do something with the authentication code returned, the URL you specify should be under your control and invoke code to do something useful.
Here's an example of what the store_tokens function should look like. It should accept two parameters, access_token and refresh_token. In the example below, the function will commit these to a local store for use when the API needs to re-authenticate:
From here:
"""An example of Box authentication with external store"""
import keyring
from boxsdk import OAuth2
from boxsdk import Client
CLIENT_ID = 'specify your Box client_id here'
CLIENT_SECRET = 'specify your Box client_secret here'
def read_tokens():
"""Reads authorisation tokens from keyring"""
# Use keyring to read the tokens
auth_token = keyring.get_password('Box_Auth', 'mybox#box.com')
refresh_token = keyring.get_password('Box_Refresh', 'mybox#box.com')
return auth_token, refresh_token
def store_tokens(access_token, refresh_token):
"""Callback function when Box SDK refreshes tokens"""
# Use keyring to store the tokens
keyring.set_password('Box_Auth', 'mybox#box.com', access_token)
keyring.set_password('Box_Refresh', 'mybox#box.com', refresh_token)
def main():
"""Authentication against Box Example"""
# Retrieve tokens from secure store
access_token, refresh_token = read_tokens()
# Set up authorisation using the tokens we've retrieved
oauth = OAuth2(
client_id=CLIENT_ID,
client_secret=CLIENT_SECRET,
access_token=access_token,
refresh_token=refresh_token,
store_tokens=store_tokens,
)
# Create the SDK client
client = Client(oauth)
# Get current user details and display
current_user = client.user(user_id='me').get()
print('Box User:', current_user.name)
if __name__ == '__main__':
main()
I suggest taking a look at the OAuth 2 tutorial. It will help give a better understanding of how OAuth works and what the various parameters are used for.
The redirect URL is set in your Box application's settings:
This is the URL where Box will send an auth code that can be used to obtain an access token. For example, if your redirect URL is set to https://myhost.com, then your server will receive a request with a URL that looks something like https://myhost.com?code=123456abcdef.
Note that your redirect URI doesn't need to be a real server. For example, apps that use a WebView will sometimes enter a fake redirect URL and then extract the auth code directly from the URL in the WebView.
The store_tokens callback is optional, but it can be used to save the access and refresh tokens in case your application needs to shutdown. It will be invoked every time the access token and refresh token changes, giving you an opportunity to save them somewhere (to disk, a DB, etc.).
You can then pass in these tokens to your OAuth2 constructor at a later time so that your users don't need to login again.
If you're just testing, you can also pass in a developer token. This tutorial explains how.
This is the most basic example that worked for me:
from boxsdk import Client, OAuth2
CLIENT_ID = ''
CLIENT_SECRET = ''
ACCESS_TOKEN = '' # this is the developer token
oauth2 = OAuth2(CLIENT_ID, CLIENT_SECRET, access_token=ACCESS_TOKEN)
client = Client(oauth2)
my = client.user(user_id='me').get()
print(my.name)
print(my.login)
print(my.avatar_url)
I am trying to access my own docs & spreadsheets via onedrive's api. I have:
import requests
client_id = 'my_id'
client_secret = 'my_secret'
scopes = 'wl.offline_access%20wl.signin%20wl.basic'
response_type = 'token' # also have tried "code"
redirect_uri = 'https://login.live.com/oauth20_desktop.srf'
base_url = 'https://apis.live.net/v5.0/'
r = requests.get('https://login.live.com/oauth20_authorize.srf?client_id=%s&scope=%s&response_type=%s&redirect_uri=%s' % (client_id, scopes, response_type, redirect_uri))
print r.text
(For my client I've also tried both "Mobile or desktop client app:" set to "Yes" and "No")
This will return the html for the user to manually click on. Since the user is me and it's my account how do I access the API without user interaction?
EDIT #1:
For those confused on what I'm looking for it would be the equivalent of Google's Service Account (OAuth2): https://console.developers.google.com/project
You cannot "bypass" the user interaction.
However you are very close to getting it to work. If you want to gain an access token in python you have to do it through the browser. You can use the web browser library to open the default web browser. It will look something like this (your app must be a desktop app):
import webbrowser
webbrowser.open("https://login.live.com/oauth20_authorize.srf?client_id=foo&scope=bar&response_type=code&redirect_uri=https://login.live.com/oauth20_desktop.srf")
This will bring you to the auth page, sign in and agree to the terms (it will differ depending on scope). It will direct you to a page where the url looks like:
https://login.live.com/oauth20_desktop.srf?code=<THISISTHECODEYOUWANT>&lc=foo
Copy this code from the browser and have your python script take it as input.
You can then make a request as described here using the code you received from the browser.
You will receive a response described here