I am learning Python and had a question regarding for and if loops. This is my scenario:
I have an endpoint that i make API-call with request.get
I need to retrieve all the historic data
I have a start_date (2017-06-17)
So i need to make multiple API-call because they have a limit of 60-days period. So i made my code like this:
date = datetime.strptime("2017-06-17", "%Y-%m-%d") # Start Date
current_date = date.date() # timedelta need date object so i make it a date object
days_after = (current_date+timedelta(days=60)).isoformat() # days_after is set to 60-days because limit in API
date_string = current_date.strftime('%Y-%m-%d') # made to string again since API need string not date object
So this is how i make the dates for 60 days period. Starting from 2017-06-17 and 60-days ahead.
This is how i make the API-request:
response = requests.get("https://reporting-api/campaign?token=xxxxxxxxxx&format=json&fromDate="+date_string+"&toDate="+days_after)
response_data = response.json() # Added this because i am writing temprorary to a JSON file
This is how i write to JSON file:
if response_data:
print("WE GOT DATA") # Debugging
data = response.json() # This is duplicate?
with open('data.json', 'w') as f: # Open my data.json file as write
json.dump(data, f) # dumps my json-data from API to the file
else:
print("NO DATA") # Debugging if no data / response. Should make a skip statement here
So my question is how can i proceed with my code so that every time i make a API-call starting from 2017-06-17 the date date_string and days_after should go 60 days forward for each API-call and append those data to data.json. I would maybe need some for loops or something?
Please note i have been using Python for 3 days now, be gentle.
Thanks!
You could use a while loop that changes the start and end date until a specified condition is met. Also, you can append the response to a file for every run. the example below I used the date of "today":
import os
from datetime import datetime, timedelta
x = 0
y = 60
date = datetime.strptime("2017-06-17", "%Y-%m-%d")
current_date = date.date()
date_start = current_date+timedelta(days=x)
while date_start < datetime.now().date():
date_start = current_date+timedelta(days=x)
days_after = current_date+timedelta(days=y)
x = x + 60
y = y + 60
response = requests.get("https://reporting-api/campaign?token=xxxxxxxxxx&format=json&fromDate="+date_start.isoformat() +"&toDate="+days_after.isoformat())
response_data = response.json()
if response_data:
print("WE GOT DATA")
data = response.json()
#create a file if not exists or append new data to it.
if os.path.exists('data.json'):
append_write = 'a' # append if already exists
else:
append_write = 'w' # make a new file if not
with open('data.json', append_write) as f:
json.dump(data, f)
else:
print("NO DATA")
Basically, on every run the time of start and end is increased by 60 days and appended to the data.json file.
Related
I am running into an issue with validating an input is a valid date, and if not coming back to the question for a retry.
I want to loop through this header and ask for input for each item.
header = [employee_id,name,address,ssn,date_of_birth,job_title,start_date,end_date]
The CSV is empty aside from the header, as I am appending these rows into via this program. I want the date_validator() to work for DOB, start_date and end_date, but so far i can get it to validate that the input is wrong, it just doesnt go back and ask for the input again.
Any help would be appreciated! thanks!
import csv
import datetime
def add_employee():
global date_answer
list = []
for i in range(len(header)):
var = header[i]
answer1 = input('Input Employees {}:'.format(var))
if "date" in header[i]:
date_answer = answer1
date_validater()
list.append(answer1)
with open('employees.csv','a',newline="") as f_object:
writer = csv.writer(f_object)
writer.writerow(list)
f_object.close()
print()
def date_validater():
# input date
date_string = date_answer
date_format = '%m/%d/%Y'
try:
dateObject = datetime.datetime.strptime(date_string, date_format)
print(dateObject)
except ValueError:
print("Incorrect data format, should be MM/DD/YYYY")
A couple hints:
Functions take parameters...
Globals are a terrible ideas most of the time. Avoid like the plague.
Pass the variables to the function to use it inside the function.
import csv
import datetime
def add_employee(header):
list = []
for i in range(len(header)):
var = header[i]
answer1 = input('Input Employees {}:'.format(var))
if "date" in header[i]:
date_validater(answer1)
list.append(answer1)
with open('employees.csv','a',newline="") as f_object:
writer = csv.writer(f_object)
writer.writerow(list)
f_object.close()
print()
def date_validater(date_string):
# input date
date_format = '%m/%d/%Y'
try:
dateObject = datetime.datetime.strptime(date_string, date_format)
print(dateObject)
except ValueError:
print("Incorrect data format, should be MM/DD/YYYY")
or something like that...
I'm trying to getting data from kraken exchange by the krakenex API. But i'm facing several problems, 'cause, I want getting the data in a range time bigger than the alllowed by the API.
The API only allows getting a dataframe with 720 rows, so 'cause that I need to do a loop while to getting more data and concat in another dataframe.
I've already read other topics about it, but I'm still not reaching good results.
import krakenex
import time
import krakenex
import pandas as pd
from pykrakenapi import KrakenAPI
from datetime import datetime
k = krakenex.API()
start = '28/01/2021 00:00:00'
start = datetime.strptime(start, "%d/%m/%Y %H:%M:%S")
start = int(time.mktime(start.timetuple()))
stop = '03/02/2021 00:00:00'
stop = datetime.strptime(stop, "%d/%m/%Y %H:%M:%S")
stop = int(time.mktime(stop.timetuple()))
prices = pd.DataFrame()
while start < stop:
time.sleep(5)
data = k.query_public('OHLC', {'pair':'XXBTZUSD', 'interval':1, 'since':start})
df = pd.DataFrame( data['result']['XXBTZUSD'])
daily_prices = df[0].to_list()
start = int(daily_prices[0])
prices = pd.concat([precos , df])
For weeks I have been working on a script that does exactly that. In my case I collect all pairs with BTC and ETH but you can use the script with any pair. To do this I used the REST API and defined some functions that automate everything. I download the data with 1 minute timeframe but it can be used for any timeframe.
First I defined a function that downloads the data in full or from a specific date, it's necessary because at the first run it will download all the data and then it will download only the new data. The parameter 'interval' defines the number of minutes of the timeframe while 'since' defines the beginning of the data to download.
def get_ohlc (pair, interval=1, since='last'):
endpoint = 'https://api.kraken.com/0/public/OHLC'
payLoad = {
'pair': pair,
'interval': interval,
'since' : since
}
response = requests.get(endpoint, payLoad)
data = response.json()
OHLC = data['result'][pair]
data = pd.DataFrame.from_records(OHLC, columns=['Time', 'Open', 'High', 'Low', 'Close', 'vwap', 'volume', 'count'])
data['Time'] = pd.to_datetime(data['Time'], unit='s')
data.set_index('Time',inplace=True)
data = data.drop(['vwap', 'volume', 'count'], axis=1)
data['Open'] = data.Open.astype(float)
data['High'] = data.High.astype(float)
data['Low'] = data.Low.astype(float)
data['Close'] = data.Close.astype(float)
return data
Then I defined a function to load the .json file that was saved into memory. The function returns the dataframe with the old data and a timestamp that indicates from where to download the new data. I also created a function for calculate the timestamp.
def load_data(pair, path):
data = pd.read_json(path + pair + '.json' , orient='split')
tmp = data.tail(1).index
tmp = tmp.strftime('%Y-%m-%d %H:%M:%S')
dt = str_to_datetime(tmp[0])
ts = dt.timestamp()
return data, ts
def str_to_datetime(datestr):
Y = int(datestr[0:4])
M = int(datestr[5:7])
D = int(datestr[8:10])
H = int(datestr[11:13])
m = int(datestr[14:16])
return datetime.datetime(Y, M, D, H, m, 0, tzinfo=tz.gettz("Etc/GMT"))
Now your main should be something like:
from countdown import countdown
import pandas as pd
import datetime
import os
path = os.getcwd() + '/historical_data/'
pair = 'XBTUSD'
while True:
if os.path.exists(path + pair + '.json') == False:
data = get_ohlc(pair, 1) # 1 minute timeframe
data.to_json(path + pair + '.json', orient='split')
else:
data1, ts = load_data(pair, path)
data2 = get_ohlc(pair, 1, ts)
data3 = pd.concat([data1, data2])
data3.drop(data3.tail(1).index,inplace=True) # delete last record because it's not ended
data3.to_json(path + pair + '.json', orient='split')
countdown(60) # update every hour
I delete the last record because when you download it it's not ended so we will download at the next update. I haven't tested if it works because I took pieces of code from my program, if it doesn't work let me know and I'll fix it.
I am struggling to understand how the date queries work in Django as I am storing a database with train times. I want to get times that are greater than the current time.
The query looks like this, but returns zero results:
latestdepartures = LatestDepartures.objects.filter(station=startstation,earliest__gte=timezone.now().astimezone(pytz.utc))
My database has the entry below for example.
When I run the query, I get the results below (first line is print(timezone.now().astimezone(pytz.utc)):
2020-08-01 15:49:06.610055+00:00
<QuerySet []>
The code which adds the data to the database looks like:
def convert_date_time(o):
if isinstance(o, datetime):
return o.__str__()
def updateservices(stationname,destination):
now = datetime.now()
# dd/mm/YY H:M:S
datenow = now.strftime("%d/%m/%Y")
board = DARWIN_SESH.get_station_board(stationname)
stationdict = dict()
stationdict['from'] = stationname
stationdict['name'] = board.location_name
stationdict['servicelist']=[]
services = board.train_services
for s in services:
traindict = dict()
service_details = DARWIN_SESH.get_service_details(s.service_id)
traindict['departuretime'] = datetime.strptime(datenow + " " + service_details.std,'%m/%d/%Y %H:%M').astimezone(pytz.utc)
traindict['callingpoints'] = []
callingpoints = service_details.subsequent_calling_points
for c in callingpoints:
if c.crs == destination:
callingpointdict = dict()
callingpointdict['code'] = c.crs
callingpointdict['name'] = c.location_name
callingpointdict['arrivaltime'] = datetime.strptime(datenow + " " + c.st,'%m/%d/%Y %H:%M').astimezone(pytz.utc)
traindict['callingpoints'].append(callingpointdict)
if len(traindict['callingpoints']) > 0:
stationdict['servicelist'].append(traindict)
#For getting the minimum departure
departures = [s['departuretime'] for s in stationdict['servicelist']]
#Store the train departure object in the database
stationdata = json.dumps(stationdict, default=convert_date_time)
LatestDepartures.objects.create(
station = stationname,
earliest = min(departures),
services = stationdata
)
return stationdata
servicedetails.std will be a time represented in 24hours in string format, for example "17:00".
Can anyone help, I am not sure if I am meant to change the date format somewhere or if it is to do with the way the datetime object is created by adding the time.
UPDATE:
Now storing the date in a different format as '%d/%m/%Y %H:%M':
Now I get dates that are greater than, but once the time current time has exceeded the earliest in the database, the query still returns results. Example output is:
2020-08-01 17:31:21.909052+00:00
print(timezone.now().astimezone(pytz.utc))
2020-08-01 18:03:00+00:00 - Time in database
I am running a code to download data and saving them in local drive. However, I am getting above mentioned error message. Please note initially I have converted date in a different format and while saving them I get this error message.
Can you please help me with this error?
'''
import quandl
import os
import pandas as pd
import datetime as dt
import glob
if name == "main":
'Creating bucket to store missing data file.'
data_missing = []
New_date = []
'Defining a path to save CSV files after downloading and also deleting all csv file at one go.'
extension = 'csv'
path = "F:/Tradepoint/MyMkt/"
if not os.path.exists(path):
os.mkdir(path)
os.chdir(path)
csv_count = [forma for forma in glob.glob('*.{}'.format(extension))]
for csv_coun in range(len(csv_count)):
os.remove(r"F:/Tradepoint/MyMkt/" + csv_count[csv_coun][0:])
'Setting up quandl configuration, reading ticker list, setting up date for which data is going to get downloaded'
quandl.ApiConfig.api_key = 'Hba3CzgNnEa2LMxR14FA'
end_date = dt.date.today()
diff_year = dt.timedelta(days=3650)
start_date = end_date - diff_year
stock_list = pd.read_csv(r"F:\Abhay_New\Abhay\Python\Project\SHARADAR_SF1.csv")
'Looping through quandl website to download data and renaming them as per requirement.'
for stock_lis in range(len(stock_list)):
data = quandl.get_table('SHARADAR/SEP', date={'gte':start_date, 'lte':end_date}, ticker=stock_list.iloc[stock_lis])
sort_by_date = data.sort_values('date')
for sort_by_dat in range(len(sort_by_date['date'])):
Date = dt.date.strftime(sort_by_date['date'][sort_by_dat],'%d-%m-%Y')
New_date.append(Date)
if len(data)>1:
Date = pd.Series(New_date).rename('Date').astype(str)
OPEN = sort_by_date['open']
HIGH = sort_by_date['high']
LOW = sort_by_date['low']
CLOSE = sort_by_date['close']
VOLUME = sort_by_date['volume']
final_data = pd.concat([Date,OPEN,HIGH,LOW,CLOSE,VOLUME],axis=1)
stk = stock_list.iloc[sort_by_dat][0]
final_data.to_csv(str(path + stk + '.csv'), sep=',', index = False, header = False)
else:
data_missing.append(stock_list.iloc[sort_by_dat])
print(data_missing)
'''
Thanks,
Abhay Dodiya
The index of both for loops is i. This causes potentially unintended behavior:
for i in range(2):
for i in range(3,11):
pass
print(i)
gives
10
10
So even after exiting the second loop, the last value from i is there. Rename the counting variable in that loop and your issue should be gone.
In your case you probably have more dates than stocks, and thus observe the error message you have.
I am new to Python and this is driving me crazy. My XML File isn't being updated. I am trying to find the day difference between today and the youngest date in an XML. Then I want to update all other dates by that amount of days. My console shows the dates are being updated, but the XML file is not being changed. I believe the error occurs in this function.
# Open File to be modified
tree = ET.parse('MAV_Case1.xml')
root = tree
datesArray = []
# Parser to convert date from ISOFormat to Date Object
# This allows us to manipulate the date range.
def getDateTimeFromISO8601String(i):
d = dateutil.parser.parse(i)
return d
#This gathers all the transDates in the XMl
def oldDate(xmlFile):
transactions = tree.iter('transaction')
for transaction in transactions:
transDate = transaction.find('transDate').text
#print(transDate)
transactionDate = getDateTimeFromISO8601String(transDate)
#print(transactionDate)
datesArray.append(transactionDate)
#print(datesArray)
newArray = datesArray
#print(newArray)
#dateArray = list(newArray)
#print(dateArray)
return newArray
#This Function converts the old dates into new ones
def newDate(newArray):
newDateArray = []
for date in newArray:
#print(date)
youngest_date = max(newArray)
#print(youngest_date)
todayDate = datetime.now()
dateDiff = abs((todayDate - youngest_date).days)
#print(dateDiff)
newDate = date + dateutil.relativedelta.relativedelta(days=dateDiff)
#print(newDate)
date = str(newDate.isoformat())
newDateArray.append(date)
#print(newDateArray)
return newDateArray
#Function Carries Updated Dates
def updateXML(newDateArray):
for transDate in root.iter('transDate'):
#print(newDate.text)
updatedDate = transDate.text
for date in newDateArray:
updatedDate = date
transDate.text = updatedDate
return transDate
updateXML(newDate(oldDate(tree)))
#Writing Back to File
now = datetime.now()
actual_time = str(now.strftime("%Y-%m-%d-%H-%M-%S"))
tree.write("Dag Account - " + str(actual_time) + ".xml", xml_declaration=True)