Develop Reference

Trying to solve a date discrepancy issue with timezone-aware data filtered from a csv between two dates

My problem is in finding the best-practice to resolve the issue of including all requested timezone-aware dates in a selected range. Depending on how I approach it, either the enddate or the startdate fail to return their date from the csv. I cannot get both to work at the same time without tricking the code (which seems like bad coding practice, and might produce issues if the user is entering dates while browsing from a different timezone locale).
In its simplest form, I am trying to return the complete year of 2021 transaction records from a csv file. The URL call to FastAPI to run the code is this:
http://127.0.0.1:8000/v1/portfolio/tastytx?year=2021
When the user enters the year parameter in the URL the enddate and startdate get set, then the csv read into a dataframe, the dataframe filtered, and a json returned. In the following code snippet, 'frame' is the dataframe from pd.read_csv code that is not shown:
import datetime as dt
import pandas as pd
from pytz import timezone
import pytz
import json
from fastapi import FastAPI, Response
startdate = year +'-01-01'
enddate = year + '-12-31'
startdate = dt.datetime.strptime(startdate, '%Y-%m-%d') #convert string to a date
startdate = startdate.replace (tzinfo=pytz.utc) #convert date to UTC and make it timezone-aware
enddate = dt.datetime.strptime(enddate, '%Y-%m-%d')
enddate = enddate.replace (tzinfo=pytz.utc)
frame['Date'] = pd.to_datetime(frame['Date'], format='%Y-%m-%d') #turns 'Date' from object into datetime64
frame['Date'] = frame['Date'].dt.tz_convert('UTC') #converts csv 'Date' column to UTC from AEST(+10)
selectdf = frame.loc[frame['Date'].between(startdate, enddate, inclusive=True)] #filters time period
return Response(selectdf.to_json(orient="records"), media_type="application/json")
The csv file containing the data has 'Date' column in AEST timezone (i.e. UTC+10).
In my code I convert everything to UTC and then do the comparison, but with the above code the date for 1st Jan 2021 is not returned, the 2nd Jan is. What is the right way to solve this, I have tried every configuration of timezone changes and so far nothing returned both 1st Jan 2021 and 31st Dec 2021 at the same time.
Sample data:
the csv file from 'Date' Column :
Date
2021-12-31T01:35:59+1000
2021-12-31T01:35:59+1000
2021-12-31T01:09:57+1000
2021-12-31T01:09:57+1000
2021-12-30T03:02:25+1000
2021-12-30T01:52:58+1000
...
2021-01-02T00:48:29+1000
2021-01-01T02:40:03+1000
2021-01-01T00:30:00+1000
2021-01-01T00:30:00+1000
There is some confusion about the issue:
to clarify the problem, the above code will return the following epoch time (startdate) as the first record: 1609512509000 and the following epoch time (enddate) as the date of the last record: 1640878559000
For me it is translating that as 2nd Jan (I am in AEST timezone in my browser) and 31st Dec respectively, and so the json returned from the above csv data is 2nd Jan to 31st Dec, thus incorrect.
If you run it in your browser it will likely return those epoch dates records relevant to your timezone. This is my problem.

I have a list of dates and I want to subtract actual date from each of them to know how many day passed. Is there any fast way to do this?

I know I should import datetime to have actual date. But the rest is black magic for me right now.
ex.
dates = ['2019-010-11', '2013-05-16', '2011-06-16', '2000-04-22']
actual_date = datetime.datetime.now()
How can I subtract this and as a result have new list with days that passed by from dates to actual_date?

If I'm understanding correctly, you need to find the current date, and then find the number of days between the current date and the dates in your list?
If so, you could try this:
from datetime import datetime, date
dates = ['2019-10-11', '2013-05-16', '2011-06-16', '2000-04-22']
actual_date = date.today()
days = []
for date in dates:
date_object = datetime.strptime(date, '%Y-%m-%d').date()
days_difference = (actual_date - date_object).days
days.append(days_difference)
print(days)
What I am doing here is:
Converting the individual date strings to a "date" object
Subtracting the this date from the actual date. This gets you the time as well, so to strip that out we add .days.
Save the outcome to a list, although of course you could do whatever you wanted with the output.

How to write a python function that returns the number of days between two dates

I am new to functions and I am trying to write a function that returns the number of days between two dates:
My attempt:
import datetime
from dateutil.parser import parse
def get_x_days_ago (date_from, current_date = None):
td = current_date - parse(date_from)
if current_date is None:
current_date = datetime.datetime.today()
else:
current_date = datetime.datetime.strptime(date_from, "%Y-%m-%d")
return td.days
print(get_x_days_ago(date_from="2021-04-10", current_date="2021-04-11"))
Expected outcome in days:
1

So there seem to be multiple issues, and as I said in the comments, a good idea would be to separate the parsing and the logic.
def get_x_days_ago(date_from, current_date = None):
if current_date is None:
current_date = datetime.datetime.today()
return (current_date - date_from).days
# Some other code, depending on where you are getting the dates from.
# Using the correct data types as the input to the get_x_days_ago (datetime.date in this case) will avoid
# polluting the actual logic with the parsing/formatting.
# If it's a web framework, convert to dates in the View, if it's CLI, convert in the CLI handling code
date_from = parse('April 11th 2020')
date_to = None # or parse('April 10th 2020')
days = get_x_days_ago(date_from, date_to)
print(days)
The error you get is from this line (as you should see in the traceback)
td = current_date - parse(date_from)
Since current_date="2021-04-11" (string), but date_from is parsed parse(date_from), you are trying to subtract date from the str.
P.S. If you have neither web nor cli, you can put this parsing code into def main, or any other point in code where you first get the initial strings representing the dates.

It looks like you're already aware that you can subtract a datetime from a datetime. I think, perhaps, you're really looking for this:
https://stackoverflow.com/a/23581184/2649560

Get previous month date from date stored in string variable

I am extracting the date from a filename and storing it in a string variable. Suppose the filename is CRM_DATA_PUBLIC_20201120_052035.txt, I have extracted the date as 20201120. Now I want to get the previous month's date from this, like 20201020 or just 202010.
I tried using date functions but it is giving error for me.
Could you please help me out here ?
Thanks in anticipation.

Try this: (changes based on a comment)
import datetime
from dateutil.relativedelta import relativedelta
filename = 'CRM_DATA_PUBLIC_20201120_052035.txt'
date = filename.split('_')[3]
#If you want the output to include the day of month as well
date = datetime.datetime.strptime(date, '%Y%m%d')
#If you want only the month
date = datetime.datetime.strptime(date, '%Y%m')
date = date - relativedelta(months=1)
date = str(date.date()).replace('-','')
print(date)
Output:
20201020

You can find your answer here https://stackoverflow.com/a/9725093/10334833
What I get from your question is already answered here but if you are still confused let me know I can help you :)

how to get number of days in month based on date string input

I am trying to automate a code. I would like to have the code pull data starting at the beginning of the month to end of the previous day. Currently I am using the following command to get the enddate:
dateEnd = pd.to_datetime('today')
How do tell the code, based on what today is to go back to the beginning of the month? AND, how do I tell the code if its the first of the month to return the previous months data?
For a bonus, once I have the start and end date, how do return find the number of days in the month? I have tried this command, but it does not want to work on a single date.
startTime_date.dt.daysinmonth

This will give you the wanted dates:
import datetime
end = datetime.date.today() - datetime.timedelta(1)
start = end.replace(day=1)
daysInMonth = (datetime.date(start.year, start.month + 1, 1) - datetime.timedelta(1)).day
start
#2018-10-01
end
#2018-10-09
daysInMonth
#31