this is a piece of code where I want to change the date and month in the link. I don't want to use the date-time module because I will be scraping and storing some information from the website for each day and for each month I will be saving the data in CSV so after every month I want to create a new CSV file
also if there would be a for loop for a month I want to use it in naming the CSV file while saving it
from selenium import webdriver
import time
path="C:\\Users\\Nihal\\Downloads\\chromedriver_win32\\chromedriver.exe"
import numpy
driver = webdriver.Chrome(path)
driver.get('https://www.wunderground.com/history/daily/in/mangalore/VOML/date/2013-4-14')
Plug the dates as tuples and use f-string to format them in URL:
from selenium import webdriver
import time
path="C:\\Users\\Nihal\\Downloads\\chromedriver_win32\\chromedriver.exe"
import numpy
driver = webdriver.Chrome(path)
dates = [(2013, 4, 14)]
for x in dates:
#https://www.wunderground.com/history/daily/in/mangalore/VOML/date/2013-4-14
driver.get(f'https://www.wunderground.com/history/daily/in/mangalore/VOML/date/{x[0]}-{x[1]}-{x[2]}')
I am not used to selenium, so I can't help you with the parsing that you do with it, but I am pretty sure that you can use the datetime module to iterate on every day of a month and on every month of the year. Here is an example of how you can iterate on this and generate the url that you need:
from datetime import timedelta, date
def daterange(start_date, end_date):
for n in range(int((end_date - start_date).days)):
yield start_date + timedelta(n)
start_date = date(2020, 8, 1)
end_date = date(2020, 8, 31)
days_to_scrape = []
for single_date in daterange(start_date, end_date):
days_to_scrape.append(f'https://www.wunderground.com/history/daily/in/mangalore/VOML/date/{single_date.strftime("%Y-%m-%d")}')
#Iteration with driver.get
If I understand, you will have no choice but to send as many requests as the number of days from which you want the data. You can then iterate on the items of the list with your scraping command. If there is another reason why you think datetime module can't do what you need it to do, please, explain it.
L.R.
P.S.
Thanks to vinzee who helped me to understand that kind of iteration with his answer: Iterating through a range of dates in Python
Related
I have searched for this topic and I found some packages that are useful. All what I am trying to get is the last price of any specific ticker such as "MSFT"
Here's a code that I found and it is good
import pandas_datareader as pdr
from datetime import datetime
ibm = pdr.get_data_yahoo(symbols='MSFT', start=datetime(2021, 3, 1), end=datetime(2021, 3, 12))
print(ibm['Adj Close'])
This works for range of dates. How can I get the last price only without hard-coding the start date or end date?
Just use tail keyword.
from datetime import datetime, date
ibm = pdr.get_data_yahoo(symbols='MSFT', start = date.today(), end = date.today())
print(ibm['Adj Close'].tail(1))
Okay so I am relatively new to programming and this has me absolutely stumped. Im scraping data from a website and the data changes every week. I want to run my scraping process each time the data changes starting back on 09-09-2015 and running to current.
I know how to do this easily running thru every number like 0909 then 0910 then 0911 but that is not what I need as that will be requesting way too many requests from the server that are pointless.
Here is the format of the URL
http://www.myexamplesite.com/?date=09092015
I know the simple:
for i in range(startDate, endDate):
url = 'http://www.myexamplesite.com/?date={}'.format(i)
driver.get(url)
But one thing i've never been able to figure out is manipulate pythons dateTime to accurately reflect the format the website uses.
i.e:
09092015
09162015
09232015
09302015
10072015
...
09272017
If all else fails I only need to do this once so it wouldnt take too long to just ignore the loop altogether and just manually enter the date I wish to scrape from and then just append all of my dataframes together. Im mainly curious on how to manipulate the datetime function in this sense for future projects that may require more data.
A good place to start are datetime, date and timedelta objects docs.
First, let's construct our starting date and ending date (today):
>>> from datetime import date, timedelta
>>> start = date(2015, 9, 9)
>>> end = date.today()
>>> start, end
(datetime.date(2015, 9, 9), datetime.date(2017, 9, 27))
Now let's define the unit of increment -- one day:
>>> day = timedelta(days=1)
>>> day
datetime.timedelta(1)
A nice thing about dates (date/datetime) and time deltas (timedelta) is they and can be added:
>>> start + day
datetime.date(2015, 9, 10)
We can also use format() to get that date in a human-readable form:
>>> "{date.day:02}{date.month:02}{date.year}".format(date=start+day)
'10092015'
So, when we put all this together:
from datetime import date, timedelta
start = date(2015, 9, 9)
end = date.today()
week = timedelta(days=7)
mydate = start
while mydate < end:
print("{date.day:02}{date.month:02}{date.year}".format(date=mydate))
mydate += week
we get a simple iteration over dates starting with 2015-09-09 and ending with today, incremented by 7 days (a week):
09092015
16092015
23092015
30092015
07102015
...
Take a look here
https://docs.python.org/2/library/datetime.html#strftime-and-strptime-behavior
You can see the table pictured here for formatting dates and times and the usage.
Of course, if the format of the dates changes in the future or you are parsing different strings, you will have to make code changes. There really is no way around that.
I'm trying to select a range of dates from a date picker like this using selenium in python
Ex: Select date range today()+1 to today()+10 days
Currently, I am able to open this page click on the calendar and then click on the custom date. After which my next step would be to select the date range and click apply.
Below is the code:
`
from selenium import webdriver
from datetime import date, timedelta
browser = webdriver.Chrome('*driver path*')
browser.get('https://demo.improvely.com/reports/webshop/ads?imt=1&utm_campaign=Date+Range+Picker&utm_source=Site+Ads&utm_medium=Banner&utm_content=Blog+Demo+Image')
date_picker = browser.find_element_by_id('daterange').click()
browser.find_element_by_xpath("/html/body/div[7]/div[1]/ul/li[7]").click()
today = date.today()
from_date = today + timedelta(days=1)
to_date = today + timedelta(days=10)
`
Please help me out with a solution to get this done.
Many thanks in advance
Below xpath will give you all the dates in current month
elements = driverInstance.find_elements_by_xpath("//div[#class='calendar left']/descendant::*[#class='table-condensed']/child::*[2]/descendant::*")
Traverse through all the dates and select desired one.
for dates in elements:
Selectdate = dates.get_attribute("innerText")
if Selectdate== from_date:
dates.click()
Hope this helps.
I have a dictionary with many sorted dates. How could I write a loop in Python that checks if a certain date is in the dictionary and if not, it returns the closest date available? I want it to work that if after subtracting one day to the date, it checks again if now it exists in the dictionary and if not, it subtracts again until it finds a existing date.
Thanks in advance
from datetime import timedelta
def function(date):
if date not in dictio:
date -= timedelta(days=1)
return date
I've made a recursive function to solve your problem:
import datetime
def find_date(date, date_dict):
if date not in date_dict.keys():
return find_date(date-datetime.timedelta(days=1), date_dict)
else:
return date
I don't know what is the content of your dictionary but the following example should show you how this works:
import numpy as np
# creates a casual dates dictionary
months = np.random.randint(3,5,20)
days = np.random.randint(1,30,20)
dates = {
datetime.date(2019,m,d): '{}_{:02}_{:02}'.format(2019,m,d)
for m,d in zip(months,days)}
# select the date to find
target_date = datetime.date(2019, np.random.randint(3,5), np.random.randint(1,30))
# print the result
print("The date I wanted: {}".format(target_date))
print("The date I got: {}".format(find_date(target_date, dates)))
What you are looking for is possibly a while loop, although beware because if it will not find the date it will run to infinite. Perhaps you want to define a limit of attempts until the script should give up?
from datetime import timedelta, date
d1 = {
date(2019, 4, 1): None
}
def function(date, dictio):
while date not in dictio:
date -= timedelta(days=1)
return date
res_date = function(date.today(), d1)
print(res_date)
Revised question with appropriate MCVE:
As part of a script I'm writing I need to have a loop that contains a different pair of dates during each iteration, these dates are the first and last available stock trading dates of each month. I have managed to find a calendar with the available dates in an index however despite my research I am not sure how to select the correct dates from this index so that they can be used in the DateTime variables start and end.
Here is as far as my research has got me and I will continue to search for and build my own solution which I will post if I manage to find one:
from __future__ import division
import numpy as np
import pandas as pd
import datetime
import pandas_market_calendars as mcal
from pandas_datareader import data as web
from datetime import date
'''
Full date range:
'''
startrange = datetime.date(2016, 1, 1)
endrange = datetime.date(2016, 12, 31)
'''
Tradable dates in the year:
'''
nyse = mcal.get_calendar('NYSE')
available = nyse.valid_days(start_date='2016-01-01', end_date='2016-12-31')
'''
The loop that needs to take first and last trading date of each month:
'''
dict1 = {}
for i in available:
start = datetime.date('''first available trade day of the month''')
end = datetime.date('''last available trade day of the month''')
diffdays = ((end - start).days)/365
dict1 [i] = diffdays
print (dict1)
That is probably because 1 January 2016 was not a trading day. To check if I am right, try giving it the date 4 January 2016, which was the following Monday. If that works, then you will have to be more sophisticated about the dates you ask for.
Look in the documentaion for dm.BbgDataManager(). It is possible that you can ask it what dates are available.