Develop Reference

How can I find a dynamic url on a webpage?

Guys I want to be able to know how you can find a dynamic url on a website. Primarily, I am looking for the search term at the end of a website. For example how would I find the link https://www.abbeywhisky.com/pages/search-results-page?q= from entering the front page of https://www.abbeywhisky.com
I am unsure if there is a way to do this by just using the landing page of a site. I would have tried to scrape the first page of the site but using the filter of "?=" does not shown any results.

How to scrape information from websites with subframes

I am trying to build a simple web scraper to extract flight information from the student Universe.
I used the selenium to navigate the webpages to get the flight information for my desired location and date. There is no problem for me to get to the right page with all the information.
However, I have difficulties in extracting the information from the webpage. I used xpath to locate those elements that contain desired info, but extracting the information is unsuccessful unless I manually scroll up and down the webpage. It seems that this has something to do with subframes embedded in the website. I tried to iterate all the iframe to see if I get information with the command driver.switch_to.frame(), but the problem remains.
It would be great if anyone could offer some help regarding how to scrape information from websites like this. The problem may not be caused by the existence of subframe. any input is appreciated.
The code I used to extract flights info is shown below, an article tag contains all the info(carrier name, departure time,arrival time and so on). What I did first is to locate this element.
def parseprice(driver):
driver.maximize_window()
parser = lxml.html.fromstring(driver.page_source,driver.current_url)
flights=parser.xpath('//article[#class="itin activeCheck"]')
driver.quit()
carriername=flights[0].xpath('//p[#id="airlineName0"]/text()')
duration=flights[0].xpath('//strong[#id="duration0"]/text()')
depttime=flights[0].xpath('//span[#id="departureTime0"]/text()')
arrtime=flights[0].xpath('//span[#id="arrivalTime0"]/text()')
price=flights[0].xpath('//p[#ng-click="pricePoint()"]//text()')
stops=flights[0].xpath('//p[#id="stops0"]//text()')
stoplis=list()
for st in stops:
res1=re.search('^(\d)+\D*',st)
if res1 is not None:
stoplis.append(int(res1.group(1)))
now=datetime.datetime.now()
now=now.timetuple()
for i in range(20):
yield{'current time':str(now[1])+'/'+str(now[2])+'/'+str(now[0]),'carrier':carriername[i],'duration':duration[i],'price':price[i],'numstops':stoplis[i],'departure_time':depttime[i],'arrival_time':arrtime[i]}

How to access the subtags within a tag using beautifulsoup in python?

I am attempting to retrieve player statistics from MLB.com for the 2016 season. I am using Beautiful Soup in Python, and I need to extract the information in the table seen here:
http://mlb.mlb.com/stats/sortable.jsp#elem=%5Bobject+Object%5D&tab_level=child&click_text=Sortable+Player+hitting&game_type='R'&season=2016&season_type=ANY&league_code='MLB'&sectionType=sp&statType=hitting&page=1&ts=1493672037085&playerType=ALL&sportCode='mlb'&split=&team_id=&active_sw=&position=&page_type=SortablePlayer&sortOrder='desc'&sortColumn=ab&results=&perPage=442&timeframe=&last_x_days=&extended=0
Here is what I have attempted:
r=requests.get(url)
soup=BeautifulSoup(r.content,'html.parser')
gdata=soup.find_all('div',{'id':'datagrid'})
print(gdata)
This should return all of the subtags within the tag, but it does not. This results in the following:
[<div id="datagrid"></div>]
Can anyone explain why this is not producing the contents of the table? Furthermore, what can I do to access the contents of the table?
Thanks

If you look at the source for the webpage, it looks like the datagrid div is actually empty & the stats are inserted dynamically as json from this URL. Maybe you can use that instead. To figure this out I looked at the page source to see that the div had no children and then used Chrome developer tools Network tab to find the request where it pulled the data:
Open the web page
Open the chrome developer tools, Command+Option+I (Mac) or Control+Shift+I (Windows, Linux).
Refresh the web page with the tools open so it processes the network requests then wait for the page to load
(optional) Type xml in the search bar on the web to narrow your search results to requests that are likely to have data
Click on each request and look at the preview of the response. At this point I just manually examined the responses to see which had your data. I got lucky and got yours on the first try since it has stats in the name.

Using BeautifulSoup to extract multiple pages of data

I am using BeautifulSoup python package to scrape a table of data from a webpage. The tables have many pages that can be clicked through, I was hoping that i could extract each page of the table by running through adjusted URL's that identify the page but this particular site is statically updating the table using javascript that changes the source code.
Does anyone know a work-around? I am new to BeautifulSoup and do not know if there is a way to do this.

Is a Web Crawler more suitable?

TL;DR Version :
I have only heard about web crawlers in intelluctual conversations Im not part of. All I want to know that can they follow a specific path like:
first page (has lot of links) -->go to links specified-->go to
links(specified, yes again)-->go to certain link-->reach final page
and download source.
I have googled a bit and came across Scrappy. But I am not sure if I fully understand web crawlers to begin with and if scrappy can help me follow the specific path I want.
Long Version
I wanted to extract some text of a group of static web pages. These web pages are very simple with just basic HTML. I used python and the urllib to access the URL,extract the text and work with it. Pretty soon I realized that I will have to basically visit all these pages and copy paste the URL into my program, which is tiresome. I wanted to know if this is more suitable for a web crawler. I want to access this
page. Then select only a few organisms (I have a list of those). On Clicking on of them you can see this page. If you look under the table - MTases active in the genome there are Enzymes which are hyperlinks. Clinking on those lead to this page. On the right hand side there is link named Sequence Data. Once clicked it leads to the page which has a small table on the lower right with yellow headers. under it it has an entry DNA (FASTA STYLE. Clicking on view will lead to the page im interested in and want to download the page source from.

I think you are definitely on the right track for looking at a web crawler to help you do this. You can also look at Norconex HTTP Collector which I know can let you follow links on a page without storing that page if is is just a listing page to you. That crawler lets you filter out pages after their links have been extracted to be followed. Ultimately, you can configure the right filters so that only the pages matching the pattern you want get downloaded for you to process (whether it is based on crawl depth, URL pattern, content pattern, etc).