Develop Reference

Adding an open close Google Chrome browser to Selenium linkedin_scraper code

I am trying to scrape some LinkedIn profiles of well known people. The code takes a bunch of LinkedIn profile URLS and then uses Selenium and scrape_linkedin to collect the information and save it into a folder as a .json file.
The problem I am running into is that LinkedIn naturally blocks the scraper from collecting some profiles. I am always able to get the first profile in the list of URLs. I put this down to the fact that it opens a new Google Chrome window and then goes to the LinkedIn page. (I could be wrong on this point however.)
What I would like to do is to add to the for loop a line which opens a new Google Chrome session and once the scraper has collected the data close the Google Chrome session such that on the next iteration in the loop it will open up a fresh new Google Chrome session.
From the package website here it states:
driver {selenium.webdriver}: driver type to use
default: selenium.webdriver.Chrome
Looking at the Selenium package website here I see:
driver = webdriver.Firefox()
...
driver.close()
So Selenium does have a close() option.
How can I add an open and close Google Chrome browser to the for loop?
I have tried alternative methods to try and collect the data such as changing the time.sleep() to 10 minutes, to changing the scroll_increment and scroll_pause but it still does not download the whole profile after the first one has been collected.
Code:
from datetime import datetime
from scrape_linkedin import ProfileScraper
import pandas as pd
import json
import os
import re
import time
my_profile_list = ['https://www.linkedin.com/in/williamhgates/', 'https://www.linkedin.com/in/christinelagarde/', 'https://www.linkedin.com/in/ursula-von-der-leyen/']
# To get LI_AT key
# Navigate to www.linkedin.com and log in
# Open browser developer tools (Ctrl-Shift-I or right click -> inspect element)
# Select the appropriate tab for your browser (Application on Chrome, Storage on Firefox)
# Click the Cookies dropdown on the left-hand menu, and select the www.linkedin.com option
# Find and copy the li_at value
myLI_AT_Key = 'INSERT LI_AT Key'
with ProfileScraper(cookie=myLI_AT_Key, scroll_increment = 50, scroll_pause = 0.8) as scraper:
for link in my_profile_list:
print('Currently scraping: ', link, 'Time: ', datetime.now())
profile = scraper.scrape(url=link)
dataJSON = profile.to_dict()
profileName = re.sub('https://www.linkedin.com/in/', '', link)
profileName = profileName.replace("?originalSubdomain=es", "")
profileName = profileName.replace("?originalSubdomain=pe", "")
profileName = profileName.replace("?locale=en_US", "")
profileName = profileName.replace("?locale=es_ES", "")
profileName = profileName.replace("?originalSubdomain=uk", "")
profileName = profileName.replace("/", "")
with open(os.path.join(os.getcwd(), 'ScrapedLinkedInprofiles', profileName + '.json'), 'w') as json_file:
json.dump(dataJSON, json_file)
time.sleep(10)
print('The first observation scraped was:', my_profile_list[0:])
print('The last observation scraped was:', my_profile_list[-1:])
print('END')

Here is a way to open and close tabs/browser.
from datetime import datetime
from scrape_linkedin import ProfileScraper
import random #new import made
from selenium import webdriver #new import made
import pandas as pd
import json
import os
import re
import time
my_profile_list = ['https://www.linkedin.com/in/williamhgates/', 'https://www.linkedin.com/in/christinelagarde/',
'https://www.linkedin.com/in/ursula-von-der-leyen/']
myLI_AT_Key = 'INSERT LI_AT Key'
for link in my_profile_list:
my_driver = webdriver.Chrome() #if you don't have Chromedrive in the environment path then use the next line instead of this
#my_driver = webdriver.Chrome(executable_path=r"C:\path\to\chromedriver.exe")
#sending our driver as the driver to be used by srape_linkedin
#you can also create driver options and pass it as an argument
ps = ProfileScraper(cookie=myLI_AT_Key, scroll_increment=random.randint(10,50), scroll_pause=0.8 + random.uniform(0.8,1),driver=my_driver) #changed name, default driver and scroll_pause time and scroll_increment made a little random
print('Currently scraping: ', link, 'Time: ', datetime.now())
profile = ps.scrape(url=link) #changed name
dataJSON = profile.to_dict()
profileName = re.sub('https://www.linkedin.com/in/', '', link)
profileName = profileName.replace("?originalSubdomain=es", "")
profileName = profileName.replace("?originalSubdomain=pe", "")
profileName = profileName.replace("?locale=en_US", "")
profileName = profileName.replace("?locale=es_ES", "")
profileName = profileName.replace("?originalSubdomain=uk", "")
profileName = profileName.replace("/", "")
with open(os.path.join(os.getcwd(), 'ScrapedLinkedInprofiles', profileName + '.json'), 'w') as json_file:
json.dump(dataJSON, json_file)
time.sleep(10 + random.randint(0,5)) #added randomness to the sleep time
#this will close your browser at the end of every iteration
my_driver.quit()
print('The first observation scraped was:', my_profile_list[0:])
print('The last observation scraped was:', my_profile_list[-1:])
print('END')
This scraper by default uses Chrome as the browser but also gives the freedom to choose what browser you want to use in all possible places like CompanyScraper, ProfileScraper, etc.
I have just changed the default arguments to be passed in the initialization of ProfileScrapper() class and made your driver run browser and close it rather than the default one, added some random time into the wait/sleep intervals as you had requested(you can tweak it as per your needs. You can change the Random Noise I have added to your comfort.
There is no need to use scrape_in_parallel() as I had suggested in my comments but if you want to then, you can define the number of browser instances(num_instances) you want to run along with your own dictionary of drivers having it's own options too(in a another dictionary) :
from scrape_linkedin import scrape_in_parallel, CompanyScraper
from selenium import webdriver
driver1 = webdriver.Chrome()
driver2 = webdriver.Chrome()
driver3 = webdriver.Chrome()
driver4 = webdriver.Chrome()
my_drivers = [driver1,driver2,driver3,driver4]
companies = ['facebook', 'google', 'amazon', 'microsoft', ...]
driver_dict = {}
for i in range(1,len(my_drivers)+1):
driver_dict[i] = my_drivers[i-1]
#Scrape all companies, output to 'companies.json' file, use 4 browser instances
scrape_in_parallel(
scraper_type=CompanyScraper,
items=companies,
output_file="companies.json",
num_instances=4,
driver= driver_dict
)
It's an open source code and since it's written solely in Python you can understand the source code very easily. It's quite an interesting scraper, thank you for letting me know about it too!
NOTE:
There are some concerning unresolved issues in this module as it's told in it's GitHub Issues tab. I would wait for a few more forks and updates if I were you if this doesn't work properly.

Upload button on Instagram won't accept Selenium send_keys

I am coding a Selenium bot (with Python) that uploads a picture to instagram from a queue in a directory. For now, I have successfully logged in on Instagram and I am now trying to interact somehow with the upload button.
I have tried to click() on it, but then a window pops up where I would normally browse my computer to find the image I want to upload. I've found that I need import autoit, but I can't understand how it works and the documentation doesn't help either, so I'd rather avoid using this.
This is what I have for now:
import os
from selenium import webdriver
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
class InstaBot():
# A COOKIES POP UP ALWAYS APPEARS UPON OPENING INSTAGRAM, SO INIT ALSO CLOSES IT
# TO UPLOAD ON INSTAGRAM, THE MOBILE VERSION IS NEEDED, WE TRY TO EMULATE A GALAXY S5
def __init__(self):
mobile_emulation = {"deviceName": "Galaxy S5"}
chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option("mobileEmulation", mobile_emulation)
# experimental options for mobile emulation added
self.driver = webdriver.Chrome(ChromeDriverManager().install(), chrome_options = chrome_options)
self.driver.get("https://www.instagram.com/")
# note that this accepts all cookies
cooki = self.driver.find_element_by_xpath('/html/body/div[2]/div/div/div/div[2]/button[1]')
cooki.click()
# FINDS THE USERNAME AND PASSWORD AND TYPES 2 INPUTS ACCORDINGLY
def loginfun(self):
entrar = WebDriverWait(self.driver, 10).until(EC.presence_of_element_located((By.XPATH, '//*[#id="react-root"]/section/main/article/div/div/div/div[2]/button')))
entrar.click()
usbar = pasbar = self.driver.find_element_by_xpath('//*[#id="loginForm"]/div[1]/div[3]/div/label/input')
usbar.send_keys(input('Username: '))
# once usbar is found, the rest will be as well
pasbar = self.driver.find_element_by_xpath('//*[#id="loginForm"]/div[1]/div[4]/div/label/input')
pasbar.send_keys(input('Password: '))
logbtn = self.driver.find_element_by_xpath('//*[#id="loginForm"]/div[1]/div[6]/button')
logbtn.click()
# CLOSES PASSWORD SAVING AND NOTIFICATION MESSAGES IN CASE THEY APPEAR
def tryclose(self):
try:
nopass = WebDriverWait(self.driver, 10).until(EC.presence_of_element_located((By.XPATH, '//*[#id="react-root"]/section/main/div/div/div/button')))
nopass.click()
except Exception:
pass
try:
nonot = WebDriverWait(self.driver, 10).until(EC.presence_of_element_located((By.XPATH, '/html/body/div[4]/div/div/div/div[3]/button[2]')))
nonot.click()
except Exception:
pass
# bla bla bla
# Basically I emulate a mobile device and log into my account,
# then I have at the bottom center of the page the upload button that
# looks like this [+]. Here is what I try:
# SELECTS THE FIRST IMAGE FROM THE PENDING DIR. USES THE UPLOAD BUTTON VIA SEND_KEYS.
# AFTER THAT, IT MOVES SAID IMAGE TO THE 'DONE' DIR.
def upload(self):
# first pending image
pend_img = (os.listdir('C:/path to my queue dir')[0])
# finds the upload button and send_keys the image to it
upbtn = self.driver.find_element_by_xpath('//*[#id="react-root"]/section/nav[2]/div/div/form/input')
upbtn.send_keys('C:/path to image in queue'+pend_img)
# moves the image to the 'done' directory
os.rename('C:/path to image in queue dir'+pend_img , \
'C:/path to image in done dir'+pend_img)
After this process, this code is able to find the image in the 'pending' (queue) directory and move it to the 'done' directory, BUT it does not interact with instagram whatsoever. So send_keys() is not working. I am a newbie in this, but I recon that the HTML path to the button upbtn may be wrong, although I cannot find any other input path or anything.
NOTE: to clarify, no errors are shown, the problem is just that send_keys does not interact with the upload button with this code.
Anyone has a fool-proof solution or an intuitive way to upload to Instagram?
Following advice from the comments, I looked for all the input tags in the HTML of the page:
After my search, these are all the XPaths to the input tags I could find, all of them have type="file":
//*[#id="react-root"]/form/input
//*[#id="react-root"]/section/main/div[1]/form/input
//*[#id="react-root"]/section/nav[1]/div/div/form/input # this one is for stories i think
//*[#id="react-root"]/section/nav[2]/div/div/form/input # it should be this one
I have tried send_keys() to all of them, yet none seems to work.

If you don't mind, I recommend a package that might help and works so good!
Instabot is capable to upload photos/videos & stories emulating a mobile device.
Installation:
pip install instabot
Implementation:
#Call bot
from instabot import Bot
Then only need a few more steps:
bot = Bot()
bot.login(username = 'user', password = 'pass')
bot.upload_photo(image_path, caption = 'Hello world')
I recommend this option due to is clean, fast and reliable
More info, visit: https://pypi.org/project/instabot/

Selenium Python Download popup pdf with specific filename

I need to download a set of individual pdf files from a webpage. It is publicly available by government (ministry of education in Turkey) so totally legal.
However my selenium browser only displays the pdf file, how can I download it and name as I wish.
(This code is also from web)
# Import your newly installed selenium package
from selenium import webdriver
from bs4 import BeautifulSoup
# Now create an 'instance' of your driver
# This path should be to wherever you downloaded the driver
driver = webdriver.Chrome(executable_path="/Users/ugur/Downloads/chromedriver")
# A new Chrome (or other browser) window should open up
download_dir = "/Users/ugur/Downloads/" # for linux/*nix, download_dir="/usr/Public"
options = webdriver.ChromeOptions()
profile = {"plugins.plugins_list": [{"enabled": False, "name": "Chrome PDF Viewer"}], # Disable Chrome's PDF Viewer
"download.default_directory": download_dir , "download.extensions_to_open": "applications/pdf"}
options.add_experimental_option("prefs", profile)
# Now just tell it wherever you want it to go
driver.get("https://odsgm.meb.gov.tr/kurslar/KazanimTestleri.aspx?sinifid=5&ders=29")
driver.find_element_by_id("ContentPlaceHolder1_dtYillikPlanlar_lnkIndir_2").click()
driver.get("https://odsgm.meb.gov.tr/kurslar/PDFFile.aspx?name=kazanimtestleri.pdf")
Thanks in advance
Extra information:
I had a python 2 code doing this perfectly. But somehow it creates empty files and I couldn't convert it to python 3. Maybe this helps (no offense but I never liked selenium)
import urllib
import urllib2
from bs4 import BeautifulSoup
import os
sinifId=5
maxOrd = 1
fileNames=[]
directory = '/Users/ugur/Downloads/Hasan'
print 'List of current files in directory '+ directory+'\n---------------------------------\n\n'
for current_file in os.listdir(directory):
if (current_file.find('pdf')>-1 and current_file.find(' ')>-1):
print current_file
order = int(current_file.split(' ',1)[0])
if order>maxOrd: maxOrd=order
fileNames.append(current_file.split(' ',2)[1])
print '\n\nStarting download \n---------------------------------\n'
ctA=int(maxOrd+1)
for ders in [29]:
urlSinif='http://odsgm.meb.gov.tr/kurslar/KazanimTestleri.aspx?sinifid='+str(sinifId)+'&ders='+str(ders)
page = urllib2.urlopen(urlSinif)
soup = BeautifulSoup(page,"lxml")
st = soup.prettify()
count=st.count('ctl00')-1
dersAdi = soup.find('a', href='/kurslar/CevapAnahtarlari.aspx?sinifid='+str(sinifId)+'&ders='+str(ders)).getText().strip()
for testNo in range(count):
if(str(sinifId)+str(ders)+str(testNo+1) in fileNames):
print str(ctA)+' '+str(sinifId)+str(ders)+str(testNo+1)+' '+dersAdi+str(testNo+1)+'.pdf'+' skipped'
else:
annex=""
if(testNo%2==1): annex="2"
eiha_url = u'http://odsgm.meb.gov.tr/kurslar/KazanimTestleri.aspx?sinifid='+str(sinifId)+'&ders='+str(ders)
data = ('__EVENTTARGET','ctl00$ContentPlaceHolder1$dtYillikPlanlar$ctl'+format(testNo, '02')+'$lnkIndir'+annex), ('__EVENTARGUMENT', '39')
print 'ctl00$ContentPlaceHolder1$dtYillikPlanlar$ctl'+format(testNo, '02')+'$lnkIndir'+annex
new_data = urllib.urlencode(data)
response = urllib2.urlopen(eiha_url, new_data)
urllib.urlretrieve (str(response.url), directory+'/{0:0>3}'.format(ctA)+' '+str(sinifId)+str(ders)+str(testNo+1)+' '+dersAdi+str(testNo+1)+'.pdf')
print str(ctA)+' '+str(sinifId)+str(ders)+str(testNo+1)+' '+dersAdi+str(testNo+1)+'.pdf'+' downloaded'
ctA=ctA+1

Add your options before launching Chrome and then specify the chrome_options parameter.
download_dir = "/Users/ugur/Downloads/"
options = webdriver.ChromeOptions()
profile = {"plugins.plugins_list": [{"enabled": False, "name": "Chrome PDF Viewer"}],
"download.default_directory": download_dir,
"download.extensions_to_open": "applications/pdf"}
options.add_experimental_option("prefs", profile)
driver = webdriver.Chrome(
executable_path="/Users/ugur/Downloads/chromedriver",
chrome_options=options
)
To answer your second question:
May I ask how to specify the filename as well?
I found this: Selenium give file name when downloading
What I do is:
file_name = ''
while file_name.lower().endswith('.pdf') is False:
time.sleep(.25)
try:
file_name = max([download_dir + '/' + f for f in os.listdir(download_dir)], key=os.path.getctime)
except ValueError:
pass

Here is the code sample I used to download pdf with a specific file name. First you need to configure chrome webdriver with required options. Then after clicking the button (to open pdf popup window), call a function to wait for download to finish and rename the downloaded file.
import os
import time
import shutil
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
# function to wait for download to finish and then rename the latest downloaded file
def wait_for_download_and_rename(newFilename):
# function to wait for all chrome downloads to finish
def chrome_downloads(drv):
if not "chrome://downloads" in drv.current_url: # if 'chrome downloads' is not current tab
drv.execute_script("window.open('');") # open a new tab
drv.switch_to.window(driver.window_handles[1]) # switch to the new tab
drv.get("chrome://downloads/") # navigate to chrome downloads
return drv.execute_script("""
return document.querySelector('downloads-manager')
.shadowRoot.querySelector('#downloadsList')
.items.filter(e => e.state === 'COMPLETE')
.map(e => e.filePath || e.file_path || e.fileUrl || e.file_url);
""")
# wait for all the downloads to be completed
dld_file_paths = WebDriverWait(driver, 120, 1).until(chrome_downloads) # returns list of downloaded file paths
# Close the current tab (chrome downloads)
if "chrome://downloads" in driver.current_url:
driver.close()
# Switch back to original tab
driver.switch_to.window(driver.window_handles[0])
# get latest downloaded file name and path
dlFilename = dld_file_paths[0] # latest downloaded file from the list
# wait till downloaded file appears in download directory
time_to_wait = 20 # adjust timeout as per your needs
time_counter = 0
while not os.path.isfile(dlFilename):
time.sleep(1)
time_counter += 1
if time_counter > time_to_wait:
break
# rename the downloaded file
shutil.move(dlFilename, os.path.join(download_dir,newFilename))
return
# specify custom download directory
download_dir = r'c:\Downloads\pdf_reports'
# for configuring chrome pdf viewer for downloading pdf popup reports
chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option('prefs', {
"download.default_directory": download_dir, # Set own Download path
"download.prompt_for_download": False, # Do not ask for download at runtime
"download.directory_upgrade": True, # Also needed to suppress download prompt
"plugins.plugins_disabled": ["Chrome PDF Viewer"], # Disable this plugin
"plugins.always_open_pdf_externally": True, # Enable this plugin
})
# get webdriver with options for configuring chrome pdf viewer
driver = webdriver.Chrome(options = chrome_options)
# open desired webpage
driver.get('https://mywebsite.com/mywebpage')
# click the button to open pdf popup
driver.find_element_by_id('someid').click()
# call the function to wait for download to finish and rename the downloaded file
wait_for_download_and_rename('My file.pdf')
# close the browser windows
driver.quit()
Set timeout (120) to the wait time as per your needs.

Non-selenium solution, You can do something like:
import requests
pdf_resp = requests.get("https://odsgm.meb.gov.tr/kurslar/PDFFile.aspx?name=kazanimtestleri.pdf")
with open("save.pdf", "wb") as f:
f.write(pdf_resp.content)
Although you might want to check the content type before to make sure it's a pdf

Automate the send email test report of test cases using selenium and python

How do i automate send email test report to recipient from selenium webdriver using python. I'm currently using pycharm IDE to automate test cases.
I want send email to someone at last when this test case ran by me. Mail should have success and failed count. I tried to find the solution but failed.This is my test case. Please give me the solution.
from selenium import webdriver
from generic_functions.FindElement import HandyWrappers
from generic_functions.takescreenshots import Screenshot
from generic_functions.error_handle import CatchExceptions
import os
import time
import unittest
class TestHome(unittest.TestCase):
driverLocation = "C:\\Users\\Sales\\Desktop\\Automation\\ABCD\\libs\\chromedriver.exe"
os.environ["webdriver.chrome.driver"] = driverLocation
driver = webdriver.Chrome(driverLocation)
driver.maximize_window()
driver.implicitly_wait(10)
def test_home(self):
try:
baseURL = "https://portal.abcd.com"
driver = self.driver
driver.get(baseURL)
hw = HandyWrappers(driver)
# regionLogin
username = hw.getElement(".//form[#id='login-form']/fieldset/section[1]/label[2]/input[#name='email']",
locatorType="xpath")
username.send_keys("abcd#live.com")
time.sleep(2)
password = hw.getElement(".//form[#id='login-form']/fieldset/section[2]/label[2]/input[#name='password']",
locatorType="xpath")
password.send_keys("abcd")
signIn = hw.getElement(".//form[#id='login-form']/footer/button[contains(text(),'Sign in')]",
locatorType="xpath")
signIn.click()
Screenshot.takeScreenshot(driver)
# endregion
time.sleep(2)
driver.execute_script("window.scrollBy(0,300);")
time.sleep(1)
# Switch to Iframe from main page
driver.switch_to.frame("ifrmClaimSummary")
# regionScrolling Right and left
driver.execute_script("window.scrollBy(100,0);")
time.sleep(1)
driver.execute_script("window.scrollBy(100,0);")
time.sleep(1)
driver.execute_script("window.scrollBy(100,0);")
time.sleep(1)
# Scrolling Left
driver.execute_script("window.scrollBy(-100,0);")
time.sleep(1)
driver.execute_script("window.scrollBy(-100,0);")
time.sleep(1)
driver.execute_script("window.scrollBy(-100,0);")
time.sleep(1)
# endregion
driver.switch_to.default_content()
driver.execute_script("window.scrollBy(0,-300);")
time.sleep(2)
driver.quit()
except Exception:
CatchExceptions.PrintException()
Screenshot.takeScreenshot(driver)
if __name__ == '__main__':
unittest.main()

Based on your comments, here's an approach you can follow to accomplish test reporting and email results:
Consider leveraging unittest for test reporting - Selenium has an example of this in their docs.
Then, you can use smtplib to send emails. There are a myriad of resources for this; here's one tutorial on smtplib with Gmail.
In a nutshell:
Use unittest to set up a test suite
Access the test results via unittest's API to do so
Build whatever string you want to send, and send it off with smtplib

Selenium Webdriver with Firebug + NetExport + FireStarter not creating a har file in Python

I am currently running Selenium with Firebug, NetExport, and (trying out) FireStarter in Python trying to get the network traffic of a URL. I expect a HAR file to appear in the directory listed, however nothing appears. When I test it in Firefox and go through the UI, a HAR file is exported and saved so I know the code itself functions as expected. After viewing multiple examples I do not see what I am missing.
I am using Firefox 29.0.1
Firebug 1.12.8
FireStarter 0.1a6
NetExport 0.9b6
Has anyone else encountered this issue? I am receiving a "webFile.txt" file being filled out correctly.
After looking up each version of the add-ons they are supposed to be compatible with the version of Firefox I am using. I tried using Firefox version 20, however that did not help. I am currently pulling source code.
In addition I have tried it with and without FireStarter, and I have tried refreshing the page manually in both cases to try to generate a HAR.
My code looks like this:
import urllib2
import sys
import re
import os
import subprocess
import hashlib
import time
import datetime
from browsermobproxy import Server
from selenium import webdriver
import selenium
a=[];
theURL='';
fireBugPath = '/Users/tai/Documents/workspace/testSelenium/testS/firebug.xpi';
netExportPath = '/Users/tai/Documents/workspace/testSelenium/testS/netExport.xpi';
fireStarterPath = '/Users/tai/Documents/workspace/testSelenium/testS/fireStarter.xpi';
profile = webdriver.firefox.firefox_profile.FirefoxProfile();
profile.add_extension( fireBugPath);
profile.add_extension(netExportPath);
profile.add_extension(fireStarterPath);
#firefox preferences
profile.set_preference("app.update.enabled", False)
profile.native_events_enabled = True
profile.set_preference("webdriver.log.file", "/Users/tai/Documents/workspace/testSelenium/testS/webFile.txt")
profile.set_preference("extensions.firebug.DBG_STARTER", True);
profile.set_preference("extensions.firebug.currentVersion", "1.12.8");
profile.set_preference("extensions.firebug.addonBarOpened", True);
profile.set_preference("extensions.firebug.addonBarOpened", True);
profile.set_preference('extensions.firebug.consoles.enableSite', True)
profile.set_preference("extensions.firebug.console.enableSites", True);
profile.set_preference("extensions.firebug.script.enableSites", True);
profile.set_preference("extensions.firebug.net.enableSites", True);
profile.set_preference("extensions.firebug.previousPlacement", 1);
profile.set_preference("extensions.firebug.allPagesActivation", "on");
profile.set_preference("extensions.firebug.onByDefault", True);
profile.set_preference("extensions.firebug.defaultPanelName", "net");
#set net export preferences
profile.set_preference("extensions.firebug.netexport.alwaysEnableAutoExport", True);
profile.set_preference("extensions.firebug.netexport.autoExportToFile", True);
profile.set_preference("extensions.firebug.netexport.saveFiles", True);
profile.set_preference("extensions.firebug.netexport.autoExportToServer", False);
profile.set_preference("extensions.firebug.netexport.Automation", True);
profile.set_preference("extensions.firebug.netexport.showPreview", False);
profile.set_preference("extensions.firebug.netexport.pageLoadedTimeout", 15000);
profile.set_preference("extensions.firebug.netexport.timeout", 10000);
profile.set_preference("extensions.firebug.netexport.defaultLogDir", "/Users/tai/Documents/workspace/testSelenium/testS/har");
profile.update_preferences();
browser = webdriver.Firefox(firefox_profile=profile);
def openURL(url,s):
theURL = url;
time.sleep(6);
#browser = webdriver.Chrome();
browser.get(url); #load the url in firefox
time.sleep(3); #wait for the page to load
browser.execute_script("window.scrollTo(0, document.body.scrollHeight/5);")
time.sleep(1); #wait for the page to load
browser.execute_script("window.scrollTo(0, document.body.scrollHeight/4);")
time.sleep(1); #wait for the page to load
browser.execute_script("window.scrollTo(0, document.body.scrollHeight/3);")
time.sleep(1); #wait for the page to load
browser.execute_script("window.scrollTo(0, document.body.scrollHeight/2);")
time.sleep(1); #wait for the page to load
browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
searchText='';
time.sleep(20); #wait for the page to load
if(s.__len__() >0):
for x in range(0, s.__len__()):
searchText+= ("" + browser.find_element_by_id(x));
else:
searchText+= browser.page_source;
a=getMatches(searchText)
#print ("\n".join(swfLinks));
print('\n'.join(removeNonURL(a)));
# print(browser.page_source);
browser.quit();
return a;
def found_window(name):
try: browser.switch_to_window(name)
except NoSuchWindowException:
return False
else:
return True # found window
def removeFirstQuote(tex):
for x in tex:
b = x[1:];
if not b in a:
a.append(b);
return a;
def getMatches(t):
return removeFirstQuote(re.findall('([\"|\'][^\"|\']*\.swf)', t));
def removeNonURL(t):
a=[];
for b in t:
if(b.lower()[:4] !="http" ):
if(b[0] == "//"):
a.append(theURL+b[2:b.__len__()]);
else:
while(b.lower()[:4] !="http" and b.__len__() >5):
b=b[1:b.__len__()];
a.append(b);
else:
a.append(b);
return a;
openURL("http://www.chron.com",a);

I fixed this issue for my own work by setting a longer wait before closing the browser. I think you are currently setting netexport to export after the program has quit, so no file is written. The line causing this is:
profile.set_preference("extensions.firebug.netexport.pageLoadedTimeout", 15000);
From the netexport source code we have that pageLoadedTimeout is the `Number of milliseconds to wait after the last page request to declare the page loaded'. So I suspect all your minor page loads are preventing netexport from having enough time to write the file. One caveat is you set the system to automatically export after 10s so I'm not sure why you are not acquiring half loaded json files.