Develop Reference

Why is DNSPython zone transfer sporadically failing

So, I'm still at the noob level when it comes to python. I know... I know... there's probably a more efficient way to do what I'm trying but still learning and hopefully, I'll get better with practice.
For a training project, I'm writing a script to do various DNS operations against a domain. I found DNSPython and it seemed to be exactly what I needed to use and I thought I was done with it but when I tried it against a different domain it keeps failing at the zone transfer.
I've got two domains hardcoded right now for testing. The megacorpone domain iw was working as I expected however, now it's failing (with no code change) in order to get it to work I had to filter the first record '#' that was returned otherwise it failed as well.
However, the zonetransfer.me domain sometimes completes the script with error but fails errors sporadically as well, but it never displays the host records for some reason and I've not been able to figure out how to fix it yet, been banging my head against it for a while now.
The megacoprone run was working every time earlier now it's not working at all. The only thing I can think of so far is that it may be a timing issue.
Run with megacoprpone
Attempting zone transfers for megacorpone.com
Traceback (most recent call last):
File "/home/kali/Exercises/Module_7/dns-axfer.py", line 56, in zoneXFR
zone = dns.zone.from_xfr(dns.query.xfr(str(server).rstrip('.'), domain))
File "/usr/lib/python3/dist-packages/dns/zone.py", line 1106, in from_xfr
for r in xfr:
File "/usr/lib/python3/dist-packages/dns/query.py", line 627, in xfr
raise TransferError(rcode)
dns.query.TransferError: Zone transfer error: REFUSED
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/kali/Exercises/Module_7/dns-axfer.py", line 73, in <module>
zoneXFR()
File "/home/kali/Exercises/Module_7/dns-axfer.py", line 66, in zoneXFR
print ("\nResults for",server, "\nZone origin:", str(zone.origin).rstrip('.'))
UnboundLocalError: local variable 'zone' referenced before assignment
Run 1 with zonetransfer.me
Attempting zone transfers for zonetransfer.me
Results for nsztm1.digi.ninja.
Zone origin: zonetransfer.me
---------------------------------------------------------------------------
Results for nsztm1.digi.ninja.
Zone origin: zonetransfer.me
---------------------------------------------------------------------------
[*] Error: <class 'dns.resolver.NoAnswer'> The DNS response does not contain an answer to the question: _acme-challenge.zonetransfer.me. IN A
Results for nsztm2.digi.ninja.
Zone origin: zonetransfer.me
---------------------------------------------------------------------------
Results for nsztm2.digi.ninja.
Zone origin: zonetransfer.me
---------------------------------------------------------------------------
[*] Error: <class 'dns.resolver.NoAnswer'> The DNS response does not contain an answer to the question: _acme-challenge.zonetransfer.me. IN A
Run 2 with no code change (zonetransfer.me)
Attempting zone transfers for zonetransfer.me
Traceback (most recent call last):
File "/home/kali/Exercises/Module_7/dns-axfer.py", line 56, in zoneXFR
zone = dns.zone.from_xfr(dns.query.xfr(str(server).rstrip('.'), domain))
File "/usr/lib/python3/dist-packages/dns/zone.py", line 1106, in from_xfr
for r in xfr:
File "/usr/lib/python3/dist-packages/dns/query.py", line 596, in xfr
_net_write(s, tcpmsg, expiration)
File "/usr/lib/python3/dist-packages/dns/query.py", line 364, in _net_write
current += sock.send(data[current:])
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/kali/Exercises/Module_7/dns-axfer.py", line 73, in <module>
zoneXFR()
File "/home/kali/Exercises/Module_7/dns-axfer.py", line 66, in zoneXFR
print ("\nResults for",server, "\nZone origin:", str(zone.origin).rstrip('.'))
UnboundLocalError: local variable 'zone' referenced before assignment
My script: bash away... I can always take constructive criticism.
#!/usr/bin/python3
import sys, argparse
import dns.query
import dns.zone
import dns.resolver
from colorama import Fore, Style
bracket = f"{Fore.BLUE}[{Fore.GREEN}*{Fore.BLUE}]{Style.RESET_ALL} "
bracket_err = f"{Fore.BLUE}[{Fore.RED}*{Fore.BLUE}]{Style.RESET_ALL} "
'''
parser = argparse.ArgumentParser()
parser.add_argument('domain')
args = parser.parse_args()
'''
# domain = (sys.argv[1])
domain = 'megacorpone.com'
#domain = 'zonetransfer.me'
def line():
print ('-' * 75)
return None
def resolveDNS(system):
resolver = dns.resolver.Resolver()
results = resolver.query(system , "A")
return results
def getNS ():
name_servers = dns.resolver.query(domain, 'NS')
print ("\nThe name servers for " + domain + " are:")
line()
for system in name_servers:
A_records = resolveDNS(str(system))
for item in A_records:
answer = ','.join([str(item)])
print (bracket, "{:30}".format(str(system).rstrip('.')), "{:15}".format(answer))
return name_servers
def getMX():
mail_server = dns.resolver.query(domain, 'MX')
print("\nMail servers for", domain)
line()
for system in mail_server:
A_records = resolveDNS(str(system.exchange))
for item in A_records:
answer = ','.join([str(item)])
print(bracket, "{:30}".format(str(system.exchange).rstrip('.')), "{:15}".format(str(answer)), '\t', "{:5}".format("Preference:"), str(system.preference))
return None
def zoneXFR():
print ("\nAttempting zone transfers for", domain,)
for server in name_servers:
try:
zone = dns.zone.from_xfr(dns.query.xfr(str(server).rstrip('.'), domain))
print ("\nResults for",server, "\nZone origin:", str(zone.origin).rstrip('.'))
line()
for host in zone:
if str(host) != '#':
A_records = resolveDNS(str(host) + "." + domain)
for item in A_records:
answer = ','.join([str(item)])
print(bracket, "{:30}".format(str(host) + "." + domain), answer)
except Exception as e:
print ("\nResults for",server, "\nZone origin:", str(zone.origin).rstrip('.'))
line()
print (bracket_err, f"{Fore.RED}Error:{Style.RESET_ALL}", e.__class__, e)
name_servers = getNS()
getMX()
zoneXFR()
print("\n")

I see that you are trying well-known name servers that are specifically set up for testing. However, for the benefit of other readers, I will add a couple explanations.
As you are probably aware, most name servers will not allow zone transfers nowadays. That being said, it is possible that each of the name servers for a given domain name will behave differently (they could have different configurations and even be running different software).
In the case of megacorpone.com there are 3 name servers listed:
ns2.megacorpone.com.
ns3.megacorpone.com.
ns1.megacorpone.com.
ns2.megacorpone.com is the only one that did allow a zone transfer.
This message
dns.query.TransferError: Zone transfer error: REFUSED
means what it means: your query was refused. Probably you talked to the wrong name server.
Then you have another error which suggest a variable scoping issue:
UnboundLocalError: local variable 'zone' referenced before assignment
You are calling functions in this order:
name_servers = getNS()
getMX()
zoneXFR()
If name_servers fails, then the subsequent call to zoneXFR will fail too. Because this code:
for server in name_servers:
will try to iterate over an empty list.
Intermittent DNS resolution failures are common so a few checks are required here. At the very least, make sure that the list of NS is not empty.
Another issue: you start a for loop outside of the try block so your control structure is broken right in the middle:
for server in name_servers:
try:
zone = dns.zone.from_xfr(dns.query.xfr(str(server).rstrip('.'), domain))
print ("\nResults for",server, "\nZone origin:", str(zone.origin).rstrip('.'))
line()
Do this instead:
try:
for server in name_servers:
zone = dns.zone.from_xfr(dns.query.xfr(str(server).rstrip('.'), domain))
print ("\nResults for",server, "\nZone origin:", str(zone.origin).rstrip('.'))
...
I suspect that your script fails intermittently because the list of name servers is not always returned in the same order. If the first NS returned is ns1.megacorpone.com. or ns3.megacorpone.com. then the code crashes. If the scripts starts with ns2.megacorpone.com (the sole NS allowing zone transfers) then it seems to work OK.
When this code fails (AXFR denied):
zone = dns.zone.from_xfr(dns.query.xfr(str(server).rstrip('.'), domain))
then zone is not defined, and that's why you cannot print it in your exception block. Instead, show the domain name or some other variables that you know are defined and valid.
So if the AXFR is denied, your script should handle this exception dns.query.TransferError, and quietly move on to the next NS if any, until the list has been exhausted.
Another bit of advice: you try to resolve resources names that are different than '#'. Instead, look at the record type. You should only resolve CNAME, MX or NS. The other common types are TXT, A, AAAA, SOA. The rest are more exotic such as NAPTR, LOC or SRV. Nothing that should be resolved I think.

Fixed up your code, doesn't look great yet, but it works
#!/usr/bin/python3
# you might want to run python3 -m pip install dnspython before running this script
import sys
import dns.query
import dns.zone
import dns.resolver
# formatting setup
from colorama import Fore, Style
bracket = f"{Fore.BLUE}[{Fore.GREEN}*{Fore.BLUE}]{Style.RESET_ALL} "
bracket_err = f"{Fore.BLUE}[{Fore.RED}*{Fore.BLUE}]{Style.RESET_ALL} "
def drawLine():
print ('-' * 75)
# read arguments
try:
domain = (sys.argv[1])
except:
print("[!] USAGE: python3 zt.py DOMAIN_NAME")
sys.exit(0)
# DNS functions
def resolveDNS(name):
resolver = dns.resolver.Resolver()
results = resolver.query(name , "A")
return results
def getNS (domain):
mapping = {}
name_servers = dns.resolver.query(domain, 'NS')
print ("\nThe name servers for " + domain + " are:")
drawLine()
for name_server in name_servers:
A_records = resolveDNS(str(name_server))
for item in A_records:
answer = ','.join([str(item)])
mapping[str(name_server)] = answer
print (bracket, "{:30}".format(str(name_server).rstrip('.')), "{:15}".format(answer))
return mapping
def zoneXFR(server):
try:
zone = dns.zone.from_xfr(dns.query.xfr(str(server).rstrip('.'), domain))
except Exception as e:
print (bracket_err, f"{Fore.RED}Error:{Style.RESET_ALL}", e.__class__, e)
else:
print ("\nResults for",server, "\nZone origin:", str(zone.origin).rstrip('.'))
drawLine()
for host in zone:
if str(host) != '#':
A_records = resolveDNS(str(host) + "." + domain)
for item in A_records:
answer = ','.join([str(item)])
print(bracket, "{:30}".format(str(host) + "." + domain), answer)
drawLine()
name_servers = getNS(domain)
for server in name_servers:
print ("\nAttempting zone transfers for", server,name_servers[server])
zoneXFR(name_servers[server])

httplib2.IncompleteRead: AttributeError: 'module' object has no attribute 'IncompleteRead'

I've been using a script that was no longer maintained, which downloads your entire Google Drive to your local storage. I mananged to fix a few issues to do with depreciation, and the script seemed to be running smoothly, however, as seemingly random times in my script, it will break, and I will receive the following error.
File "drive.py", line 169, in download_file
except httplib2.IncompleteRead:
AttributeError: 'module' object has no attribute 'IncompleteRead'
These are the modules I am using
import gflags, httplib2, logging, os, pprint, sys, re, time
import pprint
from apiclient.discovery import build
from apiclient.discovery import build
from oauth2client.file import Storage
from oauth2client.client import AccessTokenRefreshError, flow_from_clientsecrets
from oauth2client.tools import run_flow
And here is the code that is causing the error
if is_google_doc(drive_file):
try:
download_url = drive_file['exportLinks']['application/pdf']
except KeyError:
download_url = None
else:
download_url = drive_file['downloadUrl']
if download_url:
try:
resp, content = service._http.request(download_url)
except httplib2.IncompleteRead:
log( 'Error while reading file %s. Retrying...' % drive_file['title'].replace( '/', '_' ) )
print 'Error while reading file %s. Retrying...' % drive_file['title'].replace( '/', '_' )
download_file( service, drive_file, dest_path )
return False
if resp.status == 200:
try:
target = open( file_location, 'w+' )
except:
log( "Could not open file %s for writing. Please check permissions." % file_location )
print "Could not open file %s for writing. Please check permissions." % file_location
return False
target.write( content )
return True
else:
log( 'An error occurred: %s' % resp )
print 'An error occurred: %s' % resp
return False
else:
# The file doesn't have any content stored on Drive.
return False
I am assuming this error has something to do with losing connection while downloading, and I am unfamilar with the httplib2 module.
The full code can be found here
Thankyou in advance to anyone who can shed some light in a possible fix.

I've been updating that drive backup script, and have encountered the same error. I haven't yet worked out what exception is being thrown, but in order to reveal what it is (and allow the script to keep running) I've made the following change:
Remove this:
- except httplib2.IncompleteRead:
- log( 'Error while reading file %s. Retrying...' % drive_file['title'].replace( '/', '_' ) )
Replace it with this:
+ except Exception as e: #httplib2.IncompleteRead: # no longer exists
+ log( traceback.format_exc(e) + ' Error while reading file %s. Retrying...' % drive_file['title'].replace( '/', '_' ) )
This does have the downside that if it encounters an exception consistently, it may enter an endless loop. However, it will then reveal the actual exception being thrown, so the "except:" can be updated appropriately.
This change is visible in the repository
here.
If I encounter the error again I'll update this answer with more detail.

How can I get around this try except block?

I wrote a try except block that I now realize was a bad idea because it keep throwing 'blind' exceptions that are hard to debug. The problem is that I do not know how to go about writing it another way besides going through each of the methods that are called and manually reading all the exceptions and making a case for each.
How would you structure this code?
def get_wiktionary_audio(self):
'''function for adding audio path to a definition, this is meant to be run before trying to get a specific URL'''
#this path is where the audio will be saved, only added the kwarg for testing with a different path
path="study_audio/%s/words" % (self.word.language.name)
try:
wiktionary_url = "http://%s.wiktionary.org/wiki/FILE:en-us-%s.ogg" % (self.word.language.wiktionary_prefix, self.word.name)
wiktionary_page = urllib2.urlopen(wiktionary_url)
wiktionary_page = fromstring(wiktionary_page.read())
file_URL = wiktionary_page.xpath("//*[contains(concat(' ', #class, ' '), ' fullMedia ')]/a/#href")[0]
file_number = len(self.search_existing_audio())
relative_path = '%s/%s%s.ogg' % (path, self.word.name, file_number)
full_path = '%s/%s' % (settings.MEDIA_ROOT, relative_path)
os.popen("wget -q -O %s 'http:%s'" % (full_path, file_URL))
except:
return False
WordAudio.objects.create(word=self.word, audio=relative_path, source=wiktionary_url)
return True

Often, exceptions come with error strings which can be used to pinpoint the problem. You can access this value like so:
try:
# code block
except Exception as e:
print str(e)
You can also print what class of exception it is along with any error messages by using the repr method:
try:
# code block
except Exception as e:
print repr(e)

One way I like to go about it is configure Python logging and log the output. This gives you a lot of flexibility in what you do with the log output. The below example logs the exception traceback.
import traceback
import logging
logger = logging.getLogger(__name__)
try:
...
except Exception as e:
logger.exception(traceback.format_exc()) # the traceback
logger.exception(e) # just the exception message

First your code is un-pythonic. You are using 'self' for a function. "self" is usually reserved for a class. So in reading your code, it feels unnatural. Second, my style is to line up "=" signs for readability. My advice is to start over -- Use standard pythonic conventions. You can get this by going through python tutorials.
Throw exception early and often -ONLY when the code stops running. You could also move some of the naming outside the try/except block.
def get_wiktionary_audio(self):
'''function for adding audio path to a definition, this is meant to be run before trying to get a specific URL'''
#this path is where the audio will be saved, only added the kwarg for testing with a different path
path = "study_audio/%s/words" % (self.word.language.name)
try:
wiktionary_url = "http://%s.wiktionary.org/wiki/FILE:en-us-%s.ogg" % (self.word.language.wiktionary_prefix, self.word.name)
wiktionary_page = urllib2.urlopen(wiktionary_url)
wiktionary_page = fromstring(wiktionary_page.read())
file_URL = wiktionary_page.xpath("//*[contains(concat(' ', #class, ' '), ' fullMedia ')]/a/#href")[0]
file_number = len(self.search_existing_audio())
relative_path = '%s/%s%s.ogg' % (path, self.word.name, file_number)
full_path = '%s/%s' % (settings.MEDIA_ROOT, relative_path)
os.popen("wget -q -O %s 'http:%s'" % (full_path, file_URL))
except Exception as e : print e
WordAudio.objects.create(word=self.word, audio=relative_path, source=wiktionary_url)
return True

Getting error in running ExportReport Python code in ArcMap

i am unable to generate pdf report using ExportReport function of Arcgis 10.2. getting error of IOError: Could not open report template
tried different template files but still getting this error.
Template directory is correct. checked all the directory positions.
Code
import arcpy
mxd = arcpy.mapping.MapDocument(r"C:\Users\Abrar ahmad\Documents\ArcGIS\New_Rwp_Cencus(12-17-2014).mxd")
df = arcpy.mapping.ListDataFrames(mxd)[0]
for lyr in arcpy.mapping.ListLayers(mxd, "",df):
if lyr.name == "New_Districts_5_without_limit":
arcpy.mapping.ExportReport(lyr,r"C:\Users\Abrar ahmad\Documents\ArcGIS\test.rlf",r"C:\Users\Abrar ahmad\Documents\ArcGIS\ProjectReport2.pdf","USE_RLF")
del mxd
Complete Error Data:
Runtime error
Traceback (most recent call last):
File "", line 7, in
File "c:\program files (x86)\arcgis\desktop10.2\arcpy\arcpy\utils.py", line 181, in fn_
return fn(*args, **kw)
File "c:\program files (x86)\arcgis\desktop10.2\arcpy\arcpy\mapping.py", line 515, in ExportReport
return report_source._arc_object.ExportReport(*gp_fixargs((report_layout_file, output_file, dataset_option, report_title, starting_page_number, page_range, report_definition_query, extent, field_map), True))
IOError: Could not open report template

Execute your code inside try/except/finally and see if this gives some hint:
import arcpy
import sys
import traceback
mxd = arcpy.mapping.MapDocument(r"C:\Users\Abrar ahmad\Documents\ArcGIS\New_Rwp_Cencus(12-17-2014).mxd")
try:
df = arcpy.mapping.ListDataFrames(mxd)[0]
for lyr in arcpy.mapping.ListLayers(mxd, "",df):
if lyr.name == "New_Districts_5_without_limit":
arcpy.mapping.ExportReport(lyr,r"C:\Users\Abrar ahmad\Documents\ArcGIS\test.rlf",r"C:\Users\Abrar ahmad\Documents\ArcGIS\ProjectReport2.pdf","USE_RLF")
except arcpy.ExecuteError:
# Get the tool error messages
msgs = arcpy.GetMessages(2)
# Return tool error messages for use with a script tool
arcpy.AddError(msgs)
# Print tool error messages for use in Python/PythonWin
print msgs
except:
# Get the traceback object
tb = sys.exc_info()[2]
tbinfo = traceback.format_tb(tb)[0]
# Concatenate information together concerning the error into a message string
pymsg = "PYTHON ERRORS:\nTraceback info:\n" + tbinfo + "\nError Info:\n" + str(sys.exc_info()[1])
msgs = "ArcPy ERRORS:\n" + arcpy.GetMessages(2) + "\n"
# Return python error messages for use in script tool or Python Window
arcpy.AddError(pymsg)
arcpy.AddError(msgs)
# Print Python error messages for use in Python / Python Window
print pymsg + "\n"
print msgs
finally:
del mxd

Bypass exception to always execute command?

The following code works almost perfect, thanks to the help received here:
import urllib.request
import zipfile
import subprocess
urls = ["http://url.com/archive1.zip", "http://url.com/archive2.zip", "http://url.com/archive3.zip"]
filename = "C:/test/test.zip"
destinationPath = "C:/test"
for url in urls:
try:
urllib.request.urlretrieve(url,filename)
sourceZip = zipfile.ZipFile(filename, 'r')
break
except ValueError:
pass
for name in sourceZip.namelist():
sourceZip.extract(name, destinationPath)
sourceZip.close()
subprocess.call(r'C:\WINDOWS\system32\cmd.exe /C "C:\test\test.exe"')
Except that when none of the url's successfully download, the final subprocess.call command never gets executed and I get the following error:
Traceback (most recent call last):
File "test.py", line 29, in <module>
for name in sourceZip.namelist():
NameError: name 'sourceZip' is not defined
I have been playing around a bit with the try: and except, but can not come to a working solution. I believe it is because the try: except command has to be inside the sourceZip loop, but since the sourceZip variable never gets declared, it fails. How would I alter this code to have the subprocess.call always get executed at the end regardless whether the download is successfull or not? (very new to Python)

Set sourceZip = None prior to the for url in urls line. Then test if sourceZip is None after the for loop to determine if you were able to successfully fetch the file. For example:
sourceZip = None
for url in urls:
try:
urllib.request.urlretrieve(url,filename)
sourceZip = zipfile.ZipFile(filename, 'r')
break
except ValueError:
pass
if sourceZip is not None:
for name in sourceZip.namelist():
sourceZip.extract(name, destinationPath)
sourceZip.close()
subprocess.call(r'C:\WINDOWS\system32\cmd.exe /C "C:\test\test.exe"')

Initialize sourceZip to None. Then check if it is not none do the extraction and closing.