Develop Reference

Python Unit Testing using Doctest not working as expected

I have created a python module to generate weather data(Latitude, Longitude, Elevation and other details) by taking particular location as input.
Updated it as per standards and "pycodestyle" package for checking PEP8 standards does not throw any error or warnings.
My Code is given below :
def fetch_location_info(input_list, err_file):
# URL which gives us Latitude, Longitude values
LatLong_URL = (
'http://maps.googleapis.com/maps/api/geocode/json?sensor=false&address='
)
# URL which gives us Elevation values
Elevation_URL = (
'https://maps.googleapis.com/maps/api/elevation/json?locations='
)
# Initializing Error Logs with relevant title for writing error records
err_line_header = "Logging Location Data Errors"
print(err_line_header, file=err_file)
# Insert a new line in the error file after the Error Header
print("\n", file=err_file)
# Fetch and Extract Location details from google maps
input_info = []
for location in input_list:
temp_info = {'Location': location}
latlong_response = requests.get(LatLong_URL + location).json()
if latlong_response.get('results'):
for latlong_results in latlong_response.get('results'):
latlong = (
latlong_results
.get('geometry', '0')
.get('location', '0')
)
temp_info['Latitude'] = latlong.get('lat', '0')
temp_info['Longitude'] = latlong.get('lng', '0')
elevation_response = requests.get(
Elevation_URL
+ str(temp_info['Latitude'])
+ ','
+ str(temp_info['Longitude'])
).json()
if elevation_response.get('results'):
for elevation_results in elevation_response.get('results'):
temp_info['Elevation'] = (
elevation_results.get('elevation', '0'))
input_info.append(temp_info)
break
else:
print("Elevation_URL is not fetching values for {}"
.format(location),
file=err_file
)
break
else:
print("LatLong_URL is not fetching values for {}"
.format(location),
file=err_file
)
print("\n", file=err_file)
return input_info
Now as a next step, I am trying to do Unit Testing using doctest. I chose to keep the test cases in a separate file. So I created the following .txt file and kept in the same directory as the code.
This is a doctest based regression suite for Test_Weather.py
Each '>>' line is run as if in a python shell, and counts as a test.
The next line, if not '>>' is the expected output of the previous line.
If anything doesn't match exactly (including trailing spaces), the test fails.
>>> from Test_Weather import fetch_location_info
>>> fetch_location_info(["Sydney,Australia"], open('data/error_log.txt', 'w'))
print(input_info)
As seen above, the expected condition should return the contents of the list / dataframe / variable that is created within the function being tested. For a try I just tried to print the contents of the list but my unit test output throws error like below since the expected value and got value is not matching :
PS C:\Users\JKC> python -m doctest testcases.txt
********************************************************************** File "testcases.txt", line 7, in testcases.txt Failed example:
fetch_location_info(["Sydney,Australia"], open('data/error_log.txt', 'w'))
Expected:
print(input_info)
Got:
[{'Location': 'Sydney,Australia', 'Latitude': -33.8688197, 'Longitude': 151. 2092955, 'Elevation': 24.5399284362793}]
So here as you can see that the Test Case worked fine but since I am not able to print the contents of the list, it is failing the test case.
My question is How can I display the contents of list in the expected section of the unit test case ?
If I am not wrong, do I need to literally mention the output value in the expected section of the unit test case ?
Any inputs will be helpful

You need to have the doctest look exactly as if you ran it at the Python REPL:
>>> from Test_Weather import fetch_location_info
>>> fetch_location_info(["Sydney,Australia"], open('data/error_log.txt', 'w'))
[{'Location': 'Sydney,Australia', 'Latitude': -33.8688197, 'Longitude': 151.2092955, 'Elevation': 24.5399284362793}]

Intramolecular protein residue contact map using biopython, KeyError: 'CA'

I am trying to identify amino acid residues in contact in the 3D protein structure. I am new to BioPython but found this helpful website http://www2.warwick.ac.uk/fac/sci/moac/people/students/peter_cock/python/protein_contact_map/
Following their lead (which I will reproduce here for completion; Note, however, that I am using a different protein):
import Bio.PDB
import numpy as np
pdb_code = "1QHW"
pdb_filename = "1qhw.pdb"
def calc_residue_dist(residue_one, residue_two) :
"""Returns the C-alpha distance between two residues"""
diff_vector = residue_one["CA"].coord - residue_two["CA"].coord
return np.sqrt(np.sum(diff_vector * diff_vector))
def calc_dist_matrix(chain_one, chain_two) :
"""Returns a matrix of C-alpha distances between two chains"""
answer = np.zeros((len(chain_one), len(chain_two)), np.float)
for row, residue_one in enumerate(chain_one) :
for col, residue_two in enumerate(chain_two) :
answer[row, col] = calc_residue_dist(residue_one, residue_two)
return answer
structure = Bio.PDB.PDBParser().get_structure(pdb_code, pdb_filename)
model = structure[0]
dist_matrix = calc_dist_matrix(model["A"], model["A"])
But when I run the above code, I get the following error message:
Traceback (most recent call last):
File "<ipython-input-26-7239fb7ebe14>", line 4, in <module>
dist_matrix = calc_dist_matrix(model["A"], model["A"])
File "<ipython-input-3-730a11883f27>", line 15, in calc_dist_matrix
answer[row, col] = calc_residue_dist(residue_one, residue_two)
File "<ipython-input-3-730a11883f27>", line 6, in calc_residue_dist
diff_vector = residue_one["CA"].coord - residue_two["CA"].coord
File "/Users/anaconda/lib/python3.6/site-packages/Bio/PDB/Entity.py", line 39, in __getitem__
return self.child_dict[id]
KeyError: 'CA'
Any suggestions on how to fix this issue?

You have heteroatoms (water, ions, etc; anything that isn't an amino acid or nucleic acid) in your structure, remove them with:
for residue in chain:
if residue.id[0] != ' ':
chain.detach_child(residue.id)
This will remove them from your entire structure. You may want to modify if want to keep the heteroatoms for further analysis.

I believe the problem is that some of the elements in model["A"] are not amino acids and therefore do not contain "CA".
To get around this, I wrote a new function which returns only the amino acid residues:
from Bio.PDB import *
chain = model["A"]
def aa_residues(chain):
aa_only = []
for i in chain:
if i.get_resname() in standard_aa_names:
aa_only.append(i)
return aa_only
AA_1 = aa_residues(model["A"])
dist_matrix = calc_dist_matrix(AA_1, AA_1)

So I've been testing (bear in mind I know very little about Bio) and it looks like whatever is in you 1qhw.pdb file is very different from the one in that example.
pdb_code = '1qhw'
structure = Bio.PDB.PDBParser().get_structure(pdb_code, pdb_filename)
model = structure[0]
next, to see what is in it, I did:
print(list(model))
Which gave me:
[<Chain id=A>]
exploring this, it appears the pdb file is a dict of dicts. So, using this id,
test = model['A']
gives me the next dict. This level is the level being passed to your function that is causing the error. Printing this with:
print(list(test))
Gave me a huge list of the data inside, including lots of residues and related info. But crucially, no CA. Try using this to see whats inside and modify the line:
diff_vector = residue_one["CA"].coord - residue_two["CA"].coord
to reflect what you are after, replacing CA where appropriate.
I hope this helps, its a little tricky to get much more specific.

Another solution to obtain the contact map for a protein chain is to use the PdbParser shipped with ConKit.
ConKit is a library specifically designed to work with predicted contacts but has the functionality to extract contacts from a PDB file:
>>> from conkit.io.PdbIO import PdbParser
>>> p = PdbParser()
>>> with open("1qhw.pdb", "r") as pdb_fhandle:
... pdb = p.read(pdb_fhandle, f_id="1QHW", atom_type="CA")
>>> print(pdb)
ContactFile(id="1QHW_0" nmaps=1
This reads your PDB file into the pdb variable, which stores an internal ContactFile hierarchy. In this example, two residues are considered to be in contact if the participating CA atoms are within 8Å of each other.
To access the information, you can then iterate through the ContactFile and access each ContactMap, which in your case corresponds to intra-molecular contacts for chain A.
>>> for cmap in pdb:
... print(cmap)
ContactMap(id="A", ncontacts=1601)
If you would have more than one chain, there would be a ContactMap for each chain, and additional ones for inter-molecular contacts between chains.
The ContactMap for chain A contains 1601 contact pairs. You can access the Contact instances in each ContactMap by either iterating or indexing. Both work fine.
>>> print(cmap[0])
Contact(id="(26, 27)" res1="S" res1_chain="A" res1_seq=26 res2="T" res2_chain="A" res2_seq=27 raw_score=0.961895)
Each level in the hierarchy has various functions with which you could manipulate contact maps. Examples can be found here.

Returning a list of items that are raising an AttributeError when using geopy and try/except?

I am attempting to get the latitude and longitude for ~400 locations using geopy. However, many of the addresses are returning 'None' as they are either not specific enough or are intersections of streets, which is causing an "AttributeError: 'None' type has no attribute 'latitude'" and an exit of the for loop.
What I would like to do is run the for loop without exiting due to error AND return a list of all of the locations prompting errors so that I can hard code a proper address for them.
My current code is as follows:
from geopy.geocoders import Nominatim
geolocator = Nominatim()
coordinates = []
def station_errors(list_of_stations):
for station in list_of_stations:
try:
location = geolocator.geocode(str(i)) #line with error, used 'i' instead of 'station'
lat_long = (location.latitude, location.longitude)
coordinates.append(list(lat_long))
except Exception:
print("Error: %s"%(station))
continue
However, this seems to be printing out every station regardless of whether geopy was able to assign it a latitude and longitude, as I've tried many of them manually. I only want the stations that raised an error/were unable to receive a lat/long coordinate.
I am fairly new to 'try, except, continue' error testing, so any advice is greatly appreciated.
EDIT:
My initial error was using 'i' instead of 'station', an error which I've left in the original question. The code that I got to work and gives me the result that I want is as follows:
def station_errors(list_of_stations):
list_of_error_stations = []
for station in list_of_stations:
try:
location = geolocator.geocode(str(station + " DC"))
lat_long = (location.latitude, location.longitude)
coordinates.append(list(lat_long))
except Exception:
list_of_error_stations.append(station)
continue
return(list_of_error_stations)

I expect that the "AttributeError: 'None' type has no attribute 'latitude'" is given in the line:
lat_long = (location.latitude, location.longitude)
thous I would transform the code in something like:
def station_errors(list_of_stations):
for station in list_of_stations:
location = geolocator.geocode(str(i))
coordinates.append([location.latitude, location.longitude] if location is not None else None)
# or coordinates.append(list(lat_long) if location is not None else [None, None])
# which ever comes more in handy
return coordinates
coords = station_errors(los)
# and then deal with the Nones (or [None, None] s ) to "hard code a proper address for them"

Pyparsing : issues with setResultsName

I'm parsing multiple choice questions with multiple answers that look like this :
ParserElement.setDefaultWhitespaceChars(u""" \t""")
in_ = """1) first stem.
= option one one key
= option one two key
- option one three distractor
= option one four key
2) second stem ?
- option two one distractor
- option two two distractor
= option one three key
3) third stem.
- option three one key
= option three two distractor
"""
The equal sign represents a correct answer, the dash a distractor.
My grammar looks like this :
newline = Suppress(u"\n")
end_number = Suppress(oneOf(u') / ('))
end_stem = Suppress(oneOf(u"? .")) + newline
end_phrase = Optional(u'.').suppress() + newline
phrase = OneOrMore(Word(alphas)) + end_phrase
prefix = Word(u"-", max=1)('distractor') ^ Word(u"=", max=1)('key')
stem = Group(OneOrMore(Word(alphas))) + end_stem
number = Word(nums) + end_number
question = number + stem('stem') +
Group(OneOrMore(Group(prefix('prefix') + phrase('phrase'))))('options')
And when I'm parsing the results:
for match, start, end in question.scanString(in_):
for o in match.options:
try:
print('key', o.prefix.key)
except:
print('distractor', o.prefix.distractor)
I get :
AttributeError: 'unicode' object has no attribute 'distractor'
I'm pretty sure the result names are chainable. If so, what am I doing wrong ? I can easily work around this but it's unsatisfactory not knowing what I did wrong and what I misunderstood.

The problem is that o is actually the prefix -- when you call o.prefix, you're actually going one level deeper then you need to, and are retrieving the string the prefix maps to, not the ParseResults object.
You can see this by modifying the code so that it prints out the parse tree:
for match, start, end in question.scanString(in_):
for o in match.options:
print o.asXML()
try:
print('key', o.prefix.key)
except:
print('distractor', o.prefix.distractor)
The code will then print out:
<prefix>
<key>=</key>
<phrase>option</phrase>
<ITEM>one</ITEM>
<ITEM>one</ITEM>
<ITEM>key</ITEM>
</prefix>
Traceback (most recent call last):
File "so07.py", line 37, in <module>
print('distractor', o.prefix.distractor)
AttributeError: 'str' object has no attribute 'distractor'
The problem then becomes clear -- if o is the prefix, then it doesn't make sense to do o.prefix. Rather, you need to simply call o.key or o.distractor.
Also, it appears that if you try and call o.key where no key exists, then pyparsing will return an empty string rather than throwing an exception.
So, your fixed code should look like this:
for match, start, end in question.scanString(in_):
for o in match.options:
if o.key != '':
print('key', o.key)
else:
print('distractor', o.distractor)

rrdscript won't plot anything

Hey guys (and off course Ladys) ,
i have this little script which should show me some
nice rrd graphs. But i seems like i cant find a way to
bring it to work to show me some stats. This is my Script:
# Function: Simple ping plotter for rrd
import rrdtool,tempfile,commands,time,sys
from model import hosts
sys.path.append('/home/dirk/devel/python/stattool/stattool/lib')
import nurrd
from nurrd import RRDplot
class rrdPing(RRDplot):
def __init__(self):
self.DAY = 86400
self.YEAR = 365 * self.DAY
self.rrdfile = 'hostname.rrd'
self.interval = 300
self.probes = 5
self.rrdList = []
def create_rrd(self, interval):
ret = rrdtool.create("%s" % self.rrdfile, "--step", "%s" % self.interval,
"DS:packets:COUNTER:600:U:U",
"RRA:AVERAGE:0.5:1:288",
"RRA:AVERAGE:0.5:1:336")
def getHosts(self, userID):
myHosts = hosts.query.filter_by(uid=userID).all()
return myHosts.pop(0)
def _doPing(self,host):
for x in xrange(0, self.probes):
ans,unans = commands.getstatusoutput("ping -c 3 -w 6 %s| grep rtt| awk -F '/' '{ print $5 }'" % host)
print x
self.probes -=1
self.rrdList.append(unans)
return self.rrdList
def plotRRD(self):
self.create_rrd(self.interval)
times = self._doPing(self.getHosts(3))
for x in xrange(0,len(times)):
loc = times.pop(0)
rrdtool.update(self.rrdfile, '%d:%d' % (int(time.time()), int(float(loc))))
print '%d:%d' % (int(time.time()), int(float(loc)))
time.sleep(5)
self.graph(60)
def graph(self, mins):
ret = rrdtool.graph( "%s.png" % self.rrdfile, "--start", "-1", "--end" , "+1","--step","300",
"--vertical-label=Bytes/s",
"DEF:inoctets=%s:packets:AVERAGE" % self.rrdfile ,
"AREA:inoctets#7113D6:In traffic",
"CDEF:inbits=inoctets,8,*",
"COMMENT:\\n",
"GPRINT:inbits:AVERAGE:Avg In traffic\: %6.2lf \\r",
"COMMENT: ",
"GPRINT:inbits:MAX:Max In traffic\: %6.2lf")
if __name__ == "__main__":
ping = rrdPing()
ping.plotRRD()
info = rrdtool.info('hostname.rrd')
print info['last_update']
Could somebody please give me some advice or some tips how to solve this?
(Sorry code is a litte mess)
Thanks in advance
Kind regards,
Dirk

Several issues.
Firstly, you appear to only be collecting a single data sample and storing it before you try to generate a graph. You will need at least two samples, separated by about 300s, before you can get a single Primary Data Point and therefore something to graph.
Secondly, you do not post nay information as to what data you are actually storing. Are you sure your rrdPing function is returning valid data to store? You are not testing the error status of the write either.
Thirdly, the data you are collecting seems to be ping times or similar, which is a GAUGE type value. Yet, your RRD DS definition uses a COUNTER type and your graph call is treating it as network traffic data. A COUNTER type assumes increasing values and converts to a rate of change, so if you give it ping RTT data you'll get either unknowns or zeroes stored, which will not show up on a graph.
Fourthly, your call to RRDGraph is specifying a start of -1 and and end of +1. From 1 second in the past to 1 second in the future? Since your step is 300s this is an odd graph. Maybe you should have--end 'now-1' --start 'end-1day' or similar?
You should make your code test the return values for any error messages produced by the RRDTool library -- this is good practice anyway. When testing, print out the values you are updating with to be sure you are giving valid values. With RRDTool, you should collect several data samples at the step interval and store them before expecting to see a line on the graph. Also, make sure you are using the correct data type, GAUGE or COUNTER.