I have created a python module to generate weather data(Latitude, Longitude, Elevation and other details) by taking particular location as input.
Updated it as per standards and "pycodestyle" package for checking PEP8 standards does not throw any error or warnings.
My Code is given below :
def fetch_location_info(input_list, err_file):
# URL which gives us Latitude, Longitude values
LatLong_URL = (
'http://maps.googleapis.com/maps/api/geocode/json?sensor=false&address='
)
# URL which gives us Elevation values
Elevation_URL = (
'https://maps.googleapis.com/maps/api/elevation/json?locations='
)
# Initializing Error Logs with relevant title for writing error records
err_line_header = "Logging Location Data Errors"
print(err_line_header, file=err_file)
# Insert a new line in the error file after the Error Header
print("\n", file=err_file)
# Fetch and Extract Location details from google maps
input_info = []
for location in input_list:
temp_info = {'Location': location}
latlong_response = requests.get(LatLong_URL + location).json()
if latlong_response.get('results'):
for latlong_results in latlong_response.get('results'):
latlong = (
latlong_results
.get('geometry', '0')
.get('location', '0')
)
temp_info['Latitude'] = latlong.get('lat', '0')
temp_info['Longitude'] = latlong.get('lng', '0')
elevation_response = requests.get(
Elevation_URL
+ str(temp_info['Latitude'])
+ ','
+ str(temp_info['Longitude'])
).json()
if elevation_response.get('results'):
for elevation_results in elevation_response.get('results'):
temp_info['Elevation'] = (
elevation_results.get('elevation', '0'))
input_info.append(temp_info)
break
else:
print("Elevation_URL is not fetching values for {}"
.format(location),
file=err_file
)
break
else:
print("LatLong_URL is not fetching values for {}"
.format(location),
file=err_file
)
print("\n", file=err_file)
return input_info
Now as a next step, I am trying to do Unit Testing using doctest. I chose to keep the test cases in a separate file. So I created the following .txt file and kept in the same directory as the code.
This is a doctest based regression suite for Test_Weather.py
Each '>>' line is run as if in a python shell, and counts as a test.
The next line, if not '>>' is the expected output of the previous line.
If anything doesn't match exactly (including trailing spaces), the test fails.
>>> from Test_Weather import fetch_location_info
>>> fetch_location_info(["Sydney,Australia"], open('data/error_log.txt', 'w'))
print(input_info)
As seen above, the expected condition should return the contents of the list / dataframe / variable that is created within the function being tested. For a try I just tried to print the contents of the list but my unit test output throws error like below since the expected value and got value is not matching :
PS C:\Users\JKC> python -m doctest testcases.txt
********************************************************************** File "testcases.txt", line 7, in testcases.txt Failed example:
fetch_location_info(["Sydney,Australia"], open('data/error_log.txt', 'w'))
Expected:
print(input_info)
Got:
[{'Location': 'Sydney,Australia', 'Latitude': -33.8688197, 'Longitude': 151. 2092955, 'Elevation': 24.5399284362793}]
So here as you can see that the Test Case worked fine but since I am not able to print the contents of the list, it is failing the test case.
My question is How can I display the contents of list in the expected section of the unit test case ?
If I am not wrong, do I need to literally mention the output value in the expected section of the unit test case ?
Any inputs will be helpful
You need to have the doctest look exactly as if you ran it at the Python REPL:
>>> from Test_Weather import fetch_location_info
>>> fetch_location_info(["Sydney,Australia"], open('data/error_log.txt', 'w'))
[{'Location': 'Sydney,Australia', 'Latitude': -33.8688197, 'Longitude': 151.2092955, 'Elevation': 24.5399284362793}]
Related
I'm trying to port to Python a SAP table download script, that already works on Excel VBA, but I want a command line version and I would prefer to avoid VBScript for a number of reasons that go beyond the goal of this post.
I'm stuck at the moment in which I need to fill the values in a table
from win32com.client import Dispatch
Functions = Dispatch("SAP.Functions")
Functions.Connection.Client = "400"
Functions.Connection.ApplicationServer = "myserver"
Functions.Connection.Language = "EN"
Functions.Connection.User = "myuser"
Functions.Connection.Password = "mypwd"
Functions.Connection.SystemNumber = "00"
Functions.Connection.UseSAPLogonIni = False
if (Functions.Connection.Logon (0,True) == True):
print("Logon OK")
RFC = Functions.Add("RFC_READ_TABLE")
RFC.exports("QUERY_TABLE").Value = "USR02"
RFC.exports("DELIMITER").Value = "~"
#RFC.exports("ROWSKIPS").Value = 2000
#RFC.exports("ROWCOUNT").Value = 10
tblOptions = RFC.Tables("OPTIONS")
#RETURNED DATA
tblData = RFC.Tables("DATA")
tblFields = RFC.Tables("FIELDS")
tblFields.AppendRow ()
print(tblFields.RowCount)
print(tblFields(1,"FIELDNAME"))
# the 2 lines above print 1 and an empty string, so the row in the table exists
Until here it is basically copied from VBA adapting the syntax.
In VBA at this point I'm able to do
tblFields(1,"FIELDNAME") = "BNAME"
if I do the same I get an error because the left part is a function and written that way it returns a string. In VBA it is probably a bi-dimensional array.
I unsuccessfully tried various approaches like
tblFields.setValue([{"FIELDNAME":"BNAME"}])
tblFields(1,"FIELDNAME").Value = "BNAME"
tblFields(1,"FIELDNAME").setValue("BNAME")
tblFields.FieldName = "BNAME" ##kinda desperate
The script works, without setting the FIELDS table, for outputs that produce rows shorter than 500 chars. This is a SAP limit in the function.
I know that this is not the best way, but I can't use the SAPNWRFC library and I can't use librfc32.dll.
I must be able to solve this way, or revert to the VB version.
Thanks to anyone who will provide a hint
After a lot of trial and error, i found a solution.
Instead of adding row by row to the "OPTIONS" or "FIELDS" tables, you can just submit a prefilled table.
This should work:
tblFields.Data = (('VBELN', '000000', '000000', '', ''),
('POSNR', '000000', '000000', '', ''))
same here:
tblOptions.Data = (("VBELN EQ '2557788'",),)
schools=['GSGS','GSGL','JKG','JMG','MCGD','MANGD','SLSA','WHGR','WOG','GCG','LP',
'PGG', 'WVSG', 'ASGE','CZG', 'EAG','GI']
for i in range (1,17):
gmaps = googlemaps.Client(key='')
distances = gmaps.distance_matrix((GSGS), (schools), mode="driving"['rows'][0]['elements'][0]['distance']['text']
print(distances)
The elements of the list are schools. I didn't want to make the list to long so I used these abbreviations.
I want to get all the distances between "GSGS" and the schools in the list. I don't know what to write inside the second bracket.
distances = gmaps.distance_matrix((GSGS), (schools)
If I run it like that, it outputs this error:
Traceback (most recent call last):
File "C:/Users/helpmecoding/PycharmProjects/untitled/distance.py", line 31, in
<module>
distances = gmaps.distance_matrix((GSGS), (schools), mode="driving")['rows'][0]['elements'][0]['distance']['text']
KeyError: 'distance'
I could do it one for one but thats not what I want. If I write another school from the list schools and delete the for loop it works fine.
I know I have to do a loop so that it cycles trough the list, but I don't know how to do it. Behind every variable for example "GSGS" is the address/location from the school.
I deleted the key just for safety.
My Dad helped me and we solved the problem. Now i have what i want :) Now i have to do a list with all distances between the schools. And if i got that i have to do the Dijkstra Algorithm to find the shortest route between them. Thanks for helping!
import googlemaps
GSGS = (address)
GSGL = (address)
. . .
. . .
. . .
schools =
(GSGS,GSGL,JKG,JMG,MCGD,MANGD,SLSA,WHGR,WOG,GCG,LP,PGG,WVSG,ASGE,CZG,EAG,GI)
school_names = ("GSGS","GSGL","JKG","JMG","MCGD","MANGD","SLSA","WHGR","WOG","GCG","LP","PGG","WVSG","ASGE","CZG","EAG","GI")
school_distances = ()
for g in range(0,len(schools)):
n = 0
for i in schools:
gmaps = googlemaps.Client(key='TOPSECRET')
distances = gmaps.distance_matrix(schools[g], i)['rows'][0]['elements'][0]['distance']['text']
if school_names[g] != school_names[n]:
print(school_names[g] + " - " + school_names[n] + " " + distances)
else:
print(school_names[g] + " - " + school_names[n] + " " + "0 km")
n = n + 1
In my experience, it is sometimes difficult to know what is going on when you use a third-party api. Though I am not a proponent of reinventing the wheel sometimes it is necessary to get a full picture of what is going on. So, I recommend giving it a shot building your own api endpoint request call and see if that works.
import requests
schools = ['GSGS','GSGL','JKG','JMG','MCGD','MANGD','SLSA','WHGR','WOG','GCG','LP','PGG', 'WVSG', 'ASGE','CZG', 'EAG','GI']
def gmap_dist(apikey, origins, destinations, **kwargs):
units = kwargs.get("units", "imperial")
mode = kwargs.get("mode", "driving")
baseurl = "https://maps.googleapis.com/maps/api/distancematrix/json?"
urlargs = {"key": apikey, "units": units, "origins": origins, "destinations": destinations, "mode": mode}
req = requests.get(baseurl, params=urlargs)
data = req.json()
print(data)
# do this for each key and index pair until you
# find the one causing the problem if it
# is not immediately evident from the whole data print
print(data["rows"])
print(rows[0])
# Check if there are elements
try:
distances = data['rows'][0]['elements'][0]['distance']
except KeyError:
raise KeyError("No elements found")
except IndexError:
raise IndexError("API Request Error. No response returned")
else:
return distances
Also as a general rule of thumb it is good to have a test case to make sure things are working as they should before testing the whole list,
#test case
try:
test = gmap_dist(apikey="", units="imperial", origins="GSGS", destinations="GSGL", mode="driving")
except Exception as err:
raise Exception(err)
else:
dists = gmap_dist(apikey="", units="imperial", origins="GSGS", destinations=schools, mode="driving")
print(dists)
Lastly, if you are testing the distance from "GSGS" to other schools, then you might want to get it out of your list of schools as the distance will be 0.
Now, I suspect that the reason you are getting this exception is because there are no json elements returned. Probably, because one of your parameters was improperly formatted.
If this function returns a KeyError still. Check the address spelling and make sure your apikey is valid. Although if it was the Apikey I would expect they would not bother to give you even empty results.
Hope this helps. Comment if it doesn't work.
I have been trying to extract input addresses from Namecoin transactions using some python code. This code works for regular transactions (where some namecoins are transferred from one address to another); however, this doesn't work on the transactions which have name operations, such as name_new. Here is some code:
raw = namecoind.getrawtransaction(tx_hash)
data = namecoind.decoderawtransaction(raw)
if 'vin' in data:
inputs = data['vin']
for input in inputs:
input_value = input.get('value')
if 'scriptSig' in input:
script_sig_asm = str(input['scriptSig'].get('asm'))
script_sig_parts = script_sig_asm.split(' ')
if len(script_sig_parts) > 1 and (len(script_sig_parts[-1]) == 130
or len(script_sig_parts[-1]) == 66):
public_key_string = script_sig_parts[-1]
try:
recipient_address = NamecoinPublicKey(public_key_string, verify=False).address()
print recipient_address
except Exception, e:
print str(e)
return
elif len(script_sig_parts) == 1:
print "coinbase transaction input"
return
#print "Inputs:"
Running this code on a regular transaction works, i,e, we get the recipient address. But running this code for a name operation such as this shows that its a coinbase transaction, that is,
len(script_sig_parts) == 1
is True, and so the recipient_address is empty.
Can anybody point me how I can get the recipient address (in the above transaction, it is: NCAzVGKq8JrsETxAkgw3MsDPinAEPwsTfn) in a Namecoin transaction which involves name operation?
Your code should work fine (but I have not tested it) on most name transactions. For instance, if you take 499a1e4c7bb1388347e1bd1142425949971eaf1fa2521af625006f4f49ce85c5 (which is the latest update of d/domob), the relevant script sig of the input is this:
"scriptSig" : {
"asm" : "3045022100c489ac648d08416d83d5e561d222049242242ded5f1c2bfd35ea48bb78b6a90f02203a28dee3d9473755fc2288bcaec9105e973e306071d28e69e593668c94b19fc101 04a85b7360b6b95459f7286111220f7a1eaef23bc9ced8a3b56cd57360374381bcabf7182941be400ccdfb761e26fa62d50b8911358aceb6aa30de9e8df5c46742",
"hex" : "483045022100c489ac648d08416d83d5e561d222049242242ded5f1c2bfd35ea48bb78b6a90f02203a28dee3d9473755fc2288bcaec9105e973e306071d28e69e593668c94b19fc1014104a85b7360b6b95459f7286111220f7a1eaef23bc9ced8a3b56cd57360374381bcabf7182941be400ccdfb761e26fa62d50b8911358aceb6aa30de9e8df5c46742"
},
As you can see, it does have two pieces in the scriptsig. (The pubkey and the signature.) For your transaction, however, the previous output is not pay-to-pubkeyhash but instead pay-to-pubkey. Here's the relevant output of the previous 92457dfc2831bdb6439fc03e72dbe3908140d43ec410f4a7396e3d65f5ab605b:
"scriptPubKey" : {
"asm" : "046a77fa46493d61985c1157a6e3e498b3b97c878c9c23e5b4729d354b574eb33a20c0483551308e2bd08295ce238e8ad09a7a2477732eb2e995a3e20455e9d137 OP_CHECKSIG",
"hex" : "41046a77fa46493d61985c1157a6e3e498b3b97c878c9c23e5b4729d354b574eb33a20c0483551308e2bd08295ce238e8ad09a7a2477732eb2e995a3e20455e9d137ac",
"reqSigs" : 1,
"type" : "pubkey",
"addresses" : [
"NCAzVGKq8JrsETxAkgw3MsDPinAEPwsTfn"
]
}
Due to the nature of the pay-to-pubkey script, the script sig does not contain the pubkey anymore (just the signature). I don't see a way to find the input address just from the script sig. If you want to handle these (rare) cases, you have to fetch the previous output and see there. (My knowledge about ECDSA is limited -- maybe there is a way to extract the pubkey from the signature.)
Updated Updated Question:
I have a JQuery $.ajax call that looks like this:
$.ajax({
url: 'http://website.domain.gov/cgi-bin/myFolder/script.py',
type: 'post',
dataType: 'json',
data:{'lat':'30.5', 'lon':'-80.2'},
success: function(response){alert(response.data.lat);
otherFunction();}
});
In script.py, which runs with no errors on its own, I do the following:
#!/usr/bin/python
import sys, json, cgi
import os
fs = cgi.FieldStorage()
d = {}
for k in fs.keys():
d[k] = fs.getvalue(k)
ilat = d['lat']
ilon = d['lon']
pntstr = ilat + ',' + ilon
my_list = []
#os.chdir('..'); os.chdir('..')
#os.chdir('a_place/where_data/exists')
# If you'd like to run this code, you probably don't have grib files or 'degrib', so another unix command will have to be utilized to open some sort of dummy data
f = os.popen('degrib '+'datafilename '+
'options '+ pntstr)
out = f.read()
# data has 5 columns, many rows
j = 0
while j < (5*num_lines_to_read):
my_list.append(out[j+4])
j += 5
f.close()
#os.chdir('..') etc until I get back to /cgi-bin/myFolder directory...
x = my_list[0]
result = {}
result['data'] = d
sys.stdout.write('Content-Type: application/json\n\n')
sys.stdout.write(json.dumps(result))
sys.stdout.close()
NOTE: 'd' is the dummy value and has nothing to do with 'my_list'...for now
When I update the web page I'm developing and try to return 'd' from Python (in the form of an alert), I get a "very helpful" server 500 error.
I've narrowed the problem down to the "x = my_list[0]" line. If I instead type "x = my_list", I get no error. sys.stdout.write() only stops working when I try to index "my_list". I tried printing the list. It's not empty and contains the expected values with expected types. I get successful output from sys.stdout.write() if I place it before "x = my_list[0]". Problem is, I'll eventually use "my_list" to create the output that is written to stdout.
Is there some obscure file I/O thing that I'm missing here?
The expression my_list[0] is throwing an exception. Either it isn't a list, or whatever it is doesn't support the array operator.
Try examining your web server log files. Try printing type(my_list) and len(my_list) after you emit the content-type header.
[Edit 2: More information and debugging in answer below...]
I'm writing a python script to export MS Access databases into a series of text files to allow for more meaningful version control (I know - why Access? Why aren't I using existing solutions? Let's just say the restrictions aren't of a technical nature).
I've successfully exported the full contents and structure of the database using ADO and ADOX via the comtypes library, but I'm getting a problem re-importing the data.
I'm exporting the contents of each table into a text file with a list on each line, like so:
[-9, u'No reply']
[1, u'My home is as clean and comfortable as I want']
[2, u'My home could be more clean or comfortable than it is']
[3, u'My home is not at all clean or comfortable']
And the following function to import the said file:
import os
import sys
import datetime
import comtypes.client as client
from ADOconsts import *
from access_consts import *
class Db:
def create_table_contents(self, verbosity = 0):
conn = client.CreateObject("ADODB.Connection")
rs = client.CreateObject("ADODB.Recordset")
conn.ConnectionString = self.new_con_string
conn.Open()
for fname in os.listdir(self.file_path):
if fname.startswith("Table_"):
tname = fname[6:-4]
if verbosity > 0:
print "Filling table %s." % tname
conn.Execute("DELETE * FROM [%s];" % tname)
rs.Open("SELECT * FROM [%s];" % tname, conn,
adOpenDynamic, adLockOptimistic)
f = open(self.file_path + os.path.sep + fname, "r")
data = f.readline()
print repr(data)
while data != '':
data = eval(data.strip())
print data[0]
print rs.Fields.Count
rs.AddNew()
for i in range(rs.Fields.Count):
if verbosity > 1:
print "Into field %s (type %s) insert value %s." % (
rs.Fields[i].Name, str(rs.Fields[i].Type),
data[i])
rs.Fields[i].Value = data[i]
data = f.readline()
print repr(data)
rs.Update()
rs.Close()
conn.Close()
Everything works fine except that numerical values (double and int) are being inserted as zeros. Any ideas on whether the problem is with my code, eval, comtypes, or ADO?
Edit: I've fixed the problem with inserting numbers - casting them as strings(!) seems to solve the problem for both double and integer fields.
However, I now have a different issue that had previously been obscured by the above: the first field in every row is being set to 0 regardless of data type... Any ideas?
And found an answer.
rs = client.CreateObject("ADODB.Recordset")
Needs to be:
rs = client.CreateObject("ADODB.Recordset", dynamic=True)
Now I just need to look into why. Just hope this question saves someone else a few hours...
Is data[i] being treated as a string? What happens if you specifically cast it as a int/double when you set rs.Fields[i].Value?
Also, what happens when you print out the contents of rs.Fields[i].Value after it is set?
Not a complete answer yet, but it appears to be a problem during the update. I've added some further debugging code in the insertion process which generates the following (example of a single row being updated):
Inserted into field ID (type 3) insert value 1, field value now 1.
Inserted into field TextField (type 202) insert value u'Blah', field value now Blah.
Inserted into field Numbers (type 5) insert value 55.0, field value now 55.0.
After update: [0, u'Blah', 55.0]
The last value in each "Inserted..." line is the result of calling rs.Fields[i].Value before calling rs.Update(). The "After..." line shows the results of calling rs.Fields[i].Value after calling rs.Update().
What's even more annoying is that it's not reliably failing. Rerunning the exact same code on the same records a few minutes later generated:
Inserted into field ID (type 3) insert value 1, field value now 1.
Inserted into field TextField (type 202) insert value u'Blah', field value now Blah.
Inserted into field Numbers (type 5) insert value 55.0, field value now 55.0.
After update: [1, u'Blah', 2.0]
As you can see, results are reliable until you commit them, then... not.