Develop Reference

shodan - country code python

I want to list once my script in python search for specific strings , but I also want to add country code first two letters , but when I try then it says invalid KeyError: 'country_code', but the api says ocation.country_code how can I achieve that?
#!/usr/bin/python
import shodan
SHODAN_API_KEY="xxxxxxxxxxxxxxxxxxxx"
api = shodan.Shodan(SHODAN_API_KEY)
try:
# Search Shodan
results = api.search('ProFTPd-1.3.3c')
# Show the results
for result in results['matches']:
print '%s' % result['ip_str']
print '%s' % result['country_code']
except shodan.APIError, e:
print 'Error: %s' % e

I think this is the method You are using in Python
https://github.com/achillean/shodan-python/blob/master/shodan/client.py#L324
and it triggers:
return self._request('/shodan/host/search', args)
Shodan API documentation:
https://developer.shodan.io/api
check out /shodan/host/search API
I just saw that the answer is in Your question but You ate one letter from location (ocation).
Try this:
print '%s' % result['location']['country_code']
So field You are looking for is there but it is in another dictionary.
I would recommend to read API documentation well next time and as Nofal Daud said, Python error are self explanatory if You have KeyError on dict it means that field is not there. Next time listen to Python it will reveal the truth.

What kind of a JSON object is this? (Gitlab API), (Python)

Im accessing the content of a file using GitLab's File Repository API, and all seems to be going fine until I hit one specific file (any text file).
{:file_name=>"requirements.txt", :file_path=>"requirements.txt", :size=>30, :encoding=>"base64", :content=>"ImZsYXNrIgoidXdzZ2kiCiJnb2dvc2VjcmV0cyIK", :ref=>"master", :blob_id=>"7f551dddd4fd8931419450fb357bad83d2efbe6a", :commit_id=>"40e161fcb323be28259712a6cf5da8fddfda80e1", :last_commit_id=>"40e161fcb323be28259712a6cf5da8fddfda80e1"}
I have never seen colons before a key, and have never seen a '=>' in a JSON object. All other JSON objects returned have been fine. The request being made to the API isn't wrong because a 200 response is being returned.
What is this??

This appears to be an open bug in Gitlab. See https://gitlab.com/gitlab-org/gitlab-ee/issues/2298.
What is the current bug behavior?
Response:
{:file_path=>"Readme.txt", :branch=>"master"}
What is the expected correct behavior?
Response:
{"file_name": "Readme.txt", "branch": "master"}

The issue is still opened on gitlab (issue 2298)
I encountered the same problem.
I confirm that only .txt files are affected. Files with unknown extensions for Git are not affected (e.g. : .myFile, .unknowExtension ...)
This can be detected with the Content-Type header, set as text/plain when Gitlab sent response in ruby hash.
Here is a simple function to parse ruby hash in PHP.
function ruby_hash_decode($ruby)
{
if (substr( $ruby, 0, 2 ) === "{:") {
$ruby = str_replace('{:', '{"', $ruby);
$ruby = str_replace('", :', '", "', $ruby);
$ruby = str_replace(', :', ', "', $ruby);
$ruby = str_replace('=>"', '":"', $ruby);
$ruby = str_replace('=>', '":', $ruby);
$ruby = \json_decode($ruby);
}
return $ruby;
}

Make a Ruby hash to Python dict or json converter. Note, the below solution is a quick n' dirty one I made in 2 minutes. You can do much better if you use regex.
s = '{:file_path=>"Readme.txt", :branch=>"master"}'
s = s[1:-1]
s = s.split(',')
s = [elem.strip() for elem in s]
s = [subs[1:] for subs in s]
s = [subs.replace('"', '') for subs in s]
my_dict = {elem.split('=>')[0]: elem.split('=>')[1] for elem in s}
my_dict looks like this now: {'file_path': 'Readme.txt', 'branch': 'master'}

parsing API with Python - how to handle JSON with BOM

I'm using Python 2.7.11 on windows to get JSON data from API (data on trees in Warsaw, Poland, but nevermind that). I want to generate output csv file with all the data provided by the api, for further analysis. I started with a script I used for another project (also discussed here on Stackoverflow and corrected for me by #Martin Taylor).That script didn't work so I tried to modify it using my very basic understanding, googling around and applying pdb debugger. At the moment, the result looks like this:
import pdb
import json
import urllib2
import csv
pdb.set_trace()
url = "https://api.um.warszawa.pl/api/action/datastore_search/?resource_id=ed6217dd-c8d0-4f7b-8bed-3b7eb81a95ba"
myfile = 'C:/dane/drzewa.csv'
csv_myfile = csv.writer(open(myfile, 'wb'))
cols = ['numer_adres', 'stan_zdrowia', 'y_wgs84', 'dzielnica', 'adres', 'lokalizacja', 'wiek_w_dni', 'srednica_k', 'pnie_obwod', 'miasto', 'jednostka', 'x_pl2000', 'wysokosc', 'y_pl2000', 'numer_inw', 'x_wgs84', '_id', 'gatunek_1', 'gatunek', 'data_wyk_pom']
csv_myfile.writerow(cols)
def api_iterate(myfile):
while True:
global url
print url
json_page = urllib2.urlopen(url)
data = json.load(json_page)
json_page.close()
for data_object in data ['result']['records']:
csv_myfile.writerow([data_object[col] for col in cols])
try:
url = data['_links']['next']
except KeyError as e:
break
with open(myfile, 'wb'):
api_iterate(myfile)
I'm a very fresh Python user so I get confused all the time. Now I got to the point when, while reading the objects in json dictionary, I get a Keyerror message associated with the 'x_wgs84' element. I suppose it has something to do with the fact that in the source url this element is preceded by a U+FEFF unicode character. I tried to get around this but I got stuck and would appreciate assistance.
I suspect the code may be corrupt in several other ways - as I mentioned, I'm a very unskilled programmer (yet).

You need to put the key with the unicode character:
To know how to do it, one easy way is to print the keys:
>>> import requests
>>> res = requests.get('https://api.um.warszawa.pl/api/action/datastore_search/?resource_id=ed6217dd-c8d0-4f7b-8bed-3b7eb81a95ba')
>>> data = res.json()
>>> records = data['result']['records']
>>> records[0]
{u'numer_adres': u'', u'stan_zdrowia': u'dobry', u'y_wgs84': u'52.21865', u'y_pl2000': u'5787241.04475524', u'adres': u'ul. ALPEJSKA', u'x_pl2000': u'7511793.96937063', u'lokalizacja': u'Ulica ALPEJSKA', u'wiek_w_dni': u'60', u'miasto': u'Warszawa', u'jednostka': u'Dzielnica Wawer', u'pnie_obwod': u'73', u'wysokosc': u'14', u'data_wyk_pom': u'20130709', u'dzielnica': u'Wawer', u'\ufeffx_wgs84': u'21.172584', u'numer_inw': u'D386200', u'_id': 125435, u'gatunek_1': u'Quercus robur', u'gatunek': u'd\u0105b szypu\u0142kowy', u'srednica_k': u'7'}
>>> records[0].keys()
[u'numer_adres', u'stan_zdrowia', u'y_wgs84', u'y_pl2000', u'adres', u'x_pl2000', u'lokalizacja', u'wiek_w_dni', u'miasto', u'jednostka', u'pnie_obwod', u'wysokosc', u'data_wyk_pom', u'dzielnica', u'\ufeffx_wgs84', u'numer_inw', u'_id', u'gatunek_1', u'gatunek', u'srednica_k']
>>> records[0][u'\ufeffx_wgs84']
u'21.172584'
As you can see, to get your key, you need to write it as '\ufeffx_wgs84' with the unicode character that is causing trouble.
Note: I don't know if you are using python2 or 3, but you might need to put a u before your string declaration in python2 to declare it as unicode string.

I keep getting [u'error']

Why am I getting this error when trying to print keys from dictionary?
import urllib
import urllib2
import json
ret = urllib2.urlopen(urllib2.Request('https://poloniex.com/public?command=returnTicker'))
a = json.loads(ret.read())
print a.keys()
ret = urllib2.urlopen(urllib2.Request('https://poloniex.com/public?command=return24Volume'))
b = json.loads(ret.read())
print b.keys()

The error is produced by the website - your code as such is ok, it produces a json-object which has apparently the structure '{ "error" : ""}'. Try printing it out & figure out what's wrong, you probably need some authentication tokens or similar stuff.
There seems to be a API wrapper available, you should consider using or at least understanding it: http://pastebin.com/8fBVpjaj
It's directly featured on polniex website, and it clearly shows the need of API secret and key.

The error is originating from the site. The dict is loaded, and expresses the error as it's only key.
Try this:
ret = urllib2.urlopen(urllib2.Request('https://poloniex.com/public?command=return24hVolume'))
b = json.loads(ret.read())
print b.keys()
Notice the 'h' in return24hVolume.

The second URL returns:
{"error":"Invalid command."}
So as Reut Sharabani points out, you have to use 'h' in the URL:
https://poloniex.com/public?command=return24hVolume

Python: Why will this string print but not write to a file?

I am new to Python and working on a utility that changes an XML file into an HTML. The XML comes from a call to request = urllib2.Request(url), where I generate the custom url earlier in the code, and then set response = urllib2.urlopen(request) and, finally, xml_response = response.read(). This works okay, as far as I can tell.
My trouble is with parsing the response. For starters, here is a partial example of the XML structure I get back:
I tried adapting the slideshow example in the minidom tutorial here to parse my XML (which is ebay search results, by the way): http://docs.python.org/2/library/xml.dom.minidom.html
My code so far looks like this, with try blocks as an attempt to diagnose issues:
doc = minidom.parseString(xml_response)
#Extract relevant information and prepare it for HTML formatting.
try:
handleDocument(doc)
except:
print "Failed to handle document!"
def getText(nodelist): #taken straight from slideshow example
rc = []
for node in nodelist:
if node.nodeType == node.TEXT_NODE:
print "A TEXT NODE!"
rc.append(node.data)
return ''.join(rc) #this is a string, right?
def handleDocument(doc):
outputFile = open("EbaySearchResults.html", "w")
outputFile.write("<html>\n")
outputFile.write("<body>\n")
try:
items = doc.getElementsByTagName("item")
except:
"Failed to get elements by tag name."
handleItems(items)
outputFile.write("</html>\n")
outputFile.write("</body>\n")
def handleItems(items):
for item in items:
title = item.getElementsByTagName("title")[0] #there should be only one title
print "<h2>%s</h2>" % getText(title.childNodes) #this works fine!
try: #none of these things work!
outputFile.write("<h2>%s</h2>" % getText(title.childNodes))
#outputFile.write("<h2>" + getText(title.childNodes) + "</h2>")
#str = getText(title.childNodes)
#outputFIle.write(string(str))
#outputFile.write(getText(title.childNodes))
except:
print "FAIL"
I do not understand why the correct title text does print to the console but throws an exception and does not work for the output file. Writing plain strings like this works fine: outputFile.write("<html>\n") What is going on with my string construction? As far as I can tell, the getText method I am using from the minidom example returns a string--which is just the sort of thing you can write to a file..?

If I print the actual stack trace...
...
except:
print "Exception when trying to write to file:"
print '-'*60
traceback.print_exc(file=sys.stdout)
print '-'*60
traceback.print_tb(sys.last_traceback)
...
...I will instantly see the problem:
------------------------------------------------------------
Traceback (most recent call last):
File "tohtml.py", line 85, in handleItems
outputFile.write(getText(title.childNodes))
NameError: global name 'outputFile' is not defined
------------------------------------------------------------
Looks like something has gone out of scope!
Fellow beginners, take note.