Develop Reference

py2neo erroring for basic requests with neo4j

I have spent the last few months running neo4j on my local machine. I have been using py2neo to orchestrate the db with python. I have just switched to running the database on an aws ec2 instance and I am recieving errors whenever I try to send requests to the database.
I am using the following simple code to test the database, this code works perfectly on my local machine.
from py2neo import *
neo4j_url = 'http://localhost:7474/'
user = 'neo4j'
pwd = 'neo4j'
graph = Graph(neo4j_url, auth=(user,pwd))
print(graph)
graph.run("Create (n:Person{name: 'Andy person'})")
This code errors with
Graph('http://localhost:7474')
Traceback (most recent call last):
File "/home/ec2-user/KG_construction/kg-construction/KG_env/lib64/python3.8/site-packages/py2neo/client/http.py", line 443, in from_json
content = json_loads(data, object_hook=JSONHydrant.json_to_packstream)
File "/usr/lib64/python3.8/json/__init__.py", line 370, in loads
return cls(**kw).decode(s)
File "/usr/lib64/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "test_connection.py", line 9, in <module>
graph.run("Create (n:Person{name: 'Andy person'})")
File "/home/ec2-user/KG_construction/kg-construction/KG_env/lib64/python3.8/site-packages/py2neo/database.py", line 405, in run
return self.auto().run(cypher, parameters, **kwparameters)
File "/home/ec2-user/KG_construction/kg-construction/KG_env/lib64/python3.8/site-packages/py2neo/database.py", line 989, in run
result = self._connector.auto_run(cypher, parameters,
File "/home/ec2-user/KG_construction/kg-construction/KG_env/lib64/python3.8/site-packages/py2neo/client/__init__.py", line 1340, in auto_run
return cx.auto_run(cypher, parameters, graph_name=graph_name, readonly=readonly)
File "/home/ec2-user/KG_construction/kg-construction/KG_env/lib64/python3.8/site-packages/py2neo/client/http.py", line 177, in auto_run
rs = HTTPResponse.from_json(r.status, r.data.decode("utf-8"))
File "/home/ec2-user/KG_construction/kg-construction/KG_env/lib64/python3.8/site-packages/py2neo/client/http.py", line 445, in from_json
raise_from(ProtocolError("Cannot decode response content as JSON"), error)
File "<string>", line 3, in raise_from
py2neo.errors.ProtocolError: Cannot decode response content as JSON
Does anyone have idea why this error is occuring on my ec2 instance but not on my local machine?

Scrapy - how to wait for json page to be fully loaded

I scrape json pages but sometimes I get this error:
ERROR: Spider error processing <GET https://reqbin.com/echo/get/json/page/2>
Traceback (most recent call last):
File "/home/user/.local/lib/python3.8/site-packages/twisted/internet/defer.py", line 857, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "/home/user/path/scraping.py", line 239, in parse_images
jsonresponse = json.loads(response.text)
File "/usr/lib/python3.8/json/init.py", line 357, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.8/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 48662 (char 48661)
So I suspect that the json page does not have the time to be fully loaded and that's why parsing of its json content fails. And if I do it manually, I mean taking the json content as a string and loading it with the json module, it works and I don't get the json.decoder.JSONDecodeError error.
What I've done so far is to set in settings.py:
DOWNLOAD_DELAY = 5
DOWNLOAD_TIMEOUT = 600
DOWNLOAD_FAIL_ON_DATALOSS = False
CONCURRENT_REQUESTS = 8
hoping that it would slow down the scraping and solve my problem but the problem still occurs.
Any idea on how to be sure that the json page loaded completely so the parsing of its content does not fail ?

you can try to increase DOWNLOAD_TIMEOUT. It usually helps. If that's not enough, you can try to reduce CONCURRENT_REQUESTS.
If that still doesn't help, try use retry request. You can write your own retry_request function and call it return self.retry_request(response).
Or do it something like that req = response.request.copy(); req.dont_filter=True And return req.
You can also use RetryMiddleware. Read more on the documentation page https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#module-scrapy.downloadermiddlewares.retry

AttributeError : object has no attribute

I got a program from a formerly colleague and now should maintain it.
This python script asks our Jira instance with a given jql ( on the API ).
The return is a list of all issues, which are matching the search criteria.
But now it's not working, and I receive on the server ( Ubuntu ) and on my local windows PC a Json error message.
note : it ran for about a year not, but back then it worked.
Here is what the script looks like :
import json
import subprocess
jiraSerachUrl = "https://ourJiraInstance.net/rest/api/2/search?jql=key%20=%20%22TEST-123%22"
jiraResponse = subprocess.Popen(["curl","-l","-s","-u", "jiraUser"+":"+"jiraUserPassword", "-X", "GET", jiraSerachUrl ],stdout=subprocess.PIPE,shell=True).communicate()[0]
## shell=True only added for Windows Instance
print(type(jiraResponse))
##print = <class 'bytes'>
print(jiraResponse)
## print = b''
jiraJsonResponse = json.loads(jiraResponse.decode('utf-8'))
print(jiraJsonResponse)
The jql/jira search address returns the following (shorted answer, all fields of the task are returned):
{"expand":"names,schema","startAt":0,"maxResults":50,"total":1,"issues":
[{"expand":"operations,versionedRepresentations,editmeta,changelog,transitions,renderedFields",
"id":"145936","self":"https://ourJiraInstance.net/rest/api/2/issue/145936","key":"TEST-123","fields":{"parent": ...
The Error on the Windows PC is the following
Traceback (most recent call last): File
"C:\Users\User\Desktop\test.py", line 10, in
jiraJsonResponse = json.loads(jiraResponse.decode('utf-8')) File "C:\Users\User\AppData\Local\Programs\Python\Python35-32\lib\json__init__.py",
line 319, in loads
return _default_decoder.decode(s) File "C:\Users\User\AppData\Local\Programs\Python\Python35-32\lib\json\decoder.py",
line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "C:\Users\User\AppData\Local\Programs\Python\Python35-32\lib\json\decoder.py",
line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char
0)
This is the error on the Ubuntu Server ( running the same script )
Traceback (most recent call last): File "searchJira.py", line 33, in
jiraJsonResponse = json.loads(jiraResponse) File "/usr/lib/python2.7/json/init.py", line 338, in loads
return _default_decoder.decode(s) File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded
So far I tried to change the Json load to simpleJson, but with the same result.
Changing the format to which Json should decode ( e.g. unicode ) took no effect.

I have tried a bit and finaly got it. by replacing curl with responses i got finally the result I wanted. my request looks now like this :
r = requests.get(jiraSerachUrl,auth=HTTPBasicAuth(user, password), verify=False)
jiraJsonResponse=json.loads(r.text)

API Coinbase ValueError from get_buy_price()

python 3.4 and Coinbase V2 API
I am working on some BTC data analysis and trying to make continuous requests to coinbase API. When running my script, it will always eventually crash on a calls to
r = client.get_spot_price()
r = client.get_buy_price()
r = client.get_sell_price()
The unusual thing is that the script will always crash at different times. Sometimes it will successfully collect data for an hour or so and then crash, other times it will crash after 5 - 10 minutes.
ERROR:
r = client.get_spot_price()
File "/home/g/.local/lib/python3.4/site-packages/coinbase/wallet/client.py", line 191, in get_spot_price
response = self._get('v2', 'prices', 'spot', data=params)
File "/home/g/.local/lib/python3.4/site-packages/coinbase/wallet/client.py", line 129, in _get
return self._request('get', *args, **kwargs)
File "/home/g/.local/lib/python3.4/site-packages/coinbase/wallet/client.py", line 116, in _request
return self._handle_response(response)
File "/home/g/.local/lib/python3.4/site-packages/coinbase/wallet/client.py", line 125, in _handle_response
raise build_api_error(response)
File "/home/g/.local/lib/python3.4/site-packages/coinbase/wallet/error.py", line 49, in build_api_error
blob = blob or response.json()
File "/home/g/.local/lib/python3.4/site-packages/requests/models.py", line 812, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib/python3.4/json/__init__.py", line 318, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.4/json/decoder.py", line 343, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.4/json/decoder.py", line 361, in raw_decode
raise ValueError(errmsg("Expecting value", s, err.value)) from None
ValueError: Expecting value: line 1 column 1 (char 0)
It seems to be crashing due to some json decoding?
Does anyone have any idea why this will only throw errors at certain times?
I have tried something like the following to avoid crashing due to this error:
#snap is tuple of data containing data from buy, sell , spot price
if not any(snap):
print('\n\n-----ENTRY ERROR---- Snap returned None \n\n')
success = False
return
but it isn't doing the trick
What are some good ways to handle this error in your opinion?
Thanks, any help is much appreciated!

For me it could be something related with that issue https://github.com/coinbase/coinbase-python/issues/15. It seems in fact to be an internal library error (as the code does raise build_api_error(response) what confirms my assertions).
Maybe it possible that the problem is related to a internet connectivity? If your network (or the server fails), it can either fail to retrieve the JSON file or can retrieve an empty one. But, the library should inform you more clearly.
So, it will try to decode an empty file inside the JSON decoder, what causes the error.
A temporary workaround would be to brace your code with a try statement and to try again if it fails.

You have to supply it with a currency to get a price.
Here is an example:
price = client.get_spot_price(currency_pair='XRP-USD')

Google Maps Business Geocoding using Python GAE

In the Google API Console, the API Access generates keys for server, browser, Android, and iOS.
Which type of api_key do I need to generate geocoding from Google App Engine Python?
If the answer is the server, do I need to find out the server's IP address for GAE, what are they? I got 74.125.129.141, which can probably change. (or would it?)
Can you help me find an example to generate the code if you have a Python GAE server I'm not sure if I'm doing the right thing?
I have this code, which displays the data from the result.content. My purpose is to save the geocode to the database using ndb.GeoPt(lat,lng):
def get_geo(address):
browser_key = "browser_key"
server_key = "server_key"
address = cgi.escape(address.replace(" ","+"))
sensor = "true"
url = "http://maps.googleapis.com/maps/api/geocode/json?key="+ server_key +"&address=" + address + "&sensor="+ sensor
result = urlfetch.fetch(url)
data = json.loads(result.content) # This is the error
geo = data
#lat = data['results'][0]['geometry']['location']['lat']
#long = data['results'][0]['geometry']['location']['lng']
#return ndb.GeoPt(str(lat) +","+ str(long))
return geo
I have this error in using the above example:
File "C:\xampp\htdocs\mapjobs\main.py", line 18, in get_geo
data = json.loads(result.content)
File "C:\Python27\lib\json\__init__.py", line 310, in loads
return _default_decoder.decode(s)
File "C:\Python27\lib\json\decoder.py", line 346, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Python27\lib\json\decoder.py", line 364, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

It should include the browser_key, and the url should be:
url = "http://maps.googleapis.com/maps/api/geocode/json?key="+ browser_key +"&address=" + address + "&sensor="+ sensor