I got a program from a formerly colleague and now should maintain it.
This python script asks our Jira instance with a given jql ( on the API ).
The return is a list of all issues, which are matching the search criteria.
But now it's not working, and I receive on the server ( Ubuntu ) and on my local windows PC a Json error message.
note : it ran for about a year not, but back then it worked.
Here is what the script looks like :
import json
import subprocess
jiraSerachUrl = "https://ourJiraInstance.net/rest/api/2/search?jql=key%20=%20%22TEST-123%22"
jiraResponse = subprocess.Popen(["curl","-l","-s","-u", "jiraUser"+":"+"jiraUserPassword", "-X", "GET", jiraSerachUrl ],stdout=subprocess.PIPE,shell=True).communicate()[0]
## shell=True only added for Windows Instance
print(type(jiraResponse))
##print = <class 'bytes'>
print(jiraResponse)
## print = b''
jiraJsonResponse = json.loads(jiraResponse.decode('utf-8'))
print(jiraJsonResponse)
The jql/jira search address returns the following (shorted answer, all fields of the task are returned):
{"expand":"names,schema","startAt":0,"maxResults":50,"total":1,"issues":
[{"expand":"operations,versionedRepresentations,editmeta,changelog,transitions,renderedFields",
"id":"145936","self":"https://ourJiraInstance.net/rest/api/2/issue/145936","key":"TEST-123","fields":{"parent": ...
The Error on the Windows PC is the following
Traceback (most recent call last): File
"C:\Users\User\Desktop\test.py", line 10, in
jiraJsonResponse = json.loads(jiraResponse.decode('utf-8')) File "C:\Users\User\AppData\Local\Programs\Python\Python35-32\lib\json__init__.py",
line 319, in loads
return _default_decoder.decode(s) File "C:\Users\User\AppData\Local\Programs\Python\Python35-32\lib\json\decoder.py",
line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "C:\Users\User\AppData\Local\Programs\Python\Python35-32\lib\json\decoder.py",
line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char
0)
This is the error on the Ubuntu Server ( running the same script )
Traceback (most recent call last): File "searchJira.py", line 33, in
jiraJsonResponse = json.loads(jiraResponse) File "/usr/lib/python2.7/json/init.py", line 338, in loads
return _default_decoder.decode(s) File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded
So far I tried to change the Json load to simpleJson, but with the same result.
Changing the format to which Json should decode ( e.g. unicode ) took no effect.
I have tried a bit and finaly got it. by replacing curl with responses i got finally the result I wanted. my request looks now like this :
r = requests.get(jiraSerachUrl,auth=HTTPBasicAuth(user, password), verify=False)
jiraJsonResponse=json.loads(r.text)
Related
I have spent the last few months running neo4j on my local machine. I have been using py2neo to orchestrate the db with python. I have just switched to running the database on an aws ec2 instance and I am recieving errors whenever I try to send requests to the database.
I am using the following simple code to test the database, this code works perfectly on my local machine.
from py2neo import *
neo4j_url = 'http://localhost:7474/'
user = 'neo4j'
pwd = 'neo4j'
graph = Graph(neo4j_url, auth=(user,pwd))
print(graph)
graph.run("Create (n:Person{name: 'Andy person'})")
This code errors with
Graph('http://localhost:7474')
Traceback (most recent call last):
File "/home/ec2-user/KG_construction/kg-construction/KG_env/lib64/python3.8/site-packages/py2neo/client/http.py", line 443, in from_json
content = json_loads(data, object_hook=JSONHydrant.json_to_packstream)
File "/usr/lib64/python3.8/json/__init__.py", line 370, in loads
return cls(**kw).decode(s)
File "/usr/lib64/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "test_connection.py", line 9, in <module>
graph.run("Create (n:Person{name: 'Andy person'})")
File "/home/ec2-user/KG_construction/kg-construction/KG_env/lib64/python3.8/site-packages/py2neo/database.py", line 405, in run
return self.auto().run(cypher, parameters, **kwparameters)
File "/home/ec2-user/KG_construction/kg-construction/KG_env/lib64/python3.8/site-packages/py2neo/database.py", line 989, in run
result = self._connector.auto_run(cypher, parameters,
File "/home/ec2-user/KG_construction/kg-construction/KG_env/lib64/python3.8/site-packages/py2neo/client/__init__.py", line 1340, in auto_run
return cx.auto_run(cypher, parameters, graph_name=graph_name, readonly=readonly)
File "/home/ec2-user/KG_construction/kg-construction/KG_env/lib64/python3.8/site-packages/py2neo/client/http.py", line 177, in auto_run
rs = HTTPResponse.from_json(r.status, r.data.decode("utf-8"))
File "/home/ec2-user/KG_construction/kg-construction/KG_env/lib64/python3.8/site-packages/py2neo/client/http.py", line 445, in from_json
raise_from(ProtocolError("Cannot decode response content as JSON"), error)
File "<string>", line 3, in raise_from
py2neo.errors.ProtocolError: Cannot decode response content as JSON
Does anyone have idea why this error is occuring on my ec2 instance but not on my local machine?
I scrape json pages but sometimes I get this error:
ERROR: Spider error processing <GET https://reqbin.com/echo/get/json/page/2>
Traceback (most recent call last):
File "/home/user/.local/lib/python3.8/site-packages/twisted/internet/defer.py", line 857, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "/home/user/path/scraping.py", line 239, in parse_images
jsonresponse = json.loads(response.text)
File "/usr/lib/python3.8/json/init.py", line 357, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.8/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 48662 (char 48661)
So I suspect that the json page does not have the time to be fully loaded and that's why parsing of its json content fails. And if I do it manually, I mean taking the json content as a string and loading it with the json module, it works and I don't get the json.decoder.JSONDecodeError error.
What I've done so far is to set in settings.py:
DOWNLOAD_DELAY = 5
DOWNLOAD_TIMEOUT = 600
DOWNLOAD_FAIL_ON_DATALOSS = False
CONCURRENT_REQUESTS = 8
hoping that it would slow down the scraping and solve my problem but the problem still occurs.
Any idea on how to be sure that the json page loaded completely so the parsing of its content does not fail ?
you can try to increase DOWNLOAD_TIMEOUT. It usually helps. If that's not enough, you can try to reduce CONCURRENT_REQUESTS.
If that still doesn't help, try use retry request. You can write your own retry_request function and call it return self.retry_request(response).
Or do it something like that req = response.request.copy(); req.dont_filter=True And return req.
You can also use RetryMiddleware. Read more on the documentation page https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#module-scrapy.downloadermiddlewares.retry
I'm trying to work with JSON data that is pulled from USGS Earthquake API. If you follow that link, you can see the raw JSON data.
The JSON looks great; however, the returned request is wrapped in an eqfeed_callback(); that is breaking the JSON deserializer in Python.
A quick look at the code I have so far:
import requests
import json
from pprint import pprint
URL = "http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/2.5_week.geojsonp"
response = requests.get(URL)
raw_json = str(response.content)
json = json.loads(raw_json)
print(json)
I get the errors:
Traceback (most recent call last):
File "run.py", line 11, in <module>
json = json.loads(raw_json)
File "C:\Program Files\Anaconda3\lib\json\__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "C:\Program Files\Anaconda3\lib\json\decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Program Files\Anaconda3\lib\json\decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Although I'm positive the issue is that it's wrapped in that function and the JSON decoder doesn't like it. So how would I go about removing the function wrapper to leave me with the clean JSON inside.
You're using the wrong URL.
JSON wrapped in a function call is JSONP, which is needed for getting around CORS when calling an API from web browsers.
The URL to get normal JSON is
URL = "http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/2.5_week.geojson"
I'm trying to get information from the Yahoo web site with the following code:
import json, requests
response = requests.get("https://finance.yahoo.com/quote/AAPL/options?p=AAPL")
output = json.loads(response.text)
print(output)
Yet i'm getting this error message:
File "C:/Users/ziggy/.spyder-py3/untitled4.py", line 4, in <module>
output = json.loads(response.text)
File "C:\Users\ziggy\Anaconda3\lib\json\__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "C:\Users\ziggy\Anaconda3\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\ziggy\Anaconda3\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
JSONDecodeError: Expecting value
Really unclear what you're trying to achieve here considering the response is an HTML page.
But, ignoring that, if we're just talking about getting rid of the error, serialize the response first with json.dumps:
import json, requests
response = requests.get("https://finance.yahoo.com/quote/AAPL/options?p=AAPL")
encoded_response = json.dumps(response.text)
output = json.loads(encoded_response)
print(output)
I'm trying to execute Overpass queries from a Python script. I'm practicing at overpass-turbo.eu and found the following query to work as intended:
[out:json][timeout:600];
{{geocodeArea:Niedersachsen}}->.searchArea;
(
node[place=city](area.searchArea);
node[place=town](area.searchArea);
);
out;
However, when I submit the exact same query from a Python script, I get an error:
import requests
overpass_query = """
[out:json][timeout:600];
{{geocodeArea:Niedersachsen}}->.searchArea;
(
node[place=city](area.searchArea);
node[place=town](area.searchArea);
);
out;
"""
overpass_url = "http://overpass-api.de/api/interpreter"
response = requests.get(overpass_url, params={'data': overpass_query})
data = response.json()
/home/enno/events/docker/etl/venv/bin/python /home/enno/events/docker/etl/test2.py
Traceback (most recent call last):
File "/home/enno/events/docker/etl/test2.py", line 16, in <module>
data = response.json()
File "/home/enno/events/docker/etl/venv/lib/python3.6/site-packages/requests/models.py", line 897, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Process finished with exit code 1
Why is this? It seems to have to do with the curly braces, but I can't figure out how to solve this.
Many thanks,
Enno
The curly braces (aka {{geocodeArea:Niedersachsen}}) are a special feature of overpass turbo and are not part of Overpass API. See extended overpass turbo queries for a list of these shortcuts.
{{geocodeArea:name}} will tell overpass turbo to perform a geocoding request using Nominatim. It will then use the first result to construct an area(id) query. You have to perform the same step (using Nominatim or any other geocoder) in your program.