Develop Reference

Cannot delete any triple from GraphDB using pythons SPARQLWrapper

I am trying to delete a simple triple from GraphDB (version = GraphDB free) using python's SPARQLWrapper and the code snippet I found here: https://github.com/RDFLib/sparqlwrapper - Update example. I always get the following exception:
SPARQLWrapper.SPARQLExceptions.QueryBadFormed: QueryBadFormed: a bad request has been sent to the endpoint, probably the sparql query is bad formed.
my code is:
sparql = SPARQLWrapper('http://192.168.0.242:7200/repositories/DataCitation')
sparql.setMethod(POST)
sparql.setQuery("""
PREFIX pub: <http://ontology.ontotext.com/taxonomy/>
delete where {
<http://ontology.ontotext.com/resource/tsk9hdnas934> pub:occupation "Cook".
}
""")
results = sparql.query()
print(results.response.read())
When I do an ask or select statement to the same endpoint I get a valid result. Only the update statements do not work.
this is the full stacktrace
/home/filip/anaconda3/envs/TripleStoreCitationFramework/bin/python /home/filip/PycharmProjects/TripleStoreCitationFramework/GraphDB/Playground.py
Traceback (most recent call last):
File "/home/filip/anaconda3/envs/TripleStoreCitationFramework/lib/python3.8/site-packages/SPARQLWrapper/Wrapper.py", line 1073, in _query
response = urlopener(request)
File "/home/filip/anaconda3/envs/TripleStoreCitationFramework/lib/python3.8/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/home/filip/anaconda3/envs/TripleStoreCitationFramework/lib/python3.8/urllib/request.py", line 531, in open
response = meth(req, response)
File "/home/filip/anaconda3/envs/TripleStoreCitationFramework/lib/python3.8/urllib/request.py", line 640, in http_response
response = self.parent.error(
File "/home/filip/anaconda3/envs/TripleStoreCitationFramework/lib/python3.8/urllib/request.py", line 569, in error
return self._call_chain(*args)
File "/home/filip/anaconda3/envs/TripleStoreCitationFramework/lib/python3.8/urllib/request.py", line 502, in _call_chain
result = func(*args)
File "/home/filip/anaconda3/envs/TripleStoreCitationFramework/lib/python3.8/urllib/request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/filip/PycharmProjects/TripleStoreCitationFramework/GraphDB/Playground.py", line 6, in <module>
citing.delete_triples("s")
File "/home/filip/PycharmProjects/TripleStoreCitationFramework/GraphDB/DataCiting.py", line 32, in delete_triples
results = sparql.query()
File "/home/filip/anaconda3/envs/TripleStoreCitationFramework/lib/python3.8/site-packages/SPARQLWrapper/Wrapper.py", line 1107, in query
return QueryResult(self._query())
File "/home/filip/anaconda3/envs/TripleStoreCitationFramework/lib/python3.8/site-packages/SPARQLWrapper/Wrapper.py", line 1077, in _query
raise QueryBadFormed(e.read())
SPARQLWrapper.SPARQLExceptions.QueryBadFormed: QueryBadFormed: a bad request has been sent to the endpoint, probably the sparql query is bad formed.
Response:
b'Missing parameter: query'

The endpoint which you need to insert statements isn't the simple SPARQL endpoint you use for ordinary queries, but, rather, the dedicated /statements endpoint:
http:///repositories//statements
This endpoint is also used for DELETE statements.
You can look up some examples in the RDF4J documentation.
Furthermore, if you are passing your data with a query string instead of it being a part of your request body, you need to be aware of the fact that it must start with a "?update=" instead of "?query=".opps

Python - RuntimeError: working outside of request context

Trying to get the GET parameters from the URL. I have it working in my __init__.py file, but in a different file its not working.
I tried to use with app.app_context(): but I am still getting the same issue.
def log_entry(entity, type, entity_id, data, error):
with app.app_context():
zip_id = request.args.get('id')
RuntimeError: working outside of request context
Any suggestions?
Additional Info:
This is using Flask web framework which is setup as a service (API).
Example URL the user would hit http://website.com/api/endpoint?id=1
As mentioned above using `zip_id = request.args.get('id') works fine in the main file but I am in runners.py (just another file with definitions in)
Full traceback:
Debugging middleware caught exception in streamed response at a point where response headers were already sent.
Traceback (most recent call last):
File "/Users/ereeve/.virtualenvs/pi-automation-api/lib/python2.7/site-packages/werkzeug/wsgi.py", line 703, in __next__
return self._next()
File "/Users/ereeve/.virtualenvs/pi-automation-api/lib/python2.7/site-packages/werkzeug/wrappers.py", line 81, in _iter_encoded
for item in iterable:
File "/Users/ereeve/Documents/TechSol/pi-automation-api/automation_api/runners.py", line 341, in create_agencies
log_entry("test", "created", 1, "{'data':'hey'}", "")
File "/Users/ereeve/Documents/TechSol/pi-automation-api/automation_api/runners.py", line 315, in log_entry
zip_id = request.args.get('id')
File "/Users/ereeve/.virtualenvs/pi-automation-api/lib/python2.7/site-packages/werkzeug/local.py", line 343, in __getattr__
return getattr(self._get_current_object(), name)
File "/Users/ereeve/.virtualenvs/pi-automation-api/lib/python2.7/site-packages/werkzeug/local.py", line 302, in _get_current_object
return self.__local()
File "/Users/ereeve/.virtualenvs/pi-automation-api/lib/python2.7/site-packages/flask/globals.py", line 20, in _lookup_req_object
raise RuntimeError('working outside of request context')
RuntimeError: working outside of request context
Def in the same file calling the log_entry def
def create_agencies(country_code, DB, session):
document = DB.find_one({'rb_account_id': RB_COUNTRIES_new[country_code]['rb_account_id']})
t2 = new_t2(session)
log_entry("test", "created", 1, "{'data':'hey'}", "")

error using rauth in app engine (Permission denied)

I have a app engine app, using oauth and rauth, i'm trying to use Facebook, Twitter and google to login.
When i run it locally works, but in production i got this error, but only with google plus, with facebook works fine.
('Connection aborted.', error(13, 'Permission denied'))
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1535, in call
rv = self.handle_exception(request, response, e)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1529, in call
rv = self.router.dispatch(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1102, in call
return handler.dispatch()
File "/base/data/home/apps/s~app-getwell/login:1.379942143707124638/handler.py", line 11, in dispatch
webapp2.RequestHandler.dispatch(self)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/s~app-getwell/login:1.379942143707124638/loginToken.py", line 69, in get
ep=log.getTokenData(code)
File "/base/data/home/apps/s~app-getwell/login:1.379942143707124638/code/oauth/conect.py", line 34, in getTokenData
session = self.getSession(conf,code)
File "/base/data/home/apps/s~app-getwell/login:1.379942143707124638/code/oauth/conect.py", line 61, in getSession
session=conf.get_auth_session(data=self.getData(code), decoder=json.loads)
File "/base/data/home/apps/s~app-getwell/login:1.379942143707124638/code/oauth/rauth/service.py", line 556, in get_auth_session
session = self.get_session(self.get_access_token(method, **kwargs))
File "/base/data/home/apps/s~app-getwell/login:1.379942143707124638/code/oauth/rauth/service.py", line 541, in get_access_token
r = self.get_raw_access_token(method, **kwargs)
File "/base/data/home/apps/s~app-getwell/login:1.379942143707124638/code/oauth/rauth/service.py", line 518, in get_raw_access_token
**kwargs)
File "/base/data/home/apps/s~app-getwell/login:1.379942143707124638/code/oauth/rauth/session.py", line 358, in request
return super(OAuth2Session, self).request(method, url, **req_kwargs)
File "/base/data/home/apps/s~app-getwell/login:1.379942143707124638/code/oauth/rauth/requests/sessions.py", line 457, in request
resp = self.send(prep, **send_kwargs)
File "/base/data/home/apps/s~app-getwell/login:1.379942143707124638/code/oauth/rauth/requests/sessions.py", line 569, in send
r = adapter.send(request, **kwargs)
File "/base/data/home/apps/s~app-getwell/login:1.379942143707124638/code/oauth/rauth/requests/adapters.py", line 407, in send
raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(13, 'Permission denied'))
I have the billing enabled, and in the yaml have added the ssl library (latests)
This is the code with wich i make the oauth calls
class login():
def __init__(self,tipo=tipoConexion.Google, redirect_uri = 'http://map.getwell.care/'):
self.__tipo=tipo
self.config=config.data(redirect_uri)
def getAuthorizationURL(self):
conf=self.getConfig()
params=self.getParams()
url = conf.get_authorize_url(**params)
return url
def getTokenData(self, code):
''' Get the data that i need from the provider '''
conf=self.getConfig()
session = self.getSession(conf,code)
tokenizerConcreto=JsonReader.factory(self.__tipo,session)
email=tokenizerConcreto.getEmail()[0]
urlPic=tokenizerConcreto.getPicture()[0]
logueado=not tokenizerConcreto.getEmail()[1]
return logueado, urlPic, email
def getParams(self):
params=None
if self.__tipo==tipoConexion.Google:
params = self.config.GooglePlusScope
if self.__tipo==tipoConexion.Facebook:
params = self.config.FacebookScope
return params
def getConfig(self):
conf=self.config.googlePlus
if self.__tipo==tipoConexion.Facebook:
conf=self.config.facebook
if self.__tipo==tipoConexion.Twitter:
conf=self.config.twitter
return conf
def getSession(self,conf, code):
session=None
if self.__tipo==tipoConexion.Google:
session=conf.get_auth_session(data=self.getData(code), decoder=json.loads)
else:
session=conf.get_auth_session(data=self.getData(code))
return session
def getData(self,code):
data=None
if self.__tipo==tipoConexion.Google:
data={
'code' : code,
'redirect_uri': self.config.redirect_uri,
'grant_type':'authorization_code'
}#
logging.info("GetWell: Data previo al error: %s" % data)
if self.__tipo==tipoConexion.Facebook:
data={
'code' : code,
'redirect_uri': self.config.redirect_uri,
}
if self.__tipo==tipoConexion.Twitter:
raise NotImplementedError
return data
and this is the code when i got the secrets keys
class tipoConexion():
Google=0
Facebook=1
Twitter=2
class data(object):
def __init__(self, url = 'http://map.getwell.care/'):
self.redirect_uri=url
def getURL(self):
return self.redirect_uri
#property
def twitter(self):
return OAuth1Service(
consumer_key='imnotatwitterman',
consumer_secret='ilovevine',
name='twitter',
access_token_url='https://api.twitter.com/oauth/access_token',
authorize_url='https://api.twitter.com/oauth/authorize',
request_token_url='https://api.twitter.com/oauth/request_token',
base_url='https://api.twitter.com/1/')
#property
def facebook(self):
return OAuth2Service(
client_id='someID',
client_secret='MyDarkSecretInFacebook',
name='facebook',
authorize_url='https://graph.facebook.com/oauth/authorize',
access_token_url='https://graph.facebook.com/oauth/access_token',
base_url='https://graph.facebook.com/')
#property
def FacebookScope(self):
return {
'scope': 'public_profile,email',
'response_type': 'code',
'redirect_uri': self.getURL()
}
#property
def googlePlus(self):
return OAuth2Service(
client_id='ThisCouldBeMyID.apps.googleusercontent.com',
client_secret='Idonthaveanysecrets',
name='googlePlus',
authorize_url='https://accounts.google.com/o/oauth2/auth',
access_token_url='https://accounts.google.com/o/oauth2/token',
base_url='https://accounts.google.com/o/oauth2/auth')
#property
def GooglePlusScope(self):
return {
'scope': 'https://www.googleapis.com/auth/plus.profile.emails.read',
'response_type': 'code',
'redirect_uri': self.getURL()
}
as i said the most strange is that works fine with facebook, but, fail with google plus (i doble check the client_id and the client_secret and are correct) if it was a problem with the sockets facebook would have to fail too
ps.
I copy the rauth files in my project and the request files inside the rauth folder

Having issues getting firebase data through Oauth pin-auth session for Nest

Base script:
from sanction import Client
# client_id & client_secret are omitted but are valid
client_pin = input('Enter PIN:')
access_token_url = 'https://api.home.nest.com/oauth2/access_token'
c = Client(
token_endpoint=access_token_url,
client_id=client_id,
client_secret=client_secret)
c.request_token(code = client_pin)
[See edits for history]
Running c.request('/devices') returned:
Traceback (most recent call last):
File "C:\py\nest_testing_sanction.py", line 36, in <module>
c.request("/devices")
File "C:\Python34\lib\site-packages\sanction-0.4.1-py3.4.egg\sanction\__init__.py", line 169, in request
File "C:\Python34\lib\site-packages\sanction-0.4.1-py3.4.egg\sanction\__init__.py", line 211, in transport_query
File "C:\Python34\lib\urllib\request.py", line 258, in __init__
self.full_url = url
File "C:\Python34\lib\urllib\request.py", line 284, in full_url
self._parse()
File "C:\Python34\lib\urllib\request.py", line 313, in _parse
raise ValueError("unknown url type: %r" % self.full_url)
ValueError: unknown url type: 'None/devices?access_token=c.[some long session token]'
Given the output it seems like I need to be putting in a generic URL so I tried c.request('wss://developer-api.nest.com'):
Traceback (most recent call last):
File "C:\py\nest_testing_sanction.py", line 36, in <module>
data = c.request(query_url)
File "C:\Python34\lib\site-packages\sanction-0.4.1-py3.4.egg\sanction\__init__.py", line 171, in request
File "C:\Python34\lib\urllib\request.py", line 153, in urlopen
return opener.open(url, data, timeout)
File "C:\Python34\lib\urllib\request.py", line 455, in open
response = self._open(req, data)
File "C:\Python34\lib\urllib\request.py", line 478, in _open
'unknown_open', req)
File "C:\Python34\lib\urllib\request.py", line 433, in _call_chain
result = func(*args)
File "C:\Python34\lib\urllib\request.py", line 1257, in unknown_open
raise URLError('unknown url type: %s' % type)
urllib.error.URLError: <urlopen error unknown url type: nonewss>
I also tried https as per:
- same result
By contrast, this works (for a firebase.io virtual device):
firebase = firebase.FirebaseApplication('https://nesttest.firebaseio.com', None)
thermostat_result = firebase.get('/devices', 'thermostats')

In Python I would use something like sanction to keep things simple. You should be able to get it to work with the Nest API using code like: (untested, using token flow rather than pin flow)
from sanction.client import Client
# instantiating a client to get the auth URI
c = Client(auth_endpoint="https://home.nest.com/login/oauth2",
client_id=config["nest.client_id"])
# instantiating a client to process OAuth2 response
c = Client(token_endpoint="https://api.home.nest.com/oauth2/access_token",
client_id=config["nest.client_id"],
client_secret=config["nest.client_secret"])
The library is well documented, so you should be able to figure it out from here if something is missing.

This is more of a comment, but the system does not let me comment just yet.
To your question about where to put the web pin simply add code = pin to the request_token call.
c.request_token(code = nest_client_pin)
This still does not fully solve the issue as I can only use a PIN once. After I have used it once, every subsequent call will fail again as you describe. Still researching that.

Receiving Email Attachments in App Engine Python errors on Unicode Text File

I have some code to parse an email and find the attachments then store them into the Datastore as db.BlobProperties (might change that to Blobstore later). The problem is that when I send a UTF8 encoded text file it generates an error.
The code basically saves the file and returns a key that is converted to a string then stored in the parent email entity. As you can see I decode the file then store it as a blob. I have sent lots of attachments and this code works on everything but Text encoded with Unicode. Is there a better way to do this? What can I do to handle Unicode text attachments?
code snipit
my_file = []
my_list = []
if hasattr(mail_message, 'attachments'):
file_name = ""
file_blob = ""
for filename, filecontents in mail_message.attachments:
file_name = filename
file_blob = filecontents.decode()
my_file.append(file_name)
my_list.append(str(store_file(self, file_name, file_blob)))
store_file
def store_file(self, file_name, file_blob):
new_file = myblob(file_name = file_name,
file_blob = file_blob)
return new_file.put()
I have tried using file_blob = str(file_blob) in the above to no avail. That just breaks the code and the file never gets stored.
log 1 of Unicode Text File
Property file_blob must be convertible to a Blob instance (Blob() argument should be str instance, not unicode)
Traceback (most recent call last):
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1530, in __call__
rv = self.router.dispatch(request, response)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/webapp/mail_handlers.py", line 65, in post
self.receive(mail.InboundEmailMessage(self.request.body))
File "/base/data/home/apps/s~ae-baseapp/1.359073377819595139/controllers/InboundHandler.py", line 51, in receive
file_list.append(str(store_file(self, file_name, file_blob)))
File "/base/data/home/apps/s~ae-baseapp/1.359073377819595139/models/MyModel.py", line 63, in store_file
file_blob = file_blob)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 974, in __init__
prop.__set__(self, value)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 614, in __set__
value = self.validate(value)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 2780, in validate
(self.name, self.data_type.__name__, err))
BadValueError: Property file_blob must be convertible to a Blob instance (Blob() argument should be str instance, not unicode)
Log 2 of removing the filecontents.decode() and replacing it with just filecontents.
Property file_blob must be convertible to a Blob instance (Blob() argument should be str instance, not EncodedPayload)
Traceback (most recent call last):
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1530, in __call__
rv = self.router.dispatch(request, response)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/webapp/mail_handlers.py", line 65, in post
self.receive(mail.InboundEmailMessage(self.request.body))
File "/base/data/home/apps/s~ae-baseapp/1.359097282640216691/controllers/InboundHandler.py", line 57, in receive
file_list.append(str(store_file(self, file_name, file_blob)))
File "/base/data/home/apps/s~ae-baseapp/1.359097282640216691/models/MyModel.py", line 64, in store_file
file_blob = file_blob)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 974, in __init__
prop.__set__(self, value)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 614, in __set__
value = self.validate(value)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 2780, in validate
(self.name, self.data_type.__name__, err))
BadValueError: Property file_blob must be convertible to a Blob instance (Blob() argument should be str instance, not EncodedPayload)

Attachment payloads are instances of the EncodedPayload class. Attachments have an encoding and an optional character set. The former refers to transfer encodings such as base64; the latter to character encodings such as UTF-8 (character set's a bit of an outdated and misleading term here). The EncodedPayload.decode() method decodes both transfer encoding and text encoding, which as you've noticed is not very helpful if you just want to get the original bytes the user attached to their message.
There's a number of approaches you can take here, but what I'd recommend is duplicating EncodedPayload's logic for decoding transfer encoding, which looks something like this:
if filecontents.encoding and filecontents.encoding.lower() != '7bit':
try:
payload = filecontents.payload.decode(filecontents.encoding)
except LookupError:
raise UnknownEncodingError('Unknown decoding %s.' % filecontents.encoding)
except (Exception, Error), e:
raise PayloadEncodingError('Could not decode payload: %s' % e)
else:
payload = filecontents.payload
Note that if the attachment was text, you need to include the character encoding when you store it, or there'll be no way to interpret it when you send it back to the user - the original text could have been encoded using any character encoding.
Likewise, you should also save the mimetype of the attachment if you can, but this doesn't appear to be exposed anywhere in the API. You might want to consider avoiding using the IncomingMessage class at all, and instead decoding the body of the POST request using Python's mime message module.

Check if this code helps:
===========================
my_file = []
my_list = []
if hasattr(mail_message, 'attachments'):
file_name = ""
for filename, filecontents in mail_message.attachments:
file_name = filename
file_blob = filecontents.payload
file_blob = file_blob.decode(filecontents.encoding)
my_file.append(file_name)
my_list.append(str(store_file(self, file_name, file_blob)))