Develop Reference

Error to connect firestore to store an API result made at Cloud Shell with Python

I just want to learn how to store data in Firestore using Python and Google Cloud Platform, so I'm calling an API to query some example data.
For it, I'm using requests library and Firebase library from google.cloud package.
Here is the code that I'm running at Cloud Shell:
import requests
from google.cloud import firestore
url = "https://api.coindesk.com/v1/bpi/currentprice.json"
r = requests.get(url)
resp: str = r.text
if not (resp=="null" or resp=="[]"):
db = firestore.Client()
doc_ref=db.collection("CoinData").add(r.json())
When the code try to connect to Firebase to add the json of the API response I got this error:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/google/api_core/grpc_helpers.py", line 66, in error_remapped_callable
return callable_(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/grpc/_channel.py", line 946, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.7/dist-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INVALID_ARGUMENT
details = "Invalid resource field value in the request."
debug_error_string = "{"created":"#1634689546.004704998","description":"Error received from peer ipv4:74.125.134.95:443","file":"src/core/lib/surface/call.cc","file_line":1070,"grpc_message":"Invalid resource field value in the request.","grpc_status":3}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/antonyare_93/cloudshell_open/pruebas/data_coin_query.py", line 11, in <module>
doc_ref=db.collection("CoinData").add(r.json())
File "/home/antonyare_93/.local/lib/python3.7/site-packages/google/cloud/firestore_v1/collection.py", line 107, in add
write_result = document_ref.create(document_data, **kwargs)
File "/home/antonyare_93/.local/lib/python3.7/site-packages/google/cloud/firestore_v1/document.py", line 99, in create
write_results = batch.commit(**kwargs)
File "/home/antonyare_93/.local/lib/python3.7/site-packages/google/cloud/firestore_v1/batch.py", line 60, in commit
request=request, metadata=self._client._rpc_metadata, **kwargs,
File "/home/antonyare_93/.local/lib/python3.7/site-packages/google/cloud/firestore_v1/services/firestore/client.py", line 815, in commit
response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)
File "/usr/local/lib/python3.7/dist-packages/google/api_core/gapic_v1/method.py", line 142, in __call__
return wrapped_func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/google/api_core/retry.py", line 288, in retry_wrapped_func
on_error=on_error,
File "/usr/local/lib/python3.7/dist-packages/google/api_core/retry.py", line 190, in retry_target
return target()
File "/usr/local/lib/python3.7/dist-packages/google/api_core/grpc_helpers.py", line 68, in error_remapped_callable
raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.InvalidArgument: 400 Invalid resource field value in the request.
Anyone knows how I can fix it?

I've just got it.
It was a little mistake, I was missing the name of the project at the firestore client method call, here's the code working:
import requests
from google.cloud import firestore
url = "https://api.coindesk.com/v1/bpi/currentprice.json"
r = requests.get(url)
resp: str = r.text
if not (resp=="null" or resp=="[]"):
db = firestore.Client(project="mytwitterapitest")
doc_ref=db.collection("CoinData").add(r.json())

AXL Python getLine using pattern instead of uuid does not work (CUCM 11.5)

I'm struggling with getting for example line info via getLine using pattern instead of uuid. According to getLine AXL schema for CUCM 11.5 it should be possible to choose either uuid or pattern as lookup criteria. The python (2.7) module that I'm using is suds-jurko 0.6.
Base code is as follows:
from suds.client import Client
import ssl
wsdl = 'file:///C:/Users/xyz/Documents/axlplugin/schema/current/AXLAPI.wsdl'
location = 'https://10.10.20.1/axl/'
username = '***'
password = '***'
ssl._create_default_https_context = ssl._create_unverified_context
client = Client(wsdl, location=location, username=username, password=password)
First of all, the proof that the pattern I want to use as a lookup string for getLine actually exists:
>>> line2 = client.service.listLine({'pattern': '1018'}, returnedTags={'description': ''})
>>> line2
(reply){
return =
(return){
line[] =
(LLine){
_uuid = "{1EC56035-6B5D-283A-4DF0-EFEFA01FCEFF}"
description = None
},
}
}
Then, I try getLine using uuid which works well:
>>> line5 = client.service.getLine('1EC56035-6B5D-283A-4DF0-EFEFA01FCEFF')
>>> line5
(reply){
return =
(return){
line =
(RLine){
_uuid = "{1EC56035-6B5D-283A-4DF0-EFEFA01FCEFF}"
pattern = "1018"
description = None
usage = "Device"
routePartitionName = ""
aarNeighborhoodName = ""
----Rest omitted for brevity---
Now, if I try to use pattern, not uuid, I have following error:
line6 = client.service.getLine({'pattern': '1018'}, {'routePartitionName': ''})
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\site-packages\suds\client.py", line 521, in __call__
return client.invoke(args, kwargs)
File "C:\Python27\lib\site-packages\suds\client.py", line 576, in invoke
soapenv = binding.get_message(self.method, args, kwargs)
File "C:\Python27\lib\site-packages\suds\bindings\binding.py", line 109, in get_message
content = self.bodycontent(method, args, kwargs)
File "C:\Python27\lib\site-packages\suds\bindings\document.py", line 95, in bodycontent
add_param, self.options().extraArgumentErrors)
File "C:\Python27\lib\site-packages\suds\argparser.py", line 83, in parse_args
return arg_parser(args, kwargs, extra_parameter_errors)
File "C:\Python27\lib\site-packages\suds\argparser.py", line 108, in __call__
self.__process_parameters()
File "C:\Python27\lib\site-packages\suds\argparser.py", line 299, in __process_parameters
self.__process_parameter(*pdef)
File "C:\Python27\lib\site-packages\suds\argparser.py", line 294, in __process_parameter
self.__in_choice_context(), value)
File "C:\Python27\lib\site-packages\suds\bindings\document.py", line 86, in add_param
p = self.mkparam(method, pdef, value)
File "C:\Python27\lib\site-packages\suds\bindings\document.py", line 130, in mkparam
return Binding.mkparam(self, method, pdef, object)
File "C:\Python27\lib\site-packages\suds\bindings\binding.py", line 225, in mkparam
return marshaller.process(content)
File "C:\Python27\lib\site-packages\suds\mx\core.py", line 59, in process
self.append(document, content)
File "C:\Python27\lib\site-packages\suds\mx\core.py", line 72, in append
self.appender.append(parent, content)
File "C:\Python27\lib\site-packages\suds\mx\appender.py", line 88, in append
appender.append(parent, content)
File "C:\Python27\lib\site-packages\suds\mx\appender.py", line 229, in append
Appender.append(self, child, cont)
File "C:\Python27\lib\site-packages\suds\mx\appender.py", line 168, in append
self.marshaller.append(parent, content)
File "C:\Python27\lib\site-packages\suds\mx\core.py", line 71, in append
if self.start(content):
File "C:\Python27\lib\site-packages\suds\mx\literal.py", line 86, in start
raise TypeNotFound(content.tag)
suds.TypeNotFound: Type not found: 'pattern'
Trying different syntax also gives error:
line6 = client.service.getLine('1018')
No handlers could be found for logger "suds.client"
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\site-packages\suds\client.py", line 521, in __call__
return client.invoke(args, kwargs)
File "C:\Python27\lib\site-packages\suds\client.py", line 581, in invoke
result = self.send(soapenv)
File "C:\Python27\lib\site-packages\suds\client.py", line 619, in send
description=tostr(e), original_soapenv=original_soapenv)
File "C:\Python27\lib\site-packages\suds\client.py", line 670, in process_reply
raise WebFault(fault, replyroot)
suds.WebFault: Server raised fault: 'Item not valid: The specified Line was not found'
Am I using incorrect syntax or it's actually not possible to use pattern however schema file states differently?
Additionally, I'm wondering how to extract only uuid value from for example listLine method, is it possible via python axl?

In Suds, if memory serves me correctly, you can use keyword arguments for get and update requests.
For example:
resp = client.service.getLine(routePartitionName='PT-ONCLUSTER', pattern='\+49301234567')
if that doesn't work, try:
resp = client.service.getLine({'routePartitionName': 'PT-ONCLUSTER', 'pattern': '\+49301234567'})
Or better yet, check out python-zeep instead. It's much faster and better maintained than suds. Some example code with Zeep:
# -*- coding: utf-8 -*-
from zeep import Client
from zeep.cache import SqliteCache
from zeep.transports import Transport
from zeep.exceptions import Fault
from zeep.plugins import HistoryPlugin
from requests import Session
from requests.auth import HTTPBasicAuth
from urllib3 import disable_warnings
from urllib3.exceptions import InsecureRequestWarning
from lxml import etree
disable_warnings(InsecureRequestWarning)
username = 'admin'
password = 'password'
# If you're not disabling SSL verification, host should be the FQDN of the server rather than IP
host = '10.1.1.1'
wsdl = 'file://C:/path/to/wsdl/AXLAPI.wsdl'
location = 'https://{host}:8443/axl/'.format(host=host)
binding = "{http://www.cisco.com/AXLAPIService/}AXLAPIBinding"
# Create a custom session to disable Certificate verification.
# In production you shouldn't do this,
# but for testing it saves having to have the certificate in the trusted store.
session = Session()
session.verify = False
session.auth = HTTPBasicAuth(username, password)
transport = Transport(cache=SqliteCache(), session=session, timeout=20)
history = HistoryPlugin()
client = Client(wsdl=wsdl, transport=transport, plugins=[history])
service = client.create_service(binding, location)
def show_history():
for item in [history.last_sent, history.last_received]:
print(etree.tostring(item["envelope"], encoding="unicode", pretty_print=True))
try:
resp = service.getLine(pattern='1018', routePartitionName='')
except Fault:
show_history()

Reverse proxy in python 3

I need a simple reverse proxy for Python 3.
I like Twisted and its simple reverse http proxy (http://twistedmatrix.com/documents/14.0.1/_downloads/reverse-proxy.py) ...
# Copyright (c) Twisted Matrix Laboratories.
# See LICENSE for details.
"""
This example demonstrates how to run a reverse proxy.
Run this example with:
$ python reverse-proxy.py
Then visit http://localhost:8080/ in your web browser.
"""
from twisted.internet import reactor
from twisted.web import proxy, server
site = server.Site(proxy.ReverseProxyResource('www.yahoo.com', 80, ''))
reactor.listenTCP(8080, site)
reactor.run()
.... but it throws error in Python 3.
Unhandled Error
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/twisted/protocols/basic.py", line 571, in dataReceived
why = self.lineReceived(line)
File "/usr/local/lib/python3.4/dist-packages/twisted/web/http.py", line 1752, in lineReceived
self.allContentReceived()
File "/usr/local/lib/python3.4/dist-packages/twisted/web/http.py", line 1845, in allContentReceived
req.requestReceived(command, path, version)
File "/usr/local/lib/python3.4/dist-packages/twisted/web/http.py", line 766, in requestReceived
self.process()
--- <exception caught here> ---
File "/usr/local/lib/python3.4/dist-packages/twisted/web/server.py", line 185, in process
resrc = self.site.getResourceFor(self)
File "/usr/local/lib/python3.4/dist-packages/twisted/web/server.py", line 791, in getResourceFor
return resource.getChildForRequest(self.resource, request)
File "/usr/local/lib/python3.4/dist-packages/twisted/web/resource.py", line 98, in getChildForRequest
resource = resource.getChildWithDefault(pathElement, request)
File "/usr/local/lib/python3.4/dist-packages/twisted/web/resource.py", line 201, in getChildWithDefault
return self.getChild(path, request)
File "/usr/local/lib/python3.4/dist-packages/twisted/web/proxy.py", line 278, in getChild
self.host, self.port, self.path + b'/' + urlquote(path, safe=b"").encode('utf-8'),
builtins.TypeError: Can't convert 'bytes' object to str implicitly
Is there something similar that works in python 3?

Having stumbled into this problem myself, I was disappointed to find no answer here. After some digging, I was able to get it working by changing the path argument in proxy.ReverseProxyResource to bytes rather than str, rendering the following line:
site = server.Site(proxy.ReverseProxyResource("www.yahoo.com", 80, b''))
This is necessary because twisted appends a trailing slash as bytes (ie b'/') internally.

I found this simple package on pypi, it seems to be working well and it is similarly simple.
https://pypi.python.org/pypi/maproxy

Python cherrypy and bottle with ssl - server fails to save a file bigger than 102199 bytes

The next problem happens only when SSL is enabled!
I am running a server with bottle version 0.12.6 and cherrypy version 3.2.2 using https.
The client code sends a file to the server and the server saves it.
when i send a file with size below 102199 bytes, it is received and saved successfully. However, When i send a file with size bigger or equal to 102199, I get the exception:
The Server Code:
from bottle import request, response,static_file, run,server_names
from OpenSSL import crypto,SSL
from bottle import Bottle, run, request, server_names, ServerAdapter
app = Bottle()
app.mount('/test' , app)
class MySSLCherryPy(ServerAdapter):
def run(self, handler):
from cherrypy import wsgiserver
server = wsgiserver.CherryPyWSGIServer((self.host, self.port), handler)
server.ssl_certificate = "./cert"
server.ssl_private_key = "./key"
try:
server.start()
finally:
server.stop()
#app.post('/upload')
def received_file():
file = request.files.file
# file.save("./newfile")
file_path="./newfile"
with open(file_path, 'w') as open_file:
open_file.write(file.read())
if __name__=='__main__':
server_names['mysslcherrypy'] = MySSLCherryPy
run(app, host='0.0.0.0', port=4430, server='mysslcherrypy')
exit(0)
Why does the server fail to get file more than a given limit? is there a limit that i need to change?
(I tried to set the constant MEMFILE_MAX at the function received_file but it didn't help)
The problem vanish if the server is http and not https!
The exception in plain text (in case you cannot view the image):
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/bottle.py", line 861, in _handle
return route.call(**args)
File "/usr/lib/python2.6/site-packages/bottle.py", line 1727, in wrapper
rv = callback(*a, **ka)
File "testser", line 28, in received_file
file = request.files.file
File "/usr/lib/python2.6/site-packages/bottle.py", line 165, in get
if key not in storage: storage[key] = self.getter(obj)
File "/usr/lib/python2.6/site-packages/bottle.py", line 1106, in files
for name, item in self.POST.allitems():
File "/usr/lib/python2.6/site-packages/bottle.py", line 165, in get
if key not in storage: storage[key] = self.getter(obj)
File "/usr/lib/python2.6/site-packages/bottle.py", line 1222, in POST
args = dict(fp=self.body, environ=safe_env, keep_blank_values=True)
File "/usr/lib/python2.6/site-packages/bottle.py", line 1193, in body
self._body.seek(0)
File "/usr/lib/python2.6/site-packages/bottle.py", line 165, in get
if key not in storage: storage[key] = self.getter(obj)
File "/usr/lib/python2.6/site-packages/bottle.py", line 1162, in _body
for part in body_iter(read_func, self.MEMFILE_MAX):
File "/usr/lib/python2.6/site-packages/bottle.py", line 1125, in _iter_body
part = read(min(maxread, bufsize))
File "/usr/lib/python2.6/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 329, in read
data = self.rfile.read(size)
File "/usr/lib/python2.6/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 1052, in read
assert n <= left, "recv(%d) returned %d bytes" % (left, n)
AssertionError: recv(47) returned 48 bytes
Solution
In the file bottle.py I changed the value of .MEMFILE_MAX to be 10000000 and by this i solved the problem. The best way to do this is from your server code by adding the next line:
bottle.BaseRequest.MEMFILE_MAX=30000000000

Error in python webservice client using suds

Im using python 2.7 and suds 0.4 in windows and linux and in both cases I get the same error when calling a method of a web service:
Traceback (most recent call last):
File "wsclient.py", line 23, in <module>
client.service.Echo()
File "build\bdist.win32\egg\suds\client.py", line 542, in __call__
File "build\bdist.win32\egg\suds\client.py", line 602, in invoke
File "build\bdist.win32\egg\suds\client.py", line 643, in send
File "build\bdist.win32\egg\suds\client.py", line 678, in succeeded
File "build\bdist.win32\egg\suds\bindings\binding.py", line 149, in get_reply
AttributeError: 'NoneType' object has no attribute 'promotePrefixes'
My code is really simple:
import suds.bindings
suds.bindings.binding.envns = ('SOAP-ENV', 'http://www.w3.org/2003/05/soap-envelope')
from suds.client import Client
url = 'http://servicios.publipayments.com/ServicioDW.svc?wsdl'
client = Client(url)
print client
client.service.Echo()
As you can see I already did what the author of suds suggest here and also did the logging in here. But the result is the same.
Any ideas will be appreciated.
Regards.

My understanding about web services was not good enough and the service implementation is using http for exposing the wsdl AND https for the service endpoint.
So after a helpful hint of the service author I declared the client as:
client = Client('http://someUrl?wsdl',
location='https://someUrl/Service.svc')
And that solved the problem. There was nothing bad with suds.