Develop Reference

How to get list of events using the Python kubernetes API?

I am trying to obtain the list of events from a minikube cluster usigh the Python Kubernetes api using the following code:
from kubernetes import config, client
config.load_kube_config()
api = client.EventsV1beta1Api()
print(api.list_event_for_all_namespaces())
I am getting the following error:
C:\Users\lameg\kubesense>python test.py
Traceback (most recent call last):
File "C:\Users\lameg\kubesense\test.py", line 6, in <module>
print(api.list_event_for_all_namespaces())
File "C:\Users\lameg\miniforge3\lib\site-packages\kubernetes\client\api\events_v1beta1_api.py", line 651, in list_event_for_all_namespaces
return self.list_event_for_all_namespaces_with_http_info(**kwargs) # noqa: E501
File "C:\Users\lameg\miniforge3\lib\site-packages\kubernetes\client\api\events_v1beta1_api.py", line 758, in list_event_for_all_namespaces_with_http_info
return self.api_client.call_api(
File "C:\Users\lameg\miniforge3\lib\site-packages\kubernetes\client\api_client.py", line 348, in call_api
return self.__call_api(resource_path, method,
File "C:\Users\lameg\miniforge3\lib\site-packages\kubernetes\client\api_client.py", line 192, in __call_api
return_data = self.deserialize(response_data, response_type)
File "C:\Users\lameg\miniforge3\lib\site-packages\kubernetes\client\api_client.py", line 264, in deserialize
return self.__deserialize(data, response_type)
File "C:\Users\lameg\miniforge3\lib\site-packages\kubernetes\client\api_client.py", line 303, in __deserialize
return self.__deserialize_model(data, klass)
File "C:\Users\lameg\miniforge3\lib\site-packages\kubernetes\client\api_client.py", line 639, in __deserialize_model
kwargs[attr] = self.__deserialize(value, attr_type)
File "C:\Users\lameg\miniforge3\lib\site-packages\kubernetes\client\api_client.py", line 280, in __deserialize
return [self.__deserialize(sub_data, sub_kls)
File "C:\Users\lameg\miniforge3\lib\site-packages\kubernetes\client\api_client.py", line 280, in <listcomp>
return [self.__deserialize(sub_data, sub_kls)
File "C:\Users\lameg\miniforge3\lib\site-packages\kubernetes\client\api_client.py", line 303, in __deserialize
return self.__deserialize_model(data, klass)
File "C:\Users\lameg\miniforge3\lib\site-packages\kubernetes\client\api_client.py", line 641, in __deserialize_model
instance = klass(**kwargs)
File "C:\Users\lameg\miniforge3\lib\site-packages\kubernetes\client\models\v1beta1_event.py", line 112, in __init__
self.event_time = event_time
File "C:\Users\lameg\miniforge3\lib\site-packages\kubernetes\client\models\v1beta1_event.py", line 291, in event_time
raise ValueError("Invalid value for `event_time`, must not be `None`") # noqa: E501
ValueError: Invalid value for `event_time`, must not be `None`
Any ideas ?

That looks like either a bug in the Python client, or a bug in the OpenAPI specification used to generate the client: clearly, null is a value for eventTime that is supported by the API.
I think the only workaround is to monkey-patch the kubernetes.client module so that it accepts null values. Something like this:
from kubernetes import config, client
config.load_kube_config()
api = client.EventsV1beta1Api()
# This is descriptor, see https://docs.python.org/3/howto/descriptor.html
class FakeEventTime:
def __get__(self, obj, objtype=None):
return obj._event_time
def __set__(self, obj, value):
obj._event_time = value
# Monkey-patch the `event_time` attribute of ` the V1beta1Event class.
client.V1beta1Event.event_time = FakeEventTime()
# Now this works.
events = api.list_event_for_all_namespaces()
The above code runs successfully against my OpenShift instance, whereas previously it would fail as you describe in your question.

Django: Attempting to write a string directly into FileField, when backed by Amazon S3

I am attempting to write a string directly into a Django FileField by way of ContentFile.
In doing so, I get a reproducible
TypeError: Unicode-objects must be encoded before hashing
error when attempting to save the contents of this file to the database, which traces through the s3boto3 lib.
The exact source of this error is difficult to suss out.
But let's state the question plainly, in Python 3, on Django 2.2.x, what is the correct way to take a csv file created with the csv lib from Python, and save that into a Django FileField backed by Amazon S3?
This question, and my approach, is inspired by this entry on SO Django - how to create a file and save it to a model's FileField? - however, given the age of the answer, some detail relevant to newer versions of Django appear to have been left out? Difficult to tell.
Example code producing the error in question, truncated for privacy and relevance
def campaign_to_csv_string(campaign_id):
csv_string = io.StringIO()
campaign = Campaign.objects.get(pk=campaign_id)
checklist = campaign.checklist
completed_jobs = JobRecord.objects.filter(appointment__campaign=campaign)
writer = csv.writer(csv_string)
# A bunch of writing to the writer here
# string looks good at this point
return csv_string.getvalue()
calling function
csv_string = campaign_to_csv_string(campaign_report.campaign.pk)
campaign_report.last_run = datetime.datetime.now()
campaign_report.report_file.save(str(campaign_report_pk) + '.report', ContentFile(csv_string))
campaign_report.processing = False
campaign_report.save()
My guess here is that s3boto3 is taking issue with ContentFile but the debugging information sent back to me gives me no clear path forward.
edit
Stack trace by request
TypeError: Unicode-objects must be encoded before hashing
File "celery/app/trace.py", line 385, in trace_task
R = retval = fun(*args, **kwargs)
File "celery/app/trace.py", line 648, in __protected_call__
return self.run(*args, **kwargs)
File "main/tasks.py", line 94, in produce_basic_campaign_report
campaign_report.report_file.save(str(campaign_report_pk) + '.report', csv_file)
File "django/db/models/fields/files.py", line 87, in save
self.name = self.storage.save(name, content, max_length=self.field.max_length)
File "django/core/files/storage.py", line 52, in save
return self._save(name, content)
File "storages/backends/s3boto3.py", line 491, in _save
self._save_content(obj, content, parameters=parameters)
File "storages/backends/s3boto3.py", line 506, in _save_content
obj.upload_fileobj(content, ExtraArgs=put_parameters)
File "boto3/s3/inject.py", line 621, in object_upload_fileobj
ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
File "boto3/s3/inject.py", line 539, in upload_fileobj
return future.result()
File "s3transfer/futures.py", line 106, in result
return self._coordinator.result()
File "s3transfer/futures.py", line 265, in result
raise self._exception
File "s3transfer/tasks.py", line 126, in __call__
return self._execute_main(kwargs)
File "s3transfer/tasks.py", line 150, in _execute_main
return_value = self._main(**kwargs)
File "s3transfer/upload.py", line 692, in _main
client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
File "botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "botocore/client.py", line 642, in _make_api_call
request_signer=self._request_signer, context=request_context)
File "botocore/hooks.py", line 360, in emit_until_response
return self._emitter.emit_until_response(aliased_event_name, **kwargs)
File "botocore/hooks.py", line 243, in emit_until_response
responses = self._emit(event_name, kwargs, stop_on_response=True)
File "botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "botocore/handlers.py", line 212, in conditionally_calculate_md5
calculate_md5(params, **kwargs)
File "botocore/handlers.py", line 190, in calculate_md5
binary_md5 = _calculate_md5_from_file(body)
File "botocore/handlers.py", line 204, in _calculate_md5_from_file
md5.update(chunk)

The csv string needs to be encoded as bytes when instantiating the ContentFile
The error can be reproduced this way:
from django.core.files.base import ContentFile
from botocore.handlers import _calculate_md5_from_file
_calculate_md5_from_file(ContentFile('throws error'))
TypeError: Unicode-objects must be encoded before hashing.
content isn't internal converted to bytes unless it is a gzip mimetype or explicitly compressed. https://github.com/jschneier/django-storages/blob/1.7.2/storages/backends/s3boto.py#L417
_calculate_md5_from_file is expecting a file containing bytes and this is the same for the underlying boto3 s3 client put_object method.
I suggest encoding csv_string as bytes.
campaign_report.report_file.save(
str(campaign_report_pk) + '.report',
ContentFile(
csv_string.encode()
)
)

google datastore(ndb) with gevent

I am trying to use gevent based server with remote datastore api and, i occasionally get
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gevent/pywsgi.py", line 884, in handle_one_response
self.run_application()
File "/home/abhinavabcd_gmail_com/samosa-scripts/samosa_messaging_framework/geventwebsocket/handler.py", line 76, in run_ap
plication
self.run_websocket()
File "/home/abhinavabcd_gmail_com/samosa-scripts/samosa_messaging_framework/geventwebsocket/handler.py", line 52, in run_we
bsocket
self.application(self.environ, lambda s, h, e=None: [])
File "server.py", line 478, in __call__
current_app.handle()
File "/home/abhinavabcd_gmail_com/samosa-scripts/samosa_messaging_framework/geventwebsocket/resource.py", line 23, in handl
e
self.protocol.on_close()
File "/home/abhinavabcd_gmail_com/samosa-scripts/samosa_messaging_framework/geventwebsocket/protocols/base.py", line 14, in
on_close
self.app.on_close(reason)
File "server.py", line 520, in on_close
current_node.destroy_connection(self.ws)
File "server.py", line 428, in destroy_connection
db.remove_connection(conn.connection_id)
File "/home/abhinavabcd_gmail_com/samosa-scripts/samosa_messaging_framework/database_ndb.py", line 141, in remove_connectio
n
key.delete()
File "/home/abhinavabcd_gmail_com/google_appengine/google/appengine/ext/ndb/model.py", line 3451, in _put
return self._put_async(**ctx_options).get_result()
File "/home/abhinavabcd_gmail_com/google_appengine/google/appengine/ext/ndb/tasklets.py", line 383, in get_result
self.check_success()
File "/home/abhinavabcd_gmail_com/google_appengine/google/appengine/ext/ndb/tasklets.py", line 378, in check_success
self.wait()
File "/home/abhinavabcd_gmail_com/google_appengine/google/appengine/ext/ndb/tasklets.py", line 362, in wait
if not ev.run1():
File "/home/abhinavabcd_gmail_com/google_appengine/google/appengine/ext/ndb/eventloop.py", line 253, in run1
delay = self.run0()
File "/home/abhinavabcd_gmail_com/google_appengine/google/appengine/ext/ndb/eventloop.py", line 238, in run0
(rpc, self.rpcs))
RuntimeError: rpc <google.appengine.api.apiproxy_stub_map.UserRPC object at 0x7fb46e754a90> was not given to wait_any as a ch
oice {<google.appengine.api.apiproxy_stub_map.UserRPC object at 0x7fb46e779f90>: (<bound method Future._on_rpc_completion of
<Future 7fb46e756f10 created by run_queue(context.py:185) for tasklet _memcache_set_tasklet(context.py:1111); pending>>, (<go
ogle.appengine.api.apiproxy_stub_map.UserRPC object at 0x7fb46e779f90>, '', <google.appengine.datastore.datastore_rpc.Connect
ion object at 0x7fb46e874250>, <generator object _memcache_set_tasklet at 0x7fb46e79b9b0>), {}), <google.appengine.api.apipro
xy_stub_map.UserRPC object at 0x7fb46e75c690>: (<bound method Future._on_rpc_completion of <Future 7fb46e7c8950 created by ru
n_queue(context.py:185) for tasklet _memcache_set_tasklet(context.py:1111); pending>>, (<google.appengine.api.apiproxy_stub_m
ap.UserRPC object at 0x7fb46e75c690>, '', <google.appengine.datastore.datastore_rpc.Connection object at 0x7fb46e7c8ad0>, <ge
nerator object _memcache_set_tasklet at 0x7fb46e6f3640>), {})}
# monkey patched already
try:
import dev_appserver
dev_appserver.fix_sys_path()
except ImportError:
print('Please make sure the App Engine SDK is in your PYTHONPATH.')
raise
email = '----SERVICE_ACCOUNT_EMAIL-------'
from google.appengine.ext.remote_api import remote_api_stub remote_api_stub.ConfigureRemoteApiForOAuth(
'{}.appspot.com'.format("XXXPROJECT_IDXXX"),
'/_ah/remote_api' ,
service_account=email,
key_file_path='KEY---1927925e9abc.p12')
import config
from google.appengine.ext import ndb
from lru_cache import LRUCache
It happens occasionally , with datastore write operations , put and delete.
Any pointers to understand/fix this appreciated.

Python Whoosh AttributeError when updating index

I'm trying to update my whoosh search index when I make changes to my database, but keep getting an error that I haven't been able to figure out.
One of my views:
from search import update_index
#view_config(route_name='update_effect_brief', permission='edit')
def update_effect_brief(request):
name = request.matchdict['pagename']
brief = request.params['effect_brief']
effect = Effect.get_by_name(name) # This is an "effect" object
effect.update_brief(brief)
update_index(effect)
return HTTPFound(location=request.route_url('view_effect', pagename=name))
My search.py file:
from whoosh.index import create_in, open_dir
def update_index(obj):
'''Update single ingredient, product or effect.'''
index = open_dir('searchindex') # searchindex is the name of the directory
with index.searcher as searcher: # crashes on this line!
writer = index.writer()
update_doc(writer, obj)
Traceback:
Traceback (most recent call last):
File "/home/home/SkinResearch/env/local/lib/python2.7/site-packages/pyramid_debugtoolbar-1.0.9-py2.7.egg/pyramid_debugtoolbar/toolbar.py", line 152, in toolbar_tween
response = _handler(request)
File "/home/home/SkinResearch/env/local/lib/python2.7/site-packages/pyramid_debugtoolbar-1.0.9-py2.7.egg/pyramid_debugtoolbar/panels/performance.py", line 55, in resource_timer_handler
result = handler(request)
File "/home/home/SkinResearch/env/local/lib/python2.7/site-packages/pyramid-1.4.5-py2.7.egg/pyramid/tweens.py", line 21, in excview_tween
response = handler(request)
File "/home/home/SkinResearch/env/local/lib/python2.7/site-packages/pyramid_tm-0.7-py2.7.egg/pyramid_tm/__init__.py", line 82, in tm_tween
reraise(*exc_info)
File "/home/home/SkinResearch/env/local/lib/python2.7/site-packages/pyramid_tm-0.7-py2.7.egg/pyramid_tm/__init__.py", line 63, in tm_tween
response = handler(request)
File "/home/home/SkinResearch/env/local/lib/python2.7/site-packages/pyramid-1.4.5-py2.7.egg/pyramid/router.py", line 161, in handle_request
response = view_callable(context, request)
File "/home/home/SkinResearch/env/local/lib/python2.7/site-packages/pyramid-1.4.5-py2.7.egg/pyramid/config/views.py", line 237, in _secured_view
return view(context, request)
File "/home/home/SkinResearch/env/local/lib/python2.7/site-packages/pyramid-1.4.5-py2.7.egg/pyramid/config/views.py", line 377, in viewresult_to_response
result = view(context, request)
File "/home/home/SkinResearch/env/local/lib/python2.7/site-packages/pyramid-1.4.5-py2.7.egg/pyramid/config/views.py", line 493, in _requestonly_view
response = view(request)
File "/home/home/SkinResearch/skinresearch/skinproject/views.py", line 544, in update_effect_brief
update_index(effect)
File "/home/home/SkinResearch/skinresearch/skinproject/search.py", line 37, in start_update
update_index(obj)
File "/home/home/SkinResearch/skinresearch/skinproject/search.py", line 92, in update_index
with index.searcher as searcher:
AttributeError: __exit__
What am I doing wrong?

You need to call the index.searcher() method to create the context manager:
with index.searcher() as searcher:
See the Searcher object section in the Whoosh quickstart, and The Searcher object documentation.
It isn't entirely clear to me why you are creating a searcher, but then create a writer in the block and update the index. Perhaps you wanted to use the writer as a context manager instead here:
with index.writer() as writer:
update_doc(writer, obj)
and leave the searcher for searching operations.

Running into xml invalid attribute name for NETCONF python library

I'm using a high level python library ncclient to edit the configuration of a NETCONF device but I run into this error:
ValueError: Invalid attribute name u'xmlns:if'
I suspect it has something to do with an xml namespace problem since the lxml library is complaining about an attribute name
All I'm doing is creating a connection to the device and then closing it
manager = ncclient.manager.connect(
host=host,
port=port,
username=username,
password=b64decode(password),
device_params={
"name": "nexus",
"ssh_subsystem_name": "xmlagent"
}
)
manager.close_session()
Here's a stack trace:
Traceback (most recent call last):
File "./switch_config.py", line 41, in <module>
main()
File "./switch_config.py", line 26, in main
manager.close_session()
File "/usr/lib/python2.6/site-packages/ncclient/manager.py", line 107, in wrapper
return self.execute(op_cls, *args, **kwds)
File "/usr/lib/python2.6/site-packages/ncclient/manager.py", line 174, in execute
raise_mode=self._raise_mode).request(*args, **kwds)
File "/usr/lib/python2.6/site-packages/ncclient/operations/session.py", line 28, in request
return self._request(new_ele("close-session"))
File "/usr/lib/python2.6/site-packages/ncclient/operations/rpc.py", line 290, in _request
req = self._wrap(op)
File "/usr/lib/python2.6/site-packages/ncclient/operations/rpc.py", line 275, in _wrap
**self._device_handler.get_xml_extra_prefix_kwargs())
File "/usr/lib/python2.6/site-packages/ncclient/xml_.py", line 153, in <lambda>
new_ele = lambda tag, attrs={}, **extra: etree.Element(qualify(tag), attrs, **extra)
File "lxml.etree.pyx", line 2812, in lxml.etree.Element (src/lxml/lxml.etree.c:61433)
File "apihelpers.pxi", line 123, in lxml.etree._makeElement (src/lxml/lxml.etree.c:13864)
File "apihelpers.pxi", line 111, in lxml.etree._makeElement (src/lxml/lxml.etree.c:13736)
File "apihelpers.pxi", line 263, in lxml.etree._initNodeAttributes (src/lxml/lxml.etree.c:15391)
File "apihelpers.pxi", line 1524, in lxml.etree._attributeValidOrRaise (src/lxml/lxml.etree.c:26886)
ValueError: Invalid attribute name u'xmlns:if'

I eventually got it work on NX-OS by:
Remove all the namespaces in ncclient/devices/nexus.py and add namespace "xmlns":"http://www.cisco.com/nxos:1.0:netconf".
def get_xml_base_namespace_dict(self):
return { "xmlns":"http://www.cisco.com/nxos:1.0:netconf" } #Add root namespace
def get_xml_extra_prefix_kwargs(self):
d = {
# "xmlns:nxos":"http://www.cisco.com/nxos:1.0", #remove other namespaces
# "xmlns:if":"http://www.cisco.com/nxos:1.0:if_manager"
}
d.update(self.get_xml_base_namespace_dict())
return d