Develop Reference

osquery-python extension causing osqueryi errors

I have a newbie question about creating osquery extensions using osquery-python. I Created a small extension that gets some additional RPM info from my linux system. Following the instructions in the docs, I added the path to the extension in /etc/osquery/extensions.load to get it to autoload. I restarted osqueryd and I see the extension running using ps ax.
If I interactively run osqueryi, I can see the table and get data. It all works perfectly.
However, when I run an osqueryi command 'one-liner' such as :
osqueryi .tables
I get a bunch of the following errors with my output:
#INFO:thrift.transport.TSocket:Could not connect to /root/.osquery/shell.em
Traceback (most recent call last):
File "build/bdist.linux-x86_64/egg/thrift/transport/TSocket.py", line 104, in open
handle.connect(sockaddr)
File "/usr/lib64/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
error: [Errno 2] No such file or directory
ERROR:thrift.transport.TSocket:Could not connect to any of ['/root/.osquery/shell.em']
What have I done wrong?

Extensions are run in a separate process. You can see the socket errors, which indicate the extension process cannot communicate with osquery process. Make sure osqueryd or osqueryi is running. Link: osquery doc page for extensions.

Sphinx: autodoc fails for certain modules due to hardware-specific libraries raising exceptions on the dev machine

I'm working in a project involving the Raspberry Pi and the PiFace interface module. The project is being developed in Python and I have run into a problem when trying to auto document some modules with Sphinx. We are using the pifacedigitalio library, however, even though I have installed said library on the development machine, Sphinx refuses to document the module and complains that it's throwing exceptions due to hardware not being present. The hardware isn't present because this is the development machine. For testing the software, we just use a conditional if hwpresent then skip but I actually want to document those parts through Sphinx. The error produced when doing make html is:
/home/user/current/working/docs/src.rst:10: WARNING: autodoc: failed to import module u'src.billete'; the following exception was raised:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/sphinx/ext/autodoc.py", line 335, in import_object
__import__(self.modname)
File "/home/user/current/working/src/bill.py", line 4, in <module>
piface_obj = piface.PiFaceDigital()
File "/usr/local/lib/python2.7/dist-packages/pifacedigitalio/core.py", line 82, in __init__
self.init_board()
File "/usr/local/lib/python2.7/dist-packages/pifacedigitalio/core.py", line 107, in init_board
h=self.hardware_addr, b=self.bus, c=self.chip_select))
NoPiFaceDigitalDetectedError: No PiFace Digital board detected (hardware_addr=0, bus=0, chip_select=0).
Is there a way to avoid Sphinx from interpreting the code and just parse the source? I tried to modify pifacedigitalio/core.py by hand using dummy returns but Sphinx just freezes then.

You have to put a directory in your PYTHONPATH that has Mock's of those modules before you run Sphinx. Check out their Doc's, but be warned that you shouldn't be screwing around with sys.modules yourself like they do in the example. Use mock.patch instead.

Add sqlalchemy library to google app engine in pycharm

I have following question to sqlalchemy in gae. I'm using the integrated google app engine sdk from Pycharm. Everything works fine, except the integration of sqlalchemy. The database itself works fine if I debug it with the Python interpreter. But when I'm trying to run it together with the google app engine I'm getting troubles with the import of the sqlalchemy library.
I already tried to solve it with virtualenv, but it didn't work either.
Here is my output log:
Connected to pydev debugger (build 131.618)
WARNING 2014-02-17 10:58:22,887 api_server.py:331] Could not initialize images API; you are likely missing the Python "PIL" module.
WARNING 2014-02-17 10:58:22,895 simple_search_stub.py:1018] Could not read search indexes from c:\users\user\appdata\local\temp\appengine.test\search_indexes
INFO 2014-02-17 10:58:22,947 api_server.py:138] Starting API server at: localhost:60878
INFO 2014-02-17 10:58:22,994 dispatcher.py:171] Starting module "default" running at: localhost:8080
INFO 2014-02-17 10:58:23,032 admin_server.py:117] Starting admin server at: localhost:8000
pydev debugger: process 4452 is connecting
pydev debugger: process 4196 is connecting
ERROR 2014-02-17 09:58:35,742 wsgi.py:262]
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line 239, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line 298, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\runtime\wsgi.py", line 84, in LoadObject
obj = import(path[0])
File "C:\Users\user\PycharmProjects\test\main.py", line 16, in
from bin.database.DBList import DBList
File "C:\Users\user\PycharmProjects\test\bin\database\DBList.py", line 17, in
from sqlalchemy import Column, Integer, String
ImportError: No module named sqlalchemy
INFO 2014-02-17 10:58:36,410 module.py:617] default: "GET / HTTP/1.1" 500 -
So how is it possible to add a third party library in gae?

This is to be expected because GAE has no idea what things your virtualenv is using. To mitigate this problem of using a third party library, all you need to do is download the source code of the library and all its dependents, and put it into a folder underneath your root project directory.
Add the folder to your PYTHONPATH, you can do this via:
import sys; sys.path.append('<your relative path to directory here>')
And then you are all set. By convention this directory that you put all your third party libraries under, is called lib.

"Cannot allocate memory" error for psycopg2 and libgcrypt.so on Debian 6, 64bit

I am having a weird production specific error related to Python psycopg2 postgresql driver and libgcrypt module.
I was running Debian 6, 32bit to host my application with uWSGI and Nginx. I decided to switch to 64bit with fresh install. Everything seemed fine except I am getting below Python traceback for every request in the uwsgi.log:
File ".../python2.7/site-packages/django/core/handlers/wsgi.py", line 272, in __call__
response = self.get_response(request)
File ".../python2.7/site-packages/django/core/handlers/base.py", line 169, in get_response
response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
File ".../python2.7/site-packages/django/core/handlers/base.py", line 203, in handle_uncaught_exception
return debug.technical_500_response(request, *exc_info)
File ".../python2.7/site-packages/django/views/debug.py", line 59, in technical_500_response
html = reporter.get_traceback_html()
and so on...
So, I went to django/core/handlers/base.py (Django version 1.3.1) line 166 which seems the last exception handler that holds everything really exceptional. I replaced it with:
except Exception, why:
and write the "why" to a text file. The exception is exactly that:
Error loading psycopg2 module: libgcrypt.so.11: failed to map segment from shared object: Cannot allocate memory
I read almost every similar bug / issue / blogpost and could only find that one that believe not exactly same issue but quite similar:
http://www.johngirvin.com/archives/dovecot-error-127-after-upgrading-to-debian-squeeze.html
I did try with:
Django==1.3.1
Django==1.3
psycopg2==2.4.2
psycopg2==2.4.1
also rebooted which gave no result. I have postgresql-8.4, libgcrypt11 and libgcrypt11-dev installed on the system. PostgreSQL has no model except a fresh ./manage.py syncdb with Django default ones. There is more than half of the memory is just empty.
And... I also tried to run ./manage.py runserver on production which works perfectly fine. The VPS is on Linode, so I don't think there is any hardware problem.
Any help will be appreciated.

It is perhaps not about Python, Django, PostgreSQL, psycopg2 nor libsomething.so.
Check if you have option like --limit-as in your uWSGI configuration. On 64bit systems address space can be twice big. 512 or 1024 should do the job so raise it as necessary.

Sync django-piston models when using egg module

I'm using django-piston and running into the following problem when running the manage.py syncdb command:
Traceback (most recent call last):
File "./manage.py", line 13, in <module>
execute_manager(settings)
File "/home/appfirst/django/django/core/management/__init__.py", line 438, in execute_manager
utility.execute()
............ lines ommitted ...........
File "/home/appfirst/django/django/utils/translation/trans_real.py", line 176, in translation
default_translation = _fetch(settings.LANGUAGE_CODE)
File "/home/appfirst/django/django/utils/translation/trans_real.py", line 160, in _fetch
apppath = os.path.join(os.path.dirname(app.__file__), 'locale')
AttributeError: 'module' object has no attribute '__file__'
In my development environment, I don't have this problem. There I have my python path set to include the django-piston directory. On production, I installed piston by compiling it to an rpm (python setup.py bdist --format=rpm) and installing it as an egg in /usr/lib/python2.6/site-packages/. This means that in development, I can do
>>> import piston
>>> piston.__file__
/some/file/path/here
But on production I get
>>> import piston
>>> piston.__file__
AttributeError: 'module' object has no attribute '__file__'
Does anyone know a way around this?

Looks like this is a known issue with Piston 0.2.3. You could apply this patch or just use version 0.2.2 by installing it as so:
pip install django-piston==0.2.2

I had the same issue, not sure what caused it, but to fix it, I uncommented the piston app out of my INSTALLED_APPS in settings, and it started working again, so it was something to do with loading piston. I'm going to remove piston from my site-packages directory and try to reload it, and see if that helps.
Also, while trying to find the answer to my question I came across a few sites with similar issues.
This site describes an issue related to __init__.py missing.
http://www.willmer.com/kb/2007/12/attributeerror-module-object-has-no-attribute-blah/
This site describes circular imports as an issue.
http://www.answermysearches.com/python-fixing-module-object-has-no-attribute/333/
And another one that I can't find, mentioned something about not being able to load the files correctly from an app installed as an egg.