Develop Reference

python remoteconfig unable to parse file from Gitlab

I am trying to get remoteconfig working, following this guide:
https://pypi.org/project/remoteconfig/
As a control, I have this code that works:
config.read('./config.ini')
for section in config:
print(section)
When I put the same config file in a remote Gitlab, this code does not work:
from remoteconfig import config
config.read('https://myorg.org/path/repo/~/blob/app/config.ini')
for section in config:
print(section)
What could I be doing wrong here? The error msg I am getting is:
configParser.MissingSectionHeaderError: File contains no section headers
So it seems like it's reaching the file path (network/connectivity OK), but not liking what's in that file or possibly the file format? The same exact file works with localconfig.

For now I am going to use the 'gitlab' pip module and simply consume the API for the file (with private_token:
f = project.files.get(file_path='path/file', ref='master'

jupyter notebook: NameError: name 'c' is not defined

Every time that i try to launch my notebook im getting the error below .
let's specify that im new worker on the project and the file config.py was created before that i joined the team.
Does anyone knows how to resolve it please?
The code actually done is
Requirements.txt
psycopg2==2.7.3.2.
SQLAlchemy==1.2.2
pandas==0.21.0
docker==3.3.0
python-json-logger
sshtunnel==0.1.4
jupyter
jupytext==1.2
geopy==2.2.0
errror detail
~/SG/notebooks/config.py in <module>
1 # Using jupytext
----> 2 c.NotebookApp.contents_manager_class = "jupytext.TextFileContentsManager"
3 c.ContentsManager.default_jupytext_formats = "ipynb,py"
NameError: name 'c' is not defined
code
the row causing the error in the notebook is
from src.util.connect_postgres import postgres_connexion
the content of the file connect_postgres
from sqlalchemy import create_engine
from config.util.database import TARGET_TEST_HOST, TARGET_PROD_HOST, \
TARGET_TEST_DB, TARGET_PROD_DB, TARGET_TEST_USER, TARGET_PROD_USER, SG_PROD_USER, SG_PROD_HOST
from config.secrets.passwords import TARGET_PROD_PWD, TARGET_TEST_PWD, SG_PROD_PWD
from sshtunnel import SSHTunnelForwarder
import psycopg2
def _create_engine_psg(user, db, host, port, pwd):
""" Returns a connection object to PostgreSQL """
url = build_postgres_url(db, host, port, pwd, user)
return create_engine(url, client_encoding='utf8')
def build_postgres_url(db, host, port, pwd, user):
url = 'postgresql://{}:{}#{}:{}/{}'.format(user, pwd, host, port, db)
return url
def postgres_connexion(env):
if env == 'prod':
return create_engine_psg_with_tunnel_ssh(TARGET_PROD_DB,
TARGET_PROD_USER, TARGET_PROD_PWD, SG_PROD_PWD,
SG_PROD_USER,
SG_PROD_HOST, TARGET_PROD_HOST)
else:
raise ValueError("'env' parameter must be 'prod'.")
config.py
c.NotebookApp.contents_manager_class = "jupytext.TextFileContentsManager"
c.ContentsManager.default_jupytext_formats = "ipynb,py"
I red that i can generate the file and then edit it.
when i tried to create the jupyter_notebook_config it is always in my personal directory of marczhr
/Users/marczhr/.jupyter/jupyter_notebook_config.py
but i want to be done in my something that i can push on git.
Hope that im clear ^^
Thank you,

Don't run the notebook from the directory with the configuration file.
The reason is that there is an import with a config module or package in the code listed. By launching the notebook from the directory with the configuration file, it will import that Jupyter configuration file, instead of the correct package or module, with the resulting error.
Instead, run it from somewhere else, or put the configuration file elsewhere.
Or perhaps best, take the two configuration lines and add them to the end of your /Users/marczhr/.jupyter/jupyter_notebook_config.py file, then remove the 2-3 line config.py file.
In the latter case, you can now launch the notebook server from anywhere, and you don't need to specify any configuration file, since Jupyter will automatically use the generated (with added lines) one.
If you want to keep the config.py file, then launch the Jupyter notebook server from another directory, and simply specify the full path, like
jupyter --config=$HOME/SG/notebooks/config.py
All in all, this is a classic nameclash upon import, because of identically named files/directories. Always be wary of that.
(I've commented on some other potential problems in the comments: that still stands, but is irrelevant to the current problem here.)

Crawling SSL site with scrapy

I've to crawl https://dms.psc.sc.gov/Web/dockets which uses TLS v1.2 using scrapy framework. But in requesting the URL it fails to load and raise [<twisted.python.failure.Failure <class 'OpenSSL.SSL.Error'>>].
There is issue discussed on git https://github.com/scrapy/scrapy/issues/981 but it did not work for me. I have scrapy v 0.24.5 and twisted version >=14.
When I try to crawl another site which also uses TLS v1.2 it works but not for the https://dms.psc.sc.gov.
How to solve this issue?

PR fixing this problem in Scrapy was already merged. Recently (in February 2016) there was another pull request fixing similar bug
I see with most recent Scrapy version I can fetch your page all right, but with older versions problem still appears.
In general, if you stumble on HTTP-s problem with Scrapy the solution is:
upgrade Scrapy to newest version
check what version of Twisted you use, if it's not most recent update to most recent Twisted version (as of time of writing versions above 14 are confirmed to be significantly better when it comes to SSL)
If you still experience problems after updating Scrapy and Twisted you may need to subclass ScrapyClientContextFactory - see answer below for details.
More details in this github issue

1.addDOWNLOADER_CLIENTCONTEXTFACTORY='testproject.CustomContext.CustomClientContextFactory'
to your settings.py
2.create file called CustomContext.py in your project directory
and add the below code
from OpenSSL import SSL
from twisted.internet.ssl import ClientContextFactory
from twisted.internet._sslverify import ClientTLSOptions
from scrapy.core.downloader.contextfactory import ScrapyClientContextFactory
class CustomClientContextFactory(ScrapyClientContextFactory):
def getContext(self, hostname=None, port=None):
ctx = ClientContextFactory.getContext(self)
# Enable all workarounds to SSL bugs as documented by
# http://www.openssl.org/docs/ssl/SSL_CTX_set_options.html
ctx.set_options(SSL.OP_ALL)
if hostname:
ClientTLSOptions(hostname, ctx)
return ctx
Note: It worked well for crawling with https sites in windows but when I tried the same in Ubuntu 14.04 it throws error as below :-
from twisted.internet._sslverify import ClientTLSOptions
exceptions.ImportError: cannot import name ClientTLSOptions
It would be great if anyone adds solution for the above error.
EDIT:
instead of using from twisted.internet._sslverify import ClientTLSOptions
I have changed it to the below
try:
# available since twisted 14.0
from twisted.internet._sslverify import ClientTLSOptions
except ImportError:
ClientTLSOptions = None

Anyone having "TypeError: unbound method getContext() must be called with ClientContextFactory instance as first argument ..."
Replace ctx = ClientContextFactory.getContext(self)
with ctx = ScrapyClientContextFactory.getContext(self)

Vinodh Velumayil' answer is right. But I had to edit this string:
ctx = ClientContextFactory.getContext(self)
to this:
inst = ClientContextFactory()
ctx = inst.getContext()

Two different python paths in showing up Django

I see two different python paths in django.
This one shows up with the development server(i.e. python manage.py runserver etc.):
['/var/www/html/django/congressticketing',
'/usr/lib/python2.4/site-packages/MySQL_python-1.2.3c1-py2.4-linux-i686.egg', '/usr/lib/python2.4/site-packages/pisa-3.0.32-py2.4.egg',
'/usr/lib/python2.4/site-packages/Whoosh-0.3.9-py2.4.egg',
'/usr/lib/python2.4/site-packages/html5lib-0.11-py2.4.egg',
'/usr/lib/python2.4/site-packages/multiprocessing-2.6.2.1-py2.4-linux-i686.egg','/usr/lib/python2.4/site-packages/anyjson-0.2.4-py2.4.egg',
'/usr/lib/python2.4/site-packages/SQLAlchemy-0.6.0-py2.4.egg',
'/usr/lib/python2.4/site-packages/mailer-0.5-py2.4.egg',
'/usr/lib/python2.4/site-packages/python_dateutil-1.5-py2.4.egg',
'/usr/lib/python2.4/site-packages/simplejson-2.1.1-py2.4-linux-i686.egg', '/usr/lib/python2.4/site-packages/dottedish-0.6-py2.4.egg',
'/usr/lib/python2.4/site-packages/simplegeneric-0.6-py2.4.egg',
'/usr/lib/python2.4/site-packages/supervisor-3.0a8-py2.4.egg',
'/usr/lib/python2.4/site-packages/elementtree-1.2.7_20070827_preview-py2.4.egg', '/usr/lib/python2.4/site-packages/meld3-0.6.6-py2.4.egg',
'/usr/lib/python2.4/site-packages/billiard-0.3.1-py2.4.egg',
'/usr/lib/python2.4/site-packages/pycrypto-2.2-py2.4-linux-i686.egg', '/usr/lib/python2.4/site-packages/Trac-0.12-py2.4.egg',
'/usr/lib/python2.4/site-packages/Genshi-0.6-py2.4.egg',
'/usr/lib/python2.4/site-packages/django_thumbnail_works-0.2.0-py2.4.egg', '/usr/lib/python2.4/site-packages/cropresize-0.1.4-py2.4.egg',
'/usr/lib/python2.4/site-packages/uuid-1.30-py2.4.egg',
'/usr/lib/python2.4/site-packages/pyparsing-1.5.5-py2.4.egg',
'/usr/lib/python2.4/site-packages/carrot-0.10.7-py2.4.egg',
'/usr/lib/python2.4/site-packages/django_compress-1.0.1-py2.4.egg', '/usr/lib/python2.4/site-packages/BeautifulSoup-3.2.0-py2.4.egg',
'/usr/lib/python2.4/site-packages/lxml-2.3beta1-py2.4-linux-i686.egg', '/usr/lib/python2.4/site-packages/django_celery-2.1.4-py2.4.egg',
'/usr/lib/python2.4/site-packages/sorl_thumbnail-10.12-py2.4.egg', '/usr/lib/python2.4/site-packages',
'/usr/lib/python24.zip',
'/usr/lib/python2.4',
'/usr/lib/python2.4/plat-linux2',
'/usr/lib/python2.4/lib-tk',
'/usr/lib/python2.4/lib-dynload',
'/usr/lib/python2.4/site-packages/Numeric',
'/usr/lib/python2.4/site-packages/PIL',
'/usr/lib/python2.4/site-packages/gtk-2.0']
This one - much shorter - shows up in production(using mod_python):
['/var/www/html/django/congressticketing',
'/usr/lib/python2.4/site-packages/MySQL_python-1.2.3c1-py2.4-linux-i686.egg', '/usr/lib/python2.4/site-packages/pisa-3.0.32-py2.4.egg',
'/usr/lib/python2.4/site-packages/Whoosh-0.3.9-py2.4.egg',
'/usr/lib/python2.4/site-packages/html5lib-0.11-py2.4.egg',
'/usr/lib/python2.4/site-packages/multiprocessing-2.6.2.1-py2.4-linux-i686.egg', '/usr/lib/python2.4/site-packages/anyjson-0.2.4-py2.4.egg',
'/usr/lib/python2.4/site-packages/SQLAlchemy-0.6.0-py2.4.egg',
'/usr/lib/python2.4/site-packages/python_dateutil-1.5-py2.4.egg',
'/usr/lib/python2.4/site-packages/simplejson-2.1.1-py2.4-linux-i686.egg', '/usr/lib/python2.4/site-packages/dottedish-0.6-py2.4.egg',
'/usr/lib/python2.4/site-packages/supervisor-3.0a8-py2.4.egg',
'/usr/lib/python2.4/site-packages/billiard-0.3.1-py2.4.egg',
'/usr/lib/python2.4/site-packages/Trac-0.12-py2.4.egg',
'/usr/lib/python2.4/site-packages/django_thumbnail_works-0.2.0-py2.4.egg', '/usr/lib/python2.4/site-packages/cropresize-0.1.4-py2.4.egg',
'/usr/lib/python2.4/site-packages/pyparsing-1.5.5-py2.4.egg',
'/usr/lib/python2.4/site-packages/carrot-0.10.7-py2.4.egg',
'/usr/lib/python2.4/site-packages/BeautifulSoup-3.2.0-py2.4.egg',
'/usr/lib/python2.4/site-packages/lxml-2.3beta1-py2.4-linux-i686.egg', '/usr/lib/python2.4/site-packages/django_celery-2.1.4-py2.4.egg',
'/usr/lib/python2.4/site-packages/sorl_thumbnail-10.12-py2.4.egg', '/usr/lib/python2.4/site-packages',
'/usr/lib/python24.zip',
'/usr/lib/python2.4',
'/usr/lib/python2.4/plat-linux2',
'/usr/lib/python2.4/lib-tk',
'/usr/lib/python2.4/lib-dynload',
'/usr/lib/python2.4/site-packages/Numeric',
'/usr/lib/python2.4/site-packages/PIL',
'/usr/lib/python2.4/site-packages/gtk-2.0']
Does anyone have any idea why this is?

It could be a permissions problem. Check the file and directory permissions on the missing site-packages. It could be that those files are not accessible to the user name under which mod_python or Apache is running.

Default route doesn't work

I'm using the standard routing module with pylons to try and setup a default route for the home page of my website.
I've followed the instructions in the docs and here http://routes.groovie.org/recipes.html but when I try http://127.0.0.1:5000/ I just get the 'Welcome to Pylons' default page.
My config/routing.py file looks like this
from pylons import config
from routes import Mapper
def make_map():
"""Create, configure and return the routes Mapper"""
map = Mapper(directory=config['pylons.paths']['controllers'],
always_scan=config['debug'])
map.minimization = False
map.connect('/error/{action}', controller='error')
map.connect('/error/{action}/{id}', controller='error')
# CUSTOM ROUTES HERE
map.connect( '', controller='main', action='index' )
map.connect('/{controller}/{action}')
map.connect('/{controller}/{action}/{id}')
return map
I've also tried
map.connect( '/', controller='main', action='index' )
and (using http://127.0.0.1:5000/homepage/)
map.connect( 'homepage', controller='main', action='index' )
But nothing works at all. I know its reloading my config file as I used
paster serve --reload development.ini
to start the server
system info
$ paster --version
PasteScript 1.7.3 from /Library/Python/2.5/site-packages/PasteScript-1.7.3-py2.5.egg (python 2.5.1 (r251:54863, Feb 6 2009, 19:02:12))

You have to delete the static page (myapp/public/index.html). Static
files take priority due to the Cascade configuration at the end of
middleware.py.