I am trying to get remoteconfig working, following this guide:
https://pypi.org/project/remoteconfig/
As a control, I have this code that works:
config.read('./config.ini')
for section in config:
print(section)
When I put the same config file in a remote Gitlab, this code does not work:
from remoteconfig import config
config.read('https://myorg.org/path/repo/~/blob/app/config.ini')
for section in config:
print(section)
What could I be doing wrong here? The error msg I am getting is:
configParser.MissingSectionHeaderError: File contains no section headers
So it seems like it's reaching the file path (network/connectivity OK), but not liking what's in that file or possibly the file format? The same exact file works with localconfig.
For now I am going to use the 'gitlab' pip module and simply consume the API for the file (with private_token:
f = project.files.get(file_path='path/file', ref='master'
Related
Every time that i try to launch my notebook im getting the error below .
let's specify that im new worker on the project and the file config.py was created before that i joined the team.
Does anyone knows how to resolve it please?
The code actually done is
Requirements.txt
psycopg2==2.7.3.2.
SQLAlchemy==1.2.2
pandas==0.21.0
docker==3.3.0
python-json-logger
sshtunnel==0.1.4
jupyter
jupytext==1.2
geopy==2.2.0
errror detail
~/SG/notebooks/config.py in <module>
1 # Using jupytext
----> 2 c.NotebookApp.contents_manager_class = "jupytext.TextFileContentsManager"
3 c.ContentsManager.default_jupytext_formats = "ipynb,py"
NameError: name 'c' is not defined
code
the row causing the error in the notebook is
from src.util.connect_postgres import postgres_connexion
the content of the file connect_postgres
from sqlalchemy import create_engine
from config.util.database import TARGET_TEST_HOST, TARGET_PROD_HOST, \
TARGET_TEST_DB, TARGET_PROD_DB, TARGET_TEST_USER, TARGET_PROD_USER, SG_PROD_USER, SG_PROD_HOST
from config.secrets.passwords import TARGET_PROD_PWD, TARGET_TEST_PWD, SG_PROD_PWD
from sshtunnel import SSHTunnelForwarder
import psycopg2
def _create_engine_psg(user, db, host, port, pwd):
""" Returns a connection object to PostgreSQL """
url = build_postgres_url(db, host, port, pwd, user)
return create_engine(url, client_encoding='utf8')
def build_postgres_url(db, host, port, pwd, user):
url = 'postgresql://{}:{}#{}:{}/{}'.format(user, pwd, host, port, db)
return url
def postgres_connexion(env):
if env == 'prod':
return create_engine_psg_with_tunnel_ssh(TARGET_PROD_DB,
TARGET_PROD_USER, TARGET_PROD_PWD, SG_PROD_PWD,
SG_PROD_USER,
SG_PROD_HOST, TARGET_PROD_HOST)
else:
raise ValueError("'env' parameter must be 'prod'.")
config.py
c.NotebookApp.contents_manager_class = "jupytext.TextFileContentsManager"
c.ContentsManager.default_jupytext_formats = "ipynb,py"
I red that i can generate the file and then edit it.
when i tried to create the jupyter_notebook_config it is always in my personal directory of marczhr
/Users/marczhr/.jupyter/jupyter_notebook_config.py
but i want to be done in my something that i can push on git.
Hope that im clear ^^
Thank you,
Don't run the notebook from the directory with the configuration file.
The reason is that there is an import with a config module or package in the code listed. By launching the notebook from the directory with the configuration file, it will import that Jupyter configuration file, instead of the correct package or module, with the resulting error.
Instead, run it from somewhere else, or put the configuration file elsewhere.
Or perhaps best, take the two configuration lines and add them to the end of your /Users/marczhr/.jupyter/jupyter_notebook_config.py file, then remove the 2-3 line config.py file.
In the latter case, you can now launch the notebook server from anywhere, and you don't need to specify any configuration file, since Jupyter will automatically use the generated (with added lines) one.
If you want to keep the config.py file, then launch the Jupyter notebook server from another directory, and simply specify the full path, like
jupyter --config=$HOME/SG/notebooks/config.py
All in all, this is a classic nameclash upon import, because of identically named files/directories. Always be wary of that.
(I've commented on some other potential problems in the comments: that still stands, but is irrelevant to the current problem here.)
Can anyone tell me what I did wrong? I am using python-conda, and the files I have from http://meop40.troja.mff.cuni.cz:11180/gw.projekt/data.stratopauza/netcdf.profily/
Why it tells me that file doesn't exist?
>>> import netCDF4
>>> import pandas as pd
>>> import matplotlib.pyplot as plt
>>> url = 'http://meop40.troja.mff.cuni.cz:11180/gw.projekt/data.stratopauza/netcdf.profily/atmPrf_C001.2010.227.00.03.G04_2013.3520_nc'
>>> nc = netCDF4.Dataset(url)
**syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: <!DOCTYPE^ HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"><html><head><title>404 Not Found</title></head><body><h1>Not Found</h1><p>The requested URL /gw.projekt/data.stratopauza/netcdf.profily/atmPrf_C001.2010.227.00.03.G04_2013.3520_nc.dds was not found on this server.</p><hr><address>Apache/2.4.12 (Ubuntu) Server at meop40.troja.mff.cuni.cz Port 11180</address></body></html>
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "netCDF4\_netCDF4.pyx", line 1811, in netCDF4._netCDF4.Dataset.__init__ (netCDF4\_netCDF4.c:12626)
IOError: NetCDF: file not found**
NetCDF4.Dataset() can only access remote NetCDF files which are served by an OPeNDAP service, which can return metadata about the file. The error message returned is incorrect and misleading.
There is a brief tutorial, which mentions this and gives basic information at: http://unidata.github.io/netcdf4-python/#section1
I downloaded the file and had no problem opening the file. You should use the method in the answer to your previous question https://stackoverflow.com/a/44622713/1211981
Update:
Go to:
http://meop40.troja.mff.cuni.cz:11180/gw.projekt/data.stratopauza/netcdf.profily/
Click one or more of the links and save to a folder where you will run your script. Change your script or python commands to:
>>> url = 'atmPrf_C001.2010.227.00.03.G04_2013.3520_nc'
>>> nc = netCDF4.Dataset(url)
netCDF4.Dataset() will take either a url or a local file name and work the same way. In this case it will recognize the file as a NetCDF / OPeNDAP compatible.
I'm getting the following error:
File "/Users/tai/Desktop/FlashY/flashy/sniffer/database.py", line 21, in <module>
import dynamoStorage
File "/Users/tai/Desktop/FlashY/flashy/sniffer/dynamoStorage.py", line 37, in <module>
swfTable = Table(decompiled_dynamo_table, connection=dynamoConn)
File "/Library/Python/2.7/site-packages/boto/dynamodb2/table.py", line 107, in __init__
self.connection = DynamoDBConnection()
File "/Library/Python/2.7/site-packages/boto/dynamodb2/layer1.py", line 183, in __init__
super(DynamoDBConnection, self).__init__(**kwargs)
File "/Library/Python/2.7/site-packages/boto/connection.py", line 1073, in __init__
profile_name=profile_name)
File "/Library/Python/2.7/site-packages/boto/connection.py", line 572, in __init__
host, config, self.provider, self._required_auth_capability())
File "/Library/Python/2.7/site-packages/boto/auth.py", line 883, in get_auth_handler
'Check your credentials' % (len(names), str(names)))
boto.exception.NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. ['HmacAuthV4Handler'] Check your credentials
When I had the auth directly in the file my keys worked so I know the keys are correct.
I have for awsAccess.py:
#aswAccess holds the names of the bash environment set keys.
#used by other classes to create a connection to aws
aws_access_key_id=os.getenv('AWS_ACCESS_KEY');
aws_secret_access_key=os.getenv('AWS_SECRET_KEY');
aws_dynamo_region=os.getenv('DYANAMO_REGION')
and I have for database.py
#for connecting to aws
aws_access_key_id=awsAccess.aws_access_key_id
aws_secret_access_key=awsAccess.aws_secret_access_key
aws_dynamo_region=awsAccess.aws_dynamo_region
aws_dynamo_table="decompiled_swf_text"
conn= S3Connection(aws_access_key_id,aws_secret_access_key);
dynamoConn = boto.connect_dynamodb(aws_access_key_id, aws_secret_access_key)
dTable = dynamoConn.get_table(aws_dynamo_table)
So I know the keys themselves are correct.
I have a .bash_profile file that looks like this (**indicating removed, also I tried with and without ""'s):
export AWS_ACCESS_KEY="myAccessKey**"
export AWS_SECRET_KEY="mySecretKey**"
export DYNAMO_REGION="us-east"
I run source ~/.bash_profile, and then tried running but get the error. I can't see why importing would alter the impact of the same key string.
Few tips:
assert in your code, that the values for ID aws_access_key and aws_secret_access_key are not empty. It is likely, you do not have them set at the place you think you have.
remove authentication arguments from boto.connect_<something>. This will let boto to use built in authentication handlers and these shall try reading the file, checking environmental variables etc. You will have your code simpler while still being able to provide all what is needed by any of boto authentication methods.
my favourit authentication method is to use ini file (usually named boto.cfg) and having BOTO_CONFIG environmental variable set to full path to this file, e.g. to `"/home/javl/.boto/boto.cfg"
note, that if you pass any of the authentication parameters to boto.connect_<something> with value null, it will work as boto will check other methods to find the value.
since about a year ago, there is an option profile, allowing to refer to specific profile in boto config file. This let me switching to different profiles anywhere in the code.
For more tips and details, see related SO answer
I had Problem with Ubuntu Wily (15.10).
Here there is an update for ConfigParser.
.get has a additional Parameter, but that are not supported with boto.pyami.config.
see here and here
I'm trying to download an MP3 file, via its URL, using Python's urllib2.
mp3file = urllib2.urlopen(url)
output = open(dst,'wb')
output.write(mp3file.read())
output.close()
I'm getting a urllib2.HTTPError: HTTP Error 403: Forbidden error.
Trying urllib also fails, but silently.
urllib.urlretrieve(url, dst)
However, if I use wget, I can download the file successfully.
I've noted the general differences between the two methods mentioned in "Difference between Python urllib.urlretrieve() and wget", but they don't seem to apply here.
Is wget doing something to negotiate permissions that urllib2 doesn't do? If so, what, and how do I replicate this in urllib2?
Could be something on the server side - blocking python user agent for example. Try using wget user agent : Wget/1.13.4 (linux-gnu) .
In Python 2:
import urllib
# Change header for User-Agent
class AppURLopener(urllib.FancyURLopener):
version = "Wget/1.13.4 (linux-gnu)"
url = "http://www.example.com/test_file"
fname = "test_file"
urllib._urlopener = AppURLopener()
urllib.urlretrieve(url, fname)
The above didn't work for me (I'm using python3.5). wget works fine.
It's not (I assume) a huge problem for me - surely I can still do a system() and use wget to get the data, with some file renaming and munging.
But in case anyone else is suffering from the same problem, these are the errors I get from the above snippet:
Traceback (most recent call last):
File "./mksynt.py", line 10, in <module>
class AppURLopener(urllib.FancyURLopener):
AttributeError: module 'urllib' has no attribute 'FancyURLopener'
I see that the original answer was only promised to work in python2.
I have gitzilla config file setup at /ect/gitzillarc on remote central repository server with permissions all read and write.
Content of the file code config is as follows
[/home/gituser/repositories/git-main/git-main.git/.git]
bugzilla_url: https://repo.example.com/bugzilla/
bugzilla_user: sboppana#example.com
bugzilla_password: s123
user_config: deny
allowed_bug_states: NEW, ASSIGNED, REOPENED
logfile: /var/log/gitzilla.log
loglevel: info`
python at 2.6.5
pybugz at 0.9.3 (tried with 0.8.0 also)
Gitzilla at gera-gitzilla-gitzilla-2.0-19-geceeaca.tar.gz
I get the error "remote: missing/incomplete bugzilla conf (no bugzilla_url)" with git push
Of course bugzilla_url value has the real name in my config file not the example name.
Tried many but couldn't get it to work. Thanks for all the help.
Adding xmlrpc.cgi along with the bugzilla URL should solve this issue. This is the workaround for the problem in Pybugz.
For instance, if your bugzilla URL is https://repo.example.com/bugzilla/ try using https://repo.example.com/bugzilla/xmlrpc.cgi This should work.