What's the simplest way to store the application secrets (passwords, access tokens) for a Python script? I thought it'd be a *.yml file like in Ruby but surprisingly I found that it wasn't the case. So what is it then? What are the most simplest solutions?
I want to put them in a separate file because that way I'll be able not to push that file to a github repository.
I think storing credentials inside another *py file is your safest bet. Then just import it. Example would look like this
config.py
username = "xy"
password = "abcd"
main.py
import config
login(config.username, config.password)
I was dealing exactly the same question and actually ended up with the same solution as kecer suggested. Since I need to use it in dozens of scripts, I've created own library. Let me share this solution with you.
credlib.py -- universal library to handle credentials
class credential:
def __init__(self, hostname, username, password):
self.hostname = hostname
self.username = username
self.password = password
mycredentials.py -- my local file to store all credentials
from credlib import credential
sys_prod = credential("srv01", "user", "pass")
sys_stg = credential("srv02", "user", "pass")
sys_db = credential("db01", "userdb", "passdb")
mysystemlib.py -- this is a general library to access my system (both new credential system and legacy is supported)
from credlib import credential
def system_login(*args): # this is new function definition
#def system_login(hostname, username, password): # this was previous function definition
if len(args) == 1 and isinstance(args[0], credential):
hostname = args[0].hostname
username = args[0].username
password = args[0].password
elif len(args) == 3:
hostname = args[0]
username = args[1]
password = args[2]
else:
raise ValueError('Invalid arguments')
do_login(hostname, username, password) # this is original system login call
main.py -- main script that combines credentials and system libs
from mycredentials import sys_stg, sys_db
import mysystemlib
...
mysystemlib.system_login(sys_stg)
Please note that the legacy hostname/username/password way still works so it does not affect old scripts:
mysystemlib.system_login("srv02", "user", "pass")
This has a lot benefits:
same credential system across all our python scripts
files with passwords are separated (files can have more strict permissions)
files are not stored in our git repositories (excluded via .gitignore) so that our python scripts/libs can be shared with others without exposing credentials (everyone defines their own credentials in their local files)
if a password needs to be changed, we do it at a single place only
Personally I prefer to use yaml files, with the pyyaml library.
Documentation here: https://pyyaml.org/wiki/PyYAMLDocumentation
Creating a .gitignore rule is very quick and painless and there is zero chances of making a mistake. You can added the rule with echo on Linux / UNIX like system with:
echo -e '*.yaml\n*.yml' >> .gitignore
Below is an example of retrieving the settings from a settings .yaml file in the same folder / location of the reader.
Code Snippets:
#!/usr/bin/env python3
import yaml
from pathlib import Path
def get_settings():
full_file_path = Path(__file__).parent.joinpath('settings.yaml')
with open(full_file_path) as settings:
settings_data = yaml.load(settings, Loader=yaml.Loader)
return settings_data
Related
Every time that i try to launch my notebook im getting the error below .
let's specify that im new worker on the project and the file config.py was created before that i joined the team.
Does anyone knows how to resolve it please?
The code actually done is
Requirements.txt
psycopg2==2.7.3.2.
SQLAlchemy==1.2.2
pandas==0.21.0
docker==3.3.0
python-json-logger
sshtunnel==0.1.4
jupyter
jupytext==1.2
geopy==2.2.0
errror detail
~/SG/notebooks/config.py in <module>
1 # Using jupytext
----> 2 c.NotebookApp.contents_manager_class = "jupytext.TextFileContentsManager"
3 c.ContentsManager.default_jupytext_formats = "ipynb,py"
NameError: name 'c' is not defined
code
the row causing the error in the notebook is
from src.util.connect_postgres import postgres_connexion
the content of the file connect_postgres
from sqlalchemy import create_engine
from config.util.database import TARGET_TEST_HOST, TARGET_PROD_HOST, \
TARGET_TEST_DB, TARGET_PROD_DB, TARGET_TEST_USER, TARGET_PROD_USER, SG_PROD_USER, SG_PROD_HOST
from config.secrets.passwords import TARGET_PROD_PWD, TARGET_TEST_PWD, SG_PROD_PWD
from sshtunnel import SSHTunnelForwarder
import psycopg2
def _create_engine_psg(user, db, host, port, pwd):
""" Returns a connection object to PostgreSQL """
url = build_postgres_url(db, host, port, pwd, user)
return create_engine(url, client_encoding='utf8')
def build_postgres_url(db, host, port, pwd, user):
url = 'postgresql://{}:{}#{}:{}/{}'.format(user, pwd, host, port, db)
return url
def postgres_connexion(env):
if env == 'prod':
return create_engine_psg_with_tunnel_ssh(TARGET_PROD_DB,
TARGET_PROD_USER, TARGET_PROD_PWD, SG_PROD_PWD,
SG_PROD_USER,
SG_PROD_HOST, TARGET_PROD_HOST)
else:
raise ValueError("'env' parameter must be 'prod'.")
config.py
c.NotebookApp.contents_manager_class = "jupytext.TextFileContentsManager"
c.ContentsManager.default_jupytext_formats = "ipynb,py"
I red that i can generate the file and then edit it.
when i tried to create the jupyter_notebook_config it is always in my personal directory of marczhr
/Users/marczhr/.jupyter/jupyter_notebook_config.py
but i want to be done in my something that i can push on git.
Hope that im clear ^^
Thank you,
Don't run the notebook from the directory with the configuration file.
The reason is that there is an import with a config module or package in the code listed. By launching the notebook from the directory with the configuration file, it will import that Jupyter configuration file, instead of the correct package or module, with the resulting error.
Instead, run it from somewhere else, or put the configuration file elsewhere.
Or perhaps best, take the two configuration lines and add them to the end of your /Users/marczhr/.jupyter/jupyter_notebook_config.py file, then remove the 2-3 line config.py file.
In the latter case, you can now launch the notebook server from anywhere, and you don't need to specify any configuration file, since Jupyter will automatically use the generated (with added lines) one.
If you want to keep the config.py file, then launch the Jupyter notebook server from another directory, and simply specify the full path, like
jupyter --config=$HOME/SG/notebooks/config.py
All in all, this is a classic nameclash upon import, because of identically named files/directories. Always be wary of that.
(I've commented on some other potential problems in the comments: that still stands, but is irrelevant to the current problem here.)
I am working on a script to transfer some files to a Cisco IOS device using netmiko FileTransfer. Below is the code that I found to accomplish this. However I cant seem to find where the source_file should be or how to specify where on the host that files lives. How do I specify where to copy that file from?
dest_file_system = 'disk0:/'
source_file = 'test1.txt' # where should this file be live?
dest_file = 'test1.txt'
with FileTransfer(ssh_conn, source_file=source_file, dest_file=dest_file,
file_system=dest_file_system) as scp_transfer:
if not scp_transfer.check_file_exists():
if not scp_transfer.verify_space_available():
raise ValueError("Insufficient space available on remote device")
print("\nTransferring file\n")
scp_transfer.transfer_file()
The source file from the scripts perspective if you simple want to call it by name should be in the same directory as the script itself. If you want to move the file to a new directory the search path in the script is relative to the directory the script is run from. Example 1 - The file_name.txt is in the same directory as your script In your script simply call the file source = "file_name.txt". Example 2 create test_folder in current directory that holds your script and call it test_folder move the file_name.txt into test_folder. In your script your source variable would no need to look like this source = "test_folder/file_name.txt"
Hopefully, till now you found the solution already. If not then You can try the netmiko file transfer feature also. That will be a more secure and efficient way to do it.
try below sample script.
from getpass import getpass
from netmiko import ConnectHandler, file_transfer
password = getpass()
cisco = {
"device_type": "cisco_ios",
"host": "cisco1.twb-tech.com",
"username": "pyclass",
"password": password,
}
source_file = "test1.txt"
dest_file = "test1.txt"
direction = "put"
file_system = "flash:"
ssh_conn = ConnectHandler(**cisco)
transfer_dict = file_transfer(
ssh_conn,
source_file=source_file,
dest_file=dest_file,
file_system=file_system,
direction=direction,
overwrite_file=True,
)
print(transfer_dict)
Simple enough, off of my last question, I am trying to make a directory change to a players desktop or file that is similar for all, as in C:\\Users\\USERNAME\\Desktop\\Tester File but the how would I make it so that USERNAME is the username of the person's computer? I tried using %USERNAME% but I don't really know how to do that, and it didn't work, and anyway the % gave an error message (I cannot remember the message, I think it was syntax error)
I also tried using ~, but it proved to be ineffective, but it may be due to my lack of experience.
EDIT
I solved this issue, thanks to some very great help from #pstatix so thank you.
By using user = getpass.getuser() I was able to do something like 'C:\Users' + user + '\Documents' it made this all user friendly! Thanks!
Have you tried the getpass module? getpass documentation here.
import getpass
usr = getpass.getuser()
print usr
Edit: For user specified example
You may also be interested in using the os module? os documentation here.
import os
usr = os.getlogin()
path = os.path.join('..', 'Users', usr, 'Desktop', 'Tester File')
os.chdir(path)
Using os.environ for environment variables may also prove useful. os.environ documentation here For example:
import os
def getUserName():
# set possible environment variables
for name in ('LOGNAME', 'USER', 'LNAME', 'USERNAME'):
usr = os.environ.get(name)
if user:
return usr #return the variable
if __name__ == '__main__':
usr = getUserName()
# do remainder below
I need to add the win_unicode_console module to my usercustomize.py file, as described by the documentation.
I've discovered my user site packages directory with:
>>> import site
>>> site.getusersitepackages()
'C:\\Users\\my name\\AppData\\Roaming\\Python\\Python35\\site-packages'
I haven't been able to get to this directory using any method. I've tried using pushd instead of cd to emulate a network drive, and I've also tried getting there using run. No matter what I do in python, or in cmd terminal. I get the response The network path was not found.
Here is an example of one I've tried in cmd:
C:\>pushd \\Users\\my name\\AppData\\Roaming\\Python\\Python35\\site-packages
The network path was not found.
What am I doing wrong, or what could be wrong with the path?
DOS style backslashes don't need to be escaped within the Windows console (else they may have used forward slashes way back when!).
Follow these steps to manually create usercustomize.py:
Start->Run:cmd
Make sure you're on the C: drive
c:
Create the directory. mkdir creates the missing parents. Obviously, change "my name" as appropriate.
mkdir C:\Users\my name\AppData\Roaming\Python\Python35\site-packages
Create usercustomize.py:
notepad C:\Users\my name\AppData\Roaming\Python\Python35\site-packages\usercustomize.py
Click "yes" to create your file.
Edit as appropriate
Or use the following script to have Python do it for you:
import site
import os
import os.path
import io
user_site_dir = site.getusersitepackages()
user_customize_filename = os.path.join(user_site_dir, 'usercustomize.py')
win_unicode_console_text = u"""
# win_unicode_console
import win_unicode_console
win_unicode_console.enable()
"""
if os.path.exists(user_site_dir):
print("User site dir already exists")
else:
print("Creating site dir")
os.makedirs(user_site_dir)
if not os.path.exists(user_customize_filename):
print("Creating {filename}".format(filename=user_customize_filename))
file_mode = 'w+t'
else:
print("{filename} already exists".format(filename=user_customize_filename))
file_mode = 'r+t'
with io.open(user_customize_filename, file_mode) as user_customize_file:
existing_text = user_customize_file.read()
if not win_unicode_console_text in existing_text:
# file pointer should already be at the end of the file after read()
user_customize_file.write(win_unicode_console_text)
print("win_unicode_console added to {filename}".format(filename=user_customize_filename))
else:
print("win_unicode_console already enabled")
I am using a python script as an external auth option in ejabberd 2.1.6.
I wanted to start encrypting the clear text passwords that come across in the auth verification, so that they are not being stored in plain text in the backend database. When I add the following code to my python script and restart ejabberd, it hangs:
import hashlib
clear = "barfoo"
salt = "foobar"
hash = hashlib.md5( salt + clear ).hexdigest()
Does hashlib require specific priviledges to run?
When I run it as a normal user (ejabberd) it works without issue. When the python script is run within the external auth of ejabberd it hangs.
I've attempted to have it write out the 'hash' to a file and it never gets there ... if i run it as the 'ejabberd' user, it writes out to file fine.
I've tried to find information about restrictions for using this library on ubuntu without any success. Any ideas?
-sd
** 22.02.2011: Here is the full script adapted from https://git.process-one.net/ejabberd/mainline/blobs/raw/2.1.x/doc/dev.html#htoc8 :
#!/usr/bin/python
import sys
from struct import *
import hashlib
def from_ejabberd():
input_length = sys.stdin.read(2)
(size,) = unpack('>h', input_length)
return sys.stdin.read(size).split(':')
def to_ejabberd(bool):
answer = 0
if bool:
answer = 1
token = pack('>hh', 2, answer)
sys.stdout.write(token)
sys.stdout.flush()
def auth(username, server, password):
clear = "barfoo"
salt = "foobar"
hash = hashlib.md5( salt + clear ).hexdigest()
if (password == hash): return True
else: return False
def isuser(username, server):
return True
def setpass(username, server, password):
return True
while True:
data = from_ejabberd()
success = False
if data[0] == "auth":
success = auth(data[1], data[2], data[3])
elif data[0] == "isuser":
success = isuser(data[1], data[2])
elif data[0] == "setpass":
success = setpass(data[1], data[2], data[3])
to_ejabberd(success)
I have been messing with the same problem. I Can't really track down the problem with openssl binings in _hashlib. Whatever the problem is, i will have to patch python distribution sources. Not really an feasible solution. So i ended up using a pycrypto wrapper for the crypto functions which don't block in this case.
pip install pycrypto
from Crypto.Hash import MD5
m = MD5.new()
m.update("%s%s" % (salt ,clear))
h.hexdigest()
I looked into the hashlib source and while it does not seem to require too much, it does import .so files as modules and one of them hits openssl. It all looks pretty safe but if ejabberd tries to fence itself against calls to 3rd party code (or if you have SELinux or something else to that effect running), stuff can conceivably get weird. I got this in a REPL:
>>> import _md5
>>> _md5.__file__
'/usr/lib/python2.7/lib-dynload/_md5module.so'
Try this on your box and then try putting
_md5 = imp.load_dynamic('_md5', '/usr/lib/python2.7/lib-dynload/_md5module.so')
Or just
import _md5
(with the appropriate path updated to yours) in your code before the offending line and with some trace statement afterwards. Try the same with _hashlib instead of _md5 (hashlib defaults to _hashlib which wraps openssl, but if it doesn't load or doesn't have the needed hash it falls back to _md5, _sha etc.). If it's not the imports that fail/hang then you can try calling _md5.new(salt + clear) and _hashlib.openssl_md5(salt + clear) and see if it's one of them.
If it is the import at fault, then possibly a similar problem was tackled here
I don't know ejabberd, so I can't relate their solution to your problem, unfortunately.
I do have to say it, though: in all python implementations I know, = instead of == in a condition would raise a SyntaxError and that's that - the program would never even enter the main while loop.
hashlib does not require anything special. What means hangs? Where does it hang? Use pdb.set_trace() to step trough the code or use 'strace' or 'ltrace' to investigate API calls.
Try to use logging module, it can help you to watch input and output data, also check scripts permissions, let executing by user which is ejabberd, or for debug just set chmod 777 external.py.