I'm using fabric to connect to remote host, when i'm there, I try to call a script that I made (It parses the file I give in argument). But when I call the script from inside my Fabfile.py, it assumes the path I gave is from the machine I launch the fabfile from (so not my remote host)
In my fabfile.py I have:
Import import servclasse
env.host='host1'
def listconf():
#here I browes to the correct folder
s=servclasse.Server("my.file") #this is where I want it to open the host1:my.file file and instanciate a classe from what it parsed
If i do this, it tries to open the file from the folder where servclass.py is. Is there a way to give a "remote path" in argument? I would rather not downloading the file.
Should I upload the script servclasse.py with the operation.put before calling it?
Edit: more info
In my servclasse I have this:
def __init__(self, path):
self.config = ConfigParser.ConfigParser(allow_no_value=True)
self.config.readfp(open(path))
The function open() was the problem.
I figured out how to do it so i'll drop it here in case someone read this topic one day :
def listconf():
#first I browes to the correct folder then
contents = StringIO.StringIO()
get("MyFile",contents)
contents.seek(0)
s=Server(contents)
and in the servclass.py
def __init__(self, objfile):
self.config = ConfigParser.ConfigParser(allow_no_value=True)
self.config.readfp(objfile)
#and i do my stuffs
Related
Every time that i try to launch my notebook im getting the error below .
let's specify that im new worker on the project and the file config.py was created before that i joined the team.
Does anyone knows how to resolve it please?
The code actually done is
Requirements.txt
psycopg2==2.7.3.2.
SQLAlchemy==1.2.2
pandas==0.21.0
docker==3.3.0
python-json-logger
sshtunnel==0.1.4
jupyter
jupytext==1.2
geopy==2.2.0
errror detail
~/SG/notebooks/config.py in <module>
1 # Using jupytext
----> 2 c.NotebookApp.contents_manager_class = "jupytext.TextFileContentsManager"
3 c.ContentsManager.default_jupytext_formats = "ipynb,py"
NameError: name 'c' is not defined
code
the row causing the error in the notebook is
from src.util.connect_postgres import postgres_connexion
the content of the file connect_postgres
from sqlalchemy import create_engine
from config.util.database import TARGET_TEST_HOST, TARGET_PROD_HOST, \
TARGET_TEST_DB, TARGET_PROD_DB, TARGET_TEST_USER, TARGET_PROD_USER, SG_PROD_USER, SG_PROD_HOST
from config.secrets.passwords import TARGET_PROD_PWD, TARGET_TEST_PWD, SG_PROD_PWD
from sshtunnel import SSHTunnelForwarder
import psycopg2
def _create_engine_psg(user, db, host, port, pwd):
""" Returns a connection object to PostgreSQL """
url = build_postgres_url(db, host, port, pwd, user)
return create_engine(url, client_encoding='utf8')
def build_postgres_url(db, host, port, pwd, user):
url = 'postgresql://{}:{}#{}:{}/{}'.format(user, pwd, host, port, db)
return url
def postgres_connexion(env):
if env == 'prod':
return create_engine_psg_with_tunnel_ssh(TARGET_PROD_DB,
TARGET_PROD_USER, TARGET_PROD_PWD, SG_PROD_PWD,
SG_PROD_USER,
SG_PROD_HOST, TARGET_PROD_HOST)
else:
raise ValueError("'env' parameter must be 'prod'.")
config.py
c.NotebookApp.contents_manager_class = "jupytext.TextFileContentsManager"
c.ContentsManager.default_jupytext_formats = "ipynb,py"
I red that i can generate the file and then edit it.
when i tried to create the jupyter_notebook_config it is always in my personal directory of marczhr
/Users/marczhr/.jupyter/jupyter_notebook_config.py
but i want to be done in my something that i can push on git.
Hope that im clear ^^
Thank you,
Don't run the notebook from the directory with the configuration file.
The reason is that there is an import with a config module or package in the code listed. By launching the notebook from the directory with the configuration file, it will import that Jupyter configuration file, instead of the correct package or module, with the resulting error.
Instead, run it from somewhere else, or put the configuration file elsewhere.
Or perhaps best, take the two configuration lines and add them to the end of your /Users/marczhr/.jupyter/jupyter_notebook_config.py file, then remove the 2-3 line config.py file.
In the latter case, you can now launch the notebook server from anywhere, and you don't need to specify any configuration file, since Jupyter will automatically use the generated (with added lines) one.
If you want to keep the config.py file, then launch the Jupyter notebook server from another directory, and simply specify the full path, like
jupyter --config=$HOME/SG/notebooks/config.py
All in all, this is a classic nameclash upon import, because of identically named files/directories. Always be wary of that.
(I've commented on some other potential problems in the comments: that still stands, but is irrelevant to the current problem here.)
We can upload file using telegram-upload library by using the following command on terminal
telegram-upload file1.mp4 /path/to/file2.mkv
But if I want to call this inside python function, How should I do it. I mean in a python function if users passes the file path as an argument, then that function should be able to upload the file to telegram server.It is not mentioned in the documentation.
In other words I want to ask how to execute or run shell commands from inside python function?
For telegram-upload you can use upload method in telegram_upload.management and
for telegram-download use download method in the same file.
Or you can see how they are implemented there.
from telegram_upload.client import Client
from telegram_upload.config import default_config, CONFIG_FILE
from telegram_upload.exceptions import catch
from telegram_upload.files import NoDirectoriesFiles, RecursiveFiles
DIRECTORY_MODES = {
'fail': NoDirectoriesFiles,
'recursive': RecursiveFiles,
}
def upload(files, to, config, delete_on_success, print_file_id, force_file, forward, caption, directories,
no_thumbnail):
"""Upload one or more files to Telegram using your personal account.
The maximum file size is 1.5 GiB and by default they will be saved in
your saved messages.
"""
client = Client(config or default_config())
client.start()
files = DIRECTORY_MODES[directories](files)
if directories == 'fail':
# Validate now
files = list(files)
client.send_files(to, files, delete_on_success, print_file_id, force_file, forward, caption, no_thumbnail)
I found the solution.Using os module we can run command line strings inside python function i.e. os.system('telegram-upload file1.mp4 /path/to/file2.mkv')
I am working on a script to transfer some files to a Cisco IOS device using netmiko FileTransfer. Below is the code that I found to accomplish this. However I cant seem to find where the source_file should be or how to specify where on the host that files lives. How do I specify where to copy that file from?
dest_file_system = 'disk0:/'
source_file = 'test1.txt' # where should this file be live?
dest_file = 'test1.txt'
with FileTransfer(ssh_conn, source_file=source_file, dest_file=dest_file,
file_system=dest_file_system) as scp_transfer:
if not scp_transfer.check_file_exists():
if not scp_transfer.verify_space_available():
raise ValueError("Insufficient space available on remote device")
print("\nTransferring file\n")
scp_transfer.transfer_file()
The source file from the scripts perspective if you simple want to call it by name should be in the same directory as the script itself. If you want to move the file to a new directory the search path in the script is relative to the directory the script is run from. Example 1 - The file_name.txt is in the same directory as your script In your script simply call the file source = "file_name.txt". Example 2 create test_folder in current directory that holds your script and call it test_folder move the file_name.txt into test_folder. In your script your source variable would no need to look like this source = "test_folder/file_name.txt"
Hopefully, till now you found the solution already. If not then You can try the netmiko file transfer feature also. That will be a more secure and efficient way to do it.
try below sample script.
from getpass import getpass
from netmiko import ConnectHandler, file_transfer
password = getpass()
cisco = {
"device_type": "cisco_ios",
"host": "cisco1.twb-tech.com",
"username": "pyclass",
"password": password,
}
source_file = "test1.txt"
dest_file = "test1.txt"
direction = "put"
file_system = "flash:"
ssh_conn = ConnectHandler(**cisco)
transfer_dict = file_transfer(
ssh_conn,
source_file=source_file,
dest_file=dest_file,
file_system=file_system,
direction=direction,
overwrite_file=True,
)
print(transfer_dict)
I have written two scripts in python that will be working with data in the same directory. One script will be set to run every 5 minutes and will save data to the directory, and then once a day the other script will zip all of that data, archive it and delete the original files. To avoid the archiving script deleting files which may be being saved by the worker script, I want to create a system-wide mutex so that the worker script knows not to run while the archiver is doing its thing.
I've done some searching and seen that on unix-based systems, the generally accepted method of doing this is to attempt to lock a file. If you get the lock then great, go ahead and run, if you can't get it then you know the other process is already running. I've written the following code:
import fcntl
import traceback
class ProcessLock:
def __init__(self, path_to_file, block):
self.file_path = path_to_file
try:
options = fcntl.LOCK_EX
if not block:
options = options | fcntl.LOCK_NB
self.file = open(path_to_file, 'w+')
self.lock = fcntl.flock(file, options)
except:
print 'caught something: {}'.format(traceback.format_exc())
self.file = None
self.lock = None
def is_locked(self):
return self.lock is not None
def unlock(self):
self.lock = None
self.file = None
def aquire_lock(lock_name):
path = '/tmp/{}.lock'.format(lock_name)
return ProcessLock(path, False)
def aquire_lock_blocking(lock_name):
path = '/tmp/{}.lock'.format(lock_name)
return ProcessLock(path, True)
However for the life of me I can't get it to actually work. I have searched and all of the samples I've seen and other questions posted on here seem to work using the code that I have got. I've also tried both flock and lockf, but neither work. The call to open works correctly, but I get the following logged out to the console:
self.lock = fcntl.flock(file, options)
TypeError: descriptor 'fileno' of 'file' object needs an argument
I don't know enough about Python to be able to know what this error means. Hopefully someone can see if I'm doing anything wrong. I'm running this in Pycharm on macOS
I am checking out a source for a given url using a python script and I want to go to the downloadedFoler/src directory and perform a mvn clean install. I want to do it in the same script. Thank in advance.
You can do the following:
import os
import subprocess
# Context Manager to change current directory.
# I looked at this implementation on stackoverflow but unfortunately do not have the link
# to credit the user who wrote this part of the code.
class changeDir:
def __init__(self, newPath):
self.newPath = os.path.expanduser(newPath)
# Change directory with the new path
def __enter__(self):
self.savedPath = os.getcwd()
os.chdir(self.newPath)
# Return back to previous directory
def __exit__(self, etype, value, traceback):
os.chdir(self.savedPath)
# folderPath = path of the folder you want to run mvn clean install on
with changeDir(folderPath):
# ****** NOTE ******: using shell=True is strongly discouraged since it possesses security risks
subprocess.call(["mvn", "clean", "install"], shell=True)