Develop Reference

how do I find a way to use ssh and rsync

I have troubles using to implement ssh and rsync including a private key in python, including Popen (subprocess).
Basically the rsync syntax to use should be:
$ rsync -az -e --log-file=$logdir/logfile.out \
'ssh -e /home/user/.ssh/id_rsa' user#server:/target-directory
What I have is this:
import subprocess
First, I build my logdir path - with variables :
logdir = [basedir + '/' + 'log' + '/' + today + '/' + 'pfdcopy_' \
+ typ + '_' + ts + '.log.txt']
Then I build the target directory:
target= ['jboss' + '#' + targetsvr + ':' + /data']
Finally, I try to run this code
p1 = subprocess.Popen(['rsync', '-az', '--log-file=%s' % \
logdir/logfile.out , '-e', 'ssh', '-i', \
'/home/user/.ssh/id_rsa', target])
It's quite complex, I know, mainly because of the variables, and the quotation marks.
Running this, I get always different syntax errors with p1.
Any help is highly appreciated. Thanks!
edited (08-10-2018):
here is my complete runnable code snippet -
from datetime import datetime
import subprocess
import os
import fnmatch
now = datetime.now()
today = now.strftime("%Y-%m-%d")
ts = now.strftime("%Y-%m-%d-%H-%M-%S")
sign_output_dir = '/Users/fanta4/Documents/python-files/outgoing'
mandator = 'BTV'
formsize = 'A4'
basedir = '/Users/fanta4/Documents'
pdf_to_send = []
targetsvr = 'nas1'
doktyp = (
'WPGEBUEHR', 'WPDURCHFU', 'WPABR', 'WPABRKF', 'WPABRVK', 'WPABRTILG', 'WPABRERTR', 'WPAMIS', 'WPSTREP',
'WPABLAUF', 'WPAVISO', 'WPAUSZUG', 'WPERTRAEG', 'WPSIKTEST', 'WPTRANS', 'WPANSCHAFF', 'KKKONTOMIT', 'KRKURSUEW',
'WPVERLUSTA', 'WPVERLUSTG')
os.chdir(sign_output_dir)
for file in os.listdir(sign_output_dir):
if fnmatch.fnmatch(file, '*.pdf'):
pdf_to_send.append(file)
os.chdir(sign_output_dir)
print('debug: doktyp ist: {}'.format(formsize))
for typ in doktyp:
if typ in str(pdf_to_send):
ts = now.strftime("%Y-%m-%d-%Hh-%Mm-%Ss")
print('typ: {:12s} exists -> will be transfered to nas1'.format(typ))
logdir = [basedir + '/' + 'log' + '/' + mandator + '/' + today + '/' + 'pfdcopy_' + typ + '_' + ts + '.log.txt']
target = ['jboss' + '#' + targetsvr + '/data' + '/' + mandator + typ]
p1 = subprocess.Popen(
['rsync', '-az', '--log-file=%s' % logdir, '-e', 'ssh', '-i', '/Users/fanta4/.ssh/id_rsa', typ, '-l', target])
p1.returncode
if p1 > 0:
print('debug: Error with rsync of typ: {} to target: {}'.format(typ, targetsvr))
else:
print('debug: rsync mandator: {:3s} with typ: {:12s} succeeded'.format(mandator, typ))
else:
print('debug: typ: {:12s} does not exist'.format(typ))
logfile = ['/data' + '/' + 'log' + '/' + mandator + '/' + ts]
print('debug: pls see logfile in: {}'.format(logfile))
If I run this code, I get:
/Users/fanta4/anaconda3/bin/python "/Users/fanta4/Library/Mobile Documents/com~apple~CloudDocs/entw/python/prog/rsync-test.py"
Traceback (most recent call last):
/Users/fanta4/Documents/python-files/outgoing
debug: doktyp ist: A4
File "/Users/fanta4/Library/Mobile
Documents/com~apple~CloudDocs/entw/python/prog/rsync-test.py", line 37, in <module>
typ: WPGEBUEHR exists -> will be transfered to nas1
['rsync', '-az', '--log-file=%s' % logdir, '-e', 'ssh', '-i', '/Users/fanta4/.ssh/id_rsa', typ, '-l', target])
File "/Users/fanta4/anaconda3/lib/python3.6/subprocess.py", line 709, in __init__
restore_signals, start_new_session)
File "/Users/fanta4/anaconda3/lib/python3.6/subprocess.py", line 1275, in _execute_child
restore_signals, start_new_session, preexec_fn)
TypeError: expected str, bytes or os.PathLike object, not list
Process finished with exit code 1

Your issue is exemplified by the following lines (when used to generate items which are later used as elements inside an argument vector):
logdir = [basedir + '/' + 'log' + '/' + mandator + '/' + today + '/' + 'pfdcopy_' + typ + '_' + ts + '.log.txt']
target = ['jboss' + '#' + targetsvr + '/data' + '/' + mandator + typ]
You're defining logdir and target as lists (with only one string inside them), whereas they need to be strings.
Just take out the square brackets that create a list, and you'll have strings instead:
logdir = basedir + '/log/' + mandator + '/' + today + '/pfdcopy_' + typ + '_' + ts + '.log.txt'
target = 'jboss#' + targetsvr + '/data/' + mandator + typ

You haven't mentioned what syntax errors you are getting. It would really for the benefit of everyone for you to include that information. I am guessing it's missing the missing quotes around one the string parameters.
p1 = subprocess.Popen([
'rsync', '-az', '--log-file=%s' % 'logdir/logfile.out',
'-e', 'ssh', '-i',
'/home/user/.ssh/id_rsa', target
])

Python script with subprocess returns must be str, not int evrything linter says this is the case [duplicate]

This question already has an answer here:
How can I concatenate str and int objects?
(1 answer)
Closed 4 years ago.
I get the following error when i run the following code.
Version info:
Python 3.6.5 (default, May 11 2018, 04:00:52) [GCC 8.1.0] on linux
Code:
Proper format at https://gist.github.com/Drunkenpanda2000/31f76521ce1166b804a539f40ec21c60
#!/usr/bin/env python
import subprocess
#will be replaced with inputs from Chef
name='test'
vcpus=1
memory=2048
iso='/var/lib/libvirt/images/Centos.iso'
discsize= 80
os_type='linux'
os_variant='centos7'
network_bridge='default'
#setting up the command
args = (
'virt-install' +
' --name=' + name +
' --vcpus=' + vcpus +
' --memory=' + memory +
' --cdrom=' + iso +
' --disk size=' + discsize +
' --os-type=' + os_type +
' --os-varient=' + os_variant +
' --network bridge=' + network_bridge +
" --extra-args 'console=ttyS0,115200n8 serial'" )
#execute the commands in bash
subprocess.call(args, shell=True)
Error
[drunkenpanda#Diablo Scripts]$ ./createvm.py Traceback (most recent call last): File "./createvm.py", line 27, in <module>
' --network bridge=' + network_bridge + TypeError: must be str, not int
New code
args = ['virt-install',
' --name',name,
' --vcpus',str(vcpus),
' --memory',str(memory),
' --cdrom',iso,
' --disk-size',str(discsize),
' --os-variant',os_variant,
' --os-type',os_type,
' --network bridge',network_bridge]
# " --extra-args 'console=ttyS0,115200n8 serial'"\
#execute the commands in bash
subprocess.call(args, shell=False)
New error
./createvmattend.1.py
usage: virt-install --name NAME --memory MB STORAGE INSTALL [options]
virt-install: error: unrecognized arguments: --name bob --vcpus 1 --memory 2048 --cdrom /var/lib/libvirt/images/Centos.iso --disk-size 80 --os-variant centos7.0 --os-type linux --network bridge virbr0

You can only string-concatenate strings, not integers.
Crude but should work:
args = (
'virt-install' +
' --name=' + name +
' --vcpus=' + str(vcpus) + # fix
' --memory=' + str(memory) + # fix
' --cdrom=' + iso +
' --disk size=' + str(discsize) + # fix
' --os-type=' + os_type +
' --os-varient=' + os_variant +
' --network bridge=' + network_bridge +
" --extra-args 'console=ttyS0,115200n8 serial'" )
If you are on python 3.6 you might want to switch to Literal String Interpolation PEP-498:
someValue = 22
c = f"This text contains {someValue}"
or you can use .format()
someValue = 22
c = "This text contains {}".format(someValue) # positional replacement of {} by var

"Extended" template-based INI file parsing in Python

I need to parse (and feed to a database) configuration files like this one (actually it is Asterisk's sip.conf):
[client-template](!,natted-template)
foo=foovalue
moo=moovalue
[client](client-template)
bar=barvalue
This syntax means that client-template is a template itself (because of ! in parentheses) based on natted-template (defined elsewhere). And client is an object definition, based on client-template.
I could use ConfigParser, but it looks like I need something more powerful or customized?

Well, now I've tried pyparsing:
import pyparsing as pp
filename = 'client.conf'
nametag = pp.Word(pp.alphanums + "-_")
variable = pp.Word(pp.alphanums)
value = pp.Word(pp.alphanums + "=")
vardef = pp.Group(variable('variable') + pp.Literal("=").suppress() + value('value'))
vardefs = pp.Group(pp.ZeroOrMore(vardef))('vardefs')
section = pp.Group(pp.Literal("[").suppress() \
+ nametag('objectname') \
+ pp.Literal("]").suppress() \
+ pp.Optional(
pp.Literal("(").suppress() \
+ pp.Optional("!")('istemplate')
+ pp.ZeroOrMore(pp.Optional(",").suppress() + nametag)('parenttemplates') \
+ pp.Literal(")").suppress()
) \
+ vardefs)
section = section + pp.Optional(pp.SkipTo(section)).suppress()
section_group = pp.Group(section + pp.ZeroOrMore(section))('sections')
config = (pp.SkipTo(section_group).suppress() \
+ section_group)
# res = config.parseString(open(filename).read())
This is a part of a solution (and this is comments unaware so far), but I can proceed with it.
Please, if there is more elegant solution, let me know.

python print a string in array format

I passed an argument to a python script like -b bench. The bench is created like this:
bench_dir = '~/myFD/'
bench_bin = bench_dir + 'src/bin/Assembler'
bench_inp1 = bench_dir + 'input/in.fa'
bench_out1 = bench_dir + 'output/data.scratch'
bench= LiveProcess()
bench.executable = bench_bin
bench.cwd = bench_dir
bench.cmd = [bench.executable] + ['-s', bench_out1, '<', bench_inp1]
The bench.cmd should looks like:
~/myFD/src/bin/Assembler -s ~/myFD/output/data.scratch < ~/myFD/input/in.fa
to do that, I use print bench.cmd but it doesn't show the above statment correctly. Instead it shows:
['~/myFD/src/bin/Assembler', '-s', '~/myFD/output/data.scratch', ' < ', '~/myFD/input/in.fa']
how can I fix that?

Try: print ' '.join(bench.cmd). This joins the list and uses a space as delimiter

You could do ' '.join(bench.cmd).

case for join: ' '.join(bench.cmd)

Are you looking for this,
>>> mylist = ['~/myFD/src/bin/Assembler', '-s', '~/myFD/output/data.scratch', ' < ', '~/myFD/input/in.fa']
>>> " ".join(mylist)
'~/myFD/src/bin/Assembler -s ~/myFD/output/data.scratch < ~/myFD/input/in.fa'
or just concatenate your strings
bench.cmd = bench.executable + ' -s ' + bench_out1 + ' < ' + bench_inp1

Scripting Python for Linux commands

I have a question. I have been really trying to learn Python. For a project, I want to make an ncurses GUI for my backup server. My backup server runs rdiff-backup, and I want to have the ncurses take in variable names and plug them into my script. I have been trying to do a lot of reading so I don't ask dumb questions.
Here is my function for running the script:
def runScript():
# Cannot concatenate 'str' and 'list' objects
#script = rdiff + rdiffArgs
script = rdiff + ' ' + rdiffVerbosity + ' ' + rdiffStatistics \
+ ' ' + clientName + '#' + clientHost + '::' + clientDir \
+ ' ' + serverDir
os.system(script)
What I originally thought would be neat was to add all the variables into a list, so I could just run say
script = rdiff + rdiffArgs
Is there a better way to do this without all the space concatenation?
Thanks for your assistance
EDIT: Let me post the whole script so far. I wasn't very clear and I really appreciate your help and patience
#!/usr/bin/env python
import os
import smtplib
# Global variables
rdiff = '/usr/bin/rdiff-backup'
rdiffVerbosity = '-v5'
rdiffStatistics = '--print-statistics'
emailSmtp = 'smtp.gmail.com'
smtpPort = '465'
emailUsername = 'reports'
emailPassword = '3kc9dl'
emailTo = 'user#domain.com'
emailFrom = 'internal#domain.com'
serverName = 'root'
serverHost = 'SV-Datasafe'
serverDir = '/srv/backup/SV-Samba01'
clientName = 'root'
clientHost = 'SV-Samba01'
clientDir = '/srv'
rdiffArgs = rdiffArgs = [rdiffVerbosity, rdiffStatistics, \
clientName + '#' + clientHost + '::' \
+clientDir + ' ' + serverDir]
time = ''
dateStamp = datetime.now()
def sendEmail():
subject = dateStamp + clientName
body = clientDir + ' on ' + clientHost + ' backed up to ' + serverName + \
' in the directory ' + serverDir + ' on ' + dateStamp
message = """\
From: %s
To: %s
Subject: %s
%s
""" % (emailFrom, emailTo, subject, body)
deliverEmail = smtplib.SMTP(emailSmtp, port=smtpPort)
deliverEmail.login(emailUsername, emailPassword)
def runScript():
# Cannot concatenate 'str' and 'list' objects
#script = rdiff + rdiffArgs
script = rdiff + ' ' + rdiffVerbosity + ' ' + rdiffStatistics \
+ ' ' + clientName + '#' + clientHost + '::' + clientDir \
+ ' ' + serverDir
os.system(script)
# TODO:: Logging

you can use format specifiers
def runScript():
script = "%s %s %s#%s %s::%s %s" %(rdiff,rdiffVerbosity,rdiffStatistics,clientName,clientHost,clientDir,serverDir)
os.system(script)
or say your rdiffArgs is already in a list
rdiffArgs = [rdiffVerbosity,rdiffStatistics,clientName,clientHost,clientDir,serverDir]
you can join them with a space
rdiffArgs = ' '.join(rdiffArgs)
lastly, just so you might want to know, you can import rdiff in your script , since rdiff-backup is written in Python
from rdiff_backup.Main import Main as backup
task=['/etc', '/tmp/backup']
backup(task)
the above backs up /etc/ to /tmp/backup. That way, you don't have to make system call to rdiff-backup. Of course, this is up to you. making system call is sometimes easier

try to use subprocess module and pass arguments as list e.g.
client = clientName + '#' + clientHost + '::' + clientDir
cmd = [rdiff, rdiffVerbosity, rdiffStatistics, client , serverDir]
p = Popen(cmd ", shell=True)
print os.waitpid(p.pid, 0)[1]
or if have args already as list use something like this
cmd = [rdiff] + args

You join paths using os.path.join
You concatenate strings like so: "".join(['a', 'b']) or ", ".join(['c', 'd'])
Which part is difficult? I am not sure I understand the question 100%
Is this it?
script = rdiff + " ".join(rdiffArgs)