I'm using ansbile_runner Python module as bellow:
import ansible_runner
r = ansible_runner.run(private_data_dir='/tmp/demo', playbook='test.yml')
When I execute the above code, it will show the output without printing in Python. What I want is to save the stdout content into a Python variable for further text processing.
Did you read the manual under https://ansible-runner.readthedocs.io/en/stable/python_interface/ ? There is an example where you add another parameter, which is called output_fd and that could be a file handler instead of sys.stdout.
Sadly, this is a parameter of the run_command function and the documentation is not very good. A look into the source code at https://github.com/ansible/ansible-runner/blob/devel/ansible_runner/interface.py could help you.
According to the implementation details in https://github.com/ansible/ansible-runner/blob/devel/ansible_runner/runner.py it looks like, the run() function always prints to stdout.
According to the interface, there is a boolean flag in run(json_mode=TRUE) that stores the response in JSON (I expect in r instead of stdout) and there is another boolean flag quiet.
I played around a little bit. The relevant option to avoid output to stdout is quiet=True as run() attribute.
Ansible_Runner catches the output and writes it to a file in the artifacts directory. Every run() command produces that directory as described in https://ansible-runner.readthedocs.io/en/stable/intro/#runner-artifacts-directory-hierarchy. So there is a file called stdout in the artifact directory. It contains the details. You can read it as JSON.
But also the returned object contains already some relevant data. Here is my example
playbook = 'playbook.yml'
private_data_dir = 'data/' # existing folder with inventory etc
work_dir = 'playbooks/' # contains playbook and roles
try:
logging.debug('Running ansible playbook {} with private data dir {} in project dir {}'.format(playbook, private_data_dir, work_dir))
runner = ansible_runner.run(
private_data_dir=private_data_dir,
project_dir=work_dir,
playbook=playbook,
quiet=True,
json_mode=True
)
processed = runner.stats.get('processed')
failed = runner.stats.get('failures')
# TODO inform backend
for host in processed:
if host in failed:
logging.error('Host {} failed'.format(host))
else:
logging.debug('Host {} backupd'.format(host))
logging.error('Playbook runs into status {} on inventory {}'.format(runner.status, inventory.get('name')))
if runner.rc != 0:
# we have an overall failure
else:
# success message
except BaseException as err:
logging.error('Could not process ansible playbook {}\n{}'.format(inventory.get('name'),err))
So this outputs all processed hosts and informs about failures per host. Concrete more output can be found in the stdout file in artifact directory.
Related
I am trying to create a Python script generation tool that creates Python files based on certain actions. For example:
action = "usb" # dynamic input to script
dev_type = "3.0" # dynamic input to script
from configuration import config
if action == "usb":
script = """
#code to do a certain usb functionality
dev_type = """ + dev_type + """
if dev_type == "2.0"
command = config.read("USB command 2.0 ")
proc = subprocess.Popen(command,stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=False)
elif dev_type == "3.0":
command = config.read("USB command 3.0 ")
proc = subprocess.Popen(command,stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=False)
"""
elif action == "lan":
script = """
#code to run a certain lan functionality for eg:
command = config.read("lan command")
proc = subprocess.Popen(command,stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=False)
"""
with open("outputfile.py", w) as fl:
fl.writelines(script)
The actual script generator is much complex.
I generate the "outputfile.py" script in my local machine inputting the action to the script and then deploy the script generated to remote machines to record the results. This is part of a larger framework and I need to keep this execution format.
Each "script" block uses a config.read function to read certain variables required for it to run from a config file. "command" in the above example.
My actual frame work has some 800 configurations in the config file - which I deploy on the remote machines along with the script to run. So there is a lot of extra configurations in that file that may not be required for a particular script to run and is not very user friendly.
What I am looking for is based on the "script" block that get written to the output file - create a custom config file that contains only the config that is required for that script to run.
For example if action is "lan", the below script block get written to the output file
script = """
#code to run a certain lan functionality for eg:
command = config.read("lan command")
proc = subprocess.Popen(command,stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=False)
"""
What I want is only the "lan command" in my custom config file.
My question is :
How do I evaluate the "script" block (inside triple quotes) to know which config are been used in that code and then to write that config to my custom config file when I write to output file ?
There could be multiple if conditions inside the "script" block also. I do not want config for "USB command 2.0" and "USB command 3.0" in the custom config file if the action = "usb" and dev_type = "3.0"
Of course I cannot execute the code inside the "script" block - and then intercept config.read() function to write the config that got called to my custom config file. Is there a better way to do it ?
I am stumped as to what this issue could be. The nagios log does not report any errors anymore but my file does not get written with anything.
def log_something(host_name host_address, attempt_number):
with open('file', 'a+') as log:
log.write('called function at %s with args %s' %s (str(datetime.datetime.now()), locals()))
def main():
parser = argparse.ArgumentParser()
parser.add_argument('host_name')
parser.add_argument('host_address')
parser.add_argument('attempt_number')
args = parser.parse_args()
log_something(args.host_name, args.host_address, args.attempt_number)
if __name__ == "__main__":
main()
And in my commands.cfg
define command {
command_name my_command
command_line $USER1$/my_command.py $HOSTNAME$ $HOSTADDRESS$ $HOSTATTEMPT$
}
And in my host config
define host {
...
event_handler my_command
}
And in the nagios log (journalctl -xe)
OST ALERT: test-router;UP;HARD;5;PING OK - Packet loss = 0%, RTA = 0.96 ms
Jan 31 15:38:47 nagios-server.nagios[9212]: HOST EVENT HANDLER: test-router;UP;HARD;5;my_command
Nothing is written to the file, no errors are reported. When there were errors in my syntax the nagios log would print the errors that were reported to stderr, one of which was a file permission issue. I fixed that by creating the file in the same folder and chmod 777 everything. Besides if that was an issue it should be logged.
Anyone have any idea what's going on here?
I figured something out. It seems as if you define where the output of nagios event handlers goes in the nagios.cfg file.
# TEMP PATH
# This is path where Nagios can create temp files for service and
# host check results, etc.
temp_path=/tmp
I think this means that the output of the plugin checks get sent to the temp folder. However, changing that path seems to have no effect and the result of my output in the python script still gets written as file in the /tmp folder. This is CentOS7.
Check you function, it seem you missing the "," in function parameters.
def log_something(host_name host_address, attempt_number):
with open('file', 'a+') as log:
I am trying to archive a remote git repo using Python code. I did it successfully using Git command line with following command.
> git archive --format=zip --remote=ssh://path/to/my/repo -o archived_file.zip
HEAD:path/to/directory filename
This command fetches the required file from the repo and stores the zip in my current working directory. Note that there is no cloning of remote repo happening.
Now I have to do it using Python code. I am using GitPython 1.0.1. I guess if it is doable using command line then it should be doable using GitPython library. According to the docs,
repo.archive(open(join(rw_dir, 'archived_file.zip'), 'wb'))
Above line of code will archive the repo. Here repo is the instance of Repo class. It can be initialized using
repo = Repo('path to the directory which has .git folder')
If I give path to my remote repo(Ex. ssh://path/to/my/repo) in above line it goes to find it in directory where the .py file containing this code is residing(Like, Path\to\python\file\ssh:\path\to\my\repo), which is not what I want. So to sum up I can archive a local repo but not a remote one using GitPython. I may be able to archive remote repo if I am able to create a repo instance pointing to the remote repo. I am very new to Git and Python.
Is there any way to archive a remote repo using Python code without cloning it in local?
This is by the way a terrible idea, since you already have begun using gitpython, and I have never tried working with that, but I just really want to let you know, that you can do it without cloning it in local, without using gitpython.
Simply run the git command, in a shell, using subprocess..
running bash commands in python
edit: added some demonstration code, of reading stdout and writing stdin.
some of this is stolen from here:
http://eyalarubas.com/python-subproc-nonblock.html
The rest is a small demo..
first two prerequisites
shell.py
import sys
while True:
s = raw_input("Enter command: ")
print "You entered: {}".format(s)
sys.stdout.flush()
nbstreamreader.py:
from threading import Thread
from Queue import Queue, Empty
class NonBlockingStreamReader:
def __init__(self, stream):
'''
stream: the stream to read from.
Usually a process' stdout or stderr.
'''
self._s = stream
self._q = Queue()
def _populateQueue(stream, queue):
'''
Collect lines from 'stream' and put them in 'quque'.
'''
while True:
line = stream.readline()
if line:
queue.put(line)
else:
raise UnexpectedEndOfStream
self._t = Thread(target = _populateQueue,
args = (self._s, self._q))
self._t.daemon = True
self._t.start() #start collecting lines from the stream
def readline(self, timeout = None):
try:
return self._q.get(block = timeout is not None,
timeout = timeout)
except Empty:
return None
class UnexpectedEndOfStream(Exception): pass
then the actual code:
from subprocess import Popen, PIPE
from time import sleep
from nbstreamreader import NonBlockingStreamReader as NBSR
# run the shell as a subprocess:
p = Popen(['python', 'shell.py'],
stdin = PIPE, stdout = PIPE, stderr = PIPE, shell = False)
# wrap p.stdout with a NonBlockingStreamReader object:
nbsr = NBSR(p.stdout)
# issue command:
p.stdin.write('command\n')
# get the output
i = 0
while True:
output = nbsr.readline(0.1)
# 0.1 secs to let the shell output the result
if not output:
print "time out the response took to long..."
#do nothing, retry reading..
continue
if "Enter command:" in output:
p.stdin.write('try it again' + str(i) + '\n')
i += 1
print output
I asked already and few people gave good advises but there were to many unknowns for me as I am beginner. Therefore I decided to ask for help again without giving bad code.
I need a script which will execute copy files to directory while the other is still running.
Basically I run first command, it generates files (until user press enter) and then those files are gone (automatically removed).
What I would like to have is to copying those files (without have to press "Enter" as well).
I made in bash however I would like to achieve this on python. Please see below:
while kill -0 $! 2>/dev/null;do
cp -v /tmp/directory/* /tmp/
done
If first script is purely command line : it should be fully manageable with a python script.
General architecture :
python scripts starts first one with subprocess module
reads output from first script until it gets the message asking for pressing enter
copies all files from source directory to destination directory
sends \r into first script input
waits first script terminates
exits
General requirements :
first script must be purely CLI one
first script must write to standart output/error and read from standard input - if it reads/writes to physical terminal (/dev/tty on Unix/Linux or con: on Dos/Windows), it won't work
the end of processing must be identifiable in standard output/error
if the two above requirement were no met, the only way would be to wait a define amount of time
Optional operation :
if there are other interactions in first script (read and/or write), it will be necessary to add the redirections in the script, it is certainly feasible, but will be a little harder
Configuration :
the command to be run
the string (from command output) that indicates first program has finished processing
the source directory
the destination directory
a pattern for file name to be copied
if time defined and no identifiable string in output : the delay to wait before copying
A script like that should be simple to write and test and able to manage the first script as you want.
Edit : here is an example of such a script, still without timeout management.
import subprocess
import os
import shutil
import re
# default values for command execution - to be configured at installation
defCommand = "test.bat"
defEnd = "Appuyez"
defSource = "."
defDest = ".."
# BEWARE : pattern is in regex format !
defPattern="x.*\.txt"
class Launcher(object):
'''
Helper to launch a command, wait for a defined string from stderr or stdout
of the command, copy files from a source folder to a destination folder,
and write a newline to the stdin of the command.
Limits : use blocking IO without timeout'''
def __init__(self, command=defCommand, end=defEnd, source=defSource,
dest=defDest, pattern = defPattern):
self.command = command
self.end = end
self.source = source
self.dest = dest
self.pattern = pattern
def start(self):
'Actualy starts the command and copies the files'
found = False
pipes = os.pipe() # use explicit pipes to mix stdout and stderr
rx = re.compile(self.pattern)
cmd = subprocess.Popen(self.command, shell=True, stdin=subprocess.PIPE,
stdout=pipes[1], stderr=pipes[1])
os.close(pipes[1])
while True:
txt = os.read(pipes[0], 1024)
#print(txt) # for debug
if str(txt).find(self.end) != -1:
found = True
break
# only try to copy files if end string found
if found:
for file in os.listdir(self.source):
if rx.match(file):
shutil.copy(os.path.join(self.source, file), self.dest)
print("Copied : %s" % (file,))
# copy done : write the newline to command input
cmd.stdin.write(b"\n")
cmd.stdin.close()
try:
cmd.wait()
print("Command terminated with %d status" % (cmd.returncode,))
except:
print("Calling terminate ...")
cmd.terminate()
os.close(pipes[0])
# allows to use the file either as an imported module or directly as a script
if __name__ == '__main__':
# parse optional parameters
import argparse
parser = argparse.ArgumentParser(description='Launch a command and copy files')
parser.add_argument('--command', '-c', nargs = 1, default = defCommand,
help="full text of the command to launch")
parser.add_argument('--endString', '-e', nargs = 1, default = defEnd,
dest="end",
help="string that denotes that command has finished processing")
parser.add_argument('--source', '-s', nargs = 1, default = defSource,
help="source folder")
parser.add_argument('--dest', '-d', nargs = 1, default = defDest,
help = "destination folder")
parser.add_argument('--pattern', '-p', nargs = 1, default = defPattern,
help = "pattern (regex format) for files to be copied")
args = parser.parse_args()
# create and start a Launcher ...
launcher = Launcher(args.command, args.end, args.source, args.dest,
args.pattern)
launcher.start()
Could I unblock a file in windows(7), which is automatically blocked by windows (downloaded from Internet) from a python script? A WindowsError is raised when such a file is encountered. I thought of catching this exception, and running a powershell script that goes something like:
Parameter Set: ByPath
Unblock-File [-Path] <String[]> [-Confirm] [-WhatIf] [ <CommonParameters>]
Parameter Set: ByLiteralPath
Unblock-File -LiteralPath <String[]> [-Confirm] [-WhatIf] [ <CommonParameters>]
I don't know powershell scripting. But if I had one I could call it from python. Could you folks help?
Yes, all you have to do is call the following command line from Python:
powershell.exe -Command Unblock-File -Path "c:\path\to\blocked file.ps1"
From this page about the Unblock-File command: https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/unblock-file?view=powershell-7.2
Internally, the Unblock-File cmdlet removes the Zone.Identifier alternate data stream, which has a value of 3 to indicate that it was downloaded from the internet.
To remove an alternate data stream ads_name from a file path\to\file.ext, simply delete path\to\file.ext:ads_name:
try:
os.remove(your_file_path + ':Zone.Identifier')
except FileNotFoundError:
# The ADS did not exist, it was already unblocked or
# was never blocked in the first place
pass
# No need to open up a PowerShell subprocess!
(And similarly, to check if a file is blocked you can use os.path.isfile(your_file_path + ':Zone.Identifier'))
In a PowerShell script, you can use Unblock-File for this, or simply Remove-Item -Path $your_file_path':Zone.Identifier'.
Remove-Item also has a specific flag for alternate data streams: Remove-Item -Stream Zone.Identifier (which you can pipe in multiple files to, or a single -Path)
Late to the party . . . .
I have found that the Block status is simply an extra 'file' (stream) attached in NTFS and it can actually be accessed and somewhat manipulated by ordinary means. These are called Alternative Data Streams.
The ADS for file blocking (internet zone designation) is called ':Zone.Identifier' and contains, I think, some useful information:
[ZoneTransfer]
ZoneId=3
ReferrerUrl=https://www.google.com/
HostUrl=https://imgs.somewhere.com/product/1969297/some-pic.jpg
All the other info I have found says to just delete this extra stream.... But, personally, I want to keep this info.... So I tried changing the ZoneId to 0, but it still shows as Blocked in Windows File Properties.
I settled on moving it to another stream name so I can still find it later.
The below script originated from a more generic script called pyADS. I only care about deleting / changing the Zone.Identifier attached stream -- which can all be done with simple Python commands. So this is a stripped-down version. It has several really nice background references listed. I am currently running the latest Windows 10 and Python 3.8+; I make no guarantees this works on older versions.
import os
'''
References:
Accessing alternative data-streams of files on an NTFS volume https://www.codeproject.com/Articles/2670/Accessing-alternative-data-streams-of-files-on-an
Original ADS class (pyADS) https://github.com/RobinDavid/pyADS
SysInternal streams applet https://learn.microsoft.com/en-us/sysinternals/downloads/streams
Windows: killing the Zone.Identifier NTFS alternate data stream https://wiert.me/2011/11/25/windows-killing-the-zone-identifier-ntfs-alternate-data-stream-from-a-file-to-prevent-security-warning-popup/
Zone.Information https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-fscc/6e3f7352-d11c-4d76-8c39-2516a9df36e8
About URL Security Zones https://learn.microsoft.com/en-us/previous-versions/windows/internet-explorer/ie-developer/platform-apis/ms537183(v=vs.85)?redirectedfrom=MSDN
GREAT info: How Windows Determines That the File.... http://woshub.com/how-windows-determines-that-the-file-has-been-downloaded-from-the-internet/
Dixin's Blog: Understanding File Blocking and Unblocking https://weblogs.asp.net/dixin/understanding-the-internet-file-blocking-and-unblocking
'''
class ADS2():
def __init__(self, filename):
self.filename = filename
def full_filename(self, stream):
return "%s:%s" % (self.filename, stream)
def add_stream_from_file(self, filename):
if os.path.exists(filename):
with open(filename, "rb") as f: content = f.read()
return self.add_stream_from_string(filename, content)
else:
print("Could not find file: {0}".format(filename))
return False
def add_stream_from_string(self, stream_name, bytes):
fullname = self.full_filename(os.path.basename(stream_name))
if os.path.exists(fullname):
print("Stream name already exists")
return False
else:
fd = open(fullname, "wb")
fd.write(bytes)
fd.close()
return True
def delete_stream(self, stream):
try:
os.remove(self.full_filename(stream))
return True
except:
return False
def get_stream_content(self, stream):
fd = open(self.full_filename(stream), "rb")
content = fd.read()
fd.close()
return content
def UnBlockFile(file, retainInfo=True):
ads = ADS2(file)
if zi := ads.get_stream_content("Zone.Identifier"):
ads.delete_stream("Zone.Identifier")
if retainInfo: ads.add_stream_from_string("Download.Info", zi)
### Usage:
from unblock_files import UnBlockFile
UnBlockFile(r"D:\Downloads\some-pic.jpg")
Before:
D:\downloads>dir /r
Volume in drive D is foo
Directory of D:\downloads
11/09/2021 10:05 AM 8 some-pic.jpg
126 some-pic.jpg:Zone.Identifier:$DATA
1 File(s) 8 bytes
D:\downloads>more <some-pic.jpg:Zone.Identifier:$DATA
[ZoneTransfer]
ZoneId=3
ReferrerUrl=https://www.google.com/
HostUrl=https://imgs.somewhere.com/product/1969297/some-pic.jpg
After:
D:\downloads>dir /r
Volume in drive D is foo
Directory of D:\downloads
11/09/2021 10:08 AM 8 some-pic.jpg
126 some-pic.jpg:Download.Info:$DATA
1 File(s) 8 bytes