Develop Reference

How to make python script over windbg to make it's use easier?

import sys
import subprocess
command = 'C:\Program Files (x86)\Windows Kits\10\Debuggers\x64 -y ' + sys.argv[1] + ' -i ' + sys.argv[2] + ' -z ' + sys.argv[3] + ' -c "!analyze" '
process = subprocess.Popen(command.split(), stdout=subprocess.PIPE)
output, error = process.communicate()
I tried this code, I am trying to take input of crash dump name and exe location and then I have to display user understandable crash analysis ouput.How to do that using python scripting? Is it easier with cpp scripting?

take input of crash dump name and exe location and then I have to display user understandable crash analysis ouput.
It seems you want to parse the text output of the !analyze command. You can do that, but you should be aware that this command can have a lot of different output.
Let me assume you're analyzing a user mode crash dump. In such a case, I would first run a few simpler commands to check whether you got a legit dump. You may try the following commands:
|| to check the dump type (should be "user")
| to get the name of the executable (should match your application)
lmvm <app> to check the version number of your executable
If everything is fine, you can go on:
.exr -1: distinguish between a crash and a hang. A 80000003 breakpoint is more likely a hang or nothing at all.
This may help you decide if you should run !analyze or !analyze -hang.
How to do that using Python scripting?
[...] \Windows Kits\10\Debuggers\x64 -y ' + [...]
This path contains backslashes, so you want to escape them or use an r-string like r"C:\Program Files (x86)\Windows Kits\10\...".
You should probably start an executable here to make it work. cdb.exe is the command line version of WinDbg.
command.split()
This will not only split the arguments, but also the path to the exectuable. Thus subprocess.popen() will try to an application called C:\Program which does not exist.
This could fail even more often, depending on the arguments with spaces in sys.argv[].
I suggest that you pass the options as they are:
command = r'C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe'
arguments = [command]
arguments.extend(['-y', sys.argv[1]]) # Symbol path
arguments.extend(['-i', sys.argv[2]]) # Image path
arguments.extend(['-z', sys.argv[3]]) # Dump file
arguments.extend(['-c', '!analyze']) # Command(s) for analysis
process = subprocess.Popen(arguments, stdout=subprocess.PIPE)
Note that there's no split() involved, which could split in wrong position.
Side note: -i may not work as expected. If you receive the crash dump from clients, they may have a different version than the one you have on disk. Set up a proper symbol server to mitigate this.
Is it easier with CPP scripting?
It will be different, not easier.
Working example
This is a Python code that considers the above. It's still a bit hacky because of the delays etc. but there's no real indicator other than time and output for deciding when a command finished. This succeeds with Python 3.8 on a crash dump of Windows Explorer.
import subprocess
import threading
import time
import re
class ReaderThread(threading.Thread):
def __init__(self, stream):
super().__init__()
self.buffer_lock = threading.Lock()
self.stream = stream # underlying stream for reading
self.output = "" # holds console output which can be retrieved by getoutput()
def run(self):
"""
Reads one from the stream line by lines and caches the result.
:return: when the underlying stream was closed.
"""
while True:
line = self.stream.readline() # readline() will block and wait for \r\n
if len(line) == 0: # this will only apply if the stream was closed. Otherwise there is always \r\n
break
with self.buffer_lock:
self.output += line
def getoutput(self, timeout=0.1):
"""
Get the console output that has been cached until now.
If there's still output incoming, it will continue waiting in 1/10 of a second until no new
output has been detected.
:return:
"""
temp = ""
while True:
time.sleep(timeout)
if self.output == temp:
break # no new output for 100 ms, assume it's complete
else:
temp = self.output
with self.buffer_lock:
temp = self.output
self.output = ""
return temp
command = r'C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe'
arguments = [command]
arguments.extend(['-y', "srv*D:\debug\symbols*https://msdl.microsoft.com/download/symbols"]) # Symbol path, may use sys.argv[1]
# arguments.extend(['-i', sys.argv[2]]) # Image path
arguments.extend(['-z', sys.argv[3]]) # Dump file
arguments.extend(['-c', ".echo LOADING DONE"])
process = subprocess.Popen(arguments, stdout=subprocess.PIPE, stdin=subprocess.PIPE, universal_newlines=True)
reader = ReaderThread(process.stdout)
reader.start()
result = ""
while not re.search("LOADING DONE", result):
result = reader.getoutput() # ignore initial output
def dbg(command):
process.stdin.write(command+"\r\n")
process.stdin.flush()
return reader.getoutput()
result = dbg("||")
if "User mini" not in result:
raise Exception("Not a user mode dump")
else:
print("Yay, it's a user mode dump")
result = dbg("|")
if "explorer" not in result:
raise Exception("Not an explorer crash")
else:
print("Yay, it's an Explorer crash")
result = dbg("lm vm explorer")
if re.search(r"^\s*File version:\s*10\.0\..*$", result, re.M):
print("That's a recent version for which we should analyze crashes")
else:
raise Exception("That user should update to a newer version before we spend effort on old bugs")
dbg("q")

if you don't want to use windbg which is a gui
use cdb.exe it is console mode windbg it will output all the results to terminal
here is a demo
F:\>cdb -c "!analyze -v;qq" -z testdmp.dmp | grep -iE "bucket|owner"
DEFAULT_BUCKET_ID: BREAKPOINT
Scope: DEFAULT_BUCKET_ID (Failure Bucket ID prefix)
BUCKET_ID
FOLLOWUP_NAME: MachineOwner
BUCKET_ID: BREAKPOINT_ntdll!LdrpDoDebuggerBreak+30
BUCKET_ID_IMAGE_STR: ntdll.dll
BUCKET_ID_MODULE_STR: ntdll
BUCKET_ID_FUNCTION_STR: LdrpDoDebuggerBreak
BUCKET_ID_OFFSET: 30
BUCKET_ID_MODTIMEDATESTAMP: c1bb301
BUCKET_ID_MODCHECKSUM: 1f647b
BUCKET_ID_MODVER_STR: 10.0.18362.778
BUCKET_ID_PREFIX_STR: BREAKPOINT_
FAILURE_BUCKET_ID: BREAKPOINT_80000003_ntdll.dll!LdrpDoDebuggerBreak
Followup: MachineOwner
grep is a general purpose string parser
it is built-in in Linux
it is available for windows in several places
if in 32 bit you can use it from gnuwin32 package / Cygwin
if in 64 bit you can find it in git
you can use the native findstr.exe also
:\>dir /b f:\git\usr\bin\gr*
grep.exe
groups.exe
or in msys / mingw / Cygwin / wsl / third party clones /
:\>dir /b /s *grep*.exe
F:\git\mingw64\bin\x86_64-w64-mingw32-agrep.exe
F:\git\mingw64\libexec\git-core\git-grep.exe
F:\git\usr\bin\grep.exe
F:\git\usr\bin\msggrep.exe
F:\msys64\mingw64\bin\msggrep.exe
F:\msys64\mingw64\bin\pcregrep.exe
F:\msys64\mingw64\bin\x86_64-w64-mingw32-agrep.exe
F:\msys64\usr\bin\grep.exe
F:\msys64\usr\bin\grepdiff.exe
F:\msys64\usr\bin\msggrep.exe
F:\msys64\usr\bin\pcregrep.exe
or you can write your own simple string parser in python / JavaScript / typescript / c / c++ / ruby / rust / whatever
here is a sample python word lookup and repeat script
import sys
for line in sys.stdin:
if "BUCKET" in line:
print(line)
lets check this out
:\>dir /b *.py
pyfi.py
:\>cat pyfi.py
import sys
for line in sys.stdin:
if "BUCKET" in line:
print(line)
:\>cdb -c "!analyze -v ;qq" -z f:\testdmp.dmp | python pyfi.py
DEFAULT_BUCKET_ID: BREAKPOINT
Scope: DEFAULT_BUCKET_ID (Failure Bucket ID prefix)
BUCKET_ID
BUCKET_ID: BREAKPOINT_ntdll!LdrpDoDebuggerBreak+30
BUCKET_ID_IMAGE_STR: ntdll.dll
BUCKET_ID_MODULE_STR: ntdll
BUCKET_ID_FUNCTION_STR: LdrpDoDebuggerBreak
BUCKET_ID_OFFSET: 30
BUCKET_ID_MODTIMEDATESTAMP: c1bb301
BUCKET_ID_MODCHECKSUM: 1f647b
BUCKET_ID_MODVER_STR: 10.0.18362.778
BUCKET_ID_PREFIX_STR: BREAKPOINT_
FAILURE_BUCKET_ID: BREAKPOINT_80000003_ntdll.dll!LdrpDoDebuggerBreak

running command before other finished in python

I asked already and few people gave good advises but there were to many unknowns for me as I am beginner. Therefore I decided to ask for help again without giving bad code.
I need a script which will execute copy files to directory while the other is still running.
Basically I run first command, it generates files (until user press enter) and then those files are gone (automatically removed).
What I would like to have is to copying those files (without have to press "Enter" as well).
I made in bash however I would like to achieve this on python. Please see below:
while kill -0 $! 2>/dev/null;do
cp -v /tmp/directory/* /tmp/
done

If first script is purely command line : it should be fully manageable with a python script.
General architecture :
python scripts starts first one with subprocess module
reads output from first script until it gets the message asking for pressing enter
copies all files from source directory to destination directory
sends \r into first script input
waits first script terminates
exits
General requirements :
first script must be purely CLI one
first script must write to standart output/error and read from standard input - if it reads/writes to physical terminal (/dev/tty on Unix/Linux or con: on Dos/Windows), it won't work
the end of processing must be identifiable in standard output/error
if the two above requirement were no met, the only way would be to wait a define amount of time
Optional operation :
if there are other interactions in first script (read and/or write), it will be necessary to add the redirections in the script, it is certainly feasible, but will be a little harder
Configuration :
the command to be run
the string (from command output) that indicates first program has finished processing
the source directory
the destination directory
a pattern for file name to be copied
if time defined and no identifiable string in output : the delay to wait before copying
A script like that should be simple to write and test and able to manage the first script as you want.
Edit : here is an example of such a script, still without timeout management.
import subprocess
import os
import shutil
import re
# default values for command execution - to be configured at installation
defCommand = "test.bat"
defEnd = "Appuyez"
defSource = "."
defDest = ".."
# BEWARE : pattern is in regex format !
defPattern="x.*\.txt"
class Launcher(object):
'''
Helper to launch a command, wait for a defined string from stderr or stdout
of the command, copy files from a source folder to a destination folder,
and write a newline to the stdin of the command.
Limits : use blocking IO without timeout'''
def __init__(self, command=defCommand, end=defEnd, source=defSource,
dest=defDest, pattern = defPattern):
self.command = command
self.end = end
self.source = source
self.dest = dest
self.pattern = pattern
def start(self):
'Actualy starts the command and copies the files'
found = False
pipes = os.pipe() # use explicit pipes to mix stdout and stderr
rx = re.compile(self.pattern)
cmd = subprocess.Popen(self.command, shell=True, stdin=subprocess.PIPE,
stdout=pipes[1], stderr=pipes[1])
os.close(pipes[1])
while True:
txt = os.read(pipes[0], 1024)
#print(txt) # for debug
if str(txt).find(self.end) != -1:
found = True
break
# only try to copy files if end string found
if found:
for file in os.listdir(self.source):
if rx.match(file):
shutil.copy(os.path.join(self.source, file), self.dest)
print("Copied : %s" % (file,))
# copy done : write the newline to command input
cmd.stdin.write(b"\n")
cmd.stdin.close()
try:
cmd.wait()
print("Command terminated with %d status" % (cmd.returncode,))
except:
print("Calling terminate ...")
cmd.terminate()
os.close(pipes[0])
# allows to use the file either as an imported module or directly as a script
if __name__ == '__main__':
# parse optional parameters
import argparse
parser = argparse.ArgumentParser(description='Launch a command and copy files')
parser.add_argument('--command', '-c', nargs = 1, default = defCommand,
help="full text of the command to launch")
parser.add_argument('--endString', '-e', nargs = 1, default = defEnd,
dest="end",
help="string that denotes that command has finished processing")
parser.add_argument('--source', '-s', nargs = 1, default = defSource,
help="source folder")
parser.add_argument('--dest', '-d', nargs = 1, default = defDest,
help = "destination folder")
parser.add_argument('--pattern', '-p', nargs = 1, default = defPattern,
help = "pattern (regex format) for files to be copied")
args = parser.parse_args()
# create and start a Launcher ...
launcher = Launcher(args.command, args.end, args.source, args.dest,
args.pattern)
launcher.start()

python popen special characters not parsing file

I am having issues with non ASCII characters.
I am using python 2.7.3
python -V
Python 2.7.3
I installed http://pymediainfo.readthedocs.org/en/latest/
via
easy_install pymediainfo
imported as below
from pymediainfo import MediaInfo
media_info = MediaInfo.parse(os.path.join(path, to, file))
using this with ascii characters is fine
for debugging I printed the 'command' from:
/usr/local/lib/python2.7/dist-packages/pymediainfo-1.3.5-py2.7.egg/pymediainfo/__init__.py
the class that 'runs' the mediainfo command in /usr/local/lib/python2.7/dist-packages/pymediainfo-1.3.5-py2.7.egg/pymediainfo/init.py
ENV_DICT = {
"PATH": "/usr/local/bin/:/usr/bin/",
"LD_LIBRARY_PATH": "/usr/local/lib/:/usr/lib/"}
#staticmethod
def parse(filename, environment=ENV_DICT):
command = ["mediainfo", "-f", "--Output=XML", filename]
print command
print repr(command)
fileno_out, fname_out = mkstemp(suffix=".xml", prefix="media-")
fileno_err, fname_err = mkstemp(suffix=".err", prefix="media-")
fp_out = os.fdopen(fileno_out, 'r+b')
fp_err = os.fdopen(fileno_err, 'r+b')
p = Popen(command, stdout=fp_out, stderr=fp_err, env=environment)
p.wait()
fp_out.seek(0)
xml_dom = MediaInfo.parse_xml_data_into_dom(fp_out.read())
fp_out.close()
fp_err.close()
return MediaInfo(xml_dom)
both print and print repr() display:
['mediainfo', '-f', '--Output=XML', "/mnt/path/Long 73\xc2\xb0 58' W.avi"]
the filename is:
Long 73° 58' W.avi
Looking at a UTF-8 table \xc2\xb0 corresponds to °
I am aware this might just be just the console not interpreting the encoding as it should but the output of mediainfo is just
<?xml version="1.0" encoding="UTF-8"?>
<Mediainfo version="0.7.58">
</Mediainfo>
which means "file not found"
os.path.isfile(os.path.join(path, to, file))
returns true for these files
and in bash using
mediainfo -f --Output=XML "/path/to/file"
works
i have goggled and searched around and cannot see the answer.
Any ideas?
Thanks
UPDATE
I used this new test script
#!/usr/bin/python
# -*- coding: utf-8 -*-
import sys
import os
import subprocess as sub
root = "/mnt/path"
for rootfldr in sorted(os.listdir(root)):
if os.path.isfile(os.path.join(root, rootfldr)):
command = ['mediainfo', '-f', '--Output=XML', rootfldr]
aa = sub.Popen(command, stdout=sub.PIPE, stderr=sub.PIPE, stdin=sub.PIPE)
result = aa.communicate()[0]
print rootfldr
print result
And the results were fine (some of then had non ASCII chars).
I then decided to change (from the pymedinfo parse function):
p = Popen(command, stdout=fp_out, stderr=fp_err, env=environment)
to
p = Popen(command, stdout=fp_out, stderr=fp_err)
and the problem was solved
I am guessing that something is missing and or wrong from
ENV_DICT = {
"PATH": "/usr/local/bin/:/usr/bin/",
"LD_LIBRARY_PATH": "/usr/local/lib/:/usr/lib/"}

The command looks ok. The filename is passed as a bytestring that contains text encoded using utf-8. If your filesystem uses utf-8 then it is correct filename:
>>> print "/mnt/path/Long 73\xc2\xb0 58' W.avi".decode('utf-8')
/mnt/path/Long 73° 58' W.avi
It might be a bug in pymediainfo. Try passing the environment argument explicitely as a workaround e.g., environment=os.environ.

Why I got "sh: 1: Syntax error: Unterminated quoted string" when I run my Python program?

I want to count how many lines of code I have written.
Here is the Python code:
import os
import sys
EXT = ['.c','.cpp','.java','.py']
def main():
l = []
if os.path.isdir(sys.argv[1]):
for root, dirs, files in os.walk(sys.argv[1]):
l.extend([os.path.join(root, name) for name in files])
else:
l.append(sys.argv[1])
params = ["'"+p+"'" for p in l if os.path.splitext(p)[1] in EXT]
result = os.popen("wc -l %s "%" ".join(params)).read()
print result
if __name__ == '__main__':
main()
Before this, it was running as expected. But today, it give me this error:
sh: 1: Syntax error: Unterminated quoted string
I don't know what happened.

Your Python script is missing a shebang line. Add the following to the top of your file:
#!/usr/bin/env python
Then you should be able to run the following, assuming your script is at /path/to/your_script.py and it has the executable bit set:
/path/to/your_script.py arg1 arg2 [...]
Alternatively:
python /path/to/your_script.py arg1 arg2 [...]
Update following comments
I suspect what has changed is that a source file containing a ' in its name has been added to the directory you are checking and the shell is choking on this.
You could add the following function to your program:
def shellquote(s):
return "'" + s.replace("'", "'\\''") + "'"
[Lifted from Greg Hewgill's answer to How to escape os.system() calls in Python? .]
And call it like this:
params = [shellquote(p) for p in l if os.path.splitext(p)[1] in EXT]

#Johnsyweb's updated answer seems to have the correct diagnostic, but the correct fix is to not use a shell to invoke wc. Try something like this instead:
cmd = ['/bin/wc', '-l'] # Need full path!
[cmd.extend(p) for p in l if os.path.splitext(p)[1] in EXT]
result = os.popen2(cmd).read()
Note that the subprocess module is the recommended solution now. Switching to that requires a less intrusive change to your current code, though; see http://docs.python.org/2/library/subprocess.html#replacing-os-popen-os-popen2-os-popen3

Looks like your Python program was parsed like a shell script. Add something like this at the header to indicate where your Python is:
#!/usr/bin/python
or you just run python a.py.

Python script for changing windows path to unix path

I want a script where I can paste a windows path as argument, and then the script converts the path to unix path and open the path using nautilus.
I want to be able to use the script as follows:
mypythonscript.py \\thewindowspath\subpath\
The script currently looks like this:
import sys, os
path = "nautilus smb:"+sys.argv[1]
path = path.replace("\\","/")
os.system(path)
I almost works :)
The problem is that I have to add ' around the argument... like this:
mypythonscript.py '\\thewindowspath\subpath\'
Anyone who knows how I can write a script that allows that argument is without ' , ... i.e. like this:
mypythonscript.py \\thewindowspath\subpath\
EDIT: I think I have to add that the problem is that without ' the \ in the argument is treated as escape character. The solution does not necessarily have to be a python script but I want (in Linux) to be able to just paste a windows path as argument to a script.

Unless you're using a really early version of Windows: "/blah/whatever/" just works for your OP.

Actually I had something like this a while ago, I made a bash script to automatically download links I copy into clipboard, here it is edited to use your program (you first need to install xclip if you don't already have it):
#!/bin/bash
old=""
new=""
old="$(xclip -out -selection c)"
while true
do
new="$(xclip -out -selection c)"
if [ "$new" != "$old" ]
then
old="$new"
echo Found: $new
mypythonscript.py $new
fi
sleep 1
done
exit 0
Now whenever you copy something new into the clipboard, your Python script will be executed with an argument of whatever is in your clipboard.

To avoid dealing with escapes in the shell you could work with the clipboard directly:
import os
try:
from Tkinter import Tk
except ImportError:
from tkinter import Tk # py3k
# get path from clipboard
path = Tk().selection_get(selection='CLIPBOARD')
# convert path and open it
cmd = 'nautilus'
os.execlp(cmd, cmd, 'smb:' + path.replace('\\', '/'))
ntpath, urlparse, os.path modules might help to handle the paths more robustly.

#!/usr/bin/python
#! python3
#! python2
# -*- coding: utf-8 -*-
"""win2ubu.py changes WINFILEPATH Printing UBUNTU_FILEPATH
Author: Joe Dorocak aka Joe Codeswell (JoeCodeswell.com)
Usage: win2ubu.py WINFILEPATH
Example: win2ubu.py "C:\\1d\ProgressiveWebAppPjs\\Polymer2.0Pjs\\PolymerRedux\\zetc\\polymer-redux-polymer-2"
prints /mnt/c/1d/ProgressiveWebAppPjs/Polymer2.0Pjs/PolymerRedux/zetc/polymer-redux-polymer-2
N.B. spaceless path needs quotes in BASH on Windows but NOT in Windows DOS prompt!
"""
import sys,os
def winPath2ubuPath(winpath):
# d,p = os.path.splitdrive(winpath) # NG only works on windows!
d,p = winpath.split(':')
ubupath = '/mnt/'+d.lower()+p.replace('\\','/')
print (ubupath)
return ubupath
NUM_ARGS = 1
def main():
args = sys.argv[1:]
if len(args) != NUM_ARGS or "-h" in args or "--help" in args:
print (__doc__)
sys.exit(2)
winPath2ubuPath(args[0])
if __name__ == '__main__':
main()

may want to try
my_argv_path = " ".join(sys.argv[1:])
as the only reason it would split the path into separate args is spaces in pasted path
(eg: C:\Program Files would end up as two args ["c:\Program","Files"])